.. SPDX-License-Identifier: GPL-2.0+
..
.. Copyright 2025 Canonical Ltd.
.. Written by Simon Glass <simon.glass@canonical.com>
Pickman - Cherry-pick Manager
=============================
Pickman is a tool to help manage cherry-picking commits between branches.
Workflow
--------
The typical workflow for using pickman is:
1. **Setup**: Add a source branch to track with ``add-source``. This records the
starting point (merge-base) for cherry-picking.
2. **Cherry-pick**: Run ``apply -p`` to cherry-pick the next set of commits (up
to the next merge commit) and create a GitLab MR. A Claude agent handles the
cherry-picks automatically, resolving simple conflicts.
3. **Review**: Once the MR is reviewed and merged, run ``commit-source`` to
update the database with the last processed commit.
4. **Repeat**: Go back to step 2 until all commits are cherry-picked.
For fully automated workflows, use ``poll`` which runs ``step`` in a loop. The
``step`` command handles the complete cycle automatically:
- Detects merged MRs and updates the database (no manual ``commit-source``)
- Processes review comments on open MRs using Claude agent
- Creates new MRs when none are pending
This allows hands-off operation: just run ``poll`` and approve/merge MRs in
GitLab as they come in.
Commit Selection
----------------
When pickman creates an MR, it groups commits into logical sets based on merge
commits in the source branch. Understanding this helps predict what will be
included in each MR.
**Algorithm**
1. **Start from the last processed commit**: Pickman reads from its database the
hash of the last commit that was successfully cherry-picked from the source
branch.
2. **Find the next merge on first-parent chain**: Walking forward from the last
processed commit along the first-parent chain (``git log --first-parent``),
pickman finds the next merge commit. The first-parent chain represents the
mainline history of the branch.
3. **Include all commits up to that merge**: Using ``git log`` (without
``--first-parent``), pickman collects ALL commits between the last processed
commit and the merge commit. This includes:
- Commits on the mainline leading up to the merge
- Commits brought in by the merge (from the merged branch)
- The merge commit itself
**Example**
Consider this history on the source branch::
* 5c8ef70 Merge tag 'xilinx-for-v2025.01-rc5-v2'
|\
| * 1b70b6c common: memtop: Fix the return type
|/
* c06705a Makefile: Match the full path to ccache
* 0b7f4c7 imx: Fix usable memory ranges
* ff1d5d8 Revert "configs: JH7110: enable EFI_LOADER"
* d701c6a net: lwip: check if network device is available
* b6691d0 net: lwip: do not return CMD_RET_USAGE
* 9378307 binman: Regenerate tools/binman/entries.rst <-- last processed
If the database shows ``9378307`` as the last processed commit, pickman will:
1. Walk first-parent from ``9378307`` and find merge ``5c8ef70``
2. Collect all commits in ``9378307..5c8ef70``:
- ``b6691d0`` net: lwip: do not return CMD_RET_USAGE
- ``d701c6a`` net: lwip: check if network device is available
- ``ff1d5d8`` Revert "configs: JH7110..."
- ``0b7f4c7`` imx: Fix usable memory ranges
- ``c06705a`` Makefile: Match the full path to ccache
- ``1b70b6c`` common: memtop: Fix the return type (from xilinx branch)
- ``5c8ef70`` Merge tag 'xilinx-for-v2025.01-rc5-v2'
The resulting MR contains 7 commits. The branch name is derived from the first
commit's short hash: ``cherry-b6691d0``.
**Why merge-based grouping?**
Merge commits typically represent logical units of work (e.g., a pull request
or a subsystem update). By stopping at each merge, pickman:
- Keeps MRs focused and reviewable
- Preserves the original grouping from upstream
- Makes it easier to identify and skip problematic sets
**No merge found**
If there are no merge commits between the last processed commit and the branch
tip, pickman includes all remaining commits in a single set. This is noted in
the output as "no merge found".
Skipping MRs
------------
During review, if a set of commits should be skipped (e.g., not applicable to
the target branch), a reviewer can comment:
- ``pickman skip``
- ``pickman: skip``
- ``@pickman skip``
- ``@pickman: skip``
Pickman will add ``[skipped]`` to the MR title. Skipped MRs:
- Are ignored when deciding whether to create new MRs
- Don't block the ``step`` or ``poll`` commands from proceeding
- Can be unskipped by commenting ``pickman unskip``
Already-Applied Detection
-------------------------
Sometimes commits have already been applied to the target branch through a
different path (e.g., directly merged or cherry-picked with different hashes).
Pickman detects this situation automatically in several ways.
**Merge-Level Detection**
When pickman looks for the next set of commits to cherry-pick, it checks if
each merge commit's subject already exists in the target branch. If a merge
is found in the target, pickman assumes all its commits have been dealt with
and:
1. Adds all commits from that merge to the database as 'applied'
2. Advances the source position past that merge
3. Continues to the next merge
If the merge subject isn't found (e.g., because commits were cherry-picked
individually rather than merged), pickman also checks if all individual
commits in the merge have matching subjects in the target. This handles
cases where the series was applied without the merge commit itself.
This allows pickman to automatically catch up with commits that were manually
cherry-picked outside the tool.
**Pre-Cherry-Pick Detection**
Before starting cherry-picks, pickman checks for potentially already-applied
commits by searching for matching commit subjects in the target branch::
git log --oneline ci/master --grep="<subject>" -1
Commits that match are marked as "maybe already applied" and passed to the
Claude agent with the hash of the potentially matching commit. The agent then:
1. Compares the actual patch content between the original and found commits
2. Uses ``git show`` and ``diff`` to analyze the changes
3. Skips commits that are similar with only minor differences (line numbers,
context, conflict resolutions)
4. Proceeds with commits that differ significantly in actual changes
**Fallback Detection**
If pre-detection missed something and the first cherry-pick fails with
conflicts, the agent performs the same subject search and patch comparison
process. If all commits in the set are verified as already applied, the agent:
1. Aborts the cherry-pick
2. Writes a signal file (``.pickman-signal``) with status ``already_applied``
3. Reports the situation
**What pickman does**
When pickman detects the ``already_applied`` signal or when the agent reports
pre-detected applied commits:
1. Marks all commits as 'skipped' in the database
2. Updates the source position to advance past these commits
3. Creates an MR with ``[skipped]`` prefix to record the attempt
4. The MR description explains that commits were already applied
This ensures:
- There's a record of what was attempted
- The source position advances so the next ``poll`` iteration processes new
commits
- No manual intervention is required to continue
- False positives are minimized by comparing actual patch content
CI Pipelines
------------
Pickman manages CI pipelines to avoid unnecessary duplicate runs. GitLab
automatically triggers an MR pipeline whenever the source branch is updated,
so pickman skips the push pipeline to avoid running two pipelines.
**How it works**
When pushing a branch (for new MRs or updates), pickman uses ``-o ci.skip``
to skip the push pipeline. GitLab then triggers an MR pipeline when it
detects the branch update on the merge request. This ensures exactly one
pipeline runs for each push.
**Summary**
=============================== ================ ==============================
Action Pipeline Skipped Reason
=============================== ================ ==============================
Initial branch push for new MR Yes MR creation triggers pipeline
Push after rebase/review Yes MR update triggers pipeline
=============================== ================ ==============================
Usage
-----
To add a source branch to track::
./tools/pickman/pickman add-source us/next
This finds the merge-base commit between the master branch (ci/master) and the
source branch, and stores it in the database as the starting point for
cherry-picking.
To list all tracked source branches::
./tools/pickman/pickman list-sources
To compare branches and show commits that need to be cherry-picked::
./tools/pickman/pickman compare
This shows:
- The number of commits in the source branch (us/next) that are not in the
master branch (ci/master)
- The last common commit between the two branches
To check current branch for problematic cherry-picks::
./tools/pickman/pickman check
This analyzes commits on the current branch and identifies cherry-picks with
large deltas compared to their original commits. By default, it:
- Shows only problematic commits (above 20% delta threshold)
- Ignores small commits (less than 10 lines changed)
- Skips merge commits (which have different delta characteristics)
- Uses color coding: red for ≥80% delta, yellow for ≥50% delta
Options:
- ``-t, --threshold``: Delta threshold as fraction (default: 0.2 = 20%)
- ``-m, --min-lines``: Minimum lines changed to check (default: 10)
- ``-v, --verbose``: Show detailed analysis for all commits
- ``--diff``: Show patch differences for problem commits
Example output::
Cherry-pick Delta% Original Subject
----------- ------ ---------- -------
aaea489b2a 100 9bab7d2a7c net: wget: let wget_with_dns work with dns disabled
e557daec17 89 f0315babfb hash: Plumb crc8 into the hash functions
2 problem commit(s) found
This helps identify cherry-picks that may have been applied incorrectly or
need manual review due to significant differences from the original commits.
To check GitLab permissions for the configured token::
./tools/pickman/pickman check-gitlab
This verifies that the GitLab token has the required permissions to push
branches and create merge requests. Use ``-r`` to specify a different remote.
To show the next set of commits to cherry-pick from a source branch::
./tools/pickman/pickman next-set us/next
This finds commits between the last cherry-picked commit and the next merge
commit in the source branch. It stops at the merge commit since that typically
represents a logical grouping of commits (e.g., a pull request).
To count the total remaining merges to process::
./tools/pickman/pickman count-merges us/next
This shows how many merge commits remain on the first-parent chain between the
last cherry-picked commit and the source branch tip.
To show the next N merges that will be applied::
./tools/pickman/pickman next-merges us/next
This shows the upcoming merge commits on the first-parent chain, useful for
seeing what's coming up. Use ``-c`` to specify the count (default 10).
Each merge shows its processing status:
- ``[DONE]``: All commits have been applied (MR merged)
- ``[IN MR]``: All commits are in an open MR (pending review/merge)
- ``[N/M to pick]``: N of M commits still need to be cherry-picked
Example output::
Next 5 merges from us/next:
1. ea3324b3444 Merge tag 'u-boot-imx-master-20250116' [DONE]
2. c11dc783fb4 Merge https://.../u-boot-riscv [IN MR]
3. 639cd409987 Merge patch series "add sha256_hmac" [IN MR]
4. 9bfc72e3b2a Merge tag 'u-boot-imx-master-20250120' [16/16 to pick]
5. 5d5ce5f2570 Merge patch series "Rework BLK" [6/6 to pick]
This helps identify which merges have already been processed and which are
next in line for ``poll`` or ``apply``.
To apply the next set of commits using a Claude agent::
./tools/pickman/pickman apply us/next
This uses the Claude Agent SDK to automate the cherry-pick process. The agent
will:
- Run git status to check the repository state
- Cherry-pick each commit in order
- Handle simple conflicts automatically
- Report status after completion
To push the branch and create a GitLab merge request::
./tools/pickman/pickman apply us/next -p
Options for the apply command:
- ``-b, --branch``: Branch name to create (default: cherry-<hash>)
- ``-p, --push``: Push branch and create GitLab MR
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)
On successful cherry-pick, an entry is appended to ``.pickman-history`` with:
- Date and source branch
- Branch name and list of commits
- The agent's conversation log
This file is committed automatically and included in the MR description when
using ``-p``.
After successfully applying commits, update the database to record progress::
./tools/pickman/pickman commit-source us/next <commit-hash>
This updates the last cherry-picked commit for the source branch, so subsequent
``next-set`` and ``apply`` commands will start from the new position.
To check open MRs for comments and address them::
./tools/pickman/pickman review
This lists open pickman MRs (those with ``[pickman]`` in the title), checks each
for unresolved comments, and uses a Claude agent to address them. The agent will:
- Make code changes based on the feedback
- Create a local branch with version suffix (e.g., ``cherry-abc123-v2``)
- Force push to the original remote branch to update the existing MR
- Use ``--keep-empty`` when rebasing to preserve empty merge commits
After processing, pickman:
- Marks comments as processed in the database (to avoid reprocessing)
- Updates the MR description with the agent's conversation log
- Appends the review handling to ``.pickman-history``
Options for the review command:
- ``-r, --remote``: Git remote (default: ci)
To automatically create an MR if none is pending::
./tools/pickman/pickman step us/next
This command performs the following:
1. Checks for merged pickman MRs and updates the database with the last
cherry-picked commit from each merged MR
2. Checks for open pickman MRs (those with ``[pickman]`` in the title)
3. If open MRs exist, processes any review comments using Claude agent
4. If open MRs are below ``--max-mrs`` limit, runs ``apply`` with ``--push``
to create a new one
This is useful for automated workflows. The ``--max-mrs`` option controls how
many MRs can be open simultaneously (default: 5), allowing parallel review of
multiple cherry-pick sets. The automatic database update on merge means you
don't need to manually run ``commit-source`` after each MR is merged, and
review comments are handled automatically.
Options for the step command:
- ``-m, --max-mrs``: Maximum open MRs allowed (default: 5)
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)
To run step continuously in a polling loop::
./tools/pickman/pickman poll us/next
This runs the ``step`` command repeatedly with a configurable interval,
creating new MRs as previous ones are merged. Press Ctrl+C to stop.
Options for the poll command:
- ``-i, --interval``: Interval between steps in seconds (default: 300)
- ``-m, --max-mrs``: Maximum open MRs allowed (default: 5)
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)
To push a branch using the GitLab API token for authentication::
./tools/pickman/pickman push-branch <branch-name>
This is useful when you want commits to appear as coming from the token owner
(e.g., a pickman bot account) rather than the user's configured git credentials.
The agent uses this command automatically when pushing review changes.
Options for the push-branch command:
- ``-r, --remote``: Git remote (default: ci)
- ``-f, --force``: Force push (overwrite remote branch)
To list pickman merge requests from GitLab::
./tools/pickman/pickman list-mrs
This shows all pickman MRs (those with ``[pickman]`` in the title) along with
their commit count and status flags.
Options for the list-mrs command:
- ``-r, --remote``: Git remote (default: ci)
- ``-s, --state``: MR state to list: opened, merged, closed, all (default: opened)
To list commits in the database that have no MR link::
./tools/pickman/pickman list-orphans
This shows commits that were recorded in the database but weren't linked to a
merge request. This can happen when commits were applied before the MR tracking
was added.
Options for the list-orphans command:
- ``-s, --status``: Filter by status: pending, applied, all (default: all)
To link orphan commits to their MRs by matching subjects::
./tools/pickman/pickman link-orphans
This fetches merged MRs from GitLab and matches orphan commits by their subject
lines. It shows which commits can be linked to which MRs, with color-coded
status showing whether commits are in the target tree.
Options for the link-orphans command:
- ``-a, --show-all``: Show merged commits in unlinked section
- ``-n, --dry-run``: Show what would be linked without updating
- ``-r, --remote``: Git remote (default: ci)
- ``-v, --verbose``: Show individual commits under each MR
To group orphan commits by their parent merge on the source branch::
./tools/pickman/pickman group-orphans
This uses git to find which merge commit on us/next each orphan belongs to,
helping to visualize which commits should be processed together. The output
shows:
- Commits grouped by their parent merge, with the merge subject
- Color-coded status (green for applied, yellow for pending)
- Ungrouped commits that couldn't be matched to a merge
- Merge commits among the orphans
Options for the group-orphans command:
- ``-s, --status``: Filter by status: pending, applied, all (default: all)
Example output::
Orphan commits grouped by merge (44 merges):
aa482995a81a: [pending] 10 commits - Merge patch series "test: Tidy up"
[pending] a8c07ac4970d alist: Mention the error condition
[pending] 13bd29849953 alist: Add a comment for alist_init_struct()
...
079214e4beea: [applied] 10 commits - Merge patch series "SMBIOS"
[applied] 8aa5f8e02f78 smbios: Refactor the smbios headfile
...
Ungrouped (42 commits):
[applied] 06b8aafd6810 fastboot: properly handle unknown partition
Syncing Manually Applied Commits
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If commits are cherry-picked outside of pickman (e.g., manually using git),
pickman won't know about them. The ``sync`` command scans all pending commits
in the database and marks any that have been applied to the target branch::
./tools/pickman/pickman sync
This checks each pending commit's subject against ci/master and marks matches
as 'applied' with the corresponding cherry-pick hash.
Options for the sync command:
- ``-n, --dry-run``: Show what would be synced without updating the database
- ``-v, --verbose``: Show details for each synced commit
Example::
./tools/pickman/pickman sync -v
Checking 15 pending commits for 'us/next'...
Found: abc123def456 -> xyz789
Add feature X
Synced 1 commits for us/next
Requirements
------------
To use the ``apply`` command, install the Claude Agent SDK::
pip install claude-agent-sdk
You will also need an Anthropic API key set in the ``ANTHROPIC_API_KEY``
environment variable.
To use the ``-p`` (push) option for GitLab integration, install python-gitlab::
pip install python-gitlab
You will also need a GitLab API token. The token can be configured in a config
file or environment variable. Pickman checks in this order:
1. Config file ``~/.config/pickman.conf``::
[gitlab]
token = glpat-xxxxxxxxxxxxxxxxxxxx
2. ``GITLAB_TOKEN`` environment variable
3. ``GITLAB_API_TOKEN`` environment variable
See `GitLab Personal Access Tokens`_ for instructions on creating a token.
The token needs ``api`` and ``write_repository`` scopes. Using a dedicated bot
account for pickman is recommended - this ensures all commits pushed by pickman
appear as coming from the bot account rather than individual users.
.. _GitLab Personal Access Tokens:
https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
Database
--------
Pickman uses a sqlite3 database (``.pickman.db``) to track state. The schema
version is stored in the ``schema_version`` table and migrations are applied
automatically when the database is opened.
Tables
~~~~~~
**source**
Tracks source branches and their cherry-pick progress.
- ``id``: Primary key
- ``name``: Branch name (e.g., 'us/next')
- ``last_commit``: Hash of the last commit cherry-picked from this branch
**pcommit**
Tracks individual commits being cherry-picked.
- ``id``: Primary key
- ``chash``: Original commit hash
- ``source_id``: Foreign key to source table
- ``mergereq_id``: Foreign key to mergereq table (optional)
- ``subject``: Commit subject line
- ``author``: Commit author
- ``status``: One of 'pending', 'applied', 'skipped', 'conflict'
- ``cherry_hash``: Hash of the cherry-picked commit (if applied)
**mergereq**
Tracks merge requests created for cherry-picked commits.
- ``id``: Primary key
- ``source_id``: Foreign key to source table
- ``branch_name``: Git branch name for this MR
- ``mr_id``: GitLab merge request ID
- ``status``: One of 'open', 'merged', 'closed'
- ``url``: URL to the merge request
- ``created_at``: Timestamp when the MR was created
**comment**
Tracks MR comments that have been processed by the review agent.
- ``id``: Primary key
- ``mr_iid``: GitLab merge request IID
- ``comment_id``: GitLab comment/note ID
- ``processed_at``: Timestamp when the comment was processed
Commit Statuses
~~~~~~~~~~~~~~~
The ``status`` field in the pcommit table tracks the state of each commit:
**pending**
The commit has been added to the database and is part of an open MR that
hasn't been merged yet. This is the initial status when commits are first
processed by the ``apply`` command.
**applied**
The commit has been successfully cherry-picked and the MR containing it
has been merged into the target branch. The ``cherry_hash`` field contains
the hash of the resulting cherry-picked commit.
**skipped**
The commit was intentionally skipped, either because:
- A reviewer commented ``pickman skip`` on the MR
- The commit was detected as already applied to the target branch
- The commit couldn't be cherry-picked and was manually skipped
**conflict**
The cherry-pick failed due to merge conflicts that couldn't be resolved
automatically by the agent. Manual intervention may be required.
Configuration
-------------
The branches to compare are configured as constants in control.py:
- ``BRANCH_MASTER``: The main branch to compare against (default: ci/master)
- ``BRANCH_SOURCE``: The source branch with commits to cherry-pick
(default: us/next)
Testing
-------
To run the functional tests::
./tools/pickman/pickman test
Or using pytest::
python3 -m pytest tools/pickman/ftest.py -v
8083bdd0b4