u-boot/tools/pickman

.. SPDX-License-Identifier: GPL-2.0+
..
.. Copyright 2025 Canonical Ltd.
.. Written by Simon Glass <simon.glass@canonical.com>

Pickman - Cherry-pick Manager
=============================

Pickman is a tool to help manage cherry-picking commits between branches.

Workflow
--------

The typical workflow for using pickman is:

1. **Setup**: Add a source branch to track with ``add-source``. This records the
   starting point (merge-base) for cherry-picking.

2. **Cherry-pick**: Run ``apply -p`` to cherry-pick the next set of commits (up
   to the next merge commit) and create a GitLab MR. A Claude agent handles the
   cherry-picks automatically, resolving simple conflicts.

3. **Review**: Once the MR is reviewed and merged, run ``commit-source`` to
   update the database with the last processed commit.

4. **Repeat**: Go back to step 2 until all commits are cherry-picked.

For fully automated workflows, use ``poll`` which runs ``step`` in a loop. The
``step`` command handles the complete cycle automatically:

- Detects merged MRs and updates the database (no manual ``commit-source``)
- Processes review comments on open MRs using Claude agent
- Creates new MRs when none are pending

This allows hands-off operation: just run ``poll`` and approve/merge MRs in
GitLab as they come in.

Commit Selection
----------------

When pickman creates an MR, it groups commits into logical sets based on merge
commits in the source branch. Understanding this helps predict what will be
included in each MR.

**Algorithm**

1. **Start from the last processed commit**: Pickman reads from its database the
   hash of the last commit that was successfully cherry-picked from the source
   branch.

2. **Find the next merge on first-parent chain**: Walking forward from the last
   processed commit along the first-parent chain (``git log --first-parent``),
   pickman finds the next merge commit. The first-parent chain represents the
   mainline history of the branch.

3. **Include all commits up to that merge**: Using ``git log`` (without
   ``--first-parent``), pickman collects ALL commits between the last processed
   commit and the merge commit. This includes:

   - Commits on the mainline leading up to the merge
   - Commits brought in by the merge (from the merged branch)
   - The merge commit itself

**Example**

Consider this history on the source branch::

    * 5c8ef70 Merge tag 'xilinx-for-v2025.01-rc5-v2'
    |\
    | * 1b70b6c common: memtop: Fix the return type
    |/
    * c06705a Makefile: Match the full path to ccache
    * 0b7f4c7 imx: Fix usable memory ranges
    * ff1d5d8 Revert "configs: JH7110: enable EFI_LOADER"
    * d701c6a net: lwip: check if network device is available
    * b6691d0 net: lwip: do not return CMD_RET_USAGE
    * 9378307 binman: Regenerate tools/binman/entries.rst  <-- last processed

If the database shows ``9378307`` as the last processed commit, pickman will:

1. Walk first-parent from ``9378307`` and find merge ``5c8ef70``
2. Collect all commits in ``9378307..5c8ef70``:

   - ``b6691d0`` net: lwip: do not return CMD_RET_USAGE
   - ``d701c6a`` net: lwip: check if network device is available
   - ``ff1d5d8`` Revert "configs: JH7110..."
   - ``0b7f4c7`` imx: Fix usable memory ranges
   - ``c06705a`` Makefile: Match the full path to ccache
   - ``1b70b6c`` common: memtop: Fix the return type (from xilinx branch)
   - ``5c8ef70`` Merge tag 'xilinx-for-v2025.01-rc5-v2'

The resulting MR contains 7 commits. The branch name is derived from the first
commit's short hash: ``cherry-b6691d0``.

**Why merge-based grouping?**

Merge commits typically represent logical units of work (e.g., a pull request
or a subsystem update). By stopping at each merge, pickman:

- Keeps MRs focused and reviewable
- Preserves the original grouping from upstream
- Makes it easier to identify and skip problematic sets

**No merge found**

If there are no merge commits between the last processed commit and the branch
tip, pickman includes all remaining commits in a single set. This is noted in
the output as "no merge found".

Skipping MRs
------------

During review, if a set of commits should be skipped (e.g., not applicable to
the target branch), a reviewer can comment:

- ``pickman skip``
- ``pickman: skip``
- ``@pickman skip``
- ``@pickman: skip``

Pickman will add ``[skipped]`` to the MR title. Skipped MRs:

- Are ignored when deciding whether to create new MRs
- Don't block the ``step`` or ``poll`` commands from proceeding
- Can be unskipped by commenting ``pickman unskip``

CI Pipelines
------------

Pickman manages CI pipelines to avoid unnecessary runs while ensuring changes
are properly verified.

**Initial MR creation**

When creating a new MR (via ``apply -p`` or ``step``), pickman pushes the
branch with ``-o ci.skip``. This skips the push pipeline because GitLab
automatically triggers an MR pipeline when the merge request is created.
Without this, two pipelines would run: one for the push and one for the MR.

**Review comment handling**

When pushing changes after addressing review comments (via ``review``,
``step``, or ``poll``), pickman does NOT skip the pipeline. A new pipeline
is needed to verify that the changes made in response to review feedback
are correct.

**Summary**

===============================  ================  =========================
Action                           Pipeline Skipped  Reason
===============================  ================  =========================
Initial branch push for new MR   Yes               MR creation triggers one
Push after review changes        No                Need to verify changes
===============================  ================  =========================

Usage
-----

To add a source branch to track::

    ./tools/pickman/pickman add-source us/next

This finds the merge-base commit between the master branch (ci/master) and the
source branch, and stores it in the database as the starting point for
cherry-picking.

To list all tracked source branches::

    ./tools/pickman/pickman list-sources

To compare branches and show commits that need to be cherry-picked::

    ./tools/pickman/pickman compare

This shows:

- The number of commits in the source branch (us/next) that are not in the
  master branch (ci/master)
- The last common commit between the two branches

To check GitLab permissions for the configured token::

    ./tools/pickman/pickman check-gitlab

This verifies that the GitLab token has the required permissions to push
branches and create merge requests. Use ``-r`` to specify a different remote.

To show the next set of commits to cherry-pick from a source branch::

    ./tools/pickman/pickman next-set us/next

This finds commits between the last cherry-picked commit and the next merge
commit in the source branch. It stops at the merge commit since that typically
represents a logical grouping of commits (e.g., a pull request).

To count the total remaining merges to process::

    ./tools/pickman/pickman count-merges us/next

This shows how many merge commits remain on the first-parent chain between the
last cherry-picked commit and the source branch tip.

To show the next N merges that will be applied::

    ./tools/pickman/pickman next-merges us/next

This shows the upcoming merge commits on the first-parent chain, useful for
seeing what's coming up. Use ``-c`` to specify the count (default 10).

To apply the next set of commits using a Claude agent::

    ./tools/pickman/pickman apply us/next

This uses the Claude Agent SDK to automate the cherry-pick process. The agent
will:

- Run git status to check the repository state
- Cherry-pick each commit in order
- Handle simple conflicts automatically
- Report status after completion

To push the branch and create a GitLab merge request::

    ./tools/pickman/pickman apply us/next -p

Options for the apply command:

- ``-b, --branch``: Branch name to create (default: cherry-<hash>)
- ``-p, --push``: Push branch and create GitLab MR
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)

On successful cherry-pick, an entry is appended to ``.pickman-history`` with:

- Date and source branch
- Branch name and list of commits
- The agent's conversation log

This file is committed automatically and included in the MR description when
using ``-p``.

After successfully applying commits, update the database to record progress::

    ./tools/pickman/pickman commit-source us/next <commit-hash>

This updates the last cherry-picked commit for the source branch, so subsequent
``next-set`` and ``apply`` commands will start from the new position.

To check open MRs for comments and address them::

    ./tools/pickman/pickman review

This lists open pickman MRs (those with ``[pickman]`` in the title), checks each
for unresolved comments, and uses a Claude agent to address them. The agent will:

- Make code changes based on the feedback
- Create a local branch with version suffix (e.g., ``cherry-abc123-v2``)
- Force push to the original remote branch to update the existing MR
- Use ``--keep-empty`` when rebasing to preserve empty merge commits

After processing, pickman:

- Marks comments as processed in the database (to avoid reprocessing)
- Updates the MR description with the agent's conversation log
- Appends the review handling to ``.pickman-history``

Options for the review command:

- ``-r, --remote``: Git remote (default: ci)

To automatically create an MR if none is pending::

    ./tools/pickman/pickman step us/next

This command performs the following:

1. Checks for merged pickman MRs and updates the database with the last
   cherry-picked commit from each merged MR
2. Checks for open pickman MRs (those with ``[pickman]`` in the title)
3. If open MRs exist, processes any review comments using Claude agent
4. If open MRs are below ``--max-mrs`` limit, runs ``apply`` with ``--push``
   to create a new one

This is useful for automated workflows. The ``--max-mrs`` option controls how
many MRs can be open simultaneously (default: 5), allowing parallel review of
multiple cherry-pick sets. The automatic database update on merge means you
don't need to manually run ``commit-source`` after each MR is merged, and
review comments are handled automatically.

Options for the step command:

- ``-m, --max-mrs``: Maximum open MRs allowed (default: 5)
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)

To run step continuously in a polling loop::

    ./tools/pickman/pickman poll us/next

This runs the ``step`` command repeatedly with a configurable interval,
creating new MRs as previous ones are merged. Press Ctrl+C to stop.

Options for the poll command:

- ``-i, --interval``: Interval between steps in seconds (default: 300)
- ``-m, --max-mrs``: Maximum open MRs allowed (default: 5)
- ``-r, --remote``: Git remote for push (default: ci)
- ``-t, --target``: Target branch for MR (default: master)

To push a branch using the GitLab API token for authentication::

    ./tools/pickman/pickman push-branch <branch-name>

This is useful when you want commits to appear as coming from the token owner
(e.g., a pickman bot account) rather than the user's configured git credentials.
The agent uses this command automatically when pushing review changes.

Options for the push-branch command:

- ``-r, --remote``: Git remote (default: ci)
- ``-f, --force``: Force push (overwrite remote branch)

Requirements
------------

To use the ``apply`` command, install the Claude Agent SDK::

    pip install claude-agent-sdk

You will also need an Anthropic API key set in the ``ANTHROPIC_API_KEY``
environment variable.

To use the ``-p`` (push) option for GitLab integration, install python-gitlab::

    pip install python-gitlab

You will also need a GitLab API token. The token can be configured in a config
file or environment variable. Pickman checks in this order:

1. Config file ``~/.config/pickman.conf``::

       [gitlab]
       token = glpat-xxxxxxxxxxxxxxxxxxxx

2. ``GITLAB_TOKEN`` environment variable
3. ``GITLAB_API_TOKEN`` environment variable

See `GitLab Personal Access Tokens`_ for instructions on creating a token.
The token needs ``api`` and ``write_repository`` scopes. Using a dedicated bot
account for pickman is recommended - this ensures all commits pushed by pickman
appear as coming from the bot account rather than individual users.

.. _GitLab Personal Access Tokens:
   https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Database
--------

Pickman uses a sqlite3 database (``.pickman.db``) to track state. The schema
version is stored in the ``schema_version`` table and migrations are applied
automatically when the database is opened.

Tables
~~~~~~

**source**
    Tracks source branches and their cherry-pick progress.

    - ``id``: Primary key
    - ``name``: Branch name (e.g., 'us/next')
    - ``last_commit``: Hash of the last commit cherry-picked from this branch

**pcommit**
    Tracks individual commits being cherry-picked.

    - ``id``: Primary key
    - ``chash``: Original commit hash
    - ``source_id``: Foreign key to source table
    - ``mergereq_id``: Foreign key to mergereq table (optional)
    - ``subject``: Commit subject line
    - ``author``: Commit author
    - ``status``: One of 'pending', 'applied', 'skipped', 'conflict'
    - ``cherry_hash``: Hash of the cherry-picked commit (if applied)

**mergereq**
    Tracks merge requests created for cherry-picked commits.

    - ``id``: Primary key
    - ``source_id``: Foreign key to source table
    - ``branch_name``: Git branch name for this MR
    - ``mr_id``: GitLab merge request ID
    - ``status``: One of 'open', 'merged', 'closed'
    - ``url``: URL to the merge request
    - ``created_at``: Timestamp when the MR was created

**comment**
    Tracks MR comments that have been processed by the review agent.

    - ``id``: Primary key
    - ``mr_iid``: GitLab merge request IID
    - ``comment_id``: GitLab comment/note ID
    - ``processed_at``: Timestamp when the comment was processed

    This table prevents the same comment from being addressed multiple times
    when running ``review`` or ``poll`` commands.

Configuration
-------------

The branches to compare are configured as constants in control.py:

- ``BRANCH_MASTER``: The main branch to compare against (default: ci/master)
- ``BRANCH_SOURCE``: The source branch with commits to cherry-pick
  (default: us/next)

Testing
-------

To run the functional tests::

    ./tools/pickman/pickman test

Or using pytest::

    python3 -m pytest tools/pickman/ftest.py -v