Running the full test suite takes a long time. It would be useful to
distribute tests across multiple sandbox instances to speed up testing.
Add support for running tests in parallel across multiple sandbox
instances. Each worker runs a subset of tests based on its worker ID.
Add -P<n>:<w> option to the ut command where n is the total number of
workers and w is this worker's ID (0 to n-1). Tests are distributed
by index modulo number of workers.
Series-to: u-boot
Series-cc: heinrich
Cover-letter:
test: Various improvements to unit-test infrastructure
This series adds several improvements to the unit-test infrastructure:
- Move disk images to the persistent-data directory so they don't
pollute the source tree
- Add a way to keep pytest-created artefacts for faster iteration on
C tests
- Add a helper to simplify running manual unit tests from Python
- Allow combined flags with the ut command (e.g. -Efm)
- Add a -E flag to emit machine-readable result lines
- Add a -P flag to distribute tests across parallel workers
- Add -m as an alias for -f (force manual tests)
These changes make it easier to run and debug tests, especially when
iterating on C test code.
END
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
The ut command shows test output but does not provide a machine-readable
indication of whether each individual test passed or failed. External
tools must rely on heuristics like scanning for failure patterns in the
output.
Add a -E flag that emits an explicit result line after each test:
Result: PASS: test_name: file.c
Result: FAIL: test_name: file.c
This allows tools to reliably determine per-test pass/fail status
without fragile pattern matching. The flag is optional to maintain
backward compatibility with existing scripts.
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
The current argument-parsing logic uses switch (str[1]) which only
processes the second character of each argument. This prevents combining
multiple single-character flags in one argument (e.g., -fm).
Refactor the code to use a for loop that iterates through all characters
in the argument. For flags that take a value (like -r and -I), use goto
to skip the rest of the argument after processing.
This allows combined flags like -fmR instead of requiring -f -m -R.
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Add -m as an alias for the -f flag which forces manual tests to run.
This provides consistency with external test runners that will use -m
for "manual" tests.
Also update the documentation to explain what manual tests are, and fix
a typo ("types" -> "times") in the -r description.
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Add C implementations of filesystem tests that can be called via
the 'ut fs' command. These tests use UTF_MANUAL flag since they require
external setup, i.e. creation of filesystem images.
This covers the existing TestFsBasic tests.
The tests use typed arguments (fs_type, fs_image, md5 values) passed
via the command line.
Add a few helpers to make the code easier to read.
Co-developed-by: Claude <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Update the 'ut' command to have a -R command option to prevent ut_fail()
and ut_failf() from clearing GD_FLG_RECORD. This is useful when testing
the test framework itself, where error messages need to be captured.
Refactor ut_fail() and ut_failf() to call ut_unsilence_console() instead
of duplicating the flag-clearing logic.
Co-developed-by: Claude <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Update the ut command to permit passing arguments to tests.
Usage: ut -f fs test_name key1=value1 key2=value2
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Add support for passing key=value arguments to unit tests. The test
framework parses arguments based on definitions provided by each test
and makes them available via uts->args[]
For now the 'ut' command does not support this new feature.
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
When a unit test fails, it returns the error code, which results in a
warning:
exit not allowed from main input shell.
Fix it by returning a failure code instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the 'ut' command handles its flags in a strange way, in that
they must come after the subcommand.
So, we must use 'ut bloblist -r2' to run the bloblist tests twice. This
is an artefact of the way tests were run, through subcommands.
It is now possible to correct this, by doing flag-processing before
running the suite.
Update the code to handle this, so that 'ut -r2 bloblist' works. Update
the 'test_suite' test to check the new arguments.
Add a sanity-check for -I while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
Move the logic from this function into run_suite(), on the way to having
flag parsing in the top-level 'ut' command instead of its children.
Signed-off-by: Simon Glass <sjg@chromium.org>
Enhance the ut command to accept a comma-separated list of test suites
to run. Report the summary information for these at the end.
Signed-off-by: Simon Glass <sjg@chromium.org>
Now that everything is using the new test-suite features, drop support
for running commands.
Fix a missing closing-bracket while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
When tests are all in the same suite it is annoying to have to read all
the common text after each name. Skip this to help the user.
Signed-off-by: Simon Glass <sjg@chromium.org>
Show the average duration of a test, so we can keep track of how it is
trending. Report the suite with the longest average test to encourage
people to improve it.
Add a function to update the stats based on the results from a single
suite and another to show the summary information.
Make this optional, since sandbox's SPL tests do not have a timer driver
and people may want to print results without times.
Signed-off-by: Simon Glass <sjg@chromium.org>
All tests should belong to a suite, but if there is a suite we don't
know about (e.g. not added to cmd_ut.c) then the totals will not add up.
Add a check for this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Rather than having the help in the longhelp, put it in the suite info
so 'ut info -s' can show it. This is tidier, particular due to the
removal of #ifdefs
This means that the help text is present in the image (although not
displayed with 'ut info -s') so the image-size increases. But with
UNIT_TEST enabled, we expect large images so this doesn't seem
important.
Signed-off-by: Simon Glass <sjg@chromium.org>
Put the suites in order by name, for easier code-maintenance. This also
helps find test results for a particular swuit in the 'ut all' output.
Signed-off-by: Simon Glass <sjg@chromium.org>
Update this function to access a unit-test state, so that the caller can
collect results from running multiple suites.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is not needed anymore. If a test suite is not built, then it will
have no linker-list entries. So we can just check for that and know that
the suite is not present.
This allows removal of the #ifdefs and the need to keep them in sync
with the associated Makefile rules, which has actually failed, since the
help does not match what commands are actually present.
Signed-off-by: Simon Glass <sjg@chromium.org>
The 'info' test is not a real test. With the new suite array we can drop
this and the associated special-case code.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
