Running the full test suite takes a long time. It would be useful to
distribute tests across multiple sandbox instances to speed up testing.
Add support for running tests in parallel across multiple sandbox
instances. Each worker runs a subset of tests based on its worker ID.
Add -P<n>:<w> option to the ut command where n is the total number of
workers and w is this worker's ID (0 to n-1). Tests are distributed
by index modulo number of workers.
Series-to: u-boot
Series-cc: heinrich
Cover-letter:
test: Various improvements to unit-test infrastructure
This series adds several improvements to the unit-test infrastructure:
- Move disk images to the persistent-data directory so they don't
pollute the source tree
- Add a way to keep pytest-created artefacts for faster iteration on
C tests
- Add a helper to simplify running manual unit tests from Python
- Allow combined flags with the ut command (e.g. -Efm)
- Add a -E flag to emit machine-readable result lines
- Add a -P flag to distribute tests across parallel workers
- Add -m as an alias for -f (force manual tests)
These changes make it easier to run and debug tests, especially when
iterating on C test code.
END
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
When iterating on C test code, the Python fixtures that create disk
images run each time, even though the images have not changed. This
slows down the development cycle unnecessarily.
Add a -P/--persist option to prevent cleanup of test-generated files
like disk images. This allows re-running C tests directly, without
re-running the Python fixture each time.
Update the ext4l test to respect the persist flag.
Co-developed-by: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Simon Glass <simon.glass@canonical.com>
Now that these files are within the repo, link the documentation to the
normal U-Boot documentation. Update directory paths and convert it to
rST format.
Signed-off-by: Simon Glass <sjg@chromium.org>
Using some form of sandbox with Python modules is a long standing best
practice with the language. There are a number of ways to have a Python
sandbox be created. At this point in time, it seems the Python community
is moving towards using the "venv" module provided with Python rather
than a separate tool. To match that we make the following changes:
- Refer to a "Python sandbox" rather than virtualenv in comments, etc.
- Install the python3-venv module in our container and not virtualenv.
- In our CI files, invoke "python -m venv" rather than "virtualenv".
- In documentation, tell users to install python3-venv and not
virtualenv.
Signed-off-by: Tom Rini <trini@konsulko.com>
We know this is U-Boot so the prefix serves no purpose other than to
make things longer and harder to read. Drop it and rename the files.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com> # test_android / test_dfu
This fixture name is quite long and results in lots of verbose code.
We know this is U-Boot so the 'u_boot_' part is not necessary.
But it is also a bit of a misnomer, since it provides access to all the
information available to tests. It is not just the console.
It would be too confusing to use con as it would be confused with
config and it is probably too short.
So shorten it to 'ubman'.
Signed-off-by: Simon Glass <sjg@chromium.org>
Link: https://lore.kernel.org/u-boot/CAFLszTgPa4aT_J9h9pqeTtLCVn4x2JvLWRcWRD8NaN3uoSAtyA@mail.gmail.com/
Execution time varies widely with the existing tests. Provides a way to
produce a summary of the time taken for each test, along with a
histogram.
This is enabled with the --timing flag.
Enable it for sandbox in CI.
Example:
Duration : Number of tests
======== : ========================================
<1ms : 1
<8ms : 1
<20ms : # 20
<30ms : ######## 127
<50ms : ######################################## 582
<75ms : ####### 102
<100ms : ## 39
<200ms : ##### 86
<300ms : # 29
<500ms : ## 42
<750ms : # 16
<1.0s : # 15
<2.0s : # 23
<3.0s : 13
<5.0s : 9
<7.5s : 1
<10.0s : 6
<20.0s : 12
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
We want tests to avoid the usage of sudo. Describe that virt-make-fs can
generate disk images without being root.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Running tests in parallel is much faster, e.g. 15 seconds to run the tests
on sandbox (only), instead of 100 seconds (on a 16-core machine). Add a
'make pcheck' option to access this feature.
Note that the tools/ tests still run each tool's tests once after the
other, although within that, they do run in parallel. So for example,
the buildman tests run in parallel, then the binman tests run in
parallel. There would be a signiificant advantage to running them all
in parallel together, but that would require a large amount of
refactoring, e.g. with more use of pytest fixtures.
Update the documentation to represent the current state.
Signed-off-by: Simon Glass <sjg@chromium.org>
When a test hands on a real board there is no way on the console to obtain
any information about why it hung.
With sandbox we can actually find out that it died and get a signal or
exit code. Add this to make it easier to figure out what happened.
So instead of:
test/py/u_boot_spawn.py:171: in expect
c = os.read(self.fd, 1024).decode(errors='replace')
E OSError: [Errno 5] Input/output error
We get:
test/py/u_boot_spawn.py:171: in expect
c = os.read(self.fd, 1024).decode(errors='replace')
E ValueError: U-Boot exited with signal 11 (Signals.SIGSEGV)
Signed-off-by: Simon Glass <sjg@chromium.org>
This produces a lot of code output which is not very helpful and is quite
annoying to wade through. Use the short format by default.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a second on writing tests, covering when to use Python and C, where
to put the tests, etc. Add a link to the existing Python test
documentation.
Signed-off-by: Simon Glass <sjg@chromium.org>
As Stephen is no longer actively maintaining the uboot-test-hooks
repository, switch to using the instance on our GitLab.
Acked-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Convert test/py/README.md to restructured text and add it to the generated
HTML documentation.
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
