scripts: Convert the build-qemu script to Python

Convert this to Python and make use of the build_helpers module. Update
that module to remove old options and improve the ordering of options.

The script doubles in size, part of which is being a lot more friendly
with virtiofsd problems, as well as adding a verbose mode.

Update the documentation as well.

Series-to: u-boot
Cover-letter:
script: Improve the QEMU scripts to Python
This series updates the two QEMU scripts available in U-Boot:

- Converts build-qemu to Python
- Converts build-efi to avoid sudo (unless SCT is used)
- Introduces a proper settings file shared between both scripts

It also includes a few documentation updates.

The next step is to get these scripts running in CI.
END

Signed-off-by: Simon Glass <sjg@chromium.org>

MAINTAINERS

@@ -1074,7 +1074,7 @@ F:	doc/develop/uefi/u-boot_on_efi.rst
 F:	drivers/block/efi-media-uclass.c
 F:	drivers/block/sb_efi_media.c
 F:	lib/efi/
-F:	scripts/build-efi.py
+F:	scripts/build-efi
 F:	test/dm/efi_media.c
 
 EFI LOGGING
@@ -1126,7 +1126,8 @@ S:	Maintained
 W:	https://docs.u-boot.org/en/latest/board/emulation/script.html
 F:	configs/qemu_x86*
 F:	doc/board/emulation/script.rst
-F:	scripts/build-qemu.sh
+F:	scripts/build-qemu
+F:	scripts/build_helper.py
 
 ENVIRONMENT
 M:	Joe Hershberger <joe.hershberger@ni.com>

doc/board/emulation/script.rst

@@ -3,25 +3,66 @@
 Script for building and running
 ===============================
 
-You may find the script `scripts/build-qemu.sh` helpful for building and testing
+You may find the script `scripts/build-qemu` helpful for building and testing
 U-Boot on QEMU.
 
-If uses a environment variables to control how it works:
+If uses a settings file `~/.u_boot_qemu` to control how it works:
 
-ubdir
+build_dir
     base directory for building U-Boot, with each board being in its own
     subdirectory
 
-imagedir
-    directory containing OS images, containin a subdirectory for each distro
-    type (e.g. ubuntu/
+image_dir
+    directory containing OS images, containing a subdirectory for each distro
+    type (e.g. `ubuntu/`)
+
+bzimage
+    path to a bzImage file to supply to boot x86 Linux
+
+efi_image_file
+    output filename for the disk image containing an EFI app / payload
+
+efi_dir
+    directory when pre-built UEFI images are kept, e.g. OVMF-pure-efi.i386.fd
+
+sct_dir
+    directory when the UEFI Self-Certification Test (SCT) is kept
+
+sct_mnt
+    temporary mount point for building SCT: note this requires sudo
+
+A sample file is written if you don't have one, e.g.::
+
+    # U-Boot QEMU-scripts config
+
+    [DEFAULT]
+    # Set to the build directory where you build U-Boot out-of-tree
+    # We avoid in-tree build because it gets confusing trying different builds
+    # Each board gets a build in a separate subdir
+    build_dir = /tmp/b
+
+    # Image directory (for OS images)
+    image_dir = ~/dev/os
+
+    # Build the kernel with: make O=/tmp/kernel
+    bzimage = /tmp/kernel/arch/x86/boot/bzImage
+
+    # EFI image-output filename
+    efi_image_file = try.img
+
+    # Directory where OVMF-pure-efi.i386.fd etc. are kept
+    efi_dir = ~/dev/efi
+
+    # Directory where SCT image (sct.img) is kept
+    sct_dir = ~/dev/efi/sct
+
+    # Directory where the SCT image is temporarily mounted for modification
+    sct_mnt = /mnt/sct
+
 
 Once configured, you can build and run QEMU for arm64 like this::
 
-    scripts/build-qemu.sh -rsw
-
-No support is currently included for specifying a root disk, so this script can
-only be used to start installers.
+    scripts/build-qemu -rsw
 
 Options
 ~~~~~~~
@@ -34,6 +75,15 @@ Options are available to control the script:
 -B
     Don't build; assume a build exists
 
+-d/--disk DISK
+    Root disk image file to use with QEMU
+
+-e/--sct-run
+    Package an run UEFI Self-Certification Test (SCT)
+
+-E/--use-tianocore
+    Run Tianocore (OVMF) instead of U-Boot
+
 -k
     Use kvm - kernel-based Virtual Machine. By default QEMU uses its own
     emulator
@@ -52,10 +102,8 @@ Options are available to control the script:
 -s
     Use serial only (no display)
 
+-S/--sct-seq SCT_SEQ
+    SCT sequence-file to be written into the SCT image if -e
+
 -w
     Use word version (32-bit). By default, 64-bit is used
-
-.. note::
-
-    Note: For now this is a shell script, but if it expands it might be better
-    as Python, accepting the slower startup.

doc/develop/uefi/u-boot_on_efi.rst

@@ -97,9 +97,8 @@ that EFI does not support booting a 64-bit application from a 32-bit
 EFI (or vice versa). Also it will often fail to print an error message if
 you get this wrong.
 
-You may find the script `scripts/build-efi.py` helpful for building and testing
-U-Boot on UEFI on QEMU. It also includes links to UEFI binaries dating from
-2021.
+You may find the script `scripts/build-efi` helpful for building and testing
+U-Boot on UEFI on QEMU.
 
 See `Example run`_ for an example run.
 
@@ -202,7 +201,7 @@ Example run
 
 This shows running with serial enabled (see `include/configs/efi-x86_app.h`)::
 
-   $ scripts/build-efi.py -wsPr
+   $ scripts/build-efi -wsPr
    Packaging efi-x86_app32
    Running qemu-system-i386
 

scripts/build-efi

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # SPDX-License-Identifier: GPL-2.0+
 """
 Script to build an EFI thing suitable for booting with QEMU, possibly running
@@ -16,22 +16,10 @@ Use ~/.build-efi to configure the various paths used by this script.
 """
 
 from argparse import ArgumentParser
-import configparser
-import glob
 import os
-import re
 import shutil
-import sys
-import time
-
-
-OUR_PATH = os.path.dirname(os.path.realpath(__file__))
-OUR1_PATH = os.path.dirname(OUR_PATH)
-
-# Bring in the patman and dtoc libraries (but don't override the first path
-# in PYTHONPATH)
-sys.path.insert(2, os.path.join(OUR1_PATH, 'tools'))
 
+from build_helper import Helper
 
 # pylint: disable=C0413
 from u_boot_pylib import command
@@ -74,56 +62,15 @@ def parse_args():
     return args
 
 
-def get_settings():
-    """Get settings from the settings file
-
-    Return:
-        ConfigParser containing settings
-    """
-    settings = configparser.ConfigParser()
-    fname = f'{os.getenv("HOME")}/.build-efi'
-    if not os.path.exists(fname):
-        print('No config file found ~/.build-efi\nCreating one...\n')
-        tools.write_file(fname, '''[build-efi]
-# Mount path for the temporary image
-mount_point = /mnt/test-efi
-
-# Image-output filename
-image_file = try.img
-
-# Set ubdir to the build directory where you build U-Boot out-of-tree
-# We avoid in-tree build because it gets confusing trying different builds
-build_dir = /tmp/b
-
-# Build the kernel with: make O=/tmp/kernel
-bzimage = /tmp/kernel/arch/x86/boot/bzImage
-
-# Place where OVMF-pure-efi.i386.fd etc. are kept
-efi_dir = .
-''', binary=False)
-    settings.read(fname)
-    return settings
-
-
 class BuildEfi:
     """Class to collect together the various bits of state while running"""
-    def __init__(self, settings, args):
-        self.settings = settings
-        self.img = self.get_setting('image_file', 'try.img')
-        self.build_dir = self.get_setting("build_dir", '/tmp')
-        self.mnt = self.get_setting("mount_point", '/mnt/test-efi')
-        self.tmp = None
+    def __init__(self, args):
+        self.helper = Helper()
+        self.helper.read_settings()
+        self.img = self.helper.get_setting('efi_image_file', 'efi.img')
+        self.build_dir = self.helper.get_setting("build_dir", '/tmp')
         self.args = args
 
-    def get_setting(self, name, fallback=None):
-        """Get a setting by name
-
-        Args:
-            name (str): Name of setting to retrieve
-            fallback (str or None): Value to return if the setting is missing
-        """
-        return self.settings.get('build-efi', name, fallback=fallback)
-
     def run_qemu(self, bitness, serial_only):
         """Run QEMU
 
@@ -132,7 +79,7 @@ class BuildEfi:
             serial_only (bool): True to run without a display
         """
         extra = []
-        efi_dir = self.get_setting("efi_dir")
+        efi_dir = self.helper.get_setting('efi_dir')
         if self.args.arm:
             qemu_arch = 'aarch64'
             extra += ['--machine', 'virt', '-cpu', 'max']
@@ -174,83 +121,18 @@ class BuildEfi:
         cmd += extra
         command.run(*cmd)
 
-    def setup_files(self, build, build_type):
+    def setup_files(self, build, build_type, dst):
         """Set up files in the staging area
 
         Args:
             build (str): Name of build being packaged, e.g. 'efi-x86_app32'
             build_type (str): Build type ('app' or 'payload')
+            dst (str): Destination directory
         """
         print(f'Packaging {build}')
-        if not os.path.exists(self.tmp):
-            os.mkdir(self.tmp)
         fname = f'u-boot-{build_type}.efi'
-        tools.write_file(f'{self.tmp}/startup.nsh', f'fs0:{fname}',
-                         binary=False)
-        shutil.copy(f'{self.build_dir}/{build}/{fname}', self.tmp)
-
-    def copy_files(self):
-        """Copy files into the filesystem"""
-        command.run('sudo', 'cp', *glob.glob(f'{self.tmp}/*'), self.mnt)
-        if self.args.kernel:
-            bzimage = self.get_setting('bzimage_file', 'bzImage')
-            command.run('sudo', 'cp', bzimage, f'{self.mnt}/vmlinuz')
-
-    def setup_raw(self):
-        """Create a filesystem on a raw device and copy in the files"""
-        command.output('mkfs.vfat', self.img)
-        command.run('sudo', 'mkdir', '-p', self.mnt)
-        command.run('sudo', 'mount', '-o', 'loop', self.img, self.mnt)
-        self.copy_files()
-        command.run('sudo', 'umount', self.mnt)
-
-    def setup_part(self):
-        """Set up a partition table
-
-        Create a partition table and put the filesystem in the first partition
-        then copy in the files
-        """
-
-        # Create a gpt partition table with one partition
-        command.run('parted', self.img, 'mklabel', 'gpt', capture_stderr=True)
-
-        # This doesn't work correctly. It creates:
-        # Number  Start   End     Size    File system  Name  Flags
-        #  1      1049kB  24.1MB  23.1MB               boot  msftdata
-        # Odd if the same is entered interactively it does set the FS type
-        command.run('parted', '-s', '-a', 'optimal', '--',
-                    self.img, 'mkpart', 'boot', 'fat32', '1MiB', '23MiB')
-
-        # Map this partition to a loop device. Output is something like:
-        # add map loop48p1 (252:3): 0 45056 linear 7:48 2048
-        out = command.output('sudo', 'kpartx', '-av', self.img)
-        m = re.search(r'(loop.*p.)', out)
-        if not m:
-            raise ValueError(f'Invalid output from kpartx: {out}')
-
-        boot_dev = m.group(1)
-        dev = f'/dev/mapper/{boot_dev}'
-
-        command.output('mkfs.vfat', dev)
-
-        command.run('sudo', 'mount', '-o', 'loop', dev, self.mnt)
-
-        try:
-            self.copy_files()
-        finally:
-            # Sync here since this makes kpartx more likely to work the first time
-            command.run('sync')
-            command.run('sudo', 'umount', self.mnt)
-
-            # For some reason this needs a sleep or it sometimes fails, if it was
-            # run recently (in the last few seconds)
-            try:
-                cmd = 'sudo', 'kpartx', '-d', self.img
-                command.output(*cmd)
-            except command.CommandExc:
-                time.sleep(0.5)
-                cmd = 'sudo', 'kpartx', '-d', self.img
-                command.output(*cmd)
+        tools.write_file(f'{dst}/startup.nsh', f'fs0:{fname}', binary=False)
+        shutil.copy(f'{self.build_dir}/{build}/{fname}', dst)
 
     def do_build(self, build):
         """Build U-Boot for the selected board"""
@@ -267,7 +149,6 @@ class BuildEfi:
         bitness = 32 if args.word else 64
         arch = 'arm' if args.arm else 'x86'
         build_type = 'payload' if args.payload else 'app'
-        self.tmp = f'{self.build_dir}/efi{bitness}{build_type}'
         build = f'efi-{arch}_{build_type}{bitness}'
 
         if not args.no_build:
@@ -276,19 +157,17 @@ class BuildEfi:
         if args.old and bitness == 32:
             build = f'efi-{arch}_{build_type}'
 
-        self.setup_files(build, build_type)
-
-        command.output('qemu-img', 'create', self.img, '24M')
-
-        if args.partition:
-            self.setup_part()
-        else:
-            self.setup_raw()
+        with self.helper.make_disk(self.img, fs_type='vfat',
+                                   use_part=args.partition) as dirpath:
+            self.setup_files(build, build_type, dirpath)
+            if self.args.kernel:
+                bzimage = self.helper.get_setting('bzimage_file', 'bzImage')
+                command.run('cp', bzimage, f'{dirpath}/vmlinuz')
 
         if args.run:
             self.run_qemu(bitness, args.serial)
 
 
 if __name__ == "__main__":
-    efi = BuildEfi(get_settings(), parse_args())
+    efi = BuildEfi(parse_args())
     efi.start()

scripts/build-qemu

@@ -0,0 +1,414 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0+
+#
+"""Script to build/run U-Boot with QEMU
+
+It assumes that
+
+- you build U-Boot in ${ubdir}/<name> where <name> is the U-Boot
+  board config
+- your OS images are in ${imagedir}/{distroname}/
+
+So far the script supports only ARM and x86
+"""
+
+import sys
+import os
+import argparse
+import subprocess
+import shlex
+import time
+from pathlib import Path
+
+from build_helper import Helper
+
+OUR_PATH = os.path.dirname(os.path.realpath(__file__))
+OUR1_PATH = os.path.dirname(OUR_PATH)
+
+# Bring in the patman and dtoc libraries (but don't override the first path
+# in PYTHONPATH)
+sys.path.insert(2, os.path.join(OUR1_PATH, 'tools'))
+
+# pylint: disable=C0413
+from u_boot_pylib import command
+from u_boot_pylib import tools
+from u_boot_pylib import tout
+
+
+def parse_args():
+    """Parses command-line arguments"""
+    parser = argparse.ArgumentParser(
+        description='Build and/or run U-Boot with QEMU',
+        formatter_class=argparse.RawTextHelpFormatter)
+    parser.add_argument('-a', '--arch', default='arm', choices=['arm', 'x86'],
+                        help='Select architecture (arm, x86) Default: arm')
+    parser.add_argument('-B', '--no-build', action='store_true',
+                        help="Don't build; assume a build exists")
+    parser.add_argument('-d', '--disk',
+                        help='Root disk image file to use with QEMU')
+    parser.add_argument('-D', '--share-dir', metavar='DIR',
+                        help='Directory to share into the guest via virtiofs')
+    parser.add_argument('-e', '--sct-run', action='store_true',
+                        help='Run UEFI Self-Certification Test (SCT)')
+    parser.add_argument('-E', '--use-tianocore', action='store_true',
+                        help='Run Tianocore (OVMF) instead of U-Boot')
+    parser.add_argument(
+        '-k', '--kvm', action='store_true',
+        help='Use KVM (Kernel-based Virtual Machine) for acceleration')
+    parser.add_argument('-o', '--os', metavar='NAME', choices=['ubuntu'],
+                        help='Run a specified Operating System')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Show executed commands')
+    parser.add_argument('-r', '--run', action='store_true',
+                        help='Run QEMU with the built/specified image')
+    parser.add_argument(
+        '-R', '--release', default='24.04.1',
+        help='Select OS release version (e.g, 24.04) Default: 24.04.1')
+    parser.add_argument(
+        '-s', '--serial-only', action='store_true',
+        help='Use serial console only (no graphical display for QEMU)')
+    parser.add_argument(
+        '-S', '--sct-seq',
+        help='SCT sequence-file to be written into the SCT image if -e')
+    parser.add_argument('-w', '--word-32bit', action='store_true',
+                        help='Use 32-bit version for the build/architecture')
+
+    return parser.parse_args()
+
+
+class BuildQemu:
+    """Build and/or run U-Boot with QEMU based on command line arguments"""
+
+    def __init__(self, args):
+        """Set up arguments and configure paths"""
+        self.args = args
+
+        self.helper = Helper()
+        self.helper.read_settings()
+        self.imagedir = Path(self.helper.get_setting('image_dir', '~/dev'))
+        self.ubdir = Path(self.helper.get_setting('build_dir', '/tmp/b'))
+        self.sctdir = Path(self.helper.get_setting('sct_dir', '~/dev/efi/sct'))
+        self.tiano = Path(self.helper.get_setting('tianocore_dir',
+                                                  '~/dev/tiano'))
+        self.mnt = Path(self.helper.get_setting('sct_mnt', '/mnt/sct'))
+
+        self.bitness = 32 if args.word_32bit else 64
+        self.qemu_extra = []
+        self.mem = '512M'  # Default QEMU memory
+
+        if args.disk:
+            self.mem = '4G'
+            self.qemu_extra.extend(['-smp', '4'])
+
+        if args.sct_run:
+            self.mem = '4G'
+            self.qemu_extra.extend(['-smp', '4'])
+            # SCT usually runs headlessly
+            self.qemu_extra.extend(['-display', 'none'])
+            # For potential interaction within SCT
+            self.qemu_extra.extend(['-device', 'qemu-xhci'])
+            self.qemu_extra.extend(['-device', 'usb-kbd'])
+            sct_image_path = self.sctdir / 'sct.img'
+            if not sct_image_path.exists():
+                tout.fatal(f'Error: SCT image {sct_image_path} not found, '
+                           'required for -e')
+            self.qemu_extra.extend([
+                '-drive', f'file={sct_image_path},format=raw,if=none,id=vda',
+                '-device', 'virtio-blk-pci,drive=vda,bootindex=1'])
+            # Basic networking for SCT, if needed
+            self.qemu_extra.extend([
+                '-device', 'virtio-net-pci,netdev=net0',
+                '-netdev', 'user,id=net0'])
+            args.serial_only = True  # SCT implies serial output
+
+        if args.os:
+            self.mem = '4G'
+            self.qemu_extra.extend(['-smp', '4'])
+
+        self.kvm_params = []
+        if args.kvm:
+            self.kvm_params = ['-enable-kvm', '-cpu', 'host']
+
+        bios_override = None
+        if args.use_tianocore:
+            bios_override = Path(self.tiano, 'OVMF-pure-efi.x64.fd')
+            if not bios_override.exists():
+                tout.fatal(
+                    'Error: Tianocore BIOS specified (-E) but not found at '
+                    f'{bios_override}')
+
+        self.seq_fname = Path(args.sct_seq) if args.sct_seq else None
+        self.img_fname = Path(args.disk) if args.disk else None
+
+        # arch-specific setup
+        if args.arch == 'arm':
+            self.board = 'qemu_arm'
+            default_bios = 'u-boot.bin'
+            self.qemu = 'qemu-system-arm'
+            self.qemu_extra.extend(['-machine', 'virt'])
+            if not args.kvm:
+                self.qemu_extra.extend(['-accel', 'tcg'])
+            qemu_arch = 'arm'
+            if self.bitness == 64:
+                self.board = 'qemu_arm64'
+                self.qemu = 'qemu-system-aarch64'
+                self.qemu_extra.extend(['-cpu', 'cortex-a57'])
+                qemu_arch = 'arm64'
+        elif args.arch == 'x86':
+            self.board = 'qemu-x86'
+            default_bios = 'u-boot.rom'
+            self.qemu = 'qemu-system-i386'
+            qemu_arch = 'i386'  # For OS image naming
+            if self.bitness == 64:
+                self.board = 'qemu-x86_64'
+                self.qemu = 'qemu-system-x86_64'
+                qemu_arch = 'amd64'
+        else:
+            raise ValueError(f"Invalid arch '{args.arch}'")
+
+        self.os_path = None
+        if args.os == 'ubuntu':
+            img_name = (f'{args.os}-{args.release}-desktop-{qemu_arch}.iso')
+            self.os_path = self.imagedir / args.os / img_name
+
+        self.build_dir = self.ubdir / self.board
+        self.bios = (bios_override if bios_override
+                     else self.build_dir / default_bios)
+
+    @staticmethod
+    def execute_command(cmd_list, desc, check=True, **kwargs):
+        """Execute a shell command and handle errors
+
+        Args:
+            cmd_list (list of str): The command and its arguments as a list
+            desc (str): A description of the command being executed
+            check (bool): Raise CalledProcessError on non-zero exit code
+            kwargs: Additional arguments for subprocess.run
+
+        Return:
+            subprocess.CompletedProcess: The result of the subprocess.run call
+
+        Raises:
+            SystemExit: If the command is not found or fails and check is True
+        """
+        tout.info(f"Executing: {desc} -> {shlex.join(cmd_list)}")
+        try:
+            # Decode stdout/stderr by default if text=True
+            if 'text' not in kwargs:
+                kwargs['text'] = True
+            return subprocess.run(cmd_list, check=check, **kwargs)
+        except FileNotFoundError:
+            tout.fatal(f"Error: Command '{cmd_list[0]}' not found")
+        except subprocess.CalledProcessError as proc:
+            tout.error(f'Error {desc}: Command failed with exit code '
+                       f'{proc.returncode}')
+            if proc.stdout:
+                tout.error(f'Stdout:\n{proc.stdout}')
+            if proc.stderr:
+                tout.error(f'Stderr:\n{proc.stderr}')
+            tout.fatal('Failed')
+
+    def build_u_boot(self):
+        """Build U-Boot using buildman
+        """
+        self.build_dir.mkdir(parents=True, exist_ok=True)
+        cmd = ['buildman', '-w', '-o', str(self.build_dir), '--board',
+               self.board, '-I']
+
+        self.execute_command(
+            cmd,
+            f'Building U-Boot for {self.board} in {self.build_dir}')
+
+    def update_sct_sequence(self):
+        """Update the SCT image with a specified sequence file
+
+        Requires sudo for loop device setup and mounting
+        """
+        if not (self.args.sct_run and self.seq_fname and
+                self.seq_fname.exists()):
+            if (self.args.sct_run and self.seq_fname and
+                    not self.seq_fname.exists()):
+                tout.warning(f'Warning: SCT sequence file {self.seq_fname}'
+                             'not found')
+            return
+
+        fname = self.sctdir / 'sct.img'
+        if not fname.exists():
+            tout.fatal(f'Error: SCT image {fname} not found')
+
+        loopdev = None
+        try:
+            # Find free loop device and attach
+            loopdev = command.output_one_line(
+                'sudo', 'losetup', '--show', '-f', '-P', str(fname))
+            partition_path_str = f'{loopdev}p1'
+
+            uid, gid = os.getuid(), os.getgid()
+            mount_cmd = ['sudo', 'mount', partition_path_str,
+                         str(self.mnt), '-o', f'uid={uid},gid={gid},rw']
+            mount_cmd.extend(['-t', 'vfat'])
+
+            self.execute_command(mount_cmd,
+                                 f'Mounting {partition_path_str} to {self.mnt}')
+
+            target_sct_path = self.mnt / self.seq_fname.name
+            self.execute_command(
+                ['sudo', 'cp', str(self.seq_fname), str(target_sct_path)],
+                f'Copying {self.seq_fname.name} to {self.mnt}'
+            )
+            tout.info(f"Copied {self.seq_fname} to {target_sct_path}")
+
+        finally:
+            if Path(self.mnt).is_mount():
+                self.execute_command(['sudo', 'umount', str(self.mnt)],
+                                     f'Unmounting {self.mnt}', check=False)
+            if loopdev:
+                self.execute_command(['sudo', 'losetup', '-d', loopdev],
+                                     f'Detaching loop device {loopdev}',
+                                     check=False)
+
+    def run_qemu(self):
+        """Construct and run the QEMU command"""
+        if not self.bios.exists():
+            tout.fatal(f"Error: BIOS file '{self.bios}' not found")
+
+        qemu_cmd = [str(self.qemu)]
+        if self.bios:
+            qemu_cmd.extend(['-bios', str(self.bios)])
+        qemu_cmd.extend(self.kvm_params)
+        qemu_cmd.extend(['-m', self.mem])
+
+        if not self.args.sct_run:
+            qemu_cmd.extend(['-netdev', 'user,id=net0,hostfwd=tcp::2222-:22',
+                             '-device', 'virtio-net-pci,netdev=net0'])
+
+        # Display and Serial
+        # If -e (sct_run) is used, "-display none" is in qemu_extra
+        # If -s (serial_only) is used, we want no display
+        has_display_option = any(
+            item.startswith('-display') for item in self.qemu_extra)
+        if self.args.serial_only and not has_display_option:
+            qemu_cmd.extend(['-display', 'none'])
+        if not any(item.startswith('-serial') for item in self.qemu_extra):
+            qemu_cmd.extend(['-serial', 'mon:stdio'])
+
+        # Add other parameters gathered from options
+        qemu_cmd.extend(self.qemu_extra)
+        if self.os_path:
+            if not self.os_path.exists():
+                tout.error(f'OS image {self.os_path} specified but not found')
+            qemu_cmd.extend([
+                '-drive',
+                f'if=virtio,file={self.os_path},format=raw,id=hd0,readonly=on'])
+
+        if self.img_fname:
+            if self.img_fname.exists():
+                qemu_cmd.extend([
+                    '-drive',
+                    f'if=virtio,file={self.img_fname},format=raw,id=hd1'])
+            else:
+                tout.warning(f"Disk image '{self.img_fname}' not found")
+
+        sock = Path('/tmp/virtiofs.sock')
+        if self.args.share_dir:
+            virtfs_dir = Path(self.args.share_dir)
+            if not virtfs_dir.is_dir():
+                tout.fatal(f'Error: VirtFS share directory {virtfs_dir} '
+                           f'is not a valid directory')
+
+            virtiofsd = Path('/usr/libexec/virtiofsd')
+            if not virtiofsd.exists():
+                tout.fatal(f'Error: virtiofsd not found at {virtiofsd}')
+
+            # Clean up potential old socket file
+            if sock.exists():
+                try:
+                    sock.unlink()
+                    tout.info(f'Removed old socket file {sock}')
+                except OSError as e:
+                    tout.warning(
+                        f'Warning: Could not remove old socket file {sock}: '
+                        f'{e}')
+
+            qemu_cmd.extend([
+                '-chardev', f'socket,id=char0,path={sock}',
+                '-device',
+                'vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=hostshare',
+                '-object',
+                f'memory-backend-file,id=mem,size={self.mem},mem-path=/dev/shm'
+                    ',share=on',
+                '-numa', 'node,memdev=mem'])
+
+            virtiofsd_cmd = [
+                str(virtiofsd),
+                '--socket-path', str(sock),
+                '--shared-dir', str(virtfs_dir),
+                '--cache', 'auto']
+            try:
+                # Use Popen to run virtiofsd in the background
+                proc = subprocess.Popen(virtiofsd_cmd, stdout=subprocess.PIPE,
+                                        stderr=subprocess.PIPE)
+                # Give virtiofsd a moment to start and create the socket
+                time.sleep(0.5)
+                if not sock.exists() and proc.poll() is not None:
+                    stdout, stderr = proc.communicate()
+                    tout.error('Error starting virtiofsd. Exit code: '
+                               f'{proc.returncode}')
+                    if stdout:
+                        tout.error(f"virtiofsd stdout:\n{stdout.decode()}")
+                    if stderr:
+                        tout.error(f"virtiofsd stderr:\n{stderr.decode()}")
+                    tout.fatal('Failed')
+
+            except (subprocess.CalledProcessError, FileNotFoundError) as exc:
+                tout.fatal(f'Failed to start virtiofsd: {exc}')
+
+        tout.info(f'QEMU:\n{shlex.join(qemu_cmd)}\n')
+        try:
+            subprocess.run(qemu_cmd, check=True)
+        except FileNotFoundError:
+            tout.fatal(f"Error: QEMU executable '{self.qemu}' not found")
+        except subprocess.CalledProcessError as e:
+            tout.fatal(f'QEMU execution failed with exit code {e.returncode}')
+        finally:
+            # Clean up virtiofsd process and socket if it was started
+            if proc:
+                tout.info('Terminating virtiofsd')
+                proc.terminate()
+                try:
+                    proc.wait(timeout=5)
+                except subprocess.TimeoutExpired:
+                    tout.warning(
+                        'virtiofsd did not terminate gracefully; killing')
+                    proc.kill()
+                if sock.exists():
+                    try:
+                        sock.unlink()
+                    except OSError as e_os:
+                        tout.warning('Warning: Could not remove virtiofs '
+                                     f'socket {sock}: {e_os}')
+
+    def start(self):
+        """Build and run QEMU"""
+        if not self.args.no_build and not self.args.use_tianocore:
+            self.build_u_boot()
+
+        # Update SCT sequence if -e and -S are given
+        if self.args.sct_run and self.seq_fname:
+            self.update_sct_sequence()
+
+        if self.args.run:
+            self.run_qemu()
+
+
+def main():
+    """Parses arguments and initiates the BuildQemu process
+    """
+    args = parse_args()
+    tout.init(tout.INFO if args.verbose else tout.WARNING)
+
+    qemu = BuildQemu(args)
+    qemu.start()
+
+if __name__ == '__main__':
+    main()

scripts/build-qemu.sh

@@ -1,211 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-#
-# Script to build U-Boot suitable for booting with QEMU, possibly running
-# it, possibly with an OS image
-
-# This just an example. It assumes that
-
-# - you build U-Boot in ${ubdir}/<name> where <name> is the U-Boot board config
-# - your OS images are in ${imagedir}/{distroname}/...
-
-# So far the script supports only ARM and x86.
-
-set -e
-
-usage() {
-	(
-	if [[ -n "$1" ]]; then
-		echo "$1"
-		echo
-	fi
-	echo "Usage: $0 -aBkrsw"
-	echo
-	echo "   -a <arch> - Select architecture (arm, x86)"
-	echo "   -B        - Don't build; assume a build exists"
-	echo "   -e        - Run UEFI Self-Certification Test (SCT)"
-	echo "   -k        - Use kvm (kernel-based Virtual Machine)"
-	echo "   -o <name> - Run Operating System ('ubuntu' only for now)"
-	echo "   -r        - Run QEMU with the image"
-	echo "   -R <os>   - Select OS release (e.g. 24.04)"
-	echo "   -s        - Use serial only (no display)"
-	echo "   -S <seq>  - Select SCT sequence-file"
-	echo "   -w        - Use word version (32-bit)" ) >&2
-	exit 1
-}
-
-# Directory tree for OS images
-imagedir=${imagedir-/vid/software/linux}
-
-# Directory for UEFI Self-Certification Test (SCT)
-sctdir=${sctdir-/vid/software/devel/uefi/sct}
-
-# Mount point for use when writing to disk images
-mnt=${mnt-/mnt}
-
-# architecture (arm or x86)
-arch=arm
-
-# 32- or 64-bit build
-bitness=64
-
-# Build U-Boot
-build=yes
-
-# Extra setings
-extra=
-
-# Operating System to boot (ubuntu)
-os=
-
-release=24.04.1
-
-# run the image with QEMU
-run=
-
-# run QEMU without a display (U-Boot must be set to stdout=serial)
-serial=
-
-# Use kvm
-kvm=
-
-# Set ubdir to the build directory where you build U-Boot out-of-tree
-# We avoid in-tree build because it gets confusing trying different builds
-ubdir=${ubdir-/tmp/b}
-
-while getopts "a:Beko:rR:sS:w" opt; do
-	case "${opt}" in
-	a)
-		arch=$OPTARG
-		;;
-	B)
-		build=
-		;;
-	e)
-		extra+=" -m 4G -smp 4"
-		extra+=" -display none"
-		extra+=" -device virtio-gpu-pci"
-		extra+=" -device qemu-xhci"
-		extra+=" -device usb-kbd"
-		extra+=" -drive file=${sctdir}/sct.img,format=raw,if=none,id=vda"
-		extra+=" -device virtio-blk-device,drive=vda,bootindex=1"
-		extra+=" -device virtio-net-device,netdev=net0"
-		extra+=" -netdev user,id=net0"
-		;;
-	k)
-		kvm="-enable-kvm"
-		;;
-	o)
-		os=$OPTARG
-
-		# Expand memory and CPUs
-		extra+=" -m 4G -smp 4"
-		;;
-	r)
-		run=1
-		;;
-	R)
-		release=$OPTARG
-		;;
-	s)
-		serial=1
-		;;
-	S)
-		seq=$OPTARG
-		;;
-	w)
-		bitness=32
-		;;
-	*)
-		usage
-		;;
-	esac
-done
-
-# Build U-Boot for the selected board
-build_u_boot() {
-	buildman -w -o $DIR --board $BOARD -I || exit $?
-}
-
-# Write the SCT test-sequence file into the SCT image
-update_sct_seq() {
-	if [[ -z "${seq}" ]]; then
-		return
-	fi
-	LOOP=$(sudo losetup --show -f -P "${sctdir}/sct.img")
-	PART="${LOOP}p1"
-	sudo mount -o loop ${PART} ${mnt} -o uid=$(id -u),gid=$(id -g)
-	cp "${seq}" "${mnt}/."
-	sudo umount ${mnt}
-	sudo losetup -d ${LOOP}
-}
-
-# Run QEMU with U-Boot
-run_qemu() {
-	if [[ -n "${os_image}" ]]; then
-		extra+=" -drive if=virtio,file=${os_image},format=raw,id=hd0"
-	fi
-	if [[ -n "${serial}" ]]; then
-		extra+=" -display none -serial mon:stdio"
-	else
-		extra+=" -serial mon:stdio"
-	fi
-	echo "Running ${qemu} ${extra}"
-	"${qemu}" -bios "$DIR/${BIOS}" \
-		-m 512 \
-		-nic none \
-		${kvm} \
-		${extra}
-}
-
-# Check architecture
-case "${arch}" in
-arm)
-	BOARD="qemu_arm"
-	BIOS="u-boot.bin"
-	qemu=qemu-system-arm
-	extra+=" -machine virt -accel tcg"
-	suffix="arm"
-	if [[ "${bitness}" == "64" ]]; then
-		BOARD="qemu_arm64"
-		qemu=qemu-system-aarch64
-		extra+=" -cpu cortex-a57"
-		suffix="arm64"
-	fi
-	;;
-x86)
-	BOARD="qemu-x86"
-	BIOS="u-boot.rom"
-	qemu=qemu-system-i386
-	suffix="i386"
-	if [[ "${bitness}" == "64" ]]; then
-		BOARD="qemu-x86_64"
-		qemu=qemu-system-x86_64
-		suffix="amd64"
-	fi
-	;;
-*)
-	usage "Unknown architecture '${arch}'"
-esac
-
-# Check OS
-case "${os}" in
-ubuntu)
-	os_image="${imagedir}/${os}/${os}-${release}-desktop-${suffix}.iso"
-	;;
-"")
-	;;
-*)
-	usage "Unknown OS '${os}'"
-esac
-
-DIR=${ubdir}/${BOARD}
-
-if [[ -n "${build}" ]]; then
-	build_u_boot
-	update_sct_seq
-fi
-
-if [[ -n "${run}" ]]; then
-	run_qemu
-fi

scripts/build_helper.py

@@ -0,0 +1,126 @@
+# SPDX-License-Identifier: GPL-2.0+
+#
+"""Common script for build- scripts
+
+"""
+
+import configparser
+import contextlib
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+OUR_PATH = os.path.dirname(os.path.realpath(__file__))
+OUR1_PATH = os.path.dirname(OUR_PATH)
+
+# Bring in the patman and test libraries (but don't override the first path in
+# PYTHONPATH)
+sys.path.insert(2, os.path.join(OUR1_PATH, 'tools'))
+sys.path.insert(2, os.path.join(OUR1_PATH, 'test/py/tests'))
+
+from u_boot_pylib import command
+from u_boot_pylib import tools
+import fs_helper
+
+
+class Helper:
+    def __init__(self):
+        self.settings = None
+
+    def read_settings(self):
+        """Get settings from the settings file"""
+        self.settings = configparser.ConfigParser()
+        fname = f'{os.getenv("HOME")}/.u_boot_qemu'
+        if not os.path.exists(fname):
+            print('No config file found: {fname}\nCreating one...\n')
+            tools.write_file(fname, '''# U-Boot QEMU-scripts config
+
+[DEFAULT]
+# Set ubdir to the build directory where you build U-Boot out-of-tree
+# We avoid in-tree build because it gets confusing trying different builds
+# Each board gets a build in a separate subdir
+build_dir = /tmp/b
+
+# Image directory (for OS images)
+image_dir = ~/dev/os
+
+# Build the kernel with: make O=/tmp/kernel
+bzimage = /tmp/kernel/arch/x86/boot/bzImage
+
+# EFI image-output filename
+efi_image_file = try.img
+
+# Directory where OVMF-pure-efi.i386.fd etc. are kept
+efi_dir = ~/dev/efi
+
+# Directory where SCT image (sct.img) is kept
+sct_dir = ~/dev/efi/sct
+
+# Directory where the SCT image is temporarily mounted for modification
+sct_mnt = /mnt/sct
+''', binary=False)
+        self.settings.read(fname)
+
+    def get_setting(self, name, fallback=None):
+        """Get a setting by name
+
+        Args:
+            name (str): Name of setting to retrieve
+            fallback (str or None): Value to return if the setting is missing
+        """
+        raw = self.settings.get('DEFAULT', name, fallback=fallback)
+        return os.path.expandvars(os.path.expanduser(raw))
+
+    def stage(self, name):
+        """Context manager to count requests across a range of patchwork calls
+
+        Args:
+            name (str): Stage name
+
+        Return:
+            _Stage: contect object
+
+        Usage:
+            with self.stage('name'):
+                ...do things
+
+            Note that the output only appears if the -N flag is used
+        """
+        return self._Stage(name)
+
+    @contextlib.contextmanager
+    def make_disk(self, fname, size_mb=20, fs_type='ext4', use_part=False):
+        """Create a raw disk image with files on it
+
+        Args:
+            fname (str): Filename to write the images to
+            fs_type (str): Filesystem type to create (ext4 or vfat)
+            size_mb (int): Size in MiB
+            use_part (bool): True to create a partition table, False to use a
+                raw disk image
+
+        Yields:
+            str: Directory to write the files into
+        """
+        with tempfile.NamedTemporaryFile() as tmp:
+            with tempfile.TemporaryDirectory(prefix='build_helper.') as dirname:
+                try:
+                    yield dirname
+                    fs_helper.mk_fs(None, fs_type, size_mb << 20, None, dirname,
+                                    fs_img=tmp.name, quiet=True)
+                finally:
+                    pass
+
+            if use_part:
+                with open(fname, 'wb') as img:
+                    img.truncate(size_mb << 20)
+                    img.seek(1 << 20, 0)
+                    img.write(tools.read_file(tmp.name))
+                subprocess.run(
+                    ['sfdisk', fname], text=True, check=True,
+                    capture_output=True,
+                    input=f'type=c, size={size_mb-1}M, start=1M,bootable')
+            else:
+                shutil.copy2(tmp.name, fname)

test/py/tests/fs_helper.py

@@ -8,8 +8,10 @@
 import re
 import os
 from subprocess import call, check_call, check_output, CalledProcessError
+from subprocess import DEVNULL
 
-def mk_fs(config, fs_type, size, prefix, src_dir=None, size_gran = 0x100000):
+def mk_fs(config, fs_type, size, prefix, src_dir=None, size_gran = 0x100000,
+          fs_img=None, quiet=False):
     """Create a file system volume
 
     Args:
@@ -19,12 +21,15 @@ def mk_fs(config, fs_type, size, prefix, src_dir=None, size_gran = 0x100000):
         prefix (str): Prefix string of volume's file name
         src_dir (str): Root directory to use, or None for none
         size_gran (int): Size granularity of file system image in bytes
+        fs_img (str or None): Filename for image, or None to invent one
+        quiet (bool): Suppress non-error output
 
     Raises:
         CalledProcessError: if any error occurs when creating the filesystem
     """
-    fs_img = f'{prefix}.{fs_type}.img'
-    fs_img = os.path.join(config.persistent_data_dir, fs_img)
+    if not fs_img:
+        leaf = f'{prefix}.{fs_type}.img'
+        fs_img = os.path.join(config.persistent_data_dir, leaf)
 
     if fs_type == 'fat12':
         mkfs_opt = '-F 12'
@@ -58,14 +63,17 @@ def mk_fs(config, fs_type, size, prefix, src_dir=None, size_gran = 0x100000):
         check_call(f'rm -f {fs_img}', shell=True)
         check_call(f'truncate -s $(( {size_gran} * {count} )) {fs_img}',
                    shell=True)
-        check_call(f'mkfs.{fs_lnxtype} {mkfs_opt} {fs_img}', shell=True)
+        check_call(f'mkfs.{fs_lnxtype} {mkfs_opt} {fs_img}', shell=True,
+                   stdout=DEVNULL if quiet else None)
         if fs_type == 'ext4':
             sb_content = check_output(f'tune2fs -l {fs_img}',
                                       shell=True).decode()
             if 'metadata_csum' in sb_content:
                 check_call(f'tune2fs -O ^metadata_csum {fs_img}', shell=True)
         elif fs_lnxtype == 'vfat' and src_dir:
-            check_call(f'mcopy -i {fs_img} -vsmpQ {src_dir}/* ::/', shell=True)
+            flags = f"-smpQ{'' if quiet else 'v'}"
+            check_call(f'mcopy -i {fs_img} {flags} {src_dir}/* ::/',
+                       shell=True)
         elif fs_lnxtype == 'exfat' and src_dir:
             check_call(f'fattools cp {src_dir}/* {fs_img}', shell=True)
         return fs_img

tools/binman/control.py

@@ -777,7 +777,7 @@ def Binman(args):
 
     if args.cmd in ['ls', 'extract', 'replace', 'tool', 'sign']:
         try:
-            tout.init(args.verbosity)
+            tout.init(args.verbosity + 1)
             if args.cmd == 'replace':
                 tools.prepare_output_dir(args.outdir, args.preserve)
             else:
@@ -835,9 +835,9 @@ def Binman(args):
         args.indir.append(board_pathname)
 
     try:
-        tout.init(args.verbosity)
+        tout.init(args.verbosity + 1)
         elf.debug = args.debug
-        cbfs_util.VERBOSE = args.verbosity > 2
+        cbfs_util.VERBOSE = args.verbosity > tout.NOTICE
         state.use_fake_dtb = args.fake_dtb
 
         # Normally we replace the 'u-boot' etype with 'u-boot-expanded', etc.

tools/u_boot_pylib/tout.py

@@ -9,7 +9,7 @@ import sys
 from u_boot_pylib import terminal
 
 # Output verbosity levels that we support
-ERROR, WARNING, NOTICE, INFO, DETAIL, DEBUG = range(6)
+FATAL, ERROR, WARNING, NOTICE, INFO, DETAIL, DEBUG = range(7)
 
 in_progress = False
 
@@ -42,12 +42,12 @@ def user_is_present():
     Returns:
         True if it thinks the user is there, and False otherwise
     """
-    return stdout_is_tty and verbose > 0
+    return stdout_is_tty and verbose > ERROR
 
 def clear_progress():
     """Clear any active progress message on the terminal."""
     global in_progress
-    if verbose > 0 and stdout_is_tty and in_progress:
+    if verbose > ERROR and stdout_is_tty and in_progress:
         _stdout.write('\r%s\r' % (" " * len (_progress)))
         _stdout.flush()
         in_progress = False
@@ -60,7 +60,7 @@ def progress(msg, warning=False, trailer='...'):
         warning: True if this is a warning."""
     global in_progress
     clear_progress()
-    if verbose > 0:
+    if verbose > ERROR:
         _progress = msg + trailer
         if stdout_is_tty:
             col = _color.YELLOW if warning else _color.GREEN
@@ -87,6 +87,8 @@ def _output(level, msg, color=None):
             print(msg, file=sys.stderr)
         else:
             print(msg)
+    if level == FATAL:
+        sys.exit(1)
 
 def do_output(level, msg):
     """Output a message to the terminal.
@@ -98,6 +100,14 @@ def do_output(level, msg):
     """
     _output(level, msg)
 
+def fatal(msg):
+    """Display an error message and exit
+
+    Args:
+        msg; Message to display.
+    """
+    _output(FATAL, msg, _color.RED)
+
 def error(msg):
     """Display an error message
 
@@ -153,13 +163,13 @@ def user_output(msg):
     Args:
         msg; Message to display.
     """
-    _output(0, msg)
+    _output(ERROR, msg)
 
 def init(_verbose=WARNING, stdout=sys.stdout, allow_colour=True):
     """Initialize a new output object.
 
     Args:
-        verbose: Verbosity level (0-4).
+        verbose: Verbosity level (0-6).
         stdout: File to use for stdout.
     """
     global verbose, _progress, _color, _stdout, stdout_is_tty