smallsolar/u-boot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch '2024-12-21-add-arm64-CI-host-support' into next

Browse Source 
First, thanks to Simon Glass and also Linaro, we now have access to a
few fast arm64 host machines in our Gitlab instance, to use as CI
runners. This series finishes the work that I pushed earlier and Simon
had started that enables arm64 hosts to be used for most things now.

The first notable change, especially if you use this on your own Gitlab
instance is that "DEFAULT_TAG" is now unused and we instead have:
- DEFAULT_ALL_TAG:
- DEFAULT_ARM64_TAG:
- DEFAULT_AMD64_TAG:
- DEFAULT_FAST_AMD64_TAG:

This lets us say that some jobs can be run on all runners, because they
are small enough that anything we'd connect to CI is fast enough and it
also does not depend on the underlying host architecture. Next we have
tags for any arm64 host, or any amd64 host. Finally, we have a tag for
fast amd64 hosts. What these last three are for is that we have a few
jobs that need to run on amd64 hosts and so we have to restrict them
there, but we also have now reworked the world build jobs to build
(almost) everything in a single job and on the fast amd64 machines this
is still as quick as the old way was, in practice.

To reach this point, we say that the Xtensa jobs can only run on amd64
hosts. Our targets only work with the binary-only toolchain and so this
is a reasonable limit and we exclude them from the world build jobs. We
also need to deal with ensuring the right toolchain is used regardless
what the host architecture is and that we don't use the host toolchain
by accident. Finally, because some of these changes needed to be worked
out in the linter, fix some of the general warnings that notes as well.

(cherry picked from commit 5947cd76ac)

Dropped tag changes:
Signed-off-by: Simon Glass <simon.glass@canonical.com>
This commit is contained in:
Tom Rini
2024-12-21 12:27:55 -06:00
committed by Simon Glass
parent 3f44038f9d
commit 874bc0dd1b
2 changed files with 56 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
.azure-pipelines.yml
View File

@@ -2,7 +2,7 @@ variables:
windows_vm: windows-2022
ubuntu_vm: ubuntu-24.04
macos_vm: macOS-14
ci_runner_image: trini/u-boot-gitlab-ci-runner:jammy-20250404-29Apr2025
ci_runner_image: trini/u-boot-gitlab-ci-runner:jammy-20240911.1-08Dec2024
# Add '-u 0' options for Azure pipelines, otherwise we get "permission
# denied" error when it tries to "useradd -m -u 1001 vsts_azpcontainer",
# since our $(ci_runner_image) user is not root.
@@ -89,10 +89,9 @@ stages:
options: $(container_option)
steps:
- script: |
set -e
python3 -m venv /tmp/venvhtml
virtualenv -p /usr/bin/python3 /tmp/venvhtml
. /tmp/venvhtml/bin/activate
pip install -r doc/sphinx/requirements.txt -r test/py/requirements.txt
pip install -r doc/sphinx/requirements.txt
make htmldocs KDOC_WERROR=1
make infodocs
@@ -132,23 +131,16 @@ stages:
git config --global user.email bmeng.cn@gmail.com
git config --global --add safe.directory $(work_dir)
export USER=azure
python3 -m venv /tmp/venv
virtualenv -p /usr/bin/python3 /tmp/venv
. /tmp/venv/bin/activate
pip install -r test/py/requirements.txt \
-r tools/binman/requirements.txt \
-r tools/buildman/requirements.txt \
-r tools/patman/requirements.txt \
-r tools/u_boot_pylib/requirements.txt
pip install -r test/py/requirements.txt
pip install -r tools/buildman/requirements.txt
export UBOOT_TRAVIS_BUILD_DIR=/tmp/tools-only
export PYTHONPATH=${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt
export PATH=${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}
./tools/buildman/buildman -T0 -o ${UBOOT_TRAVIS_BUILD_DIR} -w --board tools-only
set -ex
export TOOLPATH="--toolpath ${UBOOT_TRAVIS_BUILD_DIR}/tools --toolpath /opt/coreboot"
./tools/binman/binman ${TOOLPATH} tool -f missing
./tools/binman/binman ${TOOLPATH} test
# Avoid "Permission denied: 'cov'" error by using a temporary file
COVERAGE_FILE=/tmp/.coverage ./tools/binman/binman ${TOOLPATH} test -T
./tools/binman/binman --toolpath ${UBOOT_TRAVIS_BUILD_DIR}/tools test
./tools/buildman/buildman -t
./tools/dtoc/dtoc -t
./tools/patman/patman test
@@ -170,14 +162,9 @@ stages:
- script: |
git config --global --add safe.directory $(work_dir)
export USER=azure
python3 -m venv /tmp/venv
. /tmp/venv/bin/activate
pip install -r test/py/requirements.txt \
-r tools/binman/requirements.txt \
-r tools/buildman/requirements.txt \
-r tools/patman/requirements.txt \
-r tools/u_boot_pylib/requirements.txt \
asteval pylint==3.3.4 pyopenssl
pip install -r test/py/requirements.txt
pip install -r tools/buildman/requirements.txt
pip install asteval pylint==2.12.2 pyopenssl
export PATH=${PATH}:~/.local/bin
echo "[MASTER]" >> .pylintrc
echo "load-plugins=pylint.extensions.docparams" >> .pylintrc
@@ -251,12 +238,13 @@ stages:
# the below corresponds to .gitlab-ci.yml "before_script"
cd \${WORK_DIR}
git config --global --add safe.directory \${WORK_DIR}
git clone --depth=1 https://source.denx.de/u-boot/u-boot-test-hooks /tmp/uboot-test-hooks
# qemu_arm64_lwip_defconfig is the same as qemu_arm64 but with NET_LWIP enabled.
# The test config and the boardenv file from qemu_arm64 can be re-used so create symlinks
ln -s conf.qemu_arm64_na test/hooks/bin/travis-ci/conf.qemu_arm64_lwip_na
ln -s u_boot_boardenv_qemu_arm64_na.py test/hooks/py/travis-ci/u_boot_boardenv_qemu_arm64_lwip_na.py
ln -s travis-ci test/hooks/bin/\`hostname\`
ln -s travis-ci test/hooks/py/\`hostname\`
ln -s conf.qemu_arm64_na /tmp/uboot-test-hooks/bin/travis-ci/conf.qemu_arm64_lwip_na
ln -s u_boot_boardenv_qemu_arm64_na.py /tmp/uboot-test-hooks/py/travis-ci/u_boot_boardenv_qemu_arm64_lwip_na.py
ln -s travis-ci /tmp/uboot-test-hooks/bin/\`hostname\`
ln -s travis-ci /tmp/uboot-test-hooks/py/\`hostname\`
if [[ "\${TEST_PY_BD}" == "qemu-riscv32_spl" ]]; then
wget -O - https://github.com/riscv-software-src/opensbi/releases/download/v1.3.1/opensbi-1.3.1-rv-bin.tar.xz | tar -C /tmp -xJ;
export OPENSBI=/tmp/opensbi-1.3.1-rv-bin/share/opensbi/ilp32/generic/firmware/fw_dynamic.bin;
@@ -276,13 +264,7 @@ stages:
if [ -n "\${BUILD_ENV}" ]; then
export \${BUILD_ENV};
fi
python3 -m venv /tmp/venv
. /tmp/venv/bin/activate
pip install -r tools/binman/requirements.txt \
-r tools/buildman/requirements.txt \
-r test/py/requirements.txt \
-r tools/u_boot_pylib/requirements.txt \
pytest-azurepipelines
pip install -r tools/buildman/requirements.txt
tools/buildman/buildman -o \${UBOOT_TRAVIS_BUILD_DIR} -w -E -W -e --board \${TEST_PY_BD} \${OVERRIDE}
cp /opt/grub/grub_x86.efi \${UBOOT_TRAVIS_BUILD_DIR}/
cp /opt/grub/grub_x64.efi \${UBOOT_TRAVIS_BUILD_DIR}/
@@ -306,19 +288,16 @@ stages:
/opt/coreboot/cbfstool \${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom remove -n fallback/payload;
/opt/coreboot/cbfstool \${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom add-flat-binary -f \${UBOOT_TRAVIS_BUILD_DIR}/u-boot.bin -n fallback/payload -c LZMA -l 0x1110000 -e 0x1110000;
fi
# If we have TF-A binaries, we need to use them.
if [[ -d /opt/tf-a/"\${TEST_PY_BD}" ]]; then
cp /opt/tf-a/"\${TEST_PY_BD}"/fip.bin /opt/tf-a/"\${TEST_PY_BD}"/bl1.bin /tmp;
export fip=/tmp/fip.bin;
export bl1=/tmp/bl1.bin;
export PATH=/opt/Base_RevC_AEMvA_pkg/models/Linux64_GCC-9.3:\${PATH};
fi
export PATH=/opt/qemu/bin:test/hooks/bin:\${PATH}
export PYTHONPATH=test/hooks/py/travis-ci
virtualenv -p /usr/bin/python3 /tmp/venv
. /tmp/venv/bin/activate
pip install -r test/py/requirements.txt
pip install pytest-azurepipelines
export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:\${PATH}
export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci
# "\${var:+"-k \$var"}" expands to "" if \$var is empty, "-k \$var" if not
./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_PY_EXTRA} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
# the below corresponds to .gitlab-ci.yml "after_script"
rm -rf /tmp/venv
rm -rf /tmp/uboot-test-hooks /tmp/venv
EOF
- task: CopyFiles@2
displayName: 'Copy test.sh for later usage'
@@ -339,25 +318,22 @@ stages:
matrix:
sandbox:
TEST_PY_BD: "sandbox"
TEST_PY_EXTRA: "--timing"
sandbox_asan:
TEST_PY_BD: "sandbox"
OVERRIDE: "-a ASAN"
TEST_PY_TEST_SPEC: "version"
sandbox_clang:
TEST_PY_BD: "sandbox"
OVERRIDE: "-O clang-18"
OVERRIDE: "-O clang-17"
sandbox_clang_asan:
TEST_PY_BD: "sandbox"
OVERRIDE: "-O clang-18 -a ASAN"
OVERRIDE: "-O clang-17 -a ASAN"
TEST_PY_TEST_SPEC: "version"
sandbox64:
TEST_PY_BD: "sandbox64"
sandbox64_clang:
TEST_PY_BD: "sandbox64"
OVERRIDE: "-O clang-18"
sandbox64_lwip:
TEST_PY_BD: "sandbox64_lwip"
OVERRIDE: "-O clang-17"
sandbox_spl:
TEST_PY_BD: "sandbox_spl"
TEST_PY_TEST_SPEC: "test_ofplatdata or test_handoff or test_spl"
@@ -432,16 +408,14 @@ stages:
TEST_PY_BD: "evb-ast2500"
TEST_PY_ID: "--id qemu"
TEST_PY_TEST_SPEC: "not sleep"
evb_ast2600:
TEST_PY_BD: "evb-ast2600"
TEST_PY_ID: "--id qemu"
TEST_PY_TEST_SPEC: "not sleep"
vexpress_ca9x4:
TEST_PY_BD: "vexpress_ca9x4"
TEST_PY_ID: "--id qemu"
TEST_PY_TEST_SPEC: "not sleep"
vexpress_fvp:
TEST_PY_BD: "vexpress_fvp"
TEST_PY_TEST_SPEC: "not sleep and not hostfs"
vexpress_fvp_bloblist:
TEST_PY_BD: "vexpress_fvp_bloblist"
TEST_PY_TEST_SPEC: "not sleep and not hostfs"
integratorcp_cm926ejs:
TEST_PY_BD: "integratorcp_cm926ejs"
TEST_PY_ID: "--id qemu"
@@ -458,12 +432,6 @@ stages:
qemu_arm_sbsa_ref:
TEST_PY_BD: "qemu-arm-sbsa"
TEST_PY_TEST_SPEC: "not sleep"
qemu_arm_spl:
TEST_PY_BD: "qemu_arm_spl"
TEST_PY_TEST_SPEC: "test_passage"
qemu_arm64_spl:
TEST_PY_BD: "qemu_arm64_spl"
TEST_PY_TEST_SPEC: "test_passage"
qemu_m68k:
TEST_PY_BD: "M5208EVBE"
TEST_PY_ID: "--id qemu"
@@ -541,7 +509,6 @@ stages:
TEST_PY_BD: "xilinx_versal_virt"
TEST_PY_ID: "--id qemu"
TEST_PY_TEST_SPEC: "not sleep"
OVERRIDE: "-a ~CONFIG_USB_DWC3"
xtfpga:
TEST_PY_BD: "xtfpga"
TEST_PY_ID: "--id qemu"
@@ -612,10 +579,7 @@ stages:
# make environment variables available as tests are running inside a container
export BUILDMAN="${BUILDMAN}"
git config --global --add safe.directory ${WORK_DIR}
python3 -m venv /tmp/venv
. /tmp/venv/bin/activate
pip install -r tools/binman/requirements.txt \
-r tools/buildman/requirements.txt
pip install -r tools/buildman/requirements.txt
EOF
cat << "EOF" >> build.sh
if [[ "${BUILDMAN}" != "" ]]; then

68
tools/docker/Dockerfile
View File

@@ -2,7 +2,7 @@
# This Dockerfile is used to build an image containing basic stuff to be used
# to build U-Boot and run our test suites.
FROM ubuntu:jammy-20250404
FROM ubuntu:jammy-20240911.1
LABEL org.opencontainers.image.authors="Tom Rini <trini@konsulko.com>"
LABEL org.opencontainers.image.description=" This image is for building U-Boot inside a container"
@@ -23,7 +23,7 @@ ENV ARCHS="aarch64 arc i386 m68k mips microblaze nios2 powerpc riscv64 riscv32 s
ENV MIRROR=https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin
# Toolchain version
ENV TCVER=14.2.0
ENV TCVER=13.2.0
RUN echo "Building on $BUILDPLATFORM, for target $TARGETPLATFORM"
@@ -32,7 +32,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y gnupg2 wget xz-utils
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
RUN echo deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-18 main | tee /etc/apt/sources.list.d/llvm.list
RUN echo deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-17 main | tee /etc/apt/sources.list.d/llvm.list
# Create a list of URLs to process, then pass them into a 'while read' loop
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then HOSTARCH=x86_64; else HOSTARCH=arm64; fi; ( \
@@ -64,22 +64,18 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
binutils-dev \
bison \
build-essential \
byacc \
cgpt \
clang-18 \
clang-17 \
coreutils \
cpio \
cryptsetup \
curl \
device-tree-compiler \
dosfstools \
e2fsprogs \
efitools \
erofs-utils \
exfatprogs \
expect \
fakeroot \
fdisk \
flex \
gawk \
gdisk \
@@ -91,13 +87,14 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
help2man \
iasl \
imagemagick \
inetutils-telnet \
iputils-ping \
libconfuse-dev \
libgit2-dev \
libjson-glib-dev \
libguestfs-tools \
libgnutls28-dev \
libgnutls30 \
liblz4-tool \
libpixman-1-dev \
libpython3-dev \
libsdl1.2-dev \
@@ -108,7 +105,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
libtool \
libudev-dev \
libusb-1.0-0-dev \
lz4 \
linux-image-generic \
lzma-alone \
lzop \
mount \
@@ -125,8 +122,9 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
python3 \
python3-dev \
python3-pip \
python3-pyelftools \
python3-sphinx \
python3-venv \
python3-virtualenv \
rpm2cpio \
sbsigntool \
socat \
@@ -138,12 +136,16 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
texinfo \
util-linux \
uuid-dev \
virtualenv \
vboot-kernel-utils \
vboot-utils \
xilinx-bootgen \
xxd \
zip
# Make kernels readable for libguestfs tools to work correctly
RUN chmod +r /boot/vmlinu*
# Build GRUB UEFI targets for ARM & RISC-V, 32-bit and 64-bit
RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \
cd /tmp/grub && \
@@ -180,7 +182,7 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \
search search_fs_file search_fs_uuid search_label serial sleep test \
true && \
make clean && \
grub_cv_cc_mcmodel=no ./configure --target=riscv64 --with-platform=efi \
./configure --target=riscv64 --with-platform=efi \
CC=gcc \
TARGET_CC=/opt/gcc-${TCVER}-nolibc/riscv64-linux/bin/riscv64-linux-gcc \
TARGET_OBJCOPY=/opt/gcc-${TCVER}-nolibc/riscv64-linux/bin/riscv64-linux-objcopy \
@@ -233,24 +235,13 @@ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \
# Build fiptool
RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/tf-a && \
cd /tmp/tf-a/ && \
git checkout v2.12.0 && \
make CROSS_COMPILE=/opt/gcc-${TCVER}-nolibc/aarch64-linux/bin/aarch64-linux- \
PLAT=fvp BL33=/dev/null -j$(nproc) all fip && \
mkdir -p /usr/local/bin /opt/tf-a/vexpress_fvp && \
cp tools/fiptool/fiptool /usr/local/bin && \
cp build/fvp/release/fip.bin build/fvp/release/bl1.bin \
/opt/tf-a/vexpress_fvp/ && \
rm -rf build/fvp && \
make CROSS_COMPILE=/opt/gcc-${TCVER}-nolibc/aarch64-linux/bin/aarch64-linux- \
PLAT=fvp BL33=/dev/null TRANSFER_LIST=1 -j$(nproc) all fip && \
mkdir -p /opt/tf-a/vexpress_fvp_bloblist && \
cp build/fvp/release/fip.bin build/fvp/release/bl1.bin \
/opt/tf-a/vexpress_fvp_bloblist/ && \
git checkout v2.10.0 && \
cd tools/fiptool && \
make -j$(nproc) && \
mkdir -p /usr/local/bin && \
cp fiptool /usr/local/bin && \
rm -rf /tmp/tf-a
# Download the Arm Architecture FVP platform. This file is double compressed.
RUN wget -O - https://developer.arm.com/-/cdn-downloads/permalink/FVPs-Architecture/FM-11.28/FVP_Base_RevC-2xAEMvA_11.28_23_Linux64.tgz | gunzip -dc | tar -C /opt -x
# Build genimage (required by some targets to generate disk images)
RUN wget -O - https://github.com/pengutronix/genimage/releases/download/v14/genimage-14.tar.xz | tar -C /tmp -xJ && \
cd /tmp/genimage-14 && \
@@ -295,8 +286,8 @@ RUN mkdir /tmp/trace && \
rm -rf /tmp/trace
# Build coreboot
RUN wget -O - https://coreboot.org/releases/coreboot-25.03.tar.xz | tar -C /tmp -xJ && \
cd /tmp/coreboot-25.03 && \
RUN wget -O - https://coreboot.org/releases/coreboot-24.08.tar.xz | tar -C /tmp -xJ && \
cd /tmp/coreboot-24.08 && \
make crossgcc-i386 CPUS=$(nproc) && \
make -C payloads/coreinfo olddefconfig && \
make -C payloads/coreinfo && \
@@ -306,34 +297,23 @@ RUN wget -O - https://coreboot.org/releases/coreboot-25.03.tar.xz | tar -C /tmp
make olddefconfig && \
make -j $(nproc) && \
sudo mkdir /opt/coreboot && \
sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/ && \
rm -rf /tmp/coreboot-25.03
sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/
# Create our user/group
RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
RUN useradd -m -U uboot
USER uboot:uboot
# Install Rust toolchain for building Rust examples
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
ENV PATH="/home/uboot/.cargo/bin:${PATH}"
# Populate the cache for pip to use. Get these via wget as the
# COPY / ADD directives don't work as we need them to.
RUN wget -O /tmp/pytest-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/test/py/requirements.txt
RUN wget -O /tmp/sphinx-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/doc/sphinx/requirements.txt
RUN wget -O /tmp/binman-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/binman/requirements.txt
RUN wget -O /tmp/buildman-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/buildman/requirements.txt
RUN wget -O /tmp/patman-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/patman/requirements.txt
RUN wget -O /tmp/u_boot_pylib-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/u_boot_pylib/requirements.txt
RUN python3 -m venv /tmp/venv && \
RUN virtualenv -p /usr/bin/python3 /tmp/venv && \
. /tmp/venv/bin/activate && \
pip install -r /tmp/pytest-requirements.txt \
-r /tmp/sphinx-requirements.txt \
-r /tmp/binman-requirements.txt \
-r /tmp/buildman-requirements.txt \
-r /tmp/patman-requirements.txt \
-r /tmp/u_boot_pylib-requirements.txt && \
-r /tmp/buildman-requirements.txt && \
deactivate && \
rm -rf /tmp/venv /tmp/*-requirements.txt