My labgrid setup
Simon Glass
sjg at chromium.org
Fri Aug 30 03:06:53 CEST 2024
Hi Tom,
On Thu, 29 Aug 2024 at 13:30, Tom Rini <trini at konsulko.com> wrote:
>
> Hey all,
>
> So now that I've posted my u-boot-test-hooks for labgrid:
> https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-trini@konsulko.com/
> the next relevant parts would be how I use that. There's three scripts,
> first of which is "uboot-hw-testcycle.sh" :
> ------------------------ >8 ------------------------
> #!/bin/bash
> set -e
>
> # Save output
> LOG=$HOME/logs/`git describe --abbrev=0`/`date +"%Y-%m-%d-%H%M"`-hw.txt
>
> # Make a base directory
> B=`mktemp -d`
> chmod a+rwX ${B}
>
> cleanup() {
> # Clean up random outputs
> [ -d test/py/.pytest_cache ] && find -type d -name __pycache__ | xargs rm -r
> sudo rm -rf ${B}
> rm -rf test/py/.pytest_cache
> }
>
> # Cleanups
> trap cleanup INT
> trap cleanup ERR
> trap cleanup EXIT
>
> # Find our arguments
> while test $# -ne 0; do
> if [ "$1" == "--no-32bit" ]; then
> NO32BIT="$1"
> shift
> elif [ "$1" == "--no-64bit" ]; then
> NO64BIT="$1"
> shift
> else
> echo "Unknown argument $1"
> exit 1
> fi
> done
>
> # Run the build
> if [ -f .gitlab-ci.yml ]; then
> IMG=`grep ci_runner_image: .azure-pipelines.yml | cut -d ' ' -f 4`
> else
> IMG=`git grep ci_runner_image: origin/master -- .azure-pipelines.yml | cut -d ' ' -f 4`
> fi
>
> # Build all platforms
> touch $LOG && chmod a+rw $LOG
> sudo docker run --rm -it --privileged \
> -v /home/trini/bin:/home/uboot/bin \
> -v /home/trini/u-boot:/home/uboot/u-boot \
> -v /home/trini/logs:/home/uboot/logs \
> -v ${B}:${B} \
> -w /home/uboot/u-boot/u-boot \
> -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/uboot/u-boot/u-boot-test-hooks/bin \
> $IMG \
> /home/uboot/bin/uboot-local-ci-in-docker.sh ${B} $NO32BIT $NO64BIT --log $LOG
>
> # Test all targets
> /home/trini/bin/uboot-local-ci-outside-docker.sh $B $NO32BIT $NO64BIT --log $LOG
>
> less $LOG
> ------------------------ >8 ------------------------
>
> Which is still a little debug-y, but is basically just build everything
> I want in the same container as CI, and then run pytest outside of that
> container. Could I do it inside the container? Maybe. This has grown
> from when I used to build everything on the host and used "flashair" SD
> cards to update my hardware. With labgrid everything might be abstracted
> enough that yes, really, I can stop doing that. I'll likely look in to
> that next just to clean things up more.
>
> Second is "uboot-local-ci-in-docker.sh" as invoked above:
> ------------------------ >8 ------------------------
> #!/bin/bash
> # Build everything we are going to test.
> set -e
>
> # Let git stop being mad about permissions.
> git config --global --add safe.directory /home/uboot/u-boot/u-boot
>
> buildaboard() {
> # Board is first, all fragments follow.
> BRD=$1
> shift 1
>
> # Second would be to use clang
> if [ "$1" == "--clang" ]; then
> echo "Building $BRD with clang" | tee -a $LOG
> OBJ=${B}/${BRD}_clang
> export HOSTCC=clang-16
> export CC=clang-16
> shift 1
> else
> echo "Building $BRD" | tee -a $LOG
> OBJ=${B}/${BRD}
> fi
> # All that remains are fragments.
> FRAGS=$@
>
> make O=${OBJ} ${BRD}_defconfig -s 2>&1 | tee -a $LOG
> pushd ${OBJ} >/dev/null
> [ ! -z "${FRAGS}" ] && ${S}/scripts/kconfig/merge_config.sh ${S}/configs/${BRD}_defconfig ${FRAGS} 2>&1 | tee -a $LOG
> make -sj$(nproc) 2>&1 | tee -a $LOG
> popd >/dev/null
> }
>
> # Prepare virtualenv
> if [ -f test/py/requirements.txt ]; then
> virtualenv -p /usr/bin/python3 /tmp/venv
> . /tmp/venv/bin/activate
> pip install --quiet -r test/py/requirements.txt
> pip install --quiet -r doc/sphinx/requirements.txt
> [ -f tools/buildman/requirements.txt ] && pip install --quiet -r tools/buildman/requirements.txt
> fi
>
> # Find our arguments
> while test $# -ne 0; do
> if [ "$1" == "--no-32bit" ]; then
> NO32BIT="true"
> shift
> elif [ "$1" == "--no-64bit" ]; then
> NO64BIT="true"
> shift
> elif [ "$1" == "--log" ]; then
> LOG=${2/trini/uboot}
> shift 2
> else
> B=$1
> shift
> fi
> done
>
> if [ -z "$LOG" ]; then
> echo "Must pass --log /path/to/log/file"
> exit 1
> fi
>
> S=`pwd`
>
> # Common to all newer TI platforms
> export BINMAN_INDIRS=/home/uboot/u-boot/ti-linux-firmware
>
> # Prepare defconfigs with addons
> echo 'CONFIG_UNIT_TEST=y' > ${B}/unittest.config
> echo 'CONFIG_CMD_BOOTMENU=y' > ${B}/general.config
> echo 'CONFIG_CMD_LOG=y' >> ${B}/general.config
> echo '# CONFIG_CMD_BOOTEFI_SELFTEST is not set' > ${B}/noefist.config
> echo 'CONFIG_CMD_BOOTEFI_HELLO=y' > ${B}/bootefi.config
> echo 'CONFIG_CMD_BOOTEFI_SELFTEST=y' >> ${B}/bootefi.config
> echo 'CONFIG_IPV6=y' > ${B}/net.config
> echo 'CONFIG_IPV6_ROUTER_DISCOVERY=y' >> ${B}/net.config
> echo 'CONFIG_CMD_TFTPPUT=y' >> ${B}/net.config
> echo 'CONFIG_FIT=y' >> ${B}/net.config
> echo 'CONFIG_FIT_SIGNATURE=y' >> ${B}/net.config
> echo '# CONFIG_AUTOBOOT_KEYED is not set' > ${B}/noautobootkeyed.config
> echo 'CONFIG_BOOTSTAGE_STASH_ADDR=0x02400000' > ${B}/bootstage-pi.config
> echo 'CONFIG_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
> echo 'CONFIG_BOOTSTAGE_STASH=y' >> ${B}/bootstage-pi.config
> echo 'CONFIG_CMD_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
>
> ## ARM
> if [ -d /opt/gcc-13.2.0-nolibc/arm-linux-gnueabi ]; then
> export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-
> else
> echo "Toolchain missing for ARM"
> exit 1
> fi
>
> if [ -z "$NO32BIT" ]; then
> # Raspberry Pi boards
> buildaboard rpi_4_32b ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
> buildaboard rpi_4_32b --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
> fi
>
> if [ -z "$NO64BIT" ]; then
> ## Build r5 cores for the TI platforms that need both
> ## Need to backport the kernel fix for cpp with clang and passing the right
> ## flags. Then need to figure out pip / venv.
> buildaboard am64x_evm_r5
> buildaboard am62x_evm_r5
> buildaboard am62x_beagleplay_r5
> fi
>
> ## AARCH64
> if [ -d /opt/gcc-13.2.0-nolibc/aarch64-linux ]; then
> export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/aarch64-linux/bin/aarch64-linux-
> else
> echo "Toolchain missing for ARM64"
> exit 1
> fi
>
> if [ -z "$NO64BIT" ]; then
> # Raspberry Pi platforms
> buildaboard rpi_4 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
> buildaboard rpi_4 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
> buildaboard rpi_arm64 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
> buildaboard rpi_arm64 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
>
> ## These TI platforms require an R5 core and then Axx core
> ## Need to backport the kernel fix for cpp with clang and passing the right
> ## flags.
> export BL31=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/bl31.bin
> export TEE=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/tee-raw.bin
> buildaboard am64x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config
>
> # The AM62x platforms share BL31/TEE.
> export BL31=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/bl31.bin
> export TEE=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/tee-raw.bin
> buildaboard am62x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config
> buildaboard am62x_beagleplay_a53 ${B}/noautobootkeyed.config ${B}/noefist.config ${B}/net.config
> fi
>
> echo "Building complete on `git rev-parse --short HEAD`"
> ------------------------ >8 ------------------------
> Which is a little debug-y but more so comments to self. I setup the
> additional changes to a platform that I want to test with, and then
> build them. For required additional binaries I point them at known good
> copies so I don't have to worry if a failure is because of U-Boot or
> some other project.
>
> Finally we have "uboot-local-ci-outside-docker.sh" which is where pytest
> is actually run.
> ------------------------ >8 ------------------------
> #!/bin/bash
> set -e
>
> # Find our arguments
> while test $# -ne 0; do
> if [ "$1" == "--no-32bit" ]; then
> NO32BIT="true"
> shift
> elif [ "$1" == "--no-64bit" ]; then
> NO64BIT="true"
> shift
> elif [ "$1" == "--log" ]; then
> LOG=$2
> shift 2
> else
> B=$1
> shift
> fi
> done
>
> if [ -z "$LOG" ]; then
> echo "Must pass --log /path/to/log/file"
> exit 1
> fi
>
> # Env
> export PATH=${PATH}:/home/trini/u-boot/u-boot-test-hooks/bin:/sbin
> export PYTHONPATH=/home/trini/u-boot/u-boot-test-hooks/py/lootbox:${PYTHONPATH}
>
> # Setup Labgrid variables and use the special virtualenv for now
> export LG_ENV=/home/trini/u-boot/lg_env.yaml
> export LG_CROSSBAR=ws://lootbox.k.g:20408/ws
> export PATH=$PATH:/home/trini/u-boot-test-hooks/bin
> source /home/trini/labgrid-venv/bin/activate
>
> runtestpy() {
> BT=$1
> shift 1
>
> if [ "$1" == "--clang" ]; then
> echo "Testing $BT built with clang"
> shift 1
> BD=${B}/${BT}
> RD=${B}/${BT}_clang_results
> PD=${B}/${BT}_clang_persistent_data
> else
> echo "Testing $BT"
> BD=${B}/${BT}
> RD=${B}/${BT}_results
> PD=${B}/${BT}_persistent_data
> fi
>
> # If we need special pytest arguments do that here.
> while getopts "sk:" options; do
> case $options in
> k)
> PA="-k $2"
> shift 2
> ;;
> esac
> done
>
> labgrid-client -c $LG_ENV -p $LG_PLACE release >/dev/null 2>&1 || true
> labgrid-client -c $LG_ENV -p $LG_PLACE acquire
> ./test/py/test.py -ra --bd $BT --build-dir $BD --result-dir $RD \
> --persistent-data-dir $PD --exitfirst $PA >> $LOG 2>&1
> labgrid-client -c $LG_ENV -p $LG_PLACE release
>
> }
>
> if [ -z "$NO64BIT" ]; then
> export LG_PLACE=rpi4
> runtestpy rpi_4
> runtestpy rpi_arm64
> runtestpy rpi_4 --clang
> runtestpy rpi_arm64 --clang
>
> export LG_PLACE=am64-sk
> runtestpy am64x_evm_a53
>
> export LG_PLACE=am62-sk
> runtestpy am62x_evm_a53
>
> export LG_PLACE=beagleplay
> runtestpy am62x_beagleplay_a53
>
> fi
>
> if [ -z "$NO32BIT" ]; then
> export LG_PLACE=rpi4
> runtestpy rpi_4_32b
> runtestpy rpi_4_32b --clang
> fi
>
> echo "Tests complete on `git rev-parse --short HEAD`"
> ------------------------ >8 ------------------------
>
> I will say, if there's one thing I don't quite like with how my
> integrations with labgrid work right now it's that acquire/release is
> done the way it is. This is where I think Simon's adding another hook is
> the right direction, if there's not some other hook that can be used and
> I'm just missing.
Thanks for posting this. I know you had described it but this makes a
lot of things clearer.
So basically you have set up builds for the different boards and use
the hook scripts to write them and do the actual power/reset/console
munging.
How do you handle interactive build/run, e.g. to run U-Boot on a
particular board quickly? Also how would gitlab work?
Re acquire/release, I added a -a option to auto-acquire the place and
release it afterwards. But if the console dies then so does Labgrid so
it doesn't release reliably...something to worry about when more
progress is made.
Regards,
Simon
More information about the U-Boot
mailing list