[PATCH v2 0/8] This series introduces U-Boot tests with tbot on real hardware.

[Heiko Schocher]

add U-Boot tests on real hardware with tbot, which is
started from gitlab CI (or other CI implementations like
jenkins, github,...)

Currently only one board in my lab at hungary is implemented, if series
gets accepted more will follow. May others are interested too,
and prepare also such setups ... or simply send me hardware, I can
add to my setup.

Current board is the imx8qxp based cxg3 board. To build and get a
working image you need some binary blobs, which are copied from the
lab host to the "CI host". After successfull build the resulting
images are copied to the lab host, and new flash.bin is booted
with uuu tool in USB SDP bootmode, and than installed with fastboot
to the emmc. Than power is switched off, bootmode set to emmc
and it gets checked, that U-Boot boots from emmc now.

Than test/py gets configured and started on the CI host which
runs the tests on the board in tha lab.

Hopefully the approach is generic enough to get other hardware
easy integrated, but if there are problems, just contact me,
if stuff is missing.

You can find a working example @gitlab in my i2c custodian branch
here:

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/jobs/1081203
(If you have no access rights, look below for a test started @github)

I dropped from above repo/branch the following commits in this series:

- tbottest: add github example
  as it is not interesting for U-Boot mainline (or?)

  Just for interest, with that the same test is started from repo @github, example:
  https://github.com/hsdenx/u-boot-test/actions/runs/14212012371/job/39821029362
  to see the full logs open there (Run U-Boot test)

- .gitlab-ci.yml: only start tbot tests
- remove .azure-pipelines.yml
  as this changes are only for speeding up my local testing.

The patches in this series:

- test/py/requirements.txt: update coverage
- tools/buildman/requirements.txt: update coverage
- test/py/requirements.txt: increase filelock version
- test/py/requirements.txt: remove unittest2 and testtools
- test: increase timeouts

are fixing small common issues I found with test/py setup ...
which could be go seperate into mainline, but in the first
shot, I want to have all patches in one series.

- CI: add U-Boot testing on real HW with tbot

  adds all the config needed for testing the imx8qxp based cxg3
  board with tbot in tbottest subdirectory. We should discuss where
  to put this code!

  May into u-boot:/test/tbottest ?

  This setup not only runs on gitlab, it also runs from github,
  jenkins ... or from a local shell without any changes in code. So you
  can use it for your normal development process (what was the goal from
  tbot at the beginning, to be more precise, it was may laziness to type
  a lot of commands on different machines again and again). So if you use
  this setup for your daily work you get the CI integration for free!

- .gitlab-ci.yml: add tbot-lab

  finally integrates the tbot test setup into .gitlab-ci.yml

  This works currently for my I2C custodian repo, with some CI secrets
  stored... we have to add this secrets to u-boot repo to get it up
  and running.

  If this is not possible may I should setup a gitlab runner?
  (Which I would like to prevent... but possible of course, and this
   would prevent the timing issues I fixed when tbot is started somewhere
   in the world)

Bonus patch with an idea for custodians I want to implement, if code
gets accepted:

- scripts: add helper for getting infos from your patchwork ToDo list

  is a preparation for activating my 8 year old setup, which
  downloaded patches from my Patchwork ToDo list, checked them
  with checkpatch.pl script  and applied them with git am, fully
  automated from within the CI !

  With that script it is very easy to reactivate this now again,
  if this is interesting for others. Currently I have not a clean
  patchwork ToDo list, which is the reson for adding this not yet.

  My plan is also to reactivate an automated git bisect session,
  in case there are problems with that patches, so test shows the
  problematic patch at the end... which I think would be a great
  feature and benefit to have in mainline CI.

  For reference an old video on youtube how this worked 8 years ago:

  https://www.youtube.com/watch?v=PhaYfqOrQOg

  It would be nice, we can make such a step configurable through
  a switch we can enable/disable before starting a gitlab CI run.

May time is now ready for it, that people have interests in such
a setup?


Changes in v2:
- patman accidentially added Signed-off-by, remove this with
  calling patman with parameter "--no-signoff"
  Sorry for that!

Heiko Schocher (8):
  test/py/requirements.txt: update coverage
  tools/buildman/requirements.txt: update coverage
  test/py/requirements.txt: increase filelock version
  test/py/requirements.txt: remove unittest2 and testtools
  test: increase timeouts
  [RFC] CI: add U-Boot testing on real HW with tbot
  .gitlab-ci.yml: add tbot-lab
  scripts: add helper for getting infos from your patchwork ToDo list

 .gitlab-ci.yml                                |  74 ++++
 MAINTAINERS                                   |   6 +
 scripts/getpatchlist.py                       | 126 +++++++
 tbottesting/README.md                         | 323 ++++++++++++++++++
 tbottesting/boardtestconfig.py                |  18 +
 tbottesting/boardtests.py                     |  50 +++
 tbottesting/preparetbot.sh                    |   5 +
 tbottesting/tbotboardtests.py                 | 127 +++++++
 tbottesting/tbotconfig-hs/boardspecific.py    | 111 ++++++
 tbottesting/tbotconfig-hs/hs/README.cxg3      |  28 ++
 tbottesting/tbotconfig-hs/hs/args/argsbase    |  11 +
 tbottesting/tbotconfig-hs/hs/args/argshs      |   1 +
 .../tbotconfig-hs/hs/args/argshs-github-ci    |   7 +
 .../tbotconfig-hs/hs/args/argshs-noeth        |   3 +
 .../tbotconfig-hs/hs/args/argshs-noeth-ssh    |   5 +
 tbottesting/tbotconfig-hs/hs/cxg3.ini         | 106 ++++++
 tbottesting/tbotconfig-hs/hs/tbot.ini         | 117 +++++++
 tbottesting/tbotconfig-hs/interactive.py      |  59 ++++
 tbottesting/tbotconfig-hs/labcallbacks.py     |  76 +++++
 tbottesting/tbotconfig-hs/tc_cxg3.py          |  82 +++++
 test/py/requirements.txt                      |   6 +-
 test/py/u_boot_console_base.py                |   4 +-
 tools/buildman/requirements.txt               |   2 +-
 23 files changed, 1340 insertions(+), 7 deletions(-)
 create mode 100755 scripts/getpatchlist.py
 create mode 100644 tbottesting/README.md
 create mode 100644 tbottesting/boardtestconfig.py
 create mode 100644 tbottesting/boardtests.py
 create mode 100755 tbottesting/preparetbot.sh
 create mode 100644 tbottesting/tbotboardtests.py
 create mode 100644 tbottesting/tbotconfig-hs/boardspecific.py
 create mode 100644 tbottesting/tbotconfig-hs/hs/README.cxg3
 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argsbase
 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs
 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-github-ci
 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-noeth
 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-noeth-ssh
 create mode 100644 tbottesting/tbotconfig-hs/hs/cxg3.ini
 create mode 100644 tbottesting/tbotconfig-hs/hs/tbot.ini
 create mode 100644 tbottesting/tbotconfig-hs/interactive.py
 create mode 100644 tbottesting/tbotconfig-hs/labcallbacks.py
 create mode 100644 tbottesting/tbotconfig-hs/tc_cxg3.py

-- 
2.20.1