[PATCH 0/3] RFC: test: Bring in the test hooks

[Heinrich Schuchardt]

On 5/2/25 21:30, Tom Rini wrote:
> On Fri, May 02, 2025 at 07:19:18PM +0200, Heinrich Schuchardt wrote:
>> On 5/2/25 17:06, Tom Rini wrote:
>>> On Fri, May 02, 2025 at 08:49:12AM -0600, Simon Glass wrote:
>>>> Hi Tom,
>>>>
>>>> On Fri, 2 May 2025 at 08:34, Tom Rini <trini at konsulko.com> wrote:
>>>>>
>>>>> On Thu, May 01, 2025 at 08:50:16PM -0600, Simon Glass wrote:
>>>>>
>>>>>> During a recent discussion with Heinrich we discussed why the hooks are
>>>>>> kept in a separate repo.
>>>>>>
>>>>>> The amount of code is small, a tenth of the size of the recently added
>>>>>> lwip, just by way of example. Testing is a critical part of U-Boot and
>>>>>> one of the things that distinguishes it from firmware projects that have
>>>>>> not kept up in this area. By having the tests somewhere else, we are
>>>>>> signalling that it is unusual, or difficult, or optional.
>>>>>>
>>>>>> The hooks mechanism also needs something of an update to take account of
>>>>>> real boards in 2025. That will be much easier to undertake if the code
>>>>>> that test/py talks to is in the same repo.
>>>>>>
>>>>>> This series brings the hook files in as first-class citizens of U-Boot.
>>>>>>
>>>>>> If we do go ahead with this, I will send a different series which has
>>>>>> separate commits (with correct author) in the u-boot-test-hooks repo.
>>>>>
>>>>> I think bringing more projects directly in to the repository is a bad
>>>>> idea. Your example of lwip isn't applicable because it's a read-only
>>>>> subtree that's maintained outside of the project (same as the dts
>>>>> subtree). But sure, lets "Say Yes". That said, we still need to:
>>>>> - Remove needless examples from the tree.
>>>>> - Not include personal labs directly in the tree.
>>>>>
>>>>> That last one is why I really think this is a bad idea. The point of
>>>>> having the hooks standalone is so that any given lab can easily add
>>>>> support for their lab and manage it, without worrying about disclosing
>>>>> internal layout. There's going to be hard coded default passwords there.
>>
>> Secrets MUST NEVER reside in git repos.
>>
>> I can't imagine that such irresponsible habits would be practiced by any
>> accountable developer.
> 
> I agree it shouldn't be done. And just this week was the most recent
> time I've seen a company do that internally, because it's internal and
> was just a temporary thing. We helped them stop needing to do that, but
> assuming it never happens is a bad idea.
> 
>> Please, have a look at
>>
>> https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions
>> https://tsi-ccdoc.readthedocs.io/en/master/ResOps/2019/gitlab/07_pass-build-secrets.html
>>
>> to see how to do it properly.
>>
>> The same is true for a private lab:
>> Use environment variables that are coming from outside the repository.
>>
>>>>> There's going to be repository secrets there. That kind of information
>>>>> really should not be in a public repository. Integrating the hooks with
>>>>> mainline will make lab management harder, not easier. The point of the
>>>>> existing labs in u-boot-test-hooks is to provide samples.
>>>>>
>>>>> I think this is all why no, we should not go down this path.
>>>>
>>>> Is it worth discussing this, or is your mind made up? I have some
>>>> thoughts on the last one.
>>>
>>> I think it's a terrible idea that I already said:
>>>> But sure, lets "Say Yes".
>>>
>>> But please do spend time explaining your thoughts and perhaps others
>>> will also agree with you and I'll feel less bad taking this in?
>>>
>>
>> Currently when changing a test hook I have to go through these steps:
>>
>> * fork u-boot-test-hooks
>> * push a commit to my fork
>> * update both .gitlab-ci.yaml and .azure-pipelines.yaml
>> * commit the change code.denx.de
>> * create a merge request for github.com/u-boot/u-boot
>>
>> If the test hooks were in the same repo as main U-Boot, I could save two
>> steps.
> 
> Yes, that's true, the steps for updating our CI are a little bit longer.
> I would be happy to make u-boot-test-hooks more widely writable (so it
> would instead be pushing to a branch for step 1+2). But I'm also
> concerned with external labs (which is both the origin of these scripts,
> and where they're also used).
> 
> Today, I have both a "konsulko" branch of u-boot-test-hooks, internally.
> That has subdirectories for both "sage" and "lootbox" (the two lab
> hosts), and in turn py and bin subdirectories for each. There's no value
> in publishing these changes as there's nothing unique about the
> platforms there, which aren't already in the mainline u-boot-test-hooks.
> 
> How should I manage these changes moving forward?

You could create two branches based on origin/master u-boot:

One for maintaining your test lab specific u-boot-test changes and one 
for your private u-boot code changes.

Check out the lab specific u-boot-test branch for steering the tests of 
the u-boot code branch.

As said don't put the secrets into any of your branches.

Best regards

Heinrich