[U-Boot] [PATCH 00/23] efi_loader implement missing functions

[Rob Clark]

On Tue, Sep 5, 2017 at 4:55 AM, Simon Glass <sjg at chromium.org> wrote:
> Hi,
>
> On 1 September 2017 at 22:45, Tom Rini <trini at konsulko.com> wrote:
>> On Wed, Aug 30, 2017 at 04:16:34AM +0800, Simon Glass wrote:
>>> Hi,
>>>
>>> On 29 August 2017 at 22:16, Rob Clark <robdclark at gmail.com> wrote:
>>> > On Tue, Aug 29, 2017 at 8:57 AM, Leif Lindholm <leif.lindholm at linaro.org> wrote:
>>> >> On Tue, Aug 29, 2017 at 02:26:48PM +0200, Alexander Graf wrote:
>>> >>> > > > I would add command
>>> >>> > > > bootefi selftest.efi
>>> >>> > > > to run the tests and provide the python wrapper code to add it to the
>>> >>> > > > test suite.
>>> >>> > >
>>> >>> > > I think that's a great idea, yes.
>>> >>> > I wonder how far we are from running UEFI tests (either the official
>>> >>> > ones, or I seem to remember hearing about some other test suite which
>>> >>> > didn't require UEFI shell)?
>>> >>>
>>> >>> Let's ask Leif, Ard and Dong.
>>> >>>
>>> >>> The official test suite definitely needs the UEFI Shell. Is the suite
>>> >>> publicly available by now?
>>> >>
>>> >> In binary form, you can access it already from the links on
>>> >> http://uefi.org/testtools
>>> >>
>>> >> Yes, 2.5 is latest release. No this is not a restriction ... the SCT
>>> >> releases have been lagging the specification releases a fair bit.
>>> >>
>>> >> The 2.5a package contains AARCH64, IA32 and X64 support (not ARM).
>>> >> Not that it couldn't contain ARM, it just hasn't been packaged.
>>> >>
>>> >>> > That seems more useful long term than re-inventing comprehensive UEFI
>>> >>> > test suite.  (Also, beyond just running shim/fallback/grub, I don't
>>> >>> > really have time to invent new tests for the stack of efi_loader
>>> >>> > patches I have.)
>>> >>>
>>> >>> Yes and no - it depends on the availability of the official suite :/.
>>> >>
>>> >> UEFI SCT is not yet open source/free software. Obviously, this is
>>> >> something Linaro has been lobbying for since day one of our
>>> >> involvement. There used to be little understanding for this, but that
>>> >> attitude has shifted substantially.
>>> >
>>> > So, if/until UEFI SCT is not an option, what about:
>>> >
>>> >   https://01.org/linux-uefi-validation
>>> >
>>> > (thx to pjones for pointing that out to me)
>>>
>>> Well in any case I'm not looking for a large functional test suite at
>>> this stage. It certainly could be useful, but not as a replacement for
>>> unit tests. The latter is for fast verification (so that everyone can
>>> run it as part of 'make tests') and easy identification of the
>>> location of bugs.
>>
>> I want to chime in here.  Unless we're talking hours of time, if "make
>> tests" takes 5 minutes to run, but that includes doing some sort of full
>> validation suite to the relevant EFI code, that seems like a win to me.
>> And if someone else is responsible for the contents of the tests and we
>> just have to confirm our expected results, that's an even bigger win.
>
> Yes it certainly is a win. But I'm already upset with how long the
> tests take to run so I would not be keen on anything that increases it
> much. How long does this test suite take to run normally?
>

I'm not sure how long SCT would take to run until we get it working (I
have a growing stack of patches on my wip-enough-uefi-for-shell
branch, and have Shell.efi running but still debugging sct.efi)..
maybe someone who has run SCT on a real UEFI implementation knows..
But that said, there seems to be some potential to run a subset of the
tests.  Either way, it seems like most of the time that travis runs
take is building a million variants of u-boot.  I'm not sure the time
spent running tests is really the issue.

BR,
-R