[PATCH v1] test: Allow simple glob pattern in the test name

[Andy Shevchenko]

On Mon, Feb 08, 2021 at 10:07:55AM -0700, Simon Glass wrote:
> On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko
> <andriy.shevchenko at linux.intel.com> wrote:
> > On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
> > > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko
> > > <andriy.shevchenko at linux.intel.com> wrote:
> > > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> > > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > > > > > goes red, which is wrong.
> > > > > >
> > > > > > One more thing, is it known bug that either in the original code, or in your
> > > > > > new branch the following test case is 100% failed?
> > > > > >
> > > > > >         /* Non-existent in DTB */
> > > > > >         ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> > > > > >
> > > > > > Can you fix this sooner than later, please?
> > > > >
> > > > > Actually it seems this very patch makes the issue visible (I suppose something
> > > > > with test case names).
> > > >
> > > > Okay, actually there *is* a problem with the test suite, i.e. you may not run
> > > > some test cases twice during the same session.
> > >
> > > If this is related to the squashfs tests? I have disabled them for now
> > > in my local tree.
> >
> > I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
> >  - first time:          Failures: 0
> >  - second time:         Failures: 9
> >  - third and next time: Failures: 12
> >
> >  Means that 12 test cases are written badly.
> 
> OK I see. It did not use to be possible to pass all tests without
> running through pytest, which does some setup. You still need to
> create a 'spi.bin' file for the SPI tests to pass.

I run it _after_ the official pytest passed. So, every file is prepared.

> I'm not sure what prevents multiple runs without quitting U-Boot, but
> I suspect it is just some state hanging around, as we don't reset
> everything. For example, sandbox provides control of what happens when
> a sysreset is performed, and perhaps that is not reset to initial
> values correct by state_reset_for_test()..

I believe this is the case in most of the cases.

-- 
With Best Regards,
Andy Shevchenko