CRASH caused by: [PATCH 06/10] setexpr: Add some tests for buffer overflow and backref

[Simon Glass]

Hi Tom, Heinrich,

On Tue, 19 Jan 2021 at 12:45, Tom Rini <trini at konsulko.com> wrote:
>
> On Tue, Jan 19, 2021 at 11:06:18AM -0700, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Sun, 17 Jan 2021 at 15:52, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> > >
> > > On 11/1/20 10:15 PM, Simon Glass wrote:
> > > > Add tests to check for buffer overflow using simple replacement as well
> > > > as back references. At present these don't fully pass.
> > > >
> > > > Signed-off-by: Simon Glass <sjg at chromium.org>
> > > > ---
> > > >
> > > >   cmd/setexpr.c      | 21 +++--------
> > > >   include/command.h  | 17 +++++++++
> > > >   test/cmd/setexpr.c | 89 ++++++++++++++++++++++++++++++++++++++++++++++
> > > >   3 files changed, 110 insertions(+), 17 deletions(-)
> >
> > Yes this is intended for sandbox.
> >
> > Do you really want to run this test on a board? If the compiler is
> > working, then the sandbox test should be enough.
> >
> > I can certainly update it to run on boards, but that was not the intent.
>
> I think it is good to run tests on hardware when there's no reason not
> to.  It ends up adding to the "if we're up and doing stuff for a while,
> is the platform stable?" form of testing, which is harder to replicate
> but useful at times.

Yes, more testing is good. One reason not to is that it adds to code
size, depending on the test type. But we can do a special build that
enables CONFIG_UNIT_TEST for a board, so that can be avoided.

Another reason is that if we do have a test failure in common code,
then having just one test that covers if means we see a single test
failure, which is easier to diagnose than a failure on 110 boards,
particularly if it is sandbox and so does not require real hardware.

Python tests can often be enabled without adding to code size, but
they are very slow. C tests require special enablement but we can run
1000s of them quickly. Maybe we should write up the trade-offs and
come up with a test policy.

Regards,
Simon