Passing boot logs to Linux?

[Simon Glass]

Hi,

On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic <dsimic at manjaro.org> wrote:
>
> On 2023-12-21 02:44, Dragan Simic wrote:
> > On 2023-12-21 02:37, Dragan Simic wrote:
> >> On 2023-12-21 02:03, Daniel Golle wrote:
> >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote:
> >>>> On 2023-12-21 00:27, Csókás Bence wrote:
> >>>> > Not every system has eMMC/uSD, and as you said, these arguments don't
> >>>> > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt
> >>>> > router for example. Whereas RAM is quite cheap nowadays.
> >>>>
> >>>> I see, but I also wonder how many such OpenWrt routers are still
> >>>> used these
> >>>> days, and, even more importantly, how many of them are regularly
> >>>> updated and
> >>>> can be expected to actually use this new feature?
> >>>
> >>> Avoid flash writes is a very important matter, even on systems with
> >>> 128 MiB of SPI-NAND flash which is by far the most common setup you
> >>> find on off-the-shelf plastic routers and access points nowadays.
> >>
> >> I agree, writing something to the SPI chips all the time, no matter
> >> how small the writes are, is a big no-no, which I already clearly
> >> expressed in one of my earlier posts.
> >>
> >>> Especially also as those devices often come without a local console,
> >>> having U-Boot's output prepended to dmesg on boot would be a very big
> >>> win.
> >>
> >> I was also thinking about that, but I'm not sure it would be accepted
> >> to the Linux kernel.  Maybe we can try getting that accepted later.
>
> Maybe, but just _maybe,_ it could be possible to add a new command-line
> option to dmesg(1) that would display the last recorded console output,
> fetched from the pstore.  That _might_ get accepted to util-linux, while
> being perfectly fine from the usability standpoint.
>
> I'm also willing to work on that, and I already contributed a few
> dmesg(1) patches to util-linux.
>
> >>>> Please, don't get me wrong, I still support having both options
> >>>> available,
> >>>> but I'm also wondering about the target demographic.
> >>>>
> >>>> > > > Plus, I don't want the console subsystem to depend on any file/disk
> >>>> > > > operations/drivers.
> >>>> > >
> >>>> > > Well, the console would still work as usual even if logging to disk
> >>>> > > would fail for any reason, which is similar to the serial console
> >>>> > > still
> >>>> > > working if the graphical console fails.  Moreover, if the disk fails,
> >>>> > > the system isn't be able to boot, so any RAM-based console logs
> >>>> > > would be
> >>>> > > lost in that case.  All this makes the RAM-based logging no more
> >>>> > > resilient to disk failures.
> >>>> >
> >>>> > Correction: if disk *reads* fail, as well as writes, then the system
> >>>> > will not boot. However, typical failure of Flash media is that it
> >>>> > becomes read-only.
> >>>>
> >>>> That's a good point, but having a read-only root filesystem usually
> >>>> also
> >>>> means having a non-operational system that can only have its stored
> >>>> data
> >>>> salvaged.  Unless the system is specifically crafted to survive such
> >>>> scenarios, of course.
> >>>
> >>> ... which holds true for any decent embedded OS, which at least
> >>> allows
> >>> limited remote access and some kind of recovery even in this
> >>> situation.
> >>
> >> Perhaps.  I'm more into running general-purpose Linux distributions on
> >> single-board computers and derived embedded devices, which are on the
> >> "thick" end of the embedded device spectrum, so to speak.
> >>
> >>>> > > I still think that using disk-based pstore is a better option.  Just
> >>>> > > as
> >>>> > > you don't want to wear out your flash disks with 30-40 KB of data, I
> >>>> > > also don't want to waste 30-40 KB of RAM.
> >>>> >
> >>>> > As I said, you could just unload the log after you're done processing
> >>>> > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my
> >>>> > laptop, but it can probably be less, maybe even 10x less, so 80-90k?),
> >>>> > so you could, in your init, process the in-RAM log, then unload it, then
> >>>> > start your other services, thereby reclaiming that RAM.
> >>>>
> >>>> Using pstore should have that unloading already covered, and the
> >>>> already
> >>>> existing systemd service is there to perform the archiving to the
> >>>> primary
> >>>> filesystem, if desired so.  It would all need to be tested in
> >>>> detail, of
> >>>> course.
> >>>
> >>> pstore/ramoops uses a statically assigned reserved memory region, so
> >>> in
> >>> the moment you want to use that feature you loose that amount of RAM
> >>> (a
> >>> few kB, so it doesn't really matter on modern systems).
> >>> As in: there is *no* dynamic allocation.
> >>>
> >>> Imho using pstore/ramoops (which is a more or less Linux-specific
> >>> debugging feature, meant to store one or more timestamped logs of
> >>> crashes) might not be the most suitable choice. I understand the
> >>> advantages of using existing infrastructure, but on the other hand
> >>> we don't need most of the complexity of pstore for the task.
> >>
> >> Hmm, let me research all that a bit more in the next couple of days,
> >> please, and I'll come back with a detailed insight.
> >>
> >>> What I'd like to see is having a couple of log lines from U-Boot
> >>> prepended to Linux' dmesg buffer, and for that we anyway will have to
> >>> come up with a way to hand over that buffer. Another reserved-memory
> >>> region would be one way, embedding the buffer as a blob into the
> >>> /chosen/ section of the device tree would be another way.
> >>
> >> As I already wrote above, it would be rather neat, but I'm afraid it
> >> wouldn't be accepted upstream.
> >
> > Sorry, I forgot this...
> >
> > As I already explained in one of my earlier posts, not all devices and
> > applications would benefit from having only "in-flight" console data
> > available, i.e. made available through a reserved region of the RAM.
> > Some devices actually need to have the recorded console outputs stored
> > on their eMMC chips or microSD cards, for post-mortem analysis of the
> > boot issues.
> >
> > That's why we actually need both options available, and that's also
> > why pstore should fit well, even if it may seem as an overkill.  I
> > hope you agree;  however, I'll do more research, as already promised.

I did send this a while back, in case it is useful:

https://www.spinics.net/lists/devicetree/msg573692.html

Regards,
Simon