Passing boot logs to Linux?

[Dragan Simic]

On 2023-12-26 23:38, Csókás Bence wrote:
> On 2023. 12. 26. 19:09, Dragan Simic wrote:
>> On 2023-12-26 10:46, Simon Glass wrote:
>>> On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic <dsimic at manjaro.org> 
>>> wrote:
>>>> On 2023-12-21 02:44, Dragan Simic wrote:
>>>> > On 2023-12-21 02:37, Dragan Simic wrote:
>>>> >> On 2023-12-21 02:03, Daniel Golle wrote:
>>>> >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote:
>>>> >>>> On 2023-12-21 00:27, Csókás Bence wrote:
>>>> >>>> > Not every system has eMMC/uSD, and as you said, these
>>>> arguments don't
>>>> >>>> > hold for a 4 MB SPI NAND, for example, one you might find in
>>>> an OpenWrt
>>>> >>>> > router for example. Whereas RAM is quite cheap nowadays.
>>>> >>>>
>>>> >>>> I see, but I also wonder how many such OpenWrt routers are still
>>>> >>>> used these
>>>> >>>> days, and, even more importantly, how many of them are regularly
>>>> >>>> updated and
>>>> >>>> can be expected to actually use this new feature?
>>>> >>>
>>>> >>> Avoid flash writes is a very important matter, even on systems with
>>>> >>> 128 MiB of SPI-NAND flash which is by far the most common setup you
>>>> >>> find on off-the-shelf plastic routers and access points nowadays.
>>>> >>
>>>> >> I agree, writing something to the SPI chips all the time, no matter
>>>> >> how small the writes are, is a big no-no, which I already clearly
>>>> >> expressed in one of my earlier posts.
>>>> >>
>>>> >>> Especially also as those devices often come without a local console,
>>>> >>> having U-Boot's output prepended to dmesg on boot would be a very
>>>> big
>>>> >>> win.
>>>> >>
>>>> >> I was also thinking about that, but I'm not sure it would be accepted
>>>> >> to the Linux kernel.  Maybe we can try getting that accepted later.
>>>> 
>>>> Maybe, but just _maybe,_ it could be possible to add a new 
>>>> command-line
>>>> option to dmesg(1) that would display the last recorded console 
>>>> output,
>>>> fetched from the pstore.  That _might_ get accepted to util-linux, 
>>>> while
>>>> being perfectly fine from the usability standpoint.
>>>> 
>>>> I'm also willing to work on that, and I already contributed a few
>>>> dmesg(1) patches to util-linux.
>>>> 
>>>> >>>> Please, don't get me wrong, I still support having both options
>>>> >>>> available,
>>>> >>>> but I'm also wondering about the target demographic.
>>>> >>>>
>>>> >>>> > > > Plus, I don't want the console subsystem to depend on any
>>>> file/disk
>>>> >>>> > > > operations/drivers.
>>>> >>>> > >
>>>> >>>> > > Well, the console would still work as usual even if logging
>>>> to disk
>>>> >>>> > > would fail for any reason, which is similar to the serial
>>>> console
>>>> >>>> > > still
>>>> >>>> > > working if the graphical console fails.  Moreover, if the
>>>> disk fails,
>>>> >>>> > > the system isn't be able to boot, so any RAM-based console logs
>>>> >>>> > > would be
>>>> >>>> > > lost in that case.  All this makes the RAM-based logging no
>>>> more
>>>> >>>> > > resilient to disk failures.
>>>> >>>> >
>>>> >>>> > Correction: if disk *reads* fail, as well as writes, then the
>>>> system
>>>> >>>> > will not boot. However, typical failure of Flash media is that it
>>>> >>>> > becomes read-only.
>>>> >>>>
>>>> >>>> That's a good point, but having a read-only root filesystem usually
>>>> >>>> also
>>>> >>>> means having a non-operational system that can only have its stored
>>>> >>>> data
>>>> >>>> salvaged.  Unless the system is specifically crafted to survive
>>>> such
>>>> >>>> scenarios, of course.
>>>> >>>
>>>> >>> ... which holds true for any decent embedded OS, which at least
>>>> >>> allows
>>>> >>> limited remote access and some kind of recovery even in this
>>>> >>> situation.
>>>> >>
>>>> >> Perhaps.  I'm more into running general-purpose Linux
>>>> distributions on
>>>> >> single-board computers and derived embedded devices, which are on the
>>>> >> "thick" end of the embedded device spectrum, so to speak.
>>>> >>
>>>> >>>> > > I still think that using disk-based pstore is a better
>>>> option.  Just
>>>> >>>> > > as
>>>> >>>> > > you don't want to wear out your flash disks with 30-40 KB of
>>>> data, I
>>>> >>>> > > also don't want to waste 30-40 KB of RAM.
>>>> >>>> >
>>>> >>>> > As I said, you could just unload the log after you're done
>>>> processing
>>>> >>>> > it. 40 KB RAM is less, than what `sshd` uses, for instance
>>>> (860k on my
>>>> >>>> > laptop, but it can probably be less, maybe even 10x less, so
>>>> 80-90k?),
>>>> >>>> > so you could, in your init, process the in-RAM log, then
>>>> unload it, then
>>>> >>>> > start your other services, thereby reclaiming that RAM.
>>>> >>>>
>>>> >>>> Using pstore should have that unloading already covered, and the
>>>> >>>> already
>>>> >>>> existing systemd service is there to perform the archiving to the
>>>> >>>> primary
>>>> >>>> filesystem, if desired so.  It would all need to be tested in
>>>> >>>> detail, of
>>>> >>>> course.
>>>> >>>
>>>> >>> pstore/ramoops uses a statically assigned reserved memory region, so
>>>> >>> in
>>>> >>> the moment you want to use that feature you loose that amount of RAM
>>>> >>> (a
>>>> >>> few kB, so it doesn't really matter on modern systems).
>>>> >>> As in: there is *no* dynamic allocation.
>>>> >>>
>>>> >>> Imho using pstore/ramoops (which is a more or less Linux-specific
>>>> >>> debugging feature, meant to store one or more timestamped logs of
>>>> >>> crashes) might not be the most suitable choice. I understand the
>>>> >>> advantages of using existing infrastructure, but on the other hand
>>>> >>> we don't need most of the complexity of pstore for the task.
>>>> >>
>>>> >> Hmm, let me research all that a bit more in the next couple of days,
>>>> >> please, and I'll come back with a detailed insight.
>>>> >>
>>>> >>> What I'd like to see is having a couple of log lines from U-Boot
>>>> >>> prepended to Linux' dmesg buffer, and for that we anyway will
>>>> have to
>>>> >>> come up with a way to hand over that buffer. Another reserved-memory
>>>> >>> region would be one way, embedding the buffer as a blob into the
>>>> >>> /chosen/ section of the device tree would be another way.
>>>> >>
>>>> >> As I already wrote above, it would be rather neat, but I'm afraid it
>>>> >> wouldn't be accepted upstream.
>>>> >
>>>> > Sorry, I forgot this...
>>>> >
>>>> > As I already explained in one of my earlier posts, not all devices and
>>>> > applications would benefit from having only "in-flight" console data
>>>> > available, i.e. made available through a reserved region of the RAM.
>>>> > Some devices actually need to have the recorded console outputs stored
>>>> > on their eMMC chips or microSD cards, for post-mortem analysis of the
>>>> > boot issues.
>>>> >
>>>> > That's why we actually need both options available, and that's also
>>>> > why pstore should fit well, even if it may seem as an overkill.  I
>>>> > hope you agree;  however, I'll do more research, as already promised.
>>> 
>>> I did send this a while back, in case it is useful:
>>> 
>>> https://www.spinics.net/lists/devicetree/msg573692.html
> 
> So, what's the status on that, did it get merged? Or did it also get
> stuck at "maybe we should use pstore"?
> 
>> This looks quite informative, thank you.  I'll make sure to read the
>> entire thread carefully.
>> 
>> Unfortunately, I've contracted some kind of flu that has rendered me
>> incapable of even thinking straight.  I'm hoping to get better in the
>> next few days.
> 
> I'm sad to hear that. Get better soon!

Thank you very much.  It's been really bad for about four days, but 
thankfully I seem to be getting a bit better now.

> In the meantime, I think I'll send a PATCH RFC (I'll CC y'all) with a
> minimal implementation, that just collects the log and writes it out to
> a block of memory, with the struct I described earlier. That way we 
> will
> be in the Merge Window, and then hopefully we can chisel out the best
> implementation by the last RC, whether that be PStores, this schema
> proposed by Simon, or the minimal implementation.
> 
> In that time, you are free to send in PStore writing support, filtering
> ANSI codes from CONSOLE_RECORD or whatever else you want me to use.

Sounds like a plan to me.