Passing boot logs to Linux?

[Dragan Simic]

On 2023-12-21 02:44, Dragan Simic wrote:
> On 2023-12-21 02:37, Dragan Simic wrote:
>> On 2023-12-21 02:03, Daniel Golle wrote:
>>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote:
>>>> On 2023-12-21 00:27, Csókás Bence wrote:
>>>> > Not every system has eMMC/uSD, and as you said, these arguments don't
>>>> > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt
>>>> > router for example. Whereas RAM is quite cheap nowadays.
>>>> 
>>>> I see, but I also wonder how many such OpenWrt routers are still 
>>>> used these
>>>> days, and, even more importantly, how many of them are regularly 
>>>> updated and
>>>> can be expected to actually use this new feature?
>>> 
>>> Avoid flash writes is a very important matter, even on systems with
>>> 128 MiB of SPI-NAND flash which is by far the most common setup you
>>> find on off-the-shelf plastic routers and access points nowadays.
>> 
>> I agree, writing something to the SPI chips all the time, no matter
>> how small the writes are, is a big no-no, which I already clearly
>> expressed in one of my earlier posts.
>> 
>>> Especially also as those devices often come without a local console,
>>> having U-Boot's output prepended to dmesg on boot would be a very big
>>> win.
>> 
>> I was also thinking about that, but I'm not sure it would be accepted
>> to the Linux kernel.  Maybe we can try getting that accepted later.

Maybe, but just _maybe,_ it could be possible to add a new command-line 
option to dmesg(1) that would display the last recorded console output, 
fetched from the pstore.  That _might_ get accepted to util-linux, while 
being perfectly fine from the usability standpoint.

I'm also willing to work on that, and I already contributed a few 
dmesg(1) patches to util-linux.

>>>> Please, don't get me wrong, I still support having both options 
>>>> available,
>>>> but I'm also wondering about the target demographic.
>>>> 
>>>> > > > Plus, I don't want the console subsystem to depend on any file/disk
>>>> > > > operations/drivers.
>>>> > >
>>>> > > Well, the console would still work as usual even if logging to disk
>>>> > > would fail for any reason, which is similar to the serial console
>>>> > > still
>>>> > > working if the graphical console fails.  Moreover, if the disk fails,
>>>> > > the system isn't be able to boot, so any RAM-based console logs
>>>> > > would be
>>>> > > lost in that case.  All this makes the RAM-based logging no more
>>>> > > resilient to disk failures.
>>>> >
>>>> > Correction: if disk *reads* fail, as well as writes, then the system
>>>> > will not boot. However, typical failure of Flash media is that it
>>>> > becomes read-only.
>>>> 
>>>> That's a good point, but having a read-only root filesystem usually 
>>>> also
>>>> means having a non-operational system that can only have its stored 
>>>> data
>>>> salvaged.  Unless the system is specifically crafted to survive such
>>>> scenarios, of course.
>>> 
>>> ... which holds true for any decent embedded OS, which at least 
>>> allows
>>> limited remote access and some kind of recovery even in this 
>>> situation.
>> 
>> Perhaps.  I'm more into running general-purpose Linux distributions on
>> single-board computers and derived embedded devices, which are on the
>> "thick" end of the embedded device spectrum, so to speak.
>> 
>>>> > > I still think that using disk-based pstore is a better option.  Just
>>>> > > as
>>>> > > you don't want to wear out your flash disks with 30-40 KB of data, I
>>>> > > also don't want to waste 30-40 KB of RAM.
>>>> >
>>>> > As I said, you could just unload the log after you're done processing
>>>> > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my
>>>> > laptop, but it can probably be less, maybe even 10x less, so 80-90k?),
>>>> > so you could, in your init, process the in-RAM log, then unload it, then
>>>> > start your other services, thereby reclaiming that RAM.
>>>> 
>>>> Using pstore should have that unloading already covered, and the 
>>>> already
>>>> existing systemd service is there to perform the archiving to the 
>>>> primary
>>>> filesystem, if desired so.  It would all need to be tested in 
>>>> detail, of
>>>> course.
>>> 
>>> pstore/ramoops uses a statically assigned reserved memory region, so 
>>> in
>>> the moment you want to use that feature you loose that amount of RAM 
>>> (a
>>> few kB, so it doesn't really matter on modern systems).
>>> As in: there is *no* dynamic allocation.
>>> 
>>> Imho using pstore/ramoops (which is a more or less Linux-specific
>>> debugging feature, meant to store one or more timestamped logs of
>>> crashes) might not be the most suitable choice. I understand the
>>> advantages of using existing infrastructure, but on the other hand
>>> we don't need most of the complexity of pstore for the task.
>> 
>> Hmm, let me research all that a bit more in the next couple of days,
>> please, and I'll come back with a detailed insight.
>> 
>>> What I'd like to see is having a couple of log lines from U-Boot
>>> prepended to Linux' dmesg buffer, and for that we anyway will have to
>>> come up with a way to hand over that buffer. Another reserved-memory
>>> region would be one way, embedding the buffer as a blob into the
>>> /chosen/ section of the device tree would be another way.
>> 
>> As I already wrote above, it would be rather neat, but I'm afraid it
>> wouldn't be accepted upstream.
> 
> Sorry, I forgot this...
> 
> As I already explained in one of my earlier posts, not all devices and
> applications would benefit from having only "in-flight" console data
> available, i.e. made available through a reserved region of the RAM.
> Some devices actually need to have the recorded console outputs stored
> on their eMMC chips or microSD cards, for post-mortem analysis of the
> boot issues.
> 
> That's why we actually need both options available, and that's also
> why pstore should fit well, even if it may seem as an overkill.  I
> hope you agree;  however, I'll do more research, as already promised.