Passing boot logs to Linux?

[Dragan Simic]

On 2023-12-21 02:37, Dragan Simic wrote:
> On 2023-12-21 02:03, Daniel Golle wrote:
>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote:
>>> On 2023-12-21 00:27, Csókás Bence wrote:
>>> > Not every system has eMMC/uSD, and as you said, these arguments don't
>>> > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt
>>> > router for example. Whereas RAM is quite cheap nowadays.
>>> 
>>> I see, but I also wonder how many such OpenWrt routers are still used 
>>> these
>>> days, and, even more importantly, how many of them are regularly 
>>> updated and
>>> can be expected to actually use this new feature?
>> 
>> Avoid flash writes is a very important matter, even on systems with
>> 128 MiB of SPI-NAND flash which is by far the most common setup you
>> find on off-the-shelf plastic routers and access points nowadays.
> 
> I agree, writing something to the SPI chips all the time, no matter
> how small the writes are, is a big no-no, which I already clearly
> expressed in one of my earlier posts.
> 
>> Especially also as those devices often come without a local console,
>> having U-Boot's output prepended to dmesg on boot would be a very big
>> win.
> 
> I was also thinking about that, but I'm not sure it would be accepted
> to the Linux kernel.  Maybe we can try getting that accepted later.
> 
>>> Please, don't get me wrong, I still support having both options 
>>> available,
>>> but I'm also wondering about the target demographic.
>>> 
>>> > > > Plus, I don't want the console subsystem to depend on any file/disk
>>> > > > operations/drivers.
>>> > >
>>> > > Well, the console would still work as usual even if logging to disk
>>> > > would fail for any reason, which is similar to the serial console
>>> > > still
>>> > > working if the graphical console fails.  Moreover, if the disk fails,
>>> > > the system isn't be able to boot, so any RAM-based console logs
>>> > > would be
>>> > > lost in that case.  All this makes the RAM-based logging no more
>>> > > resilient to disk failures.
>>> >
>>> > Correction: if disk *reads* fail, as well as writes, then the system
>>> > will not boot. However, typical failure of Flash media is that it
>>> > becomes read-only.
>>> 
>>> That's a good point, but having a read-only root filesystem usually 
>>> also
>>> means having a non-operational system that can only have its stored 
>>> data
>>> salvaged.  Unless the system is specifically crafted to survive such
>>> scenarios, of course.
>> 
>> ... which holds true for any decent embedded OS, which at least allows
>> limited remote access and some kind of recovery even in this 
>> situation.
> 
> Perhaps.  I'm more into running general-purpose Linux distributions on
> single-board computers and derived embedded devices, which are on the
> "thick" end of the embedded device spectrum, so to speak.
> 
>>> > > I still think that using disk-based pstore is a better option.  Just
>>> > > as
>>> > > you don't want to wear out your flash disks with 30-40 KB of data, I
>>> > > also don't want to waste 30-40 KB of RAM.
>>> >
>>> > As I said, you could just unload the log after you're done processing
>>> > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my
>>> > laptop, but it can probably be less, maybe even 10x less, so 80-90k?),
>>> > so you could, in your init, process the in-RAM log, then unload it, then
>>> > start your other services, thereby reclaiming that RAM.
>>> 
>>> Using pstore should have that unloading already covered, and the 
>>> already
>>> existing systemd service is there to perform the archiving to the 
>>> primary
>>> filesystem, if desired so.  It would all need to be tested in detail, 
>>> of
>>> course.
>> 
>> pstore/ramoops uses a statically assigned reserved memory region, so 
>> in
>> the moment you want to use that feature you loose that amount of RAM 
>> (a
>> few kB, so it doesn't really matter on modern systems).
>> As in: there is *no* dynamic allocation.
>> 
>> Imho using pstore/ramoops (which is a more or less Linux-specific
>> debugging feature, meant to store one or more timestamped logs of
>> crashes) might not be the most suitable choice. I understand the
>> advantages of using existing infrastructure, but on the other hand
>> we don't need most of the complexity of pstore for the task.
> 
> Hmm, let me research all that a bit more in the next couple of days,
> please, and I'll come back with a detailed insight.
> 
>> What I'd like to see is having a couple of log lines from U-Boot
>> prepended to Linux' dmesg buffer, and for that we anyway will have to
>> come up with a way to hand over that buffer. Another reserved-memory
>> region would be one way, embedding the buffer as a blob into the
>> /chosen/ section of the device tree would be another way.
> 
> As I already wrote above, it would be rather neat, but I'm afraid it
> wouldn't be accepted upstream.

Sorry, I forgot this...

As I already explained in one of my earlier posts, not all devices and 
applications would benefit from having only "in-flight" console data 
available, i.e. made available through a reserved region of the RAM.  
Some devices actually need to have the recorded console outputs stored 
on their eMMC chips or microSD cards, for post-mortem analysis of the 
boot issues.

That's why we actually need both options available, and that's also why 
pstore should fit well, even if it may seem as an overkill.  I hope you 
agree;  however, I'll do more research, as already promised.