Passing boot logs to Linux?

[Dragan Simic]

On 2023-12-21 00:27, Csókás Bence wrote:
> On 2023. 12. 20. 20:58, Dragan Simic wrote:
>> On 2023-12-20 20:24, Csókás Bence wrote:
>>> On 2023. 12. 20. 9:29, Dragan Simic wrote:
>>>> On 2023-12-20 08:52, Csókás Bence wrote:
>>>>> That's what I read as well. Is there support for U-boot to write 
>>>>> and
>>>>> Linux to read PStores?
>>>> 
>>>> No and yes, but U-Boot can already read pstore.  Please see
>>>> doc/usage/cmd/pstore.rst for the U-Boot part, and
>>>> Documentation/admin-guide/pstore-blk.rst for the Linux kernel part.
>>> 
>>> Irrelevant, as we only want to write out the console log to U-Boot, 
>>> and
>>> not the other way around (that's for collecting panic logs, which is
>>> already implemented).
>> 
>> Quite frankly, this is a bit confusing to me.  Why should it be
>> irrelevant, if we want to store the U-Boot console logs into the 
>> pstore,
>> and read them later in Linux?  That's exactly what you asked about, 
>> and
>> it was a good question.
> 
> Because we want to *write* PStore from U-Boot. The read being already
> implemented is not very useful, we could only re-read an 
> already-written
> console log (which is not helpful for us).

Well, it all depends on one's viewpoint.  See, even if we had no read 
support already implemented in U-Boot and wanted to just write the data, 
we'd almost surely have to implement reading as well, so the written 
stuff can be inspected in the U-Boot console.

>>>> Another benefit of using pstore would be no permanently wasted RAM 
>>>> for
>>>> the recorded console contents.  Also, having the data recorded to a
>>>> storage device also goes along with providing permanent records.
>>> 
>>> I'm positively *not gonna save boot logs to disk*, as most embedded
>>> systems have Flash-based storage media; writing to them on every boot
>>> would be devastating.
>> 
>> Writing 30-40 KB of data to an even remotely modern flash-based 
>> storage
>> media on each boot is pretty much nothing.  OTOH, writing 30-40 KB of
>> data to an SPI chip on each boot would be rather bad, but we aren't
>> going to do that for sure.
>> 
>> To put things into perspective, I'm writing this on a Pinebook Pro 
>> with
>> an eMMC chip, and it has stored many GBs of data over a few years.  
>> Even
>> high-endurance microSD cards are rather reliable in that regard.
> 
> Not every system has eMMC/uSD, and as you said, these arguments don't
> hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt
> router for example. Whereas RAM is quite cheap nowadays.

I see, but I also wonder how many such OpenWrt routers are still used 
these days, and, even more importantly, how many of them are regularly 
updated and can be expected to actually use this new feature?

Please, don't get me wrong, I still support having both options 
available, but I'm also wondering about the target demographic.

>>> Plus, I don't want the console subsystem to depend on any file/disk
>>> operations/drivers.
>> 
>> Well, the console would still work as usual even if logging to disk
>> would fail for any reason, which is similar to the serial console 
>> still
>> working if the graphical console fails.  Moreover, if the disk fails,
>> the system isn't be able to boot, so any RAM-based console logs would 
>> be
>> lost in that case.  All this makes the RAM-based logging no more
>> resilient to disk failures.
> 
> Correction: if disk *reads* fail, as well as writes, then the system
> will not boot. However, typical failure of Flash media is that it
> becomes read-only.

That's a good point, but having a read-only root filesystem usually also 
means having a non-operational system that can only have its stored data 
salvaged.  Unless the system is specifically crafted to survive such 
scenarios, of course.

>> I still think that using disk-based pstore is a better option.  Just 
>> as
>> you don't want to wear out your flash disks with 30-40 KB of data, I
>> also don't want to waste 30-40 KB of RAM.
> 
> As I said, you could just unload the log after you're done processing
> it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my
> laptop, but it can probably be less, maybe even 10x less, so 80-90k?),
> so you could, in your init, process the in-RAM log, then unload it, 
> then
> start your other services, thereby reclaiming that RAM.

Using pstore should have that unloading already covered, and the already 
existing systemd service is there to perform the archiving to the 
primary filesystem, if desired so.  It would all need to be tested in 
detail, of course.

>> Also, embedded devices such
>> as the PinePhone would benefit additionally from using disk-based
>> logging, because people often can't do anything in the field if 
>> booting
>> fails for some reason, so RAM-based logs would end up being lost,
>> preventing any post-mortem debugging later.
> 
> Post-mortem analysis could be interesting, true.

Yes, and it's often the only way to get to the bottom of the underlying 
issues, which are sometimes intermittent and can't be reproduced easily 
later.

>> Perhaps we can actually offer both options, i.e. both logging to RAM 
>> and
>> to disk, so everyone can choose what they prefer.  I'd also be fine 
>> with
>> working on such a dual approach, but it would obviously be more 
>> complex.
> 
> Perhaps that is the best option. But, as Sean has pointed out, U-Boot
> doesn't support disk-backed PStore anyways... So then, should we
> implement RAM-based PStore writing for the time being, and then maybe
> expand it later?

I'd vote for implementing both options at once, so we can reduce the 
overall amount of the required testing.  It will surely result in more 
time required for the development, but the total amount of time should 
be significantly shorter.