CM5 with random delays on boot

[Tholag]

Hello everyone!
A random delay in bootup bugs me or my RPI CM5 for a few weeks.
My best guess for now is that U-Boot triggers some process in rp1, the shdc-controller or the mmc.But I don’t know how to diagnose any further, so I send the details I can provide below.
Maybe someone can help?
Thanks!
Tholag


System:
Raspberry CM5 with 2GB RAM/16GB mmc, custom developer board, yocto/poky styhead and u-boot 2025.04


Symptom:
Using U-Boot, at about 2 out of three boot ups the kernel is in an exact 130 seconds wait state for root:
Waiting for root device /dev/mmcblk0p2...

It does not happen without U-Boot or with Raspbian, so I think a defect of the mmc is unlikely.
The delay occurs completely random, neither uptime, soft-reboot, cold boot, sync before reboot, filesystem activity before reboot make any difference. But it is always exact 130 seconds long (see below why...).


Diagnosis:

There are 13 mmc timeouts (10 seconds each?) on delayed boot-ups and none of them on regular boot-ups:
cat /sys/kernel/debug/mmc0/err_stats# Request Timedout:  13
Adding the `initcall_debug` kernel parameter, a late rp1 firmware probe also came to my attention:
[    2.736439] Waiting for root device /dev/mmcblk0p2...[   12.764515] probe of rp1_firmware returned 19 after 9 usecs[  133.949568] mmc0: Command Queue Engine enabled, 16 tags[  133.954800] mmc0: new HS400 Enhanced strobe MMC card at address 000
I found this in the rpi-kernel repo on GitHub, maybe it is related or helps otherwise:
https://github.com/raspberrypi/linux/issues/6642


Example bootlog: https://pastebin.com/LUYS3Dex



Unsuccessful approaches so far:
- Updating CM5 firmware (up to the current release of march-27th 2025 from https://github.com/raspberrypi/rpi-eeprom)- Mounting root via UUID instead of device- disabling the command queue engine in kernel with assorted sdhci.debug_quirks- rootwait, rootdelay and some other kernel parameters - „sync“ before reboot- No difference whatsoever if warm reboot, cold start, different ways of (clean) shutdown- Filesystems are good- Updating to u-boot 2025.04 from 2024.07