[QUESTION] Experiencing errors with RPI CM4 4GB (but not 2GB)

[David Caputo]

Hello - this is my first message to this list, so I apologize in advance if
I've got some of the customs wrong in doing so.  I have searched prior
messages and am running a very recent build
(commit 08cca188c120b8337600091c94dc0e211cd03e10) which incorporates all of
the patches relating to similar issues from what I can tell.  My
environment is running yocto dunfell on kernel 5.4.85.

When I take a stock 2GB CM4 module (empty EMMC, latest EEPROM) and boot via
pcie/xhci (USB VL805-based PCIE card on CM4IO board) U-Boot initializes the
USB device, finds and runs the U-Boot script, and starts the kernel.
However when I take a stock 4GB CM4 module (also empty EMMC, latest EEPROM)
I get errors - see console output for both scenarios below.

CM4-2GB Successful Boot:
DRAM:  1.9 GiB
RPI Compute Module 4 (0xb03140)
MMC:   mmcnr at 7e300000: 1, emmc2 at 7e340000: 0
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet at 7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb at 7e980000: USB DWC2
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus usb at 7e980000 for devces... 1 USB Device(s) found
scanning bus xhci_pci for devices... 3 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

Device 0: Vendor:  USB Rev: 1.00 Prod:  SanDisk 3.2Gen1
            Tye: Removable Hard Disk
            Capacity: 29340.0 MB = 28.6 GB (60088320 x 512)
... is now current device
Scanning usb 0:1...
Found U-Boot script /boot.scr
616 bytes read in 2 ms (300.8 KiB/s)
## Exeuting script at 02400000
Scanning mmc usb devices 0 1 2
Failed to load 'resin-image-flasher'
24 bytes read in 13 ms (1000 Bytes/s)
Card did not respond to voltage select! : -110
Can't set block device
Can't set block devic
24 bytes read in 4 ms (5.9 KiB/s)
Found resin flasher on usb 0
Loading resinOS_uEnv.txt from usb device 0 partition 1
0 bytes read in 3 ms (0 Bytes/s)
Import resinOS_uEnv.txt in environmen
Loading extra_uEnv.txt from usb device 0 partition 1
0 bytes read in 2 ms (0 Bytes/s)
Import extra_uEnv.txt in environment
Loading bootcount.env from us device 0 partition 1
Failed to load 'bootcount.env'
No bootcount.env file. Setting bootcount=0 in environment
24 bytes read in 4 ms (5.9 KiB/s)
12468223 bytes read in 112 ms (106.2 MiB/s)
Uncompressed size: 28365312 = 0x1B0D200
## Flattened Device ree blob at 2eff4100
   Booting using the fdt blob at 0x2eff4100
   Using Device Tree in place at 000000002eff4100, end 00000002f002f80

Starting kernel ...

CM4-4GB Unsuccessful Boot:
U-Boot 2021.04-rc2 (Feb 26 2021 - 20:11:08 +0000)

DRAM:  3.8 GiB
RPI Compute Module 4 (0xc03140)
MMC:   mmcnr at 7e300000: 1, emmc2 at 7e340000: 0
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet at 7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb at 7e980000: USB DWC2
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
Port not available.
scanning bus usb at 7e980000 for devices... 1 USB Device(s) found
       scanning usb for storag devices... 0 Storage Device(s) found
"Synchronous Abort" handler, esr 0x96000004
elr: 000000000009c320 lr : 0000000000092090 (reloc)
elr: 000000037c76320 lr : 0000000037c6c090
x0 : d519b040aa010000 x1 : 000000000000005c
x2 : 0000000000800000 x3 : 0000000037cd340
x4 : b900080152b00000 x5 : 000000000000005c
x6 : 0000000037cd3460 x7 : b900080152afff90
x8 : 0000000000000000 x9 : 000000000000008
x10: 000000000000ad84 x11: 0000000037849d8c
x12: 000000000000ad88 x13: 0000000037849dd8
x14: 000000003789ef0 x15: 0000000000000002
x16: 0000000037c78438 x17: 0000000000000000
x18: 0000000037855d90 x19: 0000000000000070
x20 0000000037cd2ea0 x21: 0000000000000000
x22: 000000000000000b x23: 00000000ffffffff
x24: 0000000000000000 x25: 000000007ce4e64
x26: 0000000037cbe3fe x27: 0000000037cc777b
x28: 0000000000000064 x29: 0000000037849460

Code: eb03005f 54ffff3 f9400ca4 17ffffe0 (f9400404)
Resetting CPU ...

resetting ...

The closest issue I found in searching the mailing list was this one:
https://lists.denx.de/pipermail/u-boot/2020-December/435143.html
But I believe these patches are already applied in the version that I am
running.  Does anyone have any idea what could be causing this?  When I
limit the 4GB module to use only 3GB of ram (via total_mem=3072 in
config.txt) it works, but obviously this is just a workaround and not an
actual solution.

I posted this same issue to the RPI forums, and an RPI engineer noted the
following which may help to pinpoint the issue:

It's more likely to be related to how u-boot configures the inbound address
window (BAR2) depending on the memory configuration. In the latest
releases, the firmware patches device-tree to set this @ 16GB. I don't know
if u-boot reads this, it's free to ignore it.

IIRC this has to be multiple of the SDRAM size because the bus address
matching just masks off bits and 16GB is the maximum size that the bus
could support. Previously, the firmware used to do something like pick the
smallest multiple of SDRAM size.

Picking older/newer versions of the firmware may help to verify but u-boot
should override the number if it wants something else.

This recent FW change allows the Pi4 R6 VLI FW loading (not relevant on
CM4) to pickup the BAR2 configuration from the chain-loader / host OS by
reading from the PCIe registers. That might have been a blocker for u-boot
in the past.