T113 sunxi SPI NAND thoughts, feedback wanted

[John Watts]

Greetings sunxi and U-Boot friends!

Over the past five months I've managed to slog through getting a
complete SPI NAND U-Boot and Linux setup running on my Mango Pi MQ.

My tree is here, but I will be slowly trying to upstream my work over
the next few months: https://github.com/Jookia/u-boot/commits/jookia_t113/

It is based on master at the moment with the following patches:
- SUNIV SPI NAND support in SPL
  (https://lore.kernel.org/u-boot/8034158c-03ab-7488-6afa-a67f04264705@gmail.com)
- UART1 and UART2 support for the Mango Pi MQ
- SPI NAND device tree addition for my device
- A new boot option: BOOT_DEVICE_SPINAND
- SPL SPI booting on the T113 (superseeded by someone else)
- SPI controller support for the T113 (superseeded by someone else)
- MTD Kconfig requirement for UBI
- spinand_ helper functions to support UBI in the SPL
- UBI SPINAND support
- UBI SPL FIT support
- musb gadget suport (superseeded by someone else)

This patch seems to be independently written for the T113 SPI support
but looks the same as what I've done:
https://lore.kernel.org/u-boot/20231111133432.755363-1-bigunclemax@gmail.com/

The same with this USB fix:
https://lore.kernel.org/u-boot/20230615190701.327852-1-CFSworks@gmail.com/

I plan to add feedback and review to both these patches. Though USB is a
separate subject, I would be interested to know what speeds people are
getting over USB gadget on the T113. DFU seems to cap out at 70KiB/s,
much lower than SSH in Linux.

Anyway, talking in IRC and reading patches on the mailing list, it seems
there's a little lack of direction for SPI NAND support in U-Boot.
Particularly around how to integrate it alongside existing NAND support
and handling bad blocks.

I'd like to first talk about the boot device situation. It works like
this:

- BOOT_DEVICE_SPI means SPI NOR memory
- BOOT_DEVICE_NAND means parallel NAND memory
- BOOT_DEVICE_ONENAND means OneNAND memory
- drivers/mtd/spi uses drivers/spi/spi-mem
- drivers/mtd/nand/spi uses drivers/spi/spi-mem
- drivers/spi/spi-mem uses drivers/spi/spi-sunxi
- common/spl/spl_spi uses drivers/spi/spi-mem and BOOT_DEVICE_SPI
- common/spl/spl_nand uses mtd/nand/raw and BOOT_DEVICE_NAND
- common/spl/spl_onenand uses drivers/mtd/onenand and BOOT_DEVICE_ONENAND
- common/spl/spl_ubi uses drivers/mtd/nand/raw or drivers/mtd/onenand
  and either BOOT_DEVICE_NAND or BOOT_DEVICE_ONENAND
- drivers/nand/raw has a sunxi NAND and sunxi NAND SPL driver
- arch/arm/mach-sunxi/spl_spi_sunxi implements its own SPI loader

A quick grep shows the following custom SPL_LOAD_IMAGE_METHODs in arch:

arch/arm/mach-sunxi/spl_spi_sunxi.c:SPL_LOAD_IMAGE_METHOD("sunxi SPI", 0, BOOT_DEVICE_SPI, spl_spi_load_image);
arch/x86/cpu/apollolake/spl.c:SPL_LOAD_IMAGE_METHOD("Mapped SPI", 2, BOOT_DEVICE_SPI_MMAP, rom_load_image);
arch/x86/cpu/apollolake/spl.c:SPL_LOAD_IMAGE_METHOD("Fast SPI", 1, BOOT_DEVICE_FAST_SPI,

Looking closer at the NAND APIs SPL uses, it calls the following
functions and defines:

nand_init();
nand_deselect();
int nand_spl_load_image(uint32_t offs, unsigned int size, void *dst);
int nand_spl_read_block(int block, int offset, int len, void *dst);
int onenand_spl_read_block(int block, int offset, int len, void *dst);
int onenand_spl_load_image(uint32_t offs, uint32_t size, void *dst);
BOOT_DEVICE_NAND
BOOT_DEVICE_ONENAND
CONFIG_SYS_ONENAND_PAGE_SIZE
CONFIG_SYS_NAND_U_BOOT_OFFS
CONFIG_SYS_NAND_BLOCK_SIZE
CONFIG_SYS_NAND_PAGE_SIZE
CONFIG_SPL_SPI_SUNXI_NAND_ASSUMED_PAGESIZE (used by SUNIV NAND patch)

In the SUNIV NAND patch there was a question of whether to add a
device-specific BOOT_DEVICE. I think the answer to that heavily depends
on whether the current OneNAND/NAND separation makes sense. The APIs are
basically the same, just with different implementations. The only thing
the BOOT_DEVICE_ does here is indicate that there is a distinct boot
ROM option and that we have booted from it. This disambiguates which
NAND we would be using on sunxi: Parallel or SPI.

In my patches I went ahead and added a new SPI NAND API:

spinand_init();
spinand_deselect();
spinand_spl_read_block(int block, int offset, int len, void *dst);
BOOT_DEVICE_SPINAND
CONFIG_SPL_SPINAND_PAGE_SIZE
CONFIG_SPL_SPINAND_BLOCK_SIZE
(I removed the SUNIV NAND config)

I implemented this in the custom sunxi loader code and made UBI use it.
It might be better off as its own thing in drivers/mtd/nand/spi. Then
perhaps arch/arm/mach-sunxi/spl_spi_sunxi could be refactored and moved
to drivers/mtd/spi/, or removed entirely.

I also want to note that with NAND there's also the question of what to
do with bad blocks: The flash I use has 128KiB block sizes. The SPL is
32KiB and U-Boot is 444KiB, needing 4 blocks.

The boot ROM will try multiple pages. To quote
https://linux-sunxi.org/Bootable_SPI_flash:

"Some SoCs can also boot from SPI NAND flash. Here the BROM tries to
read a valid first stage bootloader starting from page number 0, 32, 64,
96, 128, 160, 192 and 224. It only reads the first 1024 bytes from every
page. Since it simply sends the standard SPI NAND flash commands, it is
a good idea to use a flash with ECC turned on by default and is
performed by the flash itself, since errors cannot otherwise be
corrected."

In other words, it looks at 64KiB offsets from 0KiB to 448KiB. The ROM
will put the offset in the upper byte like it does with MMC.

For my solution I put the SPL at 0KiB, 128KiB, 256KiB and 384KiB. 
I then created a 128MiB UBI partition with a 4MiB U-Boot partition and
dumped the image there. UBI scans each erase block, so that's around
65536 blocks for my UBI partition. I think?

This works surprisingly well and doesn't seem to have a noticeable
impact on boot despite U-Boot being shoved to the end of the NAND in my
setup. I managed to set up fast mapping later but the naive approach
here seems to work without any trouble.

A smarter flash system could try to find a continous set of good blocks,
flash U-Boot there and then flash an SPL pointing to it. But if you're
using NAND you're probably going to be running UBI anyway for Linux, and
the code for this is already here and works. Creating custom software
for this that runs on the target for flashing seems like a pain.

I'd like some feedback or discussion about what is the best soltuion
here to go for. Maybe I'm missing something obvious?

I'd also be interested in which patches from my tree would be a good
start for contributing. I haven't sent any patches to U-Boot before and
starting with a large refactoring probably isn't a good idea when I have
some smaller patches. I might toss these up on the lists soon:

- UART1 and UART2 support for the Mango Pi MQ
- MTD Kconfig requirement for UBI
- UBI SPL FIT support

Thanks for your time,
John.