[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Wolfgang Denk]

Dear Aaron Williams,

In message <201102071402.37099.Aaron.Williams at caviumnetworks.com> you wrote:
> 
> One of our challenges is the fact that the Octeon is a 64-bit multi-core MIPS 
> processor that requires 64-bit memory access. In our case, various I/O 
> subsystems are mapped to physical addresses outside of the 4GB address space. 
> Due to this, we don't even support a 32-bit Linux kernel but require a 64-bit 
> kernel. We also support loading applications linked with our SDK onto various 
> cores of our processor, which are basically special ELF images.

There are many systems which have a physical address space that
requires more than 32 bit.  So far, we still run a 32 bit U-Boot on
these.  I understand that you are using a 64 bit port of U-Boot?

> Another challenge is the fact that we load U-Boot at the top of physical RAM 
> which requires that we use TLB mapping due to the fact that it is quite common 
> for us to support 4GB or more DRAM in our embedded environment. We need to do 
> this since a number of customers have custom operating systems and 
> applications that load in the lower 4GB and need all the space they can get. 

Existing U-Boot deals with this by mapping just the lower and the
upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
config option.

> The TLB mapping is not nearly as invasive as one might think and it actually 
> simplifies the case where we could be loading the same u-boot image from 
> different addresses, such as NOR flash, failsafe, NAND and booting over 
> PCIe/X. The only places where this does have an impact is in device drivers 
> that perform DMA and/or PCI access since we need to make sure to use wrappers 
> for mapping the virtual addresses to physical addresses. In our case, I am 
> using bus_to_phys and use the 64-bit result to fill in the appropriate 
> descriptors. So far I have made changes to EHCI and the e1000 drivers, but 
> will likely be working on others like AHCI for SATA. We also don't know where 
> we will load U-Boot into physical memory until run-time, even on a per-board 
> basis since many boards support DIMMs rather than memory on the board.

This is a pretty standard situation - that is what the original design
did, and what all PowerPC (and now all working ARM) boards are doing,
too.

> I've run into a number of corner cases and bugs in U-Boot which I have fixed 
> and hope to soon release patches which are not specific to Octeon:
> 
> 1. 4GB and larger NAND flash doesn't work. I have a generic patch for this I 
> will try and get out this week. Tested with Micron MT29F32G08CBABA.
> 2. Support for TI TMP42X temperature monitor
> 3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff (our 
> hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000 depending on 
> the PCIE bus number). I have a generic fix for this I also hope to release 
> this week.
> 4. Support for Spansion S29GL064N which reports a manufacturer ID of 0x0000 
> and requires AMD fixups.
> 5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a 16-
> bit bus when it has an 8-bit BUS.
> 6. Ability for software to change the u-boot prompt at run-time/boot time. We 
> use this to indicate whether we are booting from RAM, FLASH, or failsafe with 
> a single U-Boot binary.

All these sound pretty harmless and independent of your Octeon
support.

> Our 64-bit platform is different enough in that it is almost a completely 
> different processor compared to the other MIPS platforms. While we were able 
> to make use of and expand mipsregs.h and many of the other include files, we 
> had to replace almost all of the C code as well as start.S. Even lib/board.c 
> could not be reused, though hopefully I can clean this up so it can be used in 
> the future.

We are in the process of overcoming the long-term repercussions of the
original ARM port which did not follow the original (PPC based)
design.  ARM is now fixed, or in the process of being cleaned up.
Unfortunately MIPS folled the ARM example, and here this work still
needs to be done.  So if you are looking for reference
implementations, you rather want to look at PPC code than ARM or MIPS
code.

One part of the unification attempts is that we are trying to
synchronize lib/board.c across architectures, with the goal to use a
single common version soon.  So again, pleasre use PPC as reference
for changes, or at least try not to introduce any additional, new
incompatibilities.

> Just for the CPU portion we have added around 29Kloc of code, not including 
> the 440Kloc of code from our SDK which we link with U-Boot (LGPL). We make use 
> of our SDK since it abstracts support for all of our chips and makes porting 
> to new chips easier, such as our upcoming 68XX series of 32 core processors.

Do I understand correctly t hat this SDK (whatever that might be) will
then have to be included with the U-Boot distribution?

What would be alternatives to static linking, such as to avoid adding
all this code?

> I have tried to minimize our changes to the common code as much as possible 
> and have been largely successul. Where I have made changes, I have separated 
> out all changes specific to our hardware using #ifdef CONFIG_OCTEON. Some of 
> these changes may be useful for other platforms if other platforms choose to 
> add 64-bit support.

Such general changes should then NOT use CONFIG_OCTEON, but some
generic variable name.

> We also have a number of custom commands we have added for our platform. Some 
> of these may be able to be made more generic, but we use our own commands for 
> things like loading our Linux kernel and simple executive applications since 
> they require some datastructures to be configured that are tied to our SDK and 
> are needed for backwards compatibility.

Many boards use board specific commands; I see no problem with having
SoC specific commands either.

> I don't think I can include our SDK as a series of patches on the mailing list 
> since it is about 26MB with some of the hardware generated files being 
> hundreds of kilobytes to 12MB for our register database file (which 
> fortunately isn't needed by u-boot!) It's available under the LGPL but not 
> easily accessible through our open-source web site without registration :(

U-Boot is supposed to be self-sufficient, i. e. to contain all parts
that are required to build a working U-Boot image.  I see a potential
area of conflicts here.

> I have also been trying to keep up with the GIT patches to u-boot and I don't 
> think it will be all that difficult to move to the latest version since it 
> sounds like most of the changes have affected PowerPC and ARM.

Indeed.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Never put off until tomorrow what you can put off indefinitely.