[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

Hi all,

I have 2010.09 U-Boot working fairly well now on our Octeon 63XX processor and 
have a number of observations about U-Boot.

One of our challenges is the fact that the Octeon is a 64-bit multi-core MIPS 
processor that requires 64-bit memory access. In our case, various I/O 
subsystems are mapped to physical addresses outside of the 4GB address space. 
Due to this, we don't even support a 32-bit Linux kernel but require a 64-bit 
kernel. We also support loading applications linked with our SDK onto various 
cores of our processor, which are basically special ELF images.

Another challenge is the fact that we load U-Boot at the top of physical RAM 
which requires that we use TLB mapping due to the fact that it is quite common 
for us to support 4GB or more DRAM in our embedded environment. We need to do 
this since a number of customers have custom operating systems and 
applications that load in the lower 4GB and need all the space they can get. 
The TLB mapping is not nearly as invasive as one might think and it actually 
simplifies the case where we could be loading the same u-boot image from 
different addresses, such as NOR flash, failsafe, NAND and booting over 
PCIe/X. The only places where this does have an impact is in device drivers 
that perform DMA and/or PCI access since we need to make sure to use wrappers 
for mapping the virtual addresses to physical addresses. In our case, I am 
using bus_to_phys and use the 64-bit result to fill in the appropriate 
descriptors. So far I have made changes to EHCI and the e1000 drivers, but 
will likely be working on others like AHCI for SATA. We also don't know where 
we will load U-Boot into physical memory until run-time, even on a per-board 
basis since many boards support DIMMs rather than memory on the board.

I've run into a number of corner cases and bugs in U-Boot which I have fixed 
and hope to soon release patches which are not specific to Octeon:

1. 4GB and larger NAND flash doesn't work. I have a generic patch for this I 
will try and get out this week. Tested with Micron MT29F32G08CBABA.
2. Support for TI TMP42X temperature monitor
3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff (our 
hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000 depending on 
the PCIE bus number). I have a generic fix for this I also hope to release 
this week.
4. Support for Spansion S29GL064N which reports a manufacturer ID of 0x0000 
and requires AMD fixups.
5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a 16-
bit bus when it has an 8-bit BUS.
6. Ability for software to change the u-boot prompt at run-time/boot time. We 
use this to indicate whether we are booting from RAM, FLASH, or failsafe with 
a single U-Boot binary.

Our 64-bit platform is different enough in that it is almost a completely 
different processor compared to the other MIPS platforms. While we were able 
to make use of and expand mipsregs.h and many of the other include files, we 
had to replace almost all of the C code as well as start.S. Even lib/board.c 
could not be reused, though hopefully I can clean this up so it can be used in 
the future.

Due to the vast differences, I have tried to update the MIPS cpu tree to be 
like ARM and PowerPC such that different CPUs have different directories and a 
common mips32 directory.

Just for the CPU portion we have added around 29Kloc of code, not including 
the 440Kloc of code from our SDK which we link with U-Boot (LGPL). We make use 
of our SDK since it abstracts support for all of our chips and makes porting 
to new chips easier, such as our upcoming 68XX series of 32 core processors.

I have tried to minimize our changes to the common code as much as possible 
and have been largely successul. Where I have made changes, I have separated 
out all changes specific to our hardware using #ifdef CONFIG_OCTEON. Some of 
these changes may be useful for other platforms if other platforms choose to 
add 64-bit support.

Moving forward I hope I can get our changes included into the main U-Boot 
release, but this is going to take a while as I still have a lot of cleanup to 
do as well as add support for more of our processors and boards.

We also have a number of custom commands we have added for our platform. Some 
of these may be able to be made more generic, but we use our own commands for 
things like loading our Linux kernel and simple executive applications since 
they require some datastructures to be configured that are tied to our SDK and 
are needed for backwards compatibility.

Most of the code for our platform is also under mips/cpu/octeon since it is 
common between many different boards. The amount of board specific code is 
actually quite small.

I still have a way to go before I can start submitting patches. I need to get 
our code committed to our main trunk in-house then I'll have to clean up some 
of the formatting and break some of our files (our DDR DRAM initialization 
code is over 300K!).

I don't think I can include our SDK as a series of patches on the mailing list 
since it is about 26MB with some of the hardware generated files being 
hundreds of kilobytes to 12MB for our register database file (which 
fortunately isn't needed by u-boot!) It's available under the LGPL but not 
easily accessible through our open-source web site without registration :(

I have also been trying to keep up with the GIT patches to u-boot and I don't 
think it will be all that difficult to move to the latest version since it 
sounds like most of the changes have affected PowerPC and ARM.

-Aaron Williams