[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Monday, February 07, 2011 02:26:15 pm Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102071402.37099.Aaron.Williams at caviumnetworks.com> you wrote:
> > One of our challenges is the fact that the Octeon is a 64-bit multi-core
> > MIPS processor that requires 64-bit memory access. In our case, various
> > I/O subsystems are mapped to physical addresses outside of the 4GB
> > address space. Due to this, we don't even support a 32-bit Linux kernel
> > but require a 64-bit kernel. We also support loading applications linked
> > with our SDK onto various cores of our processor, which are basically
> > special ELF images.
> 
> There are many systems which have a physical address space that
> requires more than 32 bit.  So far, we still run a 32 bit U-Boot on
> these.  I understand that you are using a 64 bit port of U-Boot?

No. We are using a 32-bit port since I think trying to make a 64-bit port of 
U-Boot would be far more involved. We do have support for loading and 
executing 64-bit ELF images, however. We use special memcpy/memset functions 
for 64-bit memory addressing in these cases.

> 
> > Another challenge is the fact that we load U-Boot at the top of physical
> > RAM which requires that we use TLB mapping due to the fact that it is
> > quite common for us to support 4GB or more DRAM in our embedded
> > environment. We need to do this since a number of customers have custom
> > operating systems and applications that load in the lower 4GB and need
> > all the space they can get.
> 
> Existing U-Boot deals with this by mapping just the lower and the
> upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
> config option.
> 
I just looked at this and this. The only place I see this used is in 
arch/powerpc/lib/board.c and it looks like it just limits the effective memory 
size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said, we need to 
move it out of the lower 4GB when there's more memory involved. We also don't 
want there being holes in the middle of the memory if we can help it, nor do 
we want to place u-boot at the lower end of memory. At least on MIPS the TLB 
support is quite simple and is performed very early on by adding a single TLB 
entry. We have customers that need to make use of the entire KUSEG, for 
example. Our SDK only reserves a very small portion of this.

> > The TLB mapping is not nearly as invasive as one might think and it
> > actually simplifies the case where we could be loading the same u-boot
> > image from different addresses, such as NOR flash, failsafe, NAND and
> > booting over PCIe/X. The only places where this does have an impact is
> > in device drivers that perform DMA and/or PCI access since we need to
> > make sure to use wrappers for mapping the virtual addresses to physical
> > addresses. In our case, I am using bus_to_phys and use the 64-bit result
> > to fill in the appropriate descriptors. So far I have made changes to
> > EHCI and the e1000 drivers, but will likely be working on others like
> > AHCI for SATA. We also don't know where we will load U-Boot into
> > physical memory until run-time, even on a per-board basis since many
> > boards support DIMMs rather than memory on the board.
> 
> This is a pretty standard situation - that is what the original design
> did, and what all PowerPC (and now all working ARM) boards are doing,
> too.

In our case we just map u-boot to always be at 0xC0000000 regardless of where 
it is located in physical memory. In my current setup it is located at 
physical address 0x10f800000. So far I've found that the e1000 and EHCI 
drivers assume that virtual addresses are physical addresses. Furthermore, the 
e1000 driver assumes that readl can be used to access the PCI BAR address 
space, which on our platform is not the case since that is located at 
0x11b00f0000000 or 0x11c00f80000000 and needs the appropriate wrapper 
function. Alternatively I could add more TLB entries to map the PCI BAR 
address space to 0xf0000000 and 0xf8000000 respectively or modify readl/writel 
to make the appropriate translations.

Would using virt_to_bus to convert pointers to DMA addresses be appropriate 
instead of the current assumption that pointers can be used as DMA addresses 
directly? This seems like a portable solution since on platforms where the 
pointer and DMA address are the same the macro would just do nothing. Even if 
we didn't use virtual addresses and were using, say, KSEG1 the pointer and 
physical address don't match.

> 
> > I've run into a number of corner cases and bugs in U-Boot which I have
> > fixed and hope to soon release patches which are not specific to Octeon:
> > 
> > 1. 4GB and larger NAND flash doesn't work. I have a generic patch for
> > this I will try and get out this week. Tested with Micron
> > MT29F32G08CBABA. 2. Support for TI TMP42X temperature monitor
> > 3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff
> > (our hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000
> > depending on the PCIE bus number). I have a generic fix for this I also
> > hope to release this week.
> > 4. Support for Spansion S29GL064N which reports a manufacturer ID of
> > 0x0000 and requires AMD fixups.
> > 5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a
> > 16- bit bus when it has an 8-bit BUS.
> > 6. Ability for software to change the u-boot prompt at run-time/boot
> > time. We use this to indicate whether we are booting from RAM, FLASH, or
> > failsafe with a single U-Boot binary.
> 
> All these sound pretty harmless and independent of your Octeon
> support.
> 
> > Our 64-bit platform is different enough in that it is almost a completely
> > different processor compared to the other MIPS platforms. While we were
> > able to make use of and expand mipsregs.h and many of the other include
> > files, we had to replace almost all of the C code as well as start.S.
> > Even lib/board.c could not be reused, though hopefully I can clean this
> > up so it can be used in the future.
> 
> We are in the process of overcoming the long-term repercussions of the
> original ARM port which did not follow the original (PPC based)
> design.  ARM is now fixed, or in the process of being cleaned up.
> Unfortunately MIPS folled the ARM example, and here this work still
> needs to be done.  So if you are looking for reference
> implementations, you rather want to look at PPC code than ARM or MIPS
> code.
> 
> One part of the unification attempts is that we are trying to
> synchronize lib/board.c across architectures, with the goal to use a
> single common version soon.  So again, pleasre use PPC as reference
> for changes, or at least try not to introduce any additional, new
> incompatibilities.

That is what I did, though for board.c I basically took our old implementation 
and cleaned it up for the new u-boot. I need to make another pass and do a 
proper implementation once I get some time.
> 
> > Just for the CPU portion we have added around 29Kloc of code, not
> > including the 440Kloc of code from our SDK which we link with U-Boot
> > (LGPL). We make use of our SDK since it abstracts support for all of our
> > chips and makes porting to new chips easier, such as our upcoming 68XX
> > series of 32 core processors.
> 
> Do I understand correctly t hat this SDK (whatever that might be) will
> then have to be included with the U-Boot distribution?
> 
> What would be alternatives to static linking, such as to avoid adding
> all this code?
> 
It would have to be included only for the Octeon processors. It is statically 
linked and we don't want to get away from this. Also, some of our u-boot files 
are in turn statically linked against some of our utilities, such as our 
utility which loads and boots u-boot over the PCI(e) bus. It contains all of 
the register definitions that are used on our board as well as functionality 
to initialize and work with different I/O blocks. It provides an abstraction 
to make it easy to deal with all of our various chips and boards as well as 
errata. At the moment in our internal tree we just create symlinks to all of 
the files. The header files I'm placing under arch/mips/include/asm/arch-
octeon and the .c files under arch/mips/cpu/mips/octeon/cvmx to separate them.

Also note that u-boot for Octeon can only be compiled with our toolchain, 
since there is some dependency on some of the include files from our GCC 
distribution as well, plus our toolchain distribution includes support for 
some of the extensions we make use of.

> > I have tried to minimize our changes to the common code as much as
> > possible and have been largely successul. Where I have made changes, I
> > have separated out all changes specific to our hardware using #ifdef
> > CONFIG_OCTEON. Some of these changes may be useful for other platforms
> > if other platforms choose to add 64-bit support.
> 
> Such general changes should then NOT use CONFIG_OCTEON, but some
> generic variable name.
> 
I agree, though some many cases they are not general, such as some of the 
support for our compact flash in cmd_ide.c and a few other areas.

> > We also have a number of custom commands we have added for our platform.
> > Some of these may be able to be made more generic, but we use our own
> > commands for things like loading our Linux kernel and simple executive
> > applications since they require some datastructures to be configured
> > that are tied to our SDK and are needed for backwards compatibility.
> 
> Many boards use board specific commands; I see no problem with having
> SoC specific commands either.

I am placing these commands under arch/mips/cpu/octeon/commands rather than 
clutter up the common code, unless you feel it's better to put all the 
commands under the common code.
> 
> > I don't think I can include our SDK as a series of patches on the mailing
> > list since it is about 26MB with some of the hardware generated files
> > being hundreds of kilobytes to 12MB for our register database file
> > (which fortunately isn't needed by u-boot!) It's available under the
> > LGPL but not easily accessible through our open-source web site without
> > registration :(
> 
> U-Boot is supposed to be self-sufficient, i. e. to contain all parts
> that are required to build a working U-Boot image.  I see a potential
> area of conflicts here.

We don't have any problem including our SDK with U-Boot. I can work on trying 
to cut down on the files that are needed. One thing to note is that we make 
patches and changes to our SDK fairly regularly to fix errata and to support 
new chips.

The main parts I am using are the sections dealing with memory, PCI/PCIE, 
model identification, Ethernet, twsi (I2C), UART, TLB (for virt_to_bus) and a 
few other areas. The memory part we use to tell our SDK which blocks of memory 
are in use by u-boot. There are many areas that are not used by u-boot but 
this may grow. For example, in the future I may make use of the usb support 
for our earlier chips that have a proprietary USB interface or our zip support 
for faster unzip, md5, sha1, etc.

The biggest portions of our SDK are generated files for dealing with errors,  
a register database (12MB!) and include files which define access functions to 
all the various registers in all of the I/O units for all of our chips (which 
also may not be needed). I need to play with this and see what files I can 
remove from the SDK.

> 
> > I have also been trying to keep up with the GIT patches to u-boot and I
> > don't think it will be all that difficult to move to the latest version
> > since it sounds like most of the changes have affected PowerPC and ARM.
> 
> Indeed.
> 
> Best regards,
> 
> Wolfgang Denk