[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Tuesday, February 08, 2011 12:53:27 am Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102071524.17440.Aaron.Williams at caviumnetworks.com> you wrote:
> > > these.  I understand that you are using a 64 bit port of U-Boot?
> > 
> > No. We are using a 32-bit port since I think trying to make a 64-bit port
> > of U-Boot would be far more involved. We do have support for loading and
> > executing 64-bit ELF images, however. We use special memcpy/memset
> > functions for 64-bit memory addressing in these cases.
> 
> OK, let's discuss this when we see your code.
> 
> > > Existing U-Boot deals with this by mapping just the lower and the
> > > upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
> > > config option.
> > 
> > I just looked at this and this. The only place I see this used is in
> > arch/powerpc/lib/board.c and it looks like it just limits the effective
> > memory size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said,
> > we need to move it out of the lower 4GB when there's more memory
> > involved. We also don't
> 
> Why?

As I have said, we need all of the lower portion of memory and do not want to 
introduce holes in the memory space for u-boot for loading 64-bit 
applications. Also, some of our customers are loading their own proprietary 
operating systems that need all of the memory. The overhead from our SDK in 
the lower memory region is only a few kilobytes.

One other nice thing about the TLB support is all of the fixups go away.
> 
> > want there being holes in the middle of the memory if we can help it, nor
> > do we want to place u-boot at the lower end of memory. At least on MIPS
> > the TLB
> 
> The question is what is the lesser of two evils: not mapping all of
> the available memory and stick with a total of mapped memory that is
> addressable in 32 bit adress space, or adding the complexity to deal
> with 64 bit address spaces - which will require changes in _lots_ of
> places.
> 
> But again I suggst to defer the discussion until we see your code.
> 
> > Would using virt_to_bus to convert pointers to DMA addresses be
> > appropriate instead of the current assumption that pointers can be used
> > as DMA addresses directly? This seems like a portable solution since on
> > platforms where the pointer and DMA address are the same the macro would
> > just do nothing. Even if we didn't use virtual addresses and were using,
> > say, KSEG1 the pointer and physical address don't match.
> 
> Yes, I think this would indeed be a better approach.

This is what I am doing, though I'm having to make changes on a driver by 
driver basis to add this. I'm using dma_addr_t for the address result of this 
then using the same method used in Linux to fill in both the upper and lower 
32-bit descriptor portions, i.e. foo->lower = dma_addr & 0xffffffff; foo-
>upper = ((dma_addr >> 16) >> 16); This is how it's done in the Linux kernel. 
So far the drivers I've modified this way seem to work fine.
> 
> > > Do I understand correctly t hat this SDK (whatever that might be) will
> > > then have to be included with the U-Boot distribution?
> > > 
> > > What would be alternatives to static linking, such as to avoid adding
> > > all this code?
> > 
> > It would have to be included only for the Octeon processors. It is
> > statically
> 
> I don't undertsand what you mean. Either it is part of the U-Boot
> source tree, or it is not.  You cannot add it "only for the Octeon
> processors" - either it's there, or it ain't.
> 
> Given that we are talking about code in the order of 30% of the total
> U-Boot code, probably not conforming to U-Boot coding standards, I am
> anything but happy about such an addition.  Assume ever SoC vendor
> comes up with similar ideas...
> 
It can be included with u-boot, though I think first I should try and see how 
much I can strip it down. We cannot separate u-boot completely from the SDK, 
though. We use it extensively for things like Ethernet and will also likely 
make use of it for USB when I add the Synopsis USB support used in our older 
chips (which is not EHCI/OHCI compatible).

> > linked and we don't want to get away from this. Also, some of our u-boot
> > files are in turn statically linked against some of our utilities, such
> > as our
> 
> I understand that you do not _want_ to change this.  My question is:
> what would you do if you _had_ to change it?

We might be able to make some changes, but it might be difficult since we have 
a lot of developers working on this around the world. I would prefer to not 
have to maintain two separate code bases. I will need to see how much I can 
strip out.

> 
> > Also note that u-boot for Octeon can only be compiled with our toolchain,
> > since there is some dependency on some of the include files from our GCC
> > distribution as well, plus our toolchain distribution includes support
> > for some of the extensions we make use of.
> 
> Can this be fixed?  I mean, copying some header files should probably
> be a solvable problem.  What about of these "extensions" - are they
> absolutely needed in U-Boot?  Usually such extensions are either
> performance optimizations which are not really needed in U-Boot, or
> other well-localized operations that can b ehandlef with small
> assembler stubs.

Support for our Octeon2 processor is not in the mainline trunk yet. We also 
have some changes needed to work with our SDK and the Linux kernel with our 
newer chips.

> 
> > > Such general changes should then NOT use CONFIG_OCTEON, but some
> > > generic variable name.
> > 
> > I agree, though some many cases they are not general, such as some of the
> > support for our compact flash in cmd_ide.c and a few other areas.
> 
> In general we do not want to see board or SoC specific changes to
> common code.

The common code is already chock full of them from what I've seen in a number 
of areas. The number of changes for our SoC is actually quite small. In some 
cases I've just changed some functions to use __attribute__((__weak__)) so we 
can define the functionality elsewhere. I count a total of 21 ifdefs in the 
common code, most of those are in cmd_ide.c, one in cmd_boot.c, one in 
cmd_elf.c, two in env_flash.c and some in main.c, which can be removed (those 
are for adding the support for dynamically changing the boot prompt) and one 
in miiphyutil.c. Right now cmd_ide.c can be cleaned up a bit. The problem is 
that cmd_ide.c includes the driver functionality in it, requiring us to do 
this.  I can clean this up by making some of the functions weak. Is this 
reasonable?

One other change I had to make was for the command table. I have to force the 
alignment to 8 bytes in the compiler or very bad things happen.
> 
> > > Many boards use board specific commands; I see no problem with having
> > > SoC specific commands either.
> > 
> > I am placing these commands under arch/mips/cpu/octeon/commands rather
> > than clutter up the common code, unless you feel it's better to put all
> > the commands under the common code.
> 
> This sounds OK with me.
> 
> > > > I don't think I can include our SDK as a series of patches on the
> > > > mailing list since it is about 26MB with some of the hardware
> > > > generated files being hundreds of kilobytes to 12MB for our register
> > > > database file (which fortunately isn't needed by u-boot!) It's
> > > > available under the LGPL but not easily accessible through our
> > > > open-source web site without registration :(
> > > 
> > > U-Boot is supposed to be self-sufficient, i. e. to contain all parts
> > > that are required to build a working U-Boot image.  I see a potential
> > > area of conflicts here.
> > 
> > We don't have any problem including our SDK with U-Boot. I can work on
> > trying
> 
> But  we probably will have problems adding tons of such code which is
> of no use to anybody else.
As I have said, I will try and cut down on what is included.
> 
> > The biggest portions of our SDK are generated files for dealing with
> > errors, a register database (12MB!) and include files which define
> > access functions to
> 
> I'd expect that only a tiny percentage of this is actually needed /
> used in U-Boot.  We should restric it to these actually used parts.
> 
Agreed.
> 
> Best regards,
> 
> Wolfgang Denk