[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Tuesday, February 08, 2011 11:39:58 pm Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102081927.36497.Aaron.Williams at caviumnetworks.com> you
> wrote:
> 
> ...
> 
> > > > memory size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I
> > > > said, we need to move it out of the lower 4GB when there's more
> > > > memory involved. We also don't
> > > 
> > > Why?
> > 
> > As I have said, we need all of the lower portion of memory and do not
> > want to introduce holes in the memory space for u-boot for loading
> > 64-bit
> 
> When I ask "why?" then I do so to understand the reasons fdor your
> design decisions (and eventually to be able to recomment alternative
> solutions).
> 
> In such a situation phrases as "we need" or "we [do not] want" are
> completely useless und unhelpful, unless they are accompanied with a
> "because ..." part that explains _why_ you [think you] "need" or
> "want" these things.
> 
We cannot have u-boot consuming space in KUSEG or in KSEG0. Placing U-Boot at 
the bottom of memory causes problems for some of our customers which load 
proprietary operating systems that don't handle this well. They also need all 
the memory they can get. By using a single TLB entry we map U-Boot to the top 
of physical memory and map it to KSEG2. This makes the entire KUSEG and KSEG0 
available for customer applications and operating systems. We don't have the 
uncached KSEG1 since the Octeon is a cache coherent design. We also can't load 
U-Boot at the top of KUSEG or KSEG0 because that would create a hole in the 
middle of memory for loading 64-bit applications. 

The other alternative would be to make U-Boot fully 64-bit. That, however,  
would be a far larger challenge since there are assumptions everywhere about 
32-bit pointers.

By utilizing the TLB we map U-Boot to the top of physical memory which leaves 
almost all of the memory available for loading large 64-bit applications and 
operating systems without creating a hole in KSEG0/KUSEG (64-bit) where U-Boot 
was loaded.

As a side benefit, the relocation code is no longer needed with the exception 
of gd and the environment.

The drawback is that drivers cannot assume a pointer is always to a physical 
address, but they shouldn't make that assumption anyway. For example, if u-
boot is running out of KSEG0 then the pointer and physical address will not 
match on our platform.

> > > I understand that you do not _want_ to change this.  My question is:
> > > what would you do if you _had_ to change it?
> > 
> > We might be able to make some changes, but it might be difficult since we
> > have a lot of developers working on this around the world. I would
> > prefer to not have to maintain two separate code bases. I will need to
> > see how much I can strip out.
> 
> Well, frankly, this is your problem, not ours. If Cavium had decided
> to discuss the design early we might have come up with a solution
> that fits without too much problems. Instead, Cavium went on and kept
> their stuff closed for years - now they say that it's difficult to
> change things. Please understand that our foxus is on the overall
> quality and maintainability of the code, so such arguments do not
> carry much weight.
> 
I can't say much about that.  That was before my time here.

> > > > since there is some dependency on some of the include files from our
> > > > GCC distribution as well, plus our toolchain distribution includes
> > > > support for some of the extensions we make use of.
> > > 
> > > Can this be fixed?  I mean, copying some header files should probably
> > > be a solvable problem.  What about of these "extensions" - are they
> > > absolutely needed in U-Boot?  Usually such extensions are either
> > > performance optimizations which are not really needed in U-Boot, or
> > > other well-localized operations that can b ehandlef with small
> > > assembler stubs.
> > 
> > Support for our Octeon2 processor is not in the mainline trunk yet. We
> > also have some changes needed to work with our SDK and the Linux kernel
> > with our newer chips.
> 
> You reply, but you do not actually answer my questions.
> 
The code uses features found in the Octeon II extensions that are not 
available in the binutils trunk yet. I know for a fact that the standard MIPS 
toolchain used to compile other MIPS boards in U-Boot will not work at all for 
the Octeon. It doesn't even support the n32 ABI which is absolutely required.

> > > In general we do not want to see board or SoC specific changes to
> > > common code.
> > 
> > The common code is already chock full of them from what I've seen in a
> > number
> 
> Yes, and we've learned that lesson, and now we strive hard not to add
> to the mess any more.
> 
For the most part I have avoided them, and many that are there can be changed 
to be more general. The IDE code will take more work since it's already a 
mess.

> > of areas. The number of changes for our SoC is actually quite small. In
> > some cases I've just changed some functions to use
> > __attribute__((__weak__)) so we can define the functionality elsewhere.
> > I count a total of 21 ifdefs in the common code, most of those are in
> > cmd_ide.c, one in cmd_boot.c, one in cmd_elf.c, two in env_flash.c and
> > some in main.c, which can be removed (those are for adding the support
> > for dynamically changing the boot prompt) and one in miiphyutil.c. Right
> > now cmd_ide.c can be cleaned up a bit. The problem is that cmd_ide.c
> > includes the driver functionality in it, requiring us to do this.  I can
> > clean this up by making some of the functions weak. Is this reasonable?
> 
> I don't know. I haven't seen any of the code yet, so I really cannot
> tell.
> 
> > One other change I had to make was for the command table. I have to force
> > the alignment to 8 bytes in the compiler or very bad things happen.
> 
> Why?

Because otherwise the code generated by the compiler to enumerate the commands 
when looking for a match will crash because the compiler increments the 
pointer by a value that does not match what the linker does. The linker's 
minimum section alignment is 64-bits for n32, but the compiler generates code 
assuming 32-bit alignment. If the commands were just an array then there would 
be no problem. The other alternative is to add padding to force the data 
structure to be a multiple of 64-bits. I have not found any other workaround.  
I don't think this is an issue for a purely 32-bit ABI, such as o32, but it is 
an issue with n32 which supports 64-bit load/store operations and registers. 
Even telling the linker to force 32-bit alignment does not help, it still 
performs 64-bit alignment when using the n32 ABI. I haven't looked into the 
details for the ABI, but I wouldn't be surprised if the linker is doing the 
right thing and the problem is just that u-boot's assumptions break in this 
case.
> 
> 
> Best regards,
> 
> Wolfgang Denk