[U-Boot] Cavium Octeon support for u-boot

[Aaron Williams]

The Octeon version of U-Boot is based off an older versin of U-Boot (2012.07) 
with a lot of features backported. For example, I have backported NVME support 
as well as the latest filesystem code. Since there have been significant 
changes to U-Boot (i.e. 64-bit support) that were missing at the time it would 
be a massive undertaking to support the current U-Boot with Octeon or to push 
the Octeon support upstream. While in general the changes to the core U-Boot 
were fairly minimal for Octeon, there were significant changes to board.c.

Hopefully we can get OcteonTX pushed upstream, however, since we are tracking 
the current U-Boot releases. I'm not sure how to go about doing this, since I 
would basically need a branch that I can maintain.

No significant new work is planned for the Octeon SDK or bootloader, however, 
other than maintenance releases or adding features as needed by customers.

Although the Octeon code makes extensive use of the device tree, it does not 
follow the dm model since that was not available when the U-Boot work took 
place.

The way the Linux kernel and simple executive applications are started is also 
unique to Octeon since multi-core support was also not present. This code is 
complex due to the way it has evolved while trying to maintain backwards 
compatibility.

Another potential barrier to upstreaming Octeon support is the fact that it 
makes heavy use of our rather large SDK and the sheer amount of code. A quick 
check with wc shows that we have around a million lines of code involved under 
arch/mips/cpu/octeon, arch/mips/include/asm/arch-octeon and board/octeon/
common. This doesn't include the various drivers, either, or our custom 
board.c file needed for the required flexibility. Around 434Klocs are the 
hardware register definition files, which admittedly are huge due to the 
autogenerated nature of them and all the comments and big/little endian 
support. These files are shared by U-Boot, the Linux kernel and simple-
executive applications.

Some of the SDK and common board code is a mess since it tries to avoid 
changes to the core of U-Boot. The networking and QLM configuration code is a 
bit of a  mess since it has evolved over many years. I'm sorry to say that the 
SFP/QSFP code I recently added just makes it worse due to the way it ties in 
phy support. There's duplication of code between U-Boot and our SDK in the 
networking code, especially for things like phy link support. Some phys also 
just don't map cleanly into the U-Boot (or Linux for that matter) model. Inphi 
(Cortina) and the Microsemi (Vitesse) in particular do not map into the one 
phy per MAC model and all hell breaks loose with QSFPs where multiple MACs 
share a phy or one MAC is split between multiple phys.

We also have implemented hooks in a number  of key places that are missing in 
standard U-Boot. For example, we use a hook in the serial driver for a number 
of boards to poll the SFP/QSFP ports to detect when a module is plugged in or 
unplugged and to update activity LEDs.

We have also implemented something similar to native application support so 
that simple-executive applications can make calls into U-Boot via a context 
switching mechanism. This is used for filesystem access as well as some phy 
operations with our 25G Liquid I/O boards.

We also run U-Boot in KSEG2 rather than KSEG0 so that U-Boot is always mapped 
to the same virtual address regardless of where it is loaded in physical 
memory. We do this for several reasons. First of all, it eliminates the need 
to perform relocation fixups. Second of all, this frees up the lower 2GB of 
memory space which can be a tight resource with a number of customers. While 
we compile it as n32, it could conceivably be compiled as n64 but continue to 
run in kseg2. n64 mode would simplify things by allowing the removal of using 
cvmx_read_csr/cvmx_write_csr to instead use readq/writeq. The mapping between 
virtual and physical addresses in our drivers would need to remain since there 
is no IOMMU.

The code for bringing multiple cores out of reset and starting SE and the 
Linux kernel would be tricky to port. There is a fair amount of assembly code 
I've maintained on Octeon in order to do this.

Some of the fixes could also be pushed upstream, such as endian fixes we have 
made in the USB and ext4 code.

Right now the main limitation to doing this is I'm pretty much the only one 
maintaining Octeon support for U-Boot across all of our boards as well as 
working on support for OcteonTX and OcteonTX2. A number of OcteonTX drivers 
could easily be modified to work with Octeon, such as SPI, eMMC, NAND, i2c, 
etc. The main difference in many cases is the fact that OcteonTX represents 
devices as PCI devices (in addition to the device tree), whereas on Octeon 
they are directly accessible and are only represented in the device tree. 
There are huge differences, however, between OcteonTX and Octeon. With 
OcteonTX, the memory and device tree are prepared by the BDK and ATF before U-
Boot is started. On Octeon, often U-Boot is the first thing that is loaded or 
it is loaded by a simple SPI or eMMC bootloader and U-Boot is responsible for 
memory initialization and a lot of other things.

-Aaron

On Wednesday, July 25, 2018 12:55:36 AM PDT Chris Packham wrote:
> External Email
> 
> Hi Aaron,
> 
> On Wed, Jul 25, 2018 at 2:06 PM Aaron Williams
> 
> <Aaron.Williams at cavium.com> wrote:
> > While mainline U-Boot does not support Octeon, we have our own fork of it
> > that I maintain. I am using the 2018.07 release with only a few minor
> > changes around the periphery to support the older version of U-Boot
> > Octeon is based off of.
> 
> Just out of interest is there any particular barrier to Octeon support
> landing upstream? At $dayjob we have 4 platforms using various Octeon
> III processors. We're in the position of being stuck on an older
> version of u-boot because we can't bring support for our boards up to
> date without having to bring all the Octeon SoC support forward as
> well. Is there any interest from Cavium to get Octeon upstreamed?

-- 
Aaron Williams
Senior Software Engineer
Cavium, Inc.
(408) 943-7198	(510) 789-8988 (cell)