[U-Boot] [EXT] Re: Cavium/Marvell Octeon Support

[Aaron Williams]

On Thursday, October 31, 2019 6:26:51 AM PDT Tom Rini wrote:
> On Wed, Oct 30, 2019 at 11:36:19PM +0000, Aaron Williams wrote:
> > On Wednesday, October 30, 2019 3:05:25 PM PDT Tom Rini wrote:
> > > External Email
> > > 
> > > ----------------------------------------------------------------------
> > > 
> > > On Wed, Oct 23, 2019 at 03:50:00AM +0000, Aaron Williams wrote:
> > > > Hi all,
> > > > 
> > > > I have been tasked with porting our Octeon U-Boot to the latest U-Boot
> > > > and merging it upstream.
> > > 
> > > [snip]
> > > 
> > > I want to jump back up back to the top of this thread.  And first I want
> > > to say that I am glad that there is official desire to upstream support.
> > > This is good.  My concern is that the plan seems to be, at a very high
> > > level, "get everything we have for every feature upstream".  But as been
> > > said elsewhere this would roughly double the total LOC for the project,
> > > and it's not like we're a new project with a small handful of things :)
> > > It's impossible for the community to review that much code in any
> > > meaningful way over anything less than a period of several years.  I
> > > know you've said that to support various customer use cases you need all
> > > sorts of other things, and while I'm certain that's true, I believe the
> > > plan needs to be to step back and pick the smallest possible testable
> > > unit, and upstream it.  And add to it, small pieces at a time.  Thanks!
> > 
> > It might be easier if I were a maintainer for our SOC to limit the needed
> > review of a fair bit of the code. I have already found I can cut out a
> > large chunk of our code by removing support for our older models. Much of
> > the code has been very well tested, for example our serdes initialization
> > and DRAM initialization code. That's not to say I can't do some cleanup.
> 
> Don't worry, I totally expect you to become the maintainer for your SoC,
> that's key to making sure the SoC-specific stuff is done right :)  But,
> you're not the first big SoC to migrate from an internal fork to
> mainline and have a seemingly impossible number of LOC to deal with.
> It's why I'm saying you need to start with something absolutely as small
> as possible, and move forward.
> 
> > The changes to U-Boot itself should be relatively small as long as I can
> > keep much of our code under arch/mips/arch-octeon and
> > arch/mips/cpu/octeon much like how our ARM code is. For anything that is
> > applicable to other architectures I will place it in the appropriate
> > locations.
> > 
> > Our ARM code is quite a bit simpler than MIPS because on ARM most of the
> > heavy lifting is done by our "BDK" bootloader as well as ATF. On ARM,
> > U-Boot doesn't need to deal with SFPs, serdes initialization, DRAM
> > initialization, hot plug or a myriad of other issues.
> > 
> > I noticed the same with X86.  If it weren't for these other layers the X86
> > code would also be quite large.
> > 
> > I have already identified quite a few very large files that will be
> > removed, such as the error handling code for or Octeon2 and earlier CPUs.
> > 
> > Additionally I have identified a number of register definition files I can
> > get rid of, including one that's 2.3MB in size! These files tend to be
> > huge because they contain definitions for every single chip and revision
> > as well as big and little endian definitions. On top of that, there are a
> > huge number of comments. Each field contains all of the text that is in
> > our hardware reference manuals. There is no dearth of comments. I should
> > be able to cut the size of the remaining files to 1/4th their current
> > size or even smaller.
> > 
> > Some files will still remain quite large, however, such as our serdes
> > initialization and DRAM initialization code (which I plan to re-architect
> > because the original author didn't believe in functions due to stack
> > limitations. (it is well commented though). If you ever want to learn all
> > the gory details of DDR4 link training and finding trends and so-forth
> > it's all in there. The current memory initialization code is over 1MB in
> > size. I plan to cut this down and break it up in a clean manner. The
> > initialization code has grown in complexity and size over the years as
> > various instabilities have been identified and fixed. The DRAM
> > initialization code for our OcteonTX2 CPU is almost as large, though this
> > code has been cleaned up and re-written. There really is no way to avoid
> > this. The OcteonTX and OcteonTX2 DDR initialization code is similar to
> > that for Octeon. In the case of U-Boot on our ARM SoCs, though,
> > initialization is done before U-Boot is loaded.
> > 
> > I'll move the init code to drivers/ram/marvell/octeon. It will be about
> > twice as large as the AXP driver (which only handles DDR3). The serdes
> > init code I figure could go under drivers/soc/marvell/octeon.
> > 
> > I noticed that there are several directories under drivers for memory.
> > There's drivers/ram, drivers/memory and drivers/ddr. These should be
> > consolidated. I think some code might be able to be common, such as the
> > SPD decoding code. It's even possible that some algorithms might be able
> > to be made common such as deskew training and read/write leveling.
> > 
> > In terms of Octeon specific features, there really aren't too many of
> > those
> > but most of the ones we have are essential in the bootloader. There's no
> > avoiding the Serdes and low-level network initialization. The serdes init
> > code works across all networking interface types (SGMII, 1000Base-X,
> > XAUI, RXAUI, XFI, XLAUI, 25G (XLAUI), SATA, PCIe, SRIO plus all the
> > variants (i.e. KR). It also configures all the clocks and equalization.
> > It's not like a simple gigabit NIC nor is it offloaded to some other
> > layer. Some of this code will come later, for example support for NUMA
> > with CN78XX (96 cores, 256GiB of RAM).
> > 
> > Currently we are using 39MB under arch/mips. I think I can easily cut this
> > down to 15MB or smaller, especially by moving some code here to the
> > appropriate driver directories (i.e. DRAM,  pcie, watchdog, etc.)
> > 
> > It will still be a large SoC, though.
> 
> Most modern SoCs are pretty large.  Taking this one step at a time and
> evaluating and re-architecting code along the way and we'll get there.
> You're probably going to run in to a lot of code that needs to be
> adapted to new frameworks, too.  What I strongly encourage from the
> example of previous SoCs that started out this way is to think of your
> internal tree as a reference only.  Sure, you'll want to grab as much of
> the complex init sequence code when moving things over, but it shouldn't
> be thought of as "move board X/Y/Z over" but "start adding board X with
> minimal peripherals" and add on top.

This is the goal. It should be easier to develop the first port without 
networking support since the image can be booted over PCIe though the 
networking support will be key because the customer disables this access. We 
plan to adapt to the new model. I've been working with it for some time with 
our OcteonTX line which was just upstreamed.

-Aaron