Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge Ramirez-Ortiz, Foundries]

On 14/05/21, Michal Simek wrote:
> 
> 
> On 5/14/21 9:38 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> > On 13/05/21, Michal Simek wrote:
> >> Hi,
> >>
> >> On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> >>> On 13/05/21, Michal Simek wrote:
> >>>> Hi Jorge,
> >>>>
> >>>> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> >>>>> Hi Michal
> >>>>>
> >>>>> We are doing some work on an MPSoC UZ3EG platform part of which
> >>>>> requires us to replace FSBL with SPL.
> >>>>
> >>>> Just for curiosity what's the reason for this requirement that fsbl is
> >>>> not enough?
> >>>
> >>> we prefer to have a common boot strategy on all the boards we support
> >>> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
> >>> process (including their firmwares); and of course our meta layers.
> >>>
> >>>>
> >>>>>
> >>>>> It seems the actual boot process is becoming an issue on these SoCs;
> >>>>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
> >>>>> extract it and program it;
> >>>>
> >>>> Actually it is not working like this. PMU FW is own part of boot.bin and
> >>>> it is not embed in SPL.
> >>>
> >>> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
> >>>
> >>>>
> >>>>
> >>>>> 2) then SPL configures the PMU using a
> >>>>> platform specific binary that gets built also with SPL; 
> >>>>
> >>>> PMU cfg object.
> >>>
> >>> yep, pm_cfg_obj.c
> >>>
> >>>>
> >>>>
> >>>> and finally,
> >>>>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> >>>>> specific, part of the XSA).
> >>>>
> >>>> yes.
> >>>>
> >>>>>
> >>>>> It is this final step in the boot sequence that is being broken by the
> >>>>> Dynamic DDR DIMM configuration feature [1]
> >>>>>
> >>>>> [1] https://www.xilinx.com/support/answers/75768.html
> >>>>
> >>>> This was developed for zcu102 and maybe others boards which have DIMM
> >>>> modules where origin part were EOL.
> >>>
> >>> right.
> >>>
> >>> but notice that there is also support on u-boot for
> >>> altera/imx/marvell/microchip; so just wondering if we should add
> >>> drivers/ddr/xilinx to this list.  
> >>
> >> Also SPD code is around too.
> >>
> >>
> >>
> >>>>
> >>>>
> >>>>> Are you aware of any work in progress to support this? Any thoughts on
> >>>>> how to work around it and train the DDR? will the functionality
> >>>>> required to implmenet Dynamic DDR DIMM configuration be added as a
> >>>>> separate file to the XSA tarball or will we need to do some native
> >>>>> implementation in SPL?
> >>>>
> >>>> I am not aware about any work on SPL side to support this. IIRC FSBL
> >>>> didn't have generic DDR configuration. It was only by reading SPD and
> >>>> aligned some parameters but it is quite a long time I have looked at it
> >>>> last time.
> >>>
> >>> ok
> >>>
> >>>>
> >>>>
> >>>>> Becase without a change in the last link in the process chain
> >>>>> described earlier (calls to psu_init()), DDR just wont be accessible
> >>>>> to U-BOOT or OP-TEE.
> >>>>>
> >>>>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> >>>>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> >>>>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
> >>>>> DDR wasnt properly trained/initialized.
> >>>>>
> >>>>> so, any thoughts or plans you can share?
> >>>>
> >>>> The question is why you need this feature to be there. It is not working
> >>>> for every DIMM module. And normally if you have custom boards you need
> >>>> just one DDR configuration (or limited number based on HW versions) and
> >>>> for that there is really no need to waste your boot up time on it.
> >>>
> >>> not sure what you mean. we need this feature because it adds the
> >>> expected flexibility to the bootloader. Sure, we can hardcode DDR
> >>> configurations but why should we when it can be resolved by
> >>> software. Am I missing your point?
> >>
> >> Ok. One is support for evaluation boards and the second for a product.
> >> In product you don't need this because I would say most of projects are
> >> not using DIMMs.
> >> If you want to support generic evaluations boards then of course it can
> >> be done.
> >>
> >>>
> >>>> You can add multiple configurations to psu_init_gpl() and based on any
> >>>> information (board rev/pins/etc) decided which one should be used.
> >>>
> >>> so what you are suggesting is that we customize psu_init_gpl() to the
> >>> target (ie, have an updated xsa file with whatever config we need for
> >>> this system).  
> >>
> >> In products yes.
> >>
> >>>
> >>> what I fail to grasp is why we can't take a step forward and do what
> >>> we do for other architectures in u-boot. and what fsbl already does by
> >>> config.
> >>>
> >>> I mean, why not? do you foresee any integration issues with the
> >>> current bootflow?
> >>
> >> Not really if you are willing to write that code just go for it.
> >> I think it is definitely feasible to do so but I haven't seen any
> >> algorithm to do it in generic way even there must be one behind in
> >> xilinx tools. There are a lot of types of memories which you can
> >> configure in xilinx design tools but it won't be any problem to limit it
> >> to some.
> > 
> > ok. unfortunately I am not sure when - we have tight deadlines on this
> > product so we will just update psu_init().
> > 
> > In fact, and this is relevant as well to keep in mind for a future
> > TO-DO list, the issue we were having was not due to the dynamic
> > configuration of DDR but to the fact that the DDR on this product is
> > ECC (and ECC, when enabled in the DDR controller must be initialized
> > _after_ psu_init() has run - which is something that FSBL does but SPL
> > doesnt).
> > 
> > So we will just disable ECC for the time being and will send a patch
> > once we come around to do it (or with a bit of luck someone will beat
> > me to it and I will just pick it up from the next u-boot release)
> 
> On Zynq ECC should be supported already. Take a look at ddrc_init() but
> I would have to look at details how is DDR initialized because this is
> clearing just 1MB which is shadow by OCM at this stage.
> 
> I would expect that procedure will be the same. Just find out that DDR
> controller is working in ECC mode and I hope that for initialization any
> DMA is used.

yeah, awesome. thanks for the info.

> 
> Thanks,
> Michal