Zynq MPSoC Dynamic DDR DIMM Configuration support

[Michal Simek]

On 5/14/21 9:38 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> On 13/05/21, Michal Simek wrote:
>> Hi,
>>
>> On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
>>> On 13/05/21, Michal Simek wrote:
>>>> Hi Jorge,
>>>>
>>>> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
>>>>> Hi Michal
>>>>>
>>>>> We are doing some work on an MPSoC UZ3EG platform part of which
>>>>> requires us to replace FSBL with SPL.
>>>>
>>>> Just for curiosity what's the reason for this requirement that fsbl is
>>>> not enough?
>>>
>>> we prefer to have a common boot strategy on all the boards we support
>>> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
>>> process (including their firmwares); and of course our meta layers.
>>>
>>>>
>>>>>
>>>>> It seems the actual boot process is becoming an issue on these SoCs;
>>>>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
>>>>> extract it and program it;
>>>>
>>>> Actually it is not working like this. PMU FW is own part of boot.bin and
>>>> it is not embed in SPL.
>>>
>>> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
>>>
>>>>
>>>>
>>>>> 2) then SPL configures the PMU using a
>>>>> platform specific binary that gets built also with SPL; 
>>>>
>>>> PMU cfg object.
>>>
>>> yep, pm_cfg_obj.c
>>>
>>>>
>>>>
>>>> and finally,
>>>>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
>>>>> specific, part of the XSA).
>>>>
>>>> yes.
>>>>
>>>>>
>>>>> It is this final step in the boot sequence that is being broken by the
>>>>> Dynamic DDR DIMM configuration feature [1]
>>>>>
>>>>> [1] https://www.xilinx.com/support/answers/75768.html
>>>>
>>>> This was developed for zcu102 and maybe others boards which have DIMM
>>>> modules where origin part were EOL.
>>>
>>> right.
>>>
>>> but notice that there is also support on u-boot for
>>> altera/imx/marvell/microchip; so just wondering if we should add
>>> drivers/ddr/xilinx to this list.  
>>
>> Also SPD code is around too.
>>
>>
>>
>>>>
>>>>
>>>>> Are you aware of any work in progress to support this? Any thoughts on
>>>>> how to work around it and train the DDR? will the functionality
>>>>> required to implmenet Dynamic DDR DIMM configuration be added as a
>>>>> separate file to the XSA tarball or will we need to do some native
>>>>> implementation in SPL?
>>>>
>>>> I am not aware about any work on SPL side to support this. IIRC FSBL
>>>> didn't have generic DDR configuration. It was only by reading SPD and
>>>> aligned some parameters but it is quite a long time I have looked at it
>>>> last time.
>>>
>>> ok
>>>
>>>>
>>>>
>>>>> Becase without a change in the last link in the process chain
>>>>> described earlier (calls to psu_init()), DDR just wont be accessible
>>>>> to U-BOOT or OP-TEE.
>>>>>
>>>>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
>>>>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
>>>>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
>>>>> DDR wasnt properly trained/initialized.
>>>>>
>>>>> so, any thoughts or plans you can share?
>>>>
>>>> The question is why you need this feature to be there. It is not working
>>>> for every DIMM module. And normally if you have custom boards you need
>>>> just one DDR configuration (or limited number based on HW versions) and
>>>> for that there is really no need to waste your boot up time on it.
>>>
>>> not sure what you mean. we need this feature because it adds the
>>> expected flexibility to the bootloader. Sure, we can hardcode DDR
>>> configurations but why should we when it can be resolved by
>>> software. Am I missing your point?
>>
>> Ok. One is support for evaluation boards and the second for a product.
>> In product you don't need this because I would say most of projects are
>> not using DIMMs.
>> If you want to support generic evaluations boards then of course it can
>> be done.
>>
>>>
>>>> You can add multiple configurations to psu_init_gpl() and based on any
>>>> information (board rev/pins/etc) decided which one should be used.
>>>
>>> so what you are suggesting is that we customize psu_init_gpl() to the
>>> target (ie, have an updated xsa file with whatever config we need for
>>> this system).  
>>
>> In products yes.
>>
>>>
>>> what I fail to grasp is why we can't take a step forward and do what
>>> we do for other architectures in u-boot. and what fsbl already does by
>>> config.
>>>
>>> I mean, why not? do you foresee any integration issues with the
>>> current bootflow?
>>
>> Not really if you are willing to write that code just go for it.
>> I think it is definitely feasible to do so but I haven't seen any
>> algorithm to do it in generic way even there must be one behind in
>> xilinx tools. There are a lot of types of memories which you can
>> configure in xilinx design tools but it won't be any problem to limit it
>> to some.
> 
> ok. unfortunately I am not sure when - we have tight deadlines on this
> product so we will just update psu_init().
> 
> In fact, and this is relevant as well to keep in mind for a future
> TO-DO list, the issue we were having was not due to the dynamic
> configuration of DDR but to the fact that the DDR on this product is
> ECC (and ECC, when enabled in the DDR controller must be initialized
> _after_ psu_init() has run - which is something that FSBL does but SPL
> doesnt).
> 
> So we will just disable ECC for the time being and will send a patch
> once we come around to do it (or with a bit of luck someone will beat
> me to it and I will just pick it up from the next u-boot release)

On Zynq ECC should be supported already. Take a look at ddrc_init() but
I would have to look at details how is DDR initialized because this is
clearing just 1MB which is shadow by OCM at this stage.

I would expect that procedure will be the same. Just find out that DDR
controller is working in ECC mode and I hope that for initialization any
DMA is used.

Thanks,
Michal