Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Robert Hancock]

Hi Michal,

On Thu, 2021-01-21 at 09:41 +0100, Michal Simek wrote:
> Hi,
> 
> On 1/20/21 9:25 PM, Robert Hancock wrote:
> > I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
> > development board. I have been testing with U-Boot 2021.01, and using
> > the generated psu_init_gpl HW initialization code generated by Vivado
> > 2020.2, rather than the versions in the U-Boot tree, as we are
> > expecting to make changes to the configuration as we move to our own
> > board design.
> > 
> > The issue I was seeing is that the SPL seemingly locking up or crashing
> > early in the boot process, just after the SPL banner was printed. I've
> > CCed "Major A" as he reported something similar on the mailing list in
> > March ('ZynqMP boot: no messages from SPL other than "Debug uart
> > enabled"') but I didn't see a resoution posted.
> > 
> > After groveling around with the Xilinx JTAG debugger for a day or so, I
> > figured out that the problem was that the SPL ended up reading from an
> > invalid pointer address somewhere during the driver model
> > initialization process, and crashing. After using memory watchpoints to
> > identify where the bogus value was being written to the pointer storage
> > location, the memory write was actually coming from the Vivado 2020.2-
> > generated psu_init_gpl code. The offending code is in a function called
> > serdes_illcalib_pcie_gen1 and looks like this:
> > 
> >           if (gen2_calib != 1) 
> >           {
> >             ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
> >             ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
> > 0x10 : 0x00;
> >             Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
> >             Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
> >             Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
> >             Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
> >           }
> >           if (gen2_calib == 1) 
> >           {
> >             ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
> >             ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
> > 0x02 : 0x01;
> >             Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
> >             Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
> >             Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
> >             Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
> >           }
> > 
> > Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
> > hardware register, so I have no idea what this is trying to achieve.
> > Possibly this is some debug code to store some calibration results that
> > was left in by mistake? Those addresses may not be used in the Xilinx
> > FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
> > 
> > For now, I've worked around it by hacking the Xil_Out32 implementation
> > provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
> > writes to memory addresses in the OCRAM range (0xFFFC0000 to
> > 0xFFFFFFFF). With that in place, things seem to be working. Probably
> > this should be addressed in the Vivado psu_init code generator, but we
> > may need some workaround like this to avoid those using the affected
> > Xilinx tool versions from hitting this issue?
> > 
> 
> I can't see any code like this in u-boot code. I can't see context of
> this code that's why simply don't know what it does and why.

I can email the psu_init_gpl code I have privately if you need it - the code is
almost 1MB so too large for the list or the usual Pastebin etc. it seems.
However, you can see similar code in this file from ZCU102 in the Xilinx
embeddedsw repo:

https://github.com/Xilinx/embeddedsw/blob/master/lib/sw_apps/zynqmp_fsbl/misc/zcu102/psu_init_gpl.c

That has writes to 0xFFFExxxx addresses in the serdes_illcalib_pcie_gen1
function, like mine has, as well as in its serdes_illcalib which don't seem to
be in the version I have. Looking at the git history, it appears those writes
were added in a commit that updated the file for the 2020.2 tools release, so
that may have been when this was introduced.

I don't see any such writes in the psu_init_gpl files in the U-Boot repo, I am
guessing because those were all generated using older tool versions that didn't
have this behavior.

Interestingly, looking at the 2020.2 Petalinux BSP for ZCU102 in its provided
xilinx-zcu102-2020.2/project-spec/hw-description/psu_init_gpl.c file, those
memory writes are there but have been commented out. Not sure if it somehow
generated that way or if someone realized that issue and did that manually?


> And also not sure what you want me to do with this. Please suggest.

The main purpose was to let the community know of this issue as others working
with ZynqMP hardware may run into this with the current Xilinx tools.

I have this patch locally to work around this issue right now. We may want
something like this in U-Boot as even if the Xilinx tools are eventually
updated to not do these writes, people using affected versions may still run
into issues. Let me know if something like this seems reasonable and I can
submit as a proper patch:

diff --git a/board/xilinx/zynqmp/xil_io.h b/board/xilinx/zynqmp/xil_io.h
index e6caa7c850..6c7c8829e9 100644
--- a/board/xilinx/zynqmp/xil_io.h
+++ b/board/xilinx/zynqmp/xil_io.h
@@ -12,6 +12,17 @@
 
 static void Xil_Out32(unsigned long addr, unsigned long val)
 {
+	/*
+	 * Vivado 2020.2 (at least) generates psu_init_gpl code that attempts
+	 * to write to OCM memory addresses. This may be some leftover debug
+	 * code meant for use with the Xilinx FSBL, but these memory addresses
+	 * are used by U-Boot SPL and overwriting them will corrupt our memory.
+	 * Ignore attempts to write to these addresses.
+	 */
+	if (addr >= 0xfffc0000 && addr <= 0xffffffff) {
+		printf("Ignoring psu_init_gpl write to 0x%08lx\n", addr);
+		return;
+	}
 	writel(val, addr);
 }
 

> 
> Thanks,
> Michal