[U-Boot] [linux-sunxi] A10 bring up and DRAM configuration procedure

[Siarhei Siamashka]

Hi Piotr,

On Sun, 3 Apr 2016 16:26:44 +0200
Piotr Król <piotr.krol at 3mdeb.com> wrote:

> Hi all,
> I'm trying to power on custom A10 hardware based on Cubieboard1. DDR3
> modules used are MT41J64M16 [1] - at least FBGA Code D9MNQ match this
> hardware.
> 
> I'm using mainline U-Boot built using Buildroot with Cubieboard defconfig.
> Unfortunately U-Boot hangs "randomly" at various locations. Farthest I can get
> is:
> 
> ## Booting kernel from Legacy Image at 48000000 ...
>    Image Name:   Linux-3.4.103
>    Image Type:   ARM Linux Kernel Image (uncompressed)
>    Data Size:    4358824 Bytes = 4.2 MiB
>    Load Address: 40008000
>    Entry Point:  40008000
>    Verifying Checksum ... 
> 
> This hangs in crc32_no_comp [2]. Although this is not 100% reproducible. Full
> bootlog [5].
> 
> I made tests using various DRAM parameters configurations and those take
> booting process farthest are with CONFIG_DRAM_CLK=240:
> 
> # below gives ~80% chance to get to Verifying Checksum
> CONFIG_DRAM_CLK=240
> CONFIG_DRAM_TIMINGS_DDR3_1066F_1333H=y
> 
> # below gives ~85% chance to get to Verifying Checksum
> CONFIG_DRAM_CLK=240
> CONFIG_DRAM_TIMINGS_VENDOR_MAGIC=y
> 
> Other configuration like CONFIG_DRAM_CLK=360 give worst results. For
> 240MHz I was even able get to "Starting kernel ...", but this is
> sporadic. I disabled in U-Boot as much I could, my defconfig is here
> [3].
> 
> From what I was able to found this is result of incorrect DRAM configuration
> [4]. Anyone can confirm that this is correct diagnosis ? If not then what test
> I can perform to gain better confidence ?

If you get somewhat better results depending on the DRAM clock
frequency settings, then I would say that it pretty much points to
the DRAM related problems. So you are apparently on the right track.

> Other question is how to proceed from this point ? I have datasheet for DDR3,
> but number of possible parameters to tweak seems to be overwhelming. Also as
> sunxi wiki claim timing from datasheet have to be converted to number of cycles
> - I'm not sure how to approach that. Is there any guide where to start ?

Well, the standard JEDEC timings are already tabulated in 

    http://git.denx.de/?p=u-boot.git;a=blob;f=board/sunxi/dram_timings_sun4i.h

Some DRAM chips may have better timings than mandated by the standard,
but this is more like performance tuning. Either way, these timings are
a "digital" thing, they specify how many cycles to wait between sending
different commands to the DRAM chip. And they are rather unlikely to be
related to your reliability problems.

It is much more likely that you have "analog" problems and the signal
gets distorted too badly on the way from the DRAM controller in the
SoC to the DRAM chips. From what I heard, the DDR3 tracks routing is a
major PITA, one needs to get both the tracks length and the tracks
impedance matched within certain tolerance levels. As you are dealing
with a custom hardware design, then maybe the DDR3 routing is a little
bit screwed up?

The DRAM controller configuration registers allow to configure the
delays and drive/termination impedance to correct the PCB imperfection
to some extent. In practice, all the "professionally" designed Allwinner
devices work with the same generic Allwinner blessed DRAM settings,
because the PCB routing is apparently usually done and tested to work
fine with these default settings. Also reducing the DRAM clock speed
usually improves reliability. It was believed that any A10/A13/A20
device should work stable if we clock the DRAM down to 360MHz.

On the hardware side, you can maybe check the ZQ and RZQ resistors
(one connected to the SoC and also one extra resistor for each DRAM
chip). They should at least exist and be preferably 240 ohm. Some board
manufacturers tune the nominal value of these resistors, apparently
as a way to fine tune the reliability. That's the configuration knob
that is easily available to board manufacturers (they don't dare to
touch the DRAM settings in U-boot and probably don't know how to do
this).

One more thing is the VDD-DLL voltage. It is served from DCDC3 (AXP209
PMIC) and is usually set to 1.250V by default. Increasing this voltage
usually improves the DRAM reliability and allows it to be clocked
higher. You can try to increase this voltage to 1.300V to check
if this helps. Don't go beyond 1.325V on Allwinner A10 though.

> sunxi wiki describe calibration, when it is possible to run system, but what
> can be done in situation when U-Boot cannot reach that state ?

The whole idea of this calibration process is to find good ZQ and TPR3
settings. The settings are assumed to be good if the board is able to
pass the lima-memtester reliability test at a higher DRAM clock speed
without failing. We exploit the fact that lima-memster is by far more
sensitive to DRAM misconfiguration than the usual software. So the
borderline unreliable DRAM settings may not exhibit any visible bad
effects (the system is able to boot, the usual linux software works
fine too), but easily detected by lima-memtester. Because the system
is sufficiently usable at such borderline unstable DRAM settings, we
can run automation scripts to do a bruteforce search, which is probing
different TPR3 values without human supervision. That's what is
explained at:

    https://linux-sunxi.org/A10_DRAM_Controller_Calibration

But in your case, you can try different ZQ and TPR3 settings manually
in the U-Boot defconfig and instead of running lima-memtester (which
first needs a bootable userland) just monitor how far the system is
able to boot as an alternative selection criteria for picking better
configuration values. If you manage to boot the system up to a
semi-working userland, then you can switch to using lima-memtester
for the final tuning and validation.

> If anyone more experienced can advise what can be done at this point I would
> appreciate.

Maybe ask the hardware people if they have done the DDR3 routing right?
Also if it is just a single board, then it might be not misconfigured,
but defective. Trying another sample could be a good idea.

> [1] http://www.micron.com/~/media/Documents/Products/Data%20Sheet/DRAM/DDR3/1Gb_DDR3_SDRAM.pdf
> [2] http://git.denx.de/?p=u-boot.git;a=blob;f=lib/crc32.c;h=97592124867abb815d576899f8789545c3aef1aa;hb=HEAD#l200
> [3] https://gist.github.com/pietrushnic/ea41a4ae38b7af8a6fa456113ce19af8
> [4] http://www.denx.de/wiki/view/DULG/UBootCrashAfterRelocation
> [5] https://gist.github.com/pietrushnic/d6a3d6c0b4a20a3226dfb67c4d129142

-- 
Best regards,
Siarhei Siamashka