How to debug HW startup?

[Mauro Condarelli]

Next episode of this telenovela:

I rebuilt u-boot for ROM at BC030000 (my code, very similar to LinkIt).
I flashed it at 30000 in SPI NOR:

=> usb start; sf probe
starting USB...
Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
USB EHCI 1.00
scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total
16 MiB
=> load usb 0:1 85000000 u-boot.secondary
390089 bytes read in 18 ms (20.7 MiB/s)
=> sf update ${fileaddr} 30000 ${filesize}
device 0 offset 0x30000, size 0x5f3c9
390089 bytes written, 0 bytes skipped in 9.751s, speed 40952 B/s
=> reset

Restarted old u-boot and jumped to new:

U-Boot 1.1.3 (Apr 23 2017 - 14:59:01)
VoCore2 > go bc030000
## Starting application at 0xBC030000 ...
board_debug_uart_init():
<debug_uart>
System stopped here several minutes, enough for me to start writing
this email, then it continued boot sequence:
             board_debug_uart_init():
board_early_init_f():
pinctrl_select_state_full('palmbus at 10000000', 'default'):
pinctrl_select_state_full('uart2 at e00', 'default'):
pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
pinctrl_select_state_full('clkctrl at 0x2c', 'default'):


U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 23:21:39 +0100)

CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
Model: VoCore2
DRAM:  128 MiB
pinctrl_select_state_full('palmbus at 10000000', 'default'):
pinctrl_select_state_full('pinctrl at 60', 'default'):
pinctrl_select_state_full('pin_state', 'default'):
pinctrl_select_state_full('uart2 at e00', 'default'):
pinctrl_select_state_full('uart2_pins', 'default'):
pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
pinctrl_select_state_full('watchdog at 100', 'default'):
WDT:   Started with servicing (60s timeout)
board_early_init_r():
arch_early_init_r():
MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
pinctrl_select_state_full('sd_iot_mode', 'default'):
pinctrl_select_state_full('clk48m at 0', 'default'):
pinctrl_select_state_full('mmc at 10130000', 'default'):
mmc at 10130000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default
environment

In:    uart2 at e00
Out:   uart2 at e00
Err:   uart2 at e00
Model: VoCore2
arch_misc_init():
=>

Code, beside being targeted to ROM and with a different SYS_TEXT_BASE,
is identical to what runs fine when started from RAM.

I also tried copying flash to ram:

=> cp.b bc030000 80010000 5f3c9
=> go 80010000
## Starting application at 0x80010000 ...
board_debug_uart_init():
<debug_uart> [04010D08][04010D08]
DDR Calibration DQS reg = 00008888
relocate_code Pointer at: 87f5c000
***********************
Watchdog Reset Occurred
***********************

... but this is almost expected because I relocated at another address
without changing SYS_TEXT_BASE.

A further measurement shows booting u-boot from flash stops for
almost 5 minutes (4m48s, using a manual stopwatch).

I sincerely have no idea where to bang my head :(

TiA
Mauro


On 1/13/20 3:14 PM, Mauro Condarelli wrote:
> Hi Stefan,
> unfortunately it does *not* work.
>
> Ram based is ok, but rom  is just as silent as mine:
>
> => usb start; sf probe;
> starting USB...
> Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
> USB EHCI 1.00
> scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
>        scanning usb for storage devices... 1 Storage Device(s) found
> pinctrl_select_state_full('spi at b00', 'default'):
> SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total
> 16 MiB
> => load usb 0:1 85000000 u-boot_linkit.bin-rom
> 455708 bytes read in 22 ms (19.8 MiB/s)
> => sf update ${fileaddr} 0 ${filesize}
> device 0 offset 0x0, size 0x6f41c
> 455708 bytes written, 0 bytes skipped in 12.288s, speed 37966 B/s
> => reset
> resetting ...
> pinctrl_select_state_full('syscon-reboot', 'default'):
> pinctrl_select_state_full('system-controller at 0', 'default'):
>
> I might have done something stupid; Now I need do stop, but I'll
> test again this evening.
>
> Many thanks
> Mauro
>
>
> On 1/13/20 1:45 PM, Stefan Roese wrote:
>> On 13.01.20 13:24, Mauro Condarelli wrote:
>>>
>>> On 1/13/20 12:39 PM, Stefan Roese wrote:
>>>> Hi Mauro,
>>>>
>>>> On 13.01.20 11:24, Mauro Condarelli wrote:
>>>>> On 1/13/20 7:53 AM, Stefan Roese wrote:
>>>>>> Hi Mauro,
>>>>>>
>>>>>> On 11.01.20 20:00, Mauro Condarelli wrote:
>>>>>>> I managed to find ONE of the reasons why my ROM build didn't run:
>>>>>>> I forgot to enable `CONFIG_BOARD_EARLY_INIT_F=y`.
>>>>>> I see. This explains of course, why your board does not boot without
>>>>>> any "preloader".
>>>>>>
>>>>>>> I wanted, nonetheless, be prepared for further mishaps, but
>>>>>>> I have some other problems (see below).
>>>>>> Are these issues now fixed? I scanned the discussion about the DEBUG
>>>>>> UART on the list. Is this working for you now or do you have any
>>>>>> other
>>>>>> issues still? Did you successfully boot your new U-Boot from SPI NOR?
>>>>> Yes and no :(
>>>>>
>>>>> I fixed my RAM-based problems, but booting from SPI NOR still
>>>>> refuses to
>>>>> utter a single byte.
>>>>> I do attach  my defconfigs and my board.c for your reading pleasure
>>>>> (in
>>>>> case you have troubles getting asleep they should provide a good
>>>>> cure).
>>>> Before looking into this in more depth (I actually do have problems
>>>> sleeping
>>>> though, so I might take a look at it later ;)), why don't you re-start
>>>> with
>>>> your defconfig by using the linkit defconfig file. If you are lucky,
>>>> the
>>>> LinkIt binary might even work for the VoCore2. Or what are the
>>>> differences
>>>> except the SoC type and WLAN? Its the same DDR2 config and the same
>>>> UART.
>>>> So it might get you pretty far - also with ROM based booting.
>>> I will try it.
>>> Just to be on the safe side, could You attach a binary (or two: ROM and
>>> RAM) to
>>> the mail? If it works it would really give me a start base.
>> See below.
>>   
>>>>> I also added a couple of printouts in drivers/pinctrl/pinctrl-uclass.c
>>>>> and it
>>>>> turns out system tries to setup pins for uart2, but fails (maybe
>>>>> because
>>>>> pinctrl at 60 is not yet setup correctly?).
>>>>> This is the output I get from RAM-based u-boot:
>>>>>
>>>>> <debug_uart> board_debug_uart_init():
>>>>> board_early_init_f():
>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>> pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>>
>>>>> U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 01:03:59 +0100)
>>>>>
>>>>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
>>>>> Model: VoCore2
>>>>> DRAM:  128 MiB
>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>> pinctrl_select_state_full('pinctrl at 60', 'default'):
>>>>> pinctrl_select_state_full('pin_state', 'default'):
>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>> pinctrl_select_state_full('uart2_pins', 'default'):
>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>> pinctrl_select_state_full('watchdog at 100', 'default'):
>>>>> WDT:   Started with servicing (60s timeout)
>>>>> board_early_init_r():
>>>>> arch_early_init_r():
>>>>> MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>> pinctrl_select_state_full('sd_iot_mode', 'default'):
>>>>> pinctrl_select_state_full('clk48m at 0', 'default'):
>>>>> pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>> mmc at 10130000: 0
>>>>> Loading Environment from FAT... *** Warning - bad CRC, using default
>>>>> environment
>>>>>
>>>>> In:    uart2 at e00
>>>>> Out:   uart2 at e00
>>>>> Err:   uart2 at e00
>>>>> Model: VoCore2
>>>>> arch_misc_init():
>>>>> =>
>>>> I don't have all these pinctrl issues on LinkIt. I suspect a config
>>>> issue and / or dts issue. Please compare.
>>> I will, but I was trying to say something different:
>>> Apparently the stock software tries to initialize uart2 pinctrl and it
>>> *seems* to succeed later in initialization.
>>> This *seems* to indicate explicit pin setup in board_early_init_f() is
>>> not strictly needed (we would lose only the first few lines even if
>>> we don't fix the error).
>>> I had a look to pinctrl-mt7628.c and it seems to do the right thing
>>> upon call to ('uart2_pins', 'default'), so anything after that *should*
>>> work even without low-level setup.
>> You might be correct here. So this explicit pin-muxing for UART2 is
>> only needed when DEBUG UART is used. This makes the code a bit clearer
>> and smaller. Lets keep it on our list to remove this, once all this
>> is resolved.
>>  
>>>>> ROM-based u-boot, as said, does not print *anything*, not even
>>>>> garbage.
>>>>> I'm beginning to suspect I have some mishap with start address or
>>>>> similar.
>>>>> I have absolutely no idea how to debug this without a JTAG probe
>>>>> (which
>>>>> I don't have and would be non-trivial to get working; soldering
>>>>> required).
>>>> Yes. JTAG requires soldering IIRC.
>>>>  
>>>>> The only idea I currently have is to modify my "old" u-boot to do
>>>>> initialization
>>>>> and then jump to beginning of "new" u-boot.
>>>>> If I can make it work in an automatic way I can try removing
>>>>> initialization steps
>>>>> till I find the "culprit".
>>>>> Any better idea would be welcome, of course!
>>>>>
>>>>> If I have to resort to that clumsy method, would be enough to put 
>>>>> "new"
>>>>> in  SPI NOR (of course at an higher address, e.g.: 30000 as "old" is
>>>>> only
>>>>> 183272 bytes long) and jump to the first location?
>>>>> I assume I will have to relocate CONFIG_SYS_TEXT_BASE=0xbc030000
>>>> This is the TEXT_BASE for ROM based booting:
>>>>
>>>> CONFIG_SYS_TEXT_BASE=0x9c000000
>>> AFAIK 0x9c000000 and 0xbc000000 map to the same (unmapped) memory
>>> region, the latter being uncached.
>>> I proposed 0xbc030000 (but it could be 0x9c030000) as base to leave some
>>> space for "old" (possibly crippled) u-boot doing initialization and then
>>> "jumping"
>>> to "new".
>> I see. Please give the LinkIt version a try first. This sounds easier
>> (if it works) and more promising for my taste.
>>  
>>>> Please see configs/linkit-smart-7688_defconfig.
>>>>
>>>>> Did I forget something?
>>>> Not sure. I still think using the linkit port as a base for SPI NOR
>>>> based booting would be a very good test at least. Do you see any
>>>> problems with this approach (board / SoC differences)?
>>> No. I see no troubles, for a basic startup.
>>> MMC would probably need some changes, but that can come later.
>> Yes. This test is just for basic very low level bootup checking. If
>> this works, then you can use it as a basis to fine-tune your VoCore2
>> version from it by changing MMC etc support.
>>
>>> As said: I would like to try a prebuilt binary as first thing, so I can
>>> be sure I have no other problems (like gcc version or other local
>>> variations); then I would try to reproduce locally the equivalent binary
>>> with no changes and finally I would try to incorporate my specific
>>> needs.
>>> What do You think?
>> Please find attached the current mainline versions for LinkIt, one
>> for RAM booting and one for ROM booting (in SPI NOR). I checked both
>> version and they work fine on my LinkIt 7688 board:
>>
>> U-Boot 2020.01-00473-g88366b96ee (Jan 13 2020 - 13:37:15 +0100)
>>
>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (4-Byte SPI Addr)
>> Model: LinkIt-Smart-7688
>> DRAM:  128 MiB
>> Loading Environment from SPI Flash... SF: Detected mx25l25635e with
>> page size 256 Bytes, erase size 64 KiB, total 32 MiB
>> OK
>> Net:   eth0: eth at 10110000
>> Hit any key to stop autoboot:  0
>> =>
>>
>>
>> Thanks,
>> Stefan