How to debug HW startup?

[Mauro Condarelli]

Hi Stefan,
update, see below.

On 1/14/20 12:08 AM, Mauro Condarelli wrote:
> Next episode of this telenovela:
>
> I rebuilt u-boot for ROM at BC030000 (my code, very similar to LinkIt).
> I flashed it at 30000 in SPI NOR:
>
> => usb start; sf probe
> starting USB...
> Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
> USB EHCI 1.00
> scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
>        scanning usb for storage devices... 1 Storage Device(s) found
> SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total
> 16 MiB
> => load usb 0:1 85000000 u-boot.secondary
> 390089 bytes read in 18 ms (20.7 MiB/s)
> => sf update ${fileaddr} 30000 ${filesize}
> device 0 offset 0x30000, size 0x5f3c9
> 390089 bytes written, 0 bytes skipped in 9.751s, speed 40952 B/s
> => reset
>
> Restarted old u-boot and jumped to new:
>
> U-Boot 1.1.3 (Apr 23 2017 - 14:59:01)
> VoCore2 > go bc030000
> ## Starting application at 0xBC030000 ...
> board_debug_uart_init():
> <debug_uart>
> System stopped here several minutes, enough for me to start writing
> this email, then it continued boot sequence:
>              board_debug_uart_init():
> board_early_init_f():
Here is the first strange thing:
apparently `board_debug_uart_init()` is called twice.

I am not 100% sure as code is full of `#ifdef`s, but it seems
first time it's called in `arch/mips/cpu/start.S`, most likely
in line 257 as I couldn't find traces of CONFIG_MIPS_INIT_STACK_IN_SRAM
(besides `arch/mips/cpu/Kconfig:381` where is defined,
defaults to `n` and none seems to touch anymore;
second strange thing is I find no trace of it in `.config`)

This, again, does not add-up (third "strange thing") because
I see nothing between calls to `debug_uart_init` (start.S:257)
and `board_init_f` (start.S:264) that could trigger a ~5min delay
(but I'm really out of my depth, here!) unless there's something
in the `printascii()`itself (e.g.: loop at serial_mtk.c:449 blocks).

> pinctrl_select_state_full('palmbus at 10000000', 'default'):
> pinctrl_select_state_full('uart2 at e00', 'default'):
> pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>
>
> U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 23:21:39 +0100)
>
> CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
> Model: VoCore2
> DRAM:  128 MiB
> pinctrl_select_state_full('palmbus at 10000000', 'default'):
> pinctrl_select_state_full('pinctrl at 60', 'default'):
> pinctrl_select_state_full('pin_state', 'default'):
> pinctrl_select_state_full('uart2 at e00', 'default'):
> pinctrl_select_state_full('uart2_pins', 'default'):
> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
> pinctrl_select_state_full('watchdog at 100', 'default'):
> WDT:   Started with servicing (60s timeout)
> board_early_init_r():
> arch_early_init_r():
> MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
> pinctrl_select_state_full('sd_iot_mode', 'default'):
> pinctrl_select_state_full('clk48m at 0', 'default'):
> pinctrl_select_state_full('mmc at 10130000', 'default'):
> mmc at 10130000: 0
> Loading Environment from FAT... *** Warning - bad CRC, using default
> environment
>
> In:    uart2 at e00
> Out:   uart2 at e00
> Err:   uart2 at e00
> Model: VoCore2
> arch_misc_init():
> =>
>
> Code, beside being targeted to ROM and with a different SYS_TEXT_BASE,
> is identical to what runs fine when started from RAM.
>
> I also tried copying flash to ram:
>
> => cp.b bc030000 80010000 5f3c9
> => go 80010000
> ## Starting application at 0x80010000 ...
> board_debug_uart_init():
> <debug_uart> [04010D08][04010D08]
> DDR Calibration DQS reg = 00008888
> relocate_code Pointer at: 87f5c000
> ***********************
> Watchdog Reset Occurred
> ***********************
>
> ... but this is almost expected because I relocated at another address
> without changing SYS_TEXT_BASE.
>
> A further measurement shows booting u-boot from flash stops for
> almost 5 minutes (4m48s, using a manual stopwatch).
>
> I sincerely have no idea where to bang my head :(
>
> TiA
> Mauro
>
>
> On 1/13/20 3:14 PM, Mauro Condarelli wrote:
>> Hi Stefan,
>> unfortunately it does *not* work.
>>
>> Ram based is ok, but rom  is just as silent as mine:
>>
>> => usb start; sf probe;
>> starting USB...
>> Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
>> USB EHCI 1.00
>> scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
>>        scanning usb for storage devices... 1 Storage Device(s) found
>> pinctrl_select_state_full('spi at b00', 'default'):
>> SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total
>> 16 MiB
>> => load usb 0:1 85000000 u-boot_linkit.bin-rom
>> 455708 bytes read in 22 ms (19.8 MiB/s)
>> => sf update ${fileaddr} 0 ${filesize}
>> device 0 offset 0x0, size 0x6f41c
>> 455708 bytes written, 0 bytes skipped in 12.288s, speed 37966 B/s
>> => reset
>> resetting ...
>> pinctrl_select_state_full('syscon-reboot', 'default'):
>> pinctrl_select_state_full('system-controller at 0', 'default'):
>>
>> I might have done something stupid; Now I need do stop, but I'll
>> test again this evening.
>>
>> Many thanks
>> Mauro
>>
>>
>> On 1/13/20 1:45 PM, Stefan Roese wrote:
>>> On 13.01.20 13:24, Mauro Condarelli wrote:
>>>> On 1/13/20 12:39 PM, Stefan Roese wrote:
>>>>> Hi Mauro,
>>>>>
>>>>> On 13.01.20 11:24, Mauro Condarelli wrote:
>>>>>> On 1/13/20 7:53 AM, Stefan Roese wrote:
>>>>>>> Hi Mauro,
>>>>>>>
>>>>>>> On 11.01.20 20:00, Mauro Condarelli wrote:
>>>>>>>> I managed to find ONE of the reasons why my ROM build didn't run:
>>>>>>>> I forgot to enable `CONFIG_BOARD_EARLY_INIT_F=y`.
>>>>>>> I see. This explains of course, why your board does not boot without
>>>>>>> any "preloader".
>>>>>>>
>>>>>>>> I wanted, nonetheless, be prepared for further mishaps, but
>>>>>>>> I have some other problems (see below).
>>>>>>> Are these issues now fixed? I scanned the discussion about the DEBUG
>>>>>>> UART on the list. Is this working for you now or do you have any
>>>>>>> other
>>>>>>> issues still? Did you successfully boot your new U-Boot from SPI NOR?
>>>>>> Yes and no :(
>>>>>>
>>>>>> I fixed my RAM-based problems, but booting from SPI NOR still
>>>>>> refuses to
>>>>>> utter a single byte.
>>>>>> I do attach  my defconfigs and my board.c for your reading pleasure
>>>>>> (in
>>>>>> case you have troubles getting asleep they should provide a good
>>>>>> cure).
>>>>> Before looking into this in more depth (I actually do have problems
>>>>> sleeping
>>>>> though, so I might take a look at it later ;)), why don't you re-start
>>>>> with
>>>>> your defconfig by using the linkit defconfig file. If you are lucky,
>>>>> the
>>>>> LinkIt binary might even work for the VoCore2. Or what are the
>>>>> differences
>>>>> except the SoC type and WLAN? Its the same DDR2 config and the same
>>>>> UART.
>>>>> So it might get you pretty far - also with ROM based booting.
>>>> I will try it.
>>>> Just to be on the safe side, could You attach a binary (or two: ROM and
>>>> RAM) to
>>>> the mail? If it works it would really give me a start base.
>>> See below.
>>>   
>>>>>> I also added a couple of printouts in drivers/pinctrl/pinctrl-uclass.c
>>>>>> and it
>>>>>> turns out system tries to setup pins for uart2, but fails (maybe
>>>>>> because
>>>>>> pinctrl at 60 is not yet setup correctly?).
>>>>>> This is the output I get from RAM-based u-boot:
>>>>>>
>>>>>> <debug_uart> board_debug_uart_init():
>>>>>> board_early_init_f():
>>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>>> pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
>>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>>>
>>>>>> U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 01:03:59 +0100)
>>>>>>
>>>>>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
>>>>>> Model: VoCore2
>>>>>> DRAM:  128 MiB
>>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>>> pinctrl_select_state_full('pinctrl at 60', 'default'):
>>>>>> pinctrl_select_state_full('pin_state', 'default'):
>>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>>> pinctrl_select_state_full('uart2_pins', 'default'):
>>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>>> pinctrl_select_state_full('watchdog at 100', 'default'):
>>>>>> WDT:   Started with servicing (60s timeout)
>>>>>> board_early_init_r():
>>>>>> arch_early_init_r():
>>>>>> MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>>> pinctrl_select_state_full('sd_iot_mode', 'default'):
>>>>>> pinctrl_select_state_full('clk48m at 0', 'default'):
>>>>>> pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>>> mmc at 10130000: 0
>>>>>> Loading Environment from FAT... *** Warning - bad CRC, using default
>>>>>> environment
>>>>>>
>>>>>> In:    uart2 at e00
>>>>>> Out:   uart2 at e00
>>>>>> Err:   uart2 at e00
>>>>>> Model: VoCore2
>>>>>> arch_misc_init():
>>>>>> =>
>>>>> I don't have all these pinctrl issues on LinkIt. I suspect a config
>>>>> issue and / or dts issue. Please compare.
>>>> I will, but I was trying to say something different:
>>>> Apparently the stock software tries to initialize uart2 pinctrl and it
>>>> *seems* to succeed later in initialization.
>>>> This *seems* to indicate explicit pin setup in board_early_init_f() is
>>>> not strictly needed (we would lose only the first few lines even if
>>>> we don't fix the error).
>>>> I had a look to pinctrl-mt7628.c and it seems to do the right thing
>>>> upon call to ('uart2_pins', 'default'), so anything after that *should*
>>>> work even without low-level setup.
>>> You might be correct here. So this explicit pin-muxing for UART2 is
>>> only needed when DEBUG UART is used. This makes the code a bit clearer
>>> and smaller. Lets keep it on our list to remove this, once all this
>>> is resolved.
>>>  
>>>>>> ROM-based u-boot, as said, does not print *anything*, not even
>>>>>> garbage.
>>>>>> I'm beginning to suspect I have some mishap with start address or
>>>>>> similar.
>>>>>> I have absolutely no idea how to debug this without a JTAG probe
>>>>>> (which
>>>>>> I don't have and would be non-trivial to get working; soldering
>>>>>> required).
>>>>> Yes. JTAG requires soldering IIRC.
>>>>>  
>>>>>> The only idea I currently have is to modify my "old" u-boot to do
>>>>>> initialization
>>>>>> and then jump to beginning of "new" u-boot.
>>>>>> If I can make it work in an automatic way I can try removing
>>>>>> initialization steps
>>>>>> till I find the "culprit".
>>>>>> Any better idea would be welcome, of course!
>>>>>>
>>>>>> If I have to resort to that clumsy method, would be enough to put 
>>>>>> "new"
>>>>>> in  SPI NOR (of course at an higher address, e.g.: 30000 as "old" is
>>>>>> only
>>>>>> 183272 bytes long) and jump to the first location?
>>>>>> I assume I will have to relocate CONFIG_SYS_TEXT_BASE=0xbc030000
>>>>> This is the TEXT_BASE for ROM based booting:
>>>>>
>>>>> CONFIG_SYS_TEXT_BASE=0x9c000000
>>>> AFAIK 0x9c000000 and 0xbc000000 map to the same (unmapped) memory
>>>> region, the latter being uncached.
>>>> I proposed 0xbc030000 (but it could be 0x9c030000) as base to leave some
>>>> space for "old" (possibly crippled) u-boot doing initialization and then
>>>> "jumping"
>>>> to "new".
>>> I see. Please give the LinkIt version a try first. This sounds easier
>>> (if it works) and more promising for my taste.
>>>  
>>>>> Please see configs/linkit-smart-7688_defconfig.
>>>>>
>>>>>> Did I forget something?
>>>>> Not sure. I still think using the linkit port as a base for SPI NOR
>>>>> based booting would be a very good test at least. Do you see any
>>>>> problems with this approach (board / SoC differences)?
>>>> No. I see no troubles, for a basic startup.
>>>> MMC would probably need some changes, but that can come later.
>>> Yes. This test is just for basic very low level bootup checking. If
>>> this works, then you can use it as a basis to fine-tune your VoCore2
>>> version from it by changing MMC etc support.
>>>
>>>> As said: I would like to try a prebuilt binary as first thing, so I can
>>>> be sure I have no other problems (like gcc version or other local
>>>> variations); then I would try to reproduce locally the equivalent binary
>>>> with no changes and finally I would try to incorporate my specific
>>>> needs.
>>>> What do You think?
>>> Please find attached the current mainline versions for LinkIt, one
>>> for RAM booting and one for ROM booting (in SPI NOR). I checked both
>>> version and they work fine on my LinkIt 7688 board:
>>>
>>> U-Boot 2020.01-00473-g88366b96ee (Jan 13 2020 - 13:37:15 +0100)
>>>
>>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (4-Byte SPI Addr)
>>> Model: LinkIt-Smart-7688
>>> DRAM:  128 MiB
>>> Loading Environment from SPI Flash... SF: Detected mx25l25635e with
>>> page size 256 Bytes, erase size 64 KiB, total 32 MiB
>>> OK
>>> Net:   eth0: eth at 10110000
>>> Hit any key to stop autoboot:  0
>>> =>
>>>
>>>
>>> Thanks,
>>> Stefan
Regards
Mauro