Debugging VoCore2 ROM Startup (was: How to debug HW startup?)

[Mauro Condarelli]

I found *one* of the bugs in startup:
To enable UART2 pinmux setting:
    void __iomem *gpio_mode;
    gpio_mode = ioremap_nocache(MT76XX_GPIO1_MODE, 0x100);
    clrbits_le32(gpio_mode, GENMASK(27, 26));
is not enough; you need also:
    setbits_le32(gpio_mode, GENMASK(3, 2));
I actually use the more explicit, but hopefully equivalent:
    #define MT76XX_GPIO1_MODE   0x10000060
    #define MIPS_KSEG1_START    0xA0000000

    uint32_t v, *a;
    a = (uint32_t *)(MIPS_KSEG1_START + MT76XX_GPIO1_MODE);
    v = *a;
    v &= 0xF3FFFFFF;
    v |= 0x0000000C;
    *a = v;

It is unclear to me how Linkit port could work.
Anyways now my "secondary from ROM" approach now
works without the long delay described below.

Unfortunately this does not seem to be the whole story because
flashing u-boot linked directly in the "right palace":

    SYS_TEXT_BASE = 0x9c000000

still refuses to display anything on serial; I assume some other
initialization is missing, but that will be another fight.

Any insight welcome.
Regards
Mauro

On 1/14/20 12:08 AM, Mauro Condarelli wrote:
> Next episode of this telenovela:
>
> I rebuilt u-boot for ROM at BC030000 (my code, very similar to LinkIt).
> I flashed it at 30000 in SPI NOR:
>
> => usb start; sf probe
> starting USB...
> Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
> USB EHCI 1.00
> scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
>        scanning usb for storage devices... 1 Storage Device(s) found
> SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total
> 16 MiB
> => load usb 0:1 85000000 u-boot.secondary
> 390089 bytes read in 18 ms (20.7 MiB/s)
> => sf update ${fileaddr} 30000 ${filesize}
> device 0 offset 0x30000, size 0x5f3c9
> 390089 bytes written, 0 bytes skipped in 9.751s, speed 40952 B/s
> => reset
>
> Restarted old u-boot and jumped to new:
>
> U-Boot 1.1.3 (Apr 23 2017 - 14:59:01)
> VoCore2 > go bc030000
> ## Starting application at 0xBC030000 ...
> board_debug_uart_init():
> <debug_uart>
> System stopped here several minutes, enough for me to start writing
> this email, then it continued boot sequence:
>              board_debug_uart_init():
> board_early_init_f():
> pinctrl_select_state_full('palmbus at 10000000', 'default'):
> pinctrl_select_state_full('uart2 at e00', 'default'):
> pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>
>
> U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 23:21:39 +0100)
>
> CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
> Model: VoCore2
> DRAM:  128 MiB
> pinctrl_select_state_full('palmbus at 10000000', 'default'):
> pinctrl_select_state_full('pinctrl at 60', 'default'):
> pinctrl_select_state_full('pin_state', 'default'):
> pinctrl_select_state_full('uart2 at e00', 'default'):
> pinctrl_select_state_full('uart2_pins', 'default'):
> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
> pinctrl_select_state_full('watchdog at 100', 'default'):
> WDT:   Started with servicing (60s timeout)
> board_early_init_r():
> arch_early_init_r():
> MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
> pinctrl_select_state_full('sd_iot_mode', 'default'):
> pinctrl_select_state_full('clk48m at 0', 'default'):
> pinctrl_select_state_full('mmc at 10130000', 'default'):
> mmc at 10130000: 0
> Loading Environment from FAT... *** Warning - bad CRC, using default
> environment
>
> In:    uart2 at e00
> Out:   uart2 at e00
> Err:   uart2 at e00
> Model: VoCore2
> arch_misc_init():
> =>
>
> Code, beside being targeted to ROM and with a different SYS_TEXT_BASE,
> is identical to what runs fine when started from RAM.
>
> I also tried copying flash to ram:
>
> => cp.b bc030000 80010000 5f3c9
> => go 80010000
> ## Starting application at 0x80010000 ...
> board_debug_uart_init():
> <debug_uart> [04010D08][04010D08]
> DDR Calibration DQS reg = 00008888
> relocate_code Pointer at: 87f5c000
> ***********************
> Watchdog Reset Occurred
> ***********************
>
> ... but this is almost expected because I relocated at another address
> without changing SYS_TEXT_BASE.
>
> A further measurement shows booting u-boot from flash stops for
> almost 5 minutes (4m48s, using a manual stopwatch).
>
> I sincerely have no idea where to bang my head :(
>
> TiA
> Mauro
>
>
> On 1/13/20 3:14 PM, Mauro Condarelli wrote:
>> Hi Stefan,
>> unfortunately it does *not* work.
>>
>> Ram based is ok, but rom  is just as silent as mine:
>>
>> => usb start; sf probe;
>> starting USB...
>> Bus ehci at 101c0000: pinctrl_select_state_full('ehci at 101c0000', 'default'):
>> USB EHCI 1.00
>> scanning bus ehci at 101c0000 for devices... 3 USB Device(s) found
>>        scanning usb for storage devices... 1 Storage Device(s) found
>> pinctrl_select_state_full('spi at b00', 'default'):
>> SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total
>> 16 MiB
>> => load usb 0:1 85000000 u-boot_linkit.bin-rom
>> 455708 bytes read in 22 ms (19.8 MiB/s)
>> => sf update ${fileaddr} 0 ${filesize}
>> device 0 offset 0x0, size 0x6f41c
>> 455708 bytes written, 0 bytes skipped in 12.288s, speed 37966 B/s
>> => reset
>> resetting ...
>> pinctrl_select_state_full('syscon-reboot', 'default'):
>> pinctrl_select_state_full('system-controller at 0', 'default'):
>>
>> I might have done something stupid; Now I need do stop, but I'll
>> test again this evening.
>>
>> Many thanks
>> Mauro
>>
>>
>> On 1/13/20 1:45 PM, Stefan Roese wrote:
>>> On 13.01.20 13:24, Mauro Condarelli wrote:
>>>> On 1/13/20 12:39 PM, Stefan Roese wrote:
>>>>> Hi Mauro,
>>>>>
>>>>> On 13.01.20 11:24, Mauro Condarelli wrote:
>>>>>> On 1/13/20 7:53 AM, Stefan Roese wrote:
>>>>>>> Hi Mauro,
>>>>>>>
>>>>>>> On 11.01.20 20:00, Mauro Condarelli wrote:
>>>>>>>> I managed to find ONE of the reasons why my ROM build didn't run:
>>>>>>>> I forgot to enable `CONFIG_BOARD_EARLY_INIT_F=y`.
>>>>>>> I see. This explains of course, why your board does not boot without
>>>>>>> any "preloader".
>>>>>>>
>>>>>>>> I wanted, nonetheless, be prepared for further mishaps, but
>>>>>>>> I have some other problems (see below).
>>>>>>> Are these issues now fixed? I scanned the discussion about the DEBUG
>>>>>>> UART on the list. Is this working for you now or do you have any
>>>>>>> other
>>>>>>> issues still? Did you successfully boot your new U-Boot from SPI NOR?
>>>>>> Yes and no :(
>>>>>>
>>>>>> I fixed my RAM-based problems, but booting from SPI NOR still
>>>>>> refuses to
>>>>>> utter a single byte.
>>>>>> I do attach  my defconfigs and my board.c for your reading pleasure
>>>>>> (in
>>>>>> case you have troubles getting asleep they should provide a good
>>>>>> cure).
>>>>> Before looking into this in more depth (I actually do have problems
>>>>> sleeping
>>>>> though, so I might take a look at it later ;)), why don't you re-start
>>>>> with
>>>>> your defconfig by using the linkit defconfig file. If you are lucky,
>>>>> the
>>>>> LinkIt binary might even work for the VoCore2. Or what are the
>>>>> differences
>>>>> except the SoC type and WLAN? Its the same DDR2 config and the same
>>>>> UART.
>>>>> So it might get you pretty far - also with ROM based booting.
>>>> I will try it.
>>>> Just to be on the safe side, could You attach a binary (or two: ROM and
>>>> RAM) to
>>>> the mail? If it works it would really give me a start base.
>>> See below.
>>>   
>>>>>> I also added a couple of printouts in drivers/pinctrl/pinctrl-uclass.c
>>>>>> and it
>>>>>> turns out system tries to setup pins for uart2, but fails (maybe
>>>>>> because
>>>>>> pinctrl at 60 is not yet setup correctly?).
>>>>>> This is the output I get from RAM-based u-boot:
>>>>>>
>>>>>> <debug_uart> board_debug_uart_init():
>>>>>> board_early_init_f():
>>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>>> pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
>>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>>>
>>>>>> U-Boot 2020.01-00370-g97a60844bd-dirty (Jan 13 2020 - 01:03:59 +0100)
>>>>>>
>>>>>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (3-Byte SPI Addr)
>>>>>> Model: VoCore2
>>>>>> DRAM:  128 MiB
>>>>>> pinctrl_select_state_full('palmbus at 10000000', 'default'):
>>>>>> pinctrl_select_state_full('pinctrl at 60', 'default'):
>>>>>> pinctrl_select_state_full('pin_state', 'default'):
>>>>>> pinctrl_select_state_full('uart2 at e00', 'default'):
>>>>>> pinctrl_select_state_full('uart2_pins', 'default'):
>>>>>> pinctrl_select_state_full('clkctrl at 0x2c', 'default'):
>>>>>> pinctrl_select_state_full('watchdog at 100', 'default'):
>>>>>> WDT:   Started with servicing (60s timeout)
>>>>>> board_early_init_r():
>>>>>> arch_early_init_r():
>>>>>> MMC:   pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>>> pinctrl_select_state_full('sd_iot_mode', 'default'):
>>>>>> pinctrl_select_state_full('clk48m at 0', 'default'):
>>>>>> pinctrl_select_state_full('mmc at 10130000', 'default'):
>>>>>> mmc at 10130000: 0
>>>>>> Loading Environment from FAT... *** Warning - bad CRC, using default
>>>>>> environment
>>>>>>
>>>>>> In:    uart2 at e00
>>>>>> Out:   uart2 at e00
>>>>>> Err:   uart2 at e00
>>>>>> Model: VoCore2
>>>>>> arch_misc_init():
>>>>>> =>
>>>>> I don't have all these pinctrl issues on LinkIt. I suspect a config
>>>>> issue and / or dts issue. Please compare.
>>>> I will, but I was trying to say something different:
>>>> Apparently the stock software tries to initialize uart2 pinctrl and it
>>>> *seems* to succeed later in initialization.
>>>> This *seems* to indicate explicit pin setup in board_early_init_f() is
>>>> not strictly needed (we would lose only the first few lines even if
>>>> we don't fix the error).
>>>> I had a look to pinctrl-mt7628.c and it seems to do the right thing
>>>> upon call to ('uart2_pins', 'default'), so anything after that *should*
>>>> work even without low-level setup.
>>> You might be correct here. So this explicit pin-muxing for UART2 is
>>> only needed when DEBUG UART is used. This makes the code a bit clearer
>>> and smaller. Lets keep it on our list to remove this, once all this
>>> is resolved.
>>>  
>>>>>> ROM-based u-boot, as said, does not print *anything*, not even
>>>>>> garbage.
>>>>>> I'm beginning to suspect I have some mishap with start address or
>>>>>> similar.
>>>>>> I have absolutely no idea how to debug this without a JTAG probe
>>>>>> (which
>>>>>> I don't have and would be non-trivial to get working; soldering
>>>>>> required).
>>>>> Yes. JTAG requires soldering IIRC.
>>>>>  
>>>>>> The only idea I currently have is to modify my "old" u-boot to do
>>>>>> initialization
>>>>>> and then jump to beginning of "new" u-boot.
>>>>>> If I can make it work in an automatic way I can try removing
>>>>>> initialization steps
>>>>>> till I find the "culprit".
>>>>>> Any better idea would be welcome, of course!
>>>>>>
>>>>>> If I have to resort to that clumsy method, would be enough to put 
>>>>>> "new"
>>>>>> in  SPI NOR (of course at an higher address, e.g.: 30000 as "old" is
>>>>>> only
>>>>>> 183272 bytes long) and jump to the first location?
>>>>>> I assume I will have to relocate CONFIG_SYS_TEXT_BASE=0xbc030000
>>>>> This is the TEXT_BASE for ROM based booting:
>>>>>
>>>>> CONFIG_SYS_TEXT_BASE=0x9c000000
>>>> AFAIK 0x9c000000 and 0xbc000000 map to the same (unmapped) memory
>>>> region, the latter being uncached.
>>>> I proposed 0xbc030000 (but it could be 0x9c030000) as base to leave some
>>>> space for "old" (possibly crippled) u-boot doing initialization and then
>>>> "jumping"
>>>> to "new".
>>> I see. Please give the LinkIt version a try first. This sounds easier
>>> (if it works) and more promising for my taste.
>>>  
>>>>> Please see configs/linkit-smart-7688_defconfig.
>>>>>
>>>>>> Did I forget something?
>>>>> Not sure. I still think using the linkit port as a base for SPI NOR
>>>>> based booting would be a very good test at least. Do you see any
>>>>> problems with this approach (board / SoC differences)?
>>>> No. I see no troubles, for a basic startup.
>>>> MMC would probably need some changes, but that can come later.
>>> Yes. This test is just for basic very low level bootup checking. If
>>> this works, then you can use it as a basis to fine-tune your VoCore2
>>> version from it by changing MMC etc support.
>>>
>>>> As said: I would like to try a prebuilt binary as first thing, so I can
>>>> be sure I have no other problems (like gcc version or other local
>>>> variations); then I would try to reproduce locally the equivalent binary
>>>> with no changes and finally I would try to incorporate my specific
>>>> needs.
>>>> What do You think?
>>> Please find attached the current mainline versions for LinkIt, one
>>> for RAM booting and one for ROM booting (in SPI NOR). I checked both
>>> version and they work fine on my LinkIt 7688 board:
>>>
>>> U-Boot 2020.01-00473-g88366b96ee (Jan 13 2020 - 13:37:15 +0100)
>>>
>>> CPU:   MT7628 Rev 1.2 - Boot from XTAL (4-Byte SPI Addr)
>>> Model: LinkIt-Smart-7688
>>> DRAM:  128 MiB
>>> Loading Environment from SPI Flash... SF: Detected mx25l25635e with
>>> page size 256 Bytes, erase size 64 KiB, total 32 MiB
>>> OK
>>> Net:   eth0: eth at 10110000
>>> Hit any key to stop autoboot:  0
>>> =>
>>>
>>>
>>> Thanks,
>>> Stefan