[PATCH] riscv: Fix build against binutils 2.38
Heinrich Schuchardt
heinrich.schuchardt at canonical.com
Sun Oct 2 01:24:08 CEST 2022
On 3/9/22 07:31, Leo Liang wrote:
> Hi Alex,
> On Thu, Mar 03, 2022 at 11:06:18AM +0000, Leo Liang wrote:
>> Hi Alex,
>> On Tue, Mar 01, 2022 at 03:21:56AM +0000, Leo Liang wrote:
>>> Hi Alex,
>>> On Mon, Feb 21, 2022 at 05:42:41PM +0100, Alexandre Ghiti wrote:
>>>> On Sat, Feb 19, 2022 at 9:52 AM Leo Liang <ycliang at andestech.com> wrote:
>>>>>
>>>>> Hi Alex,
>>>>> On Thu, Feb 17, 2022 at 11:28:46AM +0100, Alexandre Ghiti wrote:
>>>>>> Hi Leo,
>>>>>>
>>>>>> On Thu, Feb 17, 2022 at 10:25 AM Leo Liang <ycliang at andestech.com> wrote:
>>>>>>>
>>>>>>> Hi Alexandre,
>>>>>>> On Fri, Jan 28, 2022 at 02:47:13PM +0100, Alexandre Ghiti wrote:
>>>>>>>> The following description is copied from the equivalent patch for the
>>>>>>>> Linux Kernel proposed by Aurelien Jarno:
>>>>>>>>
>>>>>>>> From version 2.38, binutils default to ISA spec version 20191213. This
>>>>>>>> means that the csr read/write (csrr*/csrw*) instructions and fence.i
>>>>>>>> instruction has separated from the `I` extension, become two standalone
>>>>>>>> extensions: Zicsr and Zifencei. As the kernel uses those instruction,
>>>>>>>> this causes the following build failure:
>>>>>>>>
>>>>>>>> arch/riscv/cpu/mtrap.S: Assembler messages:
>>>>>>>> arch/riscv/cpu/mtrap.S:65: Error: unrecognized opcode `csrr a0,scause'
>>>>>>>> arch/riscv/cpu/mtrap.S:66: Error: unrecognized opcode `csrr a1,sepc'
>>>>>>>> arch/riscv/cpu/mtrap.S:67: Error: unrecognized opcode `csrr a2,stval'
>>>>>>>> arch/riscv/cpu/mtrap.S:70: Error: unrecognized opcode `csrw sepc,a0'
>>>>>>>>
>>>>>>>> Signed-off-by: Alexandre Ghiti <alexandre.ghiti at canonical.com>
>>>>>>>> ---
>>>>>>>> arch/riscv/Makefile | 11 ++++++++++-
>>>>>>>> 1 file changed, 10 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> This patch seems to fail CI somehow.
>>>>>>> (https://source.denx.de/u-boot/custodians/u-boot-riscv/-/pipelines/11004)
>>>>>>>
>>>>>>> Could you take a look at it ?
>>>>>>
>>>>>> I have just tried on master (commit ab8903a24db1) and it failed for
>>>>>> the same reason, so this is not related to this patch. Nevertheless,
>>>>>> I'll try to bisect the problem :)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>
>>>>> Thanks for putting the effort into it!
>>>>>
>>>>> AFAIK, this patch does nothing related to the error message "undefined reference to `__ashldi3'" from the failed CI.
>>>>>
>>>>> Nonetheless, I have tried a few times myself,
>>>>> and found that CI could pass with ab8903a24db1 but cannot pass with this patch on my side.
>>>>> https://source.denx.de/u-boot/custodians/u-boot-riscv/-/commits/staging
>>>>> https://source.denx.de/u-boot/custodians/u-boot-riscv/-/pipelines
>>>>>
>>>>
>>>> To me it is an issue with the toolchain: libgcc is missing those
>>>> symbols. If I use an Ubuntu toolchain, it fails no matter which commit
>>>> I am on (I tested as far as v2021.10). But if I use a toolchain from
>>>> https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/,
>>>> it works fine.
>>>>
>>>> What I don't understand is how you manage to have different build
>>>> results with the same docker image: can you confirm that you use the
>>>> same toolchains in the following builds?
>>>>
>>>> https://source.denx.de/u-boot/custodians/u-boot-riscv/-/jobs/393701
>>>> https://source.denx.de/u-boot/custodians/u-boot-riscv/-/jobs/393783
>>>>
>>>
>>> Sorry for the late reply.
>>> I have checked the toolchain version of these two builds,
>>> they are using the same toolchain[1] from Tom's docker image on docker hub[2].
>>>
>>> Also the fail is reproducible using this docker image with the following commands:
>>>
>>> leo at host sudo docker run -it --name leo-test trini/u-boot-gitlab-ci-runner:focal-20220113-03Feb2022 /bin/bash
>>> uboot at 356268d27bf0:~$ git clone https://source.denx.de/u-boot/custodians/u-boot-riscv.git && cd u-boot-riscv
>>> uboot at 356268d27bf0:~/u-boot-riscv$ git checkout staging
>>> uboot at 356268d27bf0:~/u-boot-riscv$ export PATH=/opt/gcc-11.1.0-nolibc/riscv64-linux/bin:$PATH
>>> uboot at 356268d27bf0:~/u-boot-riscv$ export CROSS_COMPILE=riscv64-linux-
>>> uboot at 356268d27bf0:~/u-boot-riscv$ make qemu-riscv32_spl_defconfig
>>> uboot at 356268d27bf0:~/u-boot-riscv$ make -j`nproc` V=1
>>>
>>> ...
>>>
>>> riscv64-linux-ld.bfd -m elf32lriscv --gc-sections -static -pie -Bstatic --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o arch/riscv/lib/built-in.o board/emulation/common/built-in.o board/emulation/qemu-riscv/built-in.o boot/built-in.o cmd/built-in.o common/built-in.o disk/built-in.o drivers/built-in.o drivers/usb/cdns3/built-in.o drivers/usb/common/built-in.o drivers/usb/dwc3/built-in.o drivers/usb/emul/built-in.o drivers/usb/eth/built-in.o drivers/usb/host/built-in.o drivers/usb/mtu3/built-in.o drivers/usb/musb-new/built-in.o drivers/usb/musb/built-in.o drivers/usb/phy/built-in.o drivers/usb/ulpi/built-in.o env/built-in.o fs/built-in.o lib/built-in.o net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map; true
>>> riscv64-linux-ld.bfd: drivers/virtio/virtio-uclass.o: in function `virtio_uclass_child_pre_probe':
>>> /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:339: undefined reference to `__lshrdi3'
>>> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:311: undefined reference to `__ashldi3'
>>> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:310: undefined reference to `__ashldi3'
>>> riscv64-linux-ld.bfd: drivers/nvme/nvme.o: in function `nvme_blk_rw':
>>> /home/uboot/u-boot-riscv/drivers/nvme/nvme.c:776: undefined reference to `__lshrdi3'
>>> riscv64-linux-ld.bfd: fs/ext4/ext4_common.o: in function `ext4fs_get_extent_block':
>>> /home/uboot/u-boot-riscv/fs/ext4/ext4_common.c:1560: undefined reference to `__ashldi3'
>>> riscv64-linux-ld.bfd: fs/ext4/dev.o: in function `ext4fs_set_blk_dev':
>>> /home/uboot/u-boot-riscv/fs/ext4/dev.c:46: undefined reference to `__lshrdi3'
>>> riscv64-linux-ld.bfd: lib/display_options.o: in function `print_size':
>>> /home/uboot/u-boot-riscv/lib/display_options.c:103: undefined reference to `__lshrdi3'
>>> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:102: undefined reference to `__ashldi3'
>>> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:95: undefined reference to `__ashldi3'
>>> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:124: undefined reference to `__lshrdi3'
>>> riscv64-linux-ld.bfd: lib/vsprintf.o: in function `number':
>>> /home/uboot/u-boot-riscv/lib/vsprintf.c:213: undefined reference to `__lshrdi3'
>>> make: *** [Makefile:1800: u-boot] Error 1
>>>
>>> Furthermore, Using the same container and perform the identical build commands listed above on master branch produce no error.
>>>
>>> What seems a bit odd is that when testing qemu-riscv32_spl,
>>> buildman still uses 64 bit toolchain for the build.
>>> I am not sure what the effect of "riscv64-linux-ld.bfd -m elf32lriscv" is.
>>>
>>> Do you have any thoughts ?
>>>
>>> [1] https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/11.1.0/x86_64-gcc-11.1.0-nolibc-riscv64-linux.tar.xz
>>> [2] https://hub.docker.com/layers/trini/u-boot-gitlab-ci-runner/focal-20220113-03Feb2022/images/sha256-b1cddec69526015c550ac8e528114c976ccd1a0e75676328c4f81b834e06b2b3?context=explore
>>>
>>> Best regards,
>>> Leo
>>>
>>
>> I re-ran again with buildman's verbose option (./tools/buildman/buildman -o /tmp/qemu-riscv32_spl/ -w -E -W -e -V -v --board qemu-riscv32_spl)
>> and found that the failed command is as below:
>>
>> riscv64-linux-ld.bfd -m elf32lriscv --gc-sections -static -pie -Bstatic --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o arch/riscv/lib/built-in.o board/emulation/common/built-in.o board/emulation/qemu-riscv/built-in.o boot/built-in.o cmd/built-in.o common/built-in.o disk/built-in.o drivers/built-in.o drivers/usb/cdns3/built-in.o drivers/usb/common/built-in.o drivers/usb/dwc3/built-in.o drivers/usb/emul/built-in.o drivers/usb/eth/built-in.o drivers/usb/host/built-in.o drivers/usb/mtu3/built-in.o drivers/usb/musb-new/built-in.o drivers/usb/musb/built-in.o drivers/usb/phy/built-in.o drivers/usb/ulpi/built-in.o env/built-in.o fs/built-in.o lib/built-in.o net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map; true
>> ^^^^^^^^^^^^^^^^^^^^^^
>>
>> The reason for this to fail is due to the wrong search path for libgcc.
>> The above command uses "-L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0",
>> where it should be "-L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0/lib32/ilp32[d]".
>>
>> This search path is generated from Makefile:865 `PLATFORM_LIBGCC := -L $(shell dirname `$(CC) $(c_flags) -print-libgcc-file-name`) -lgcc`.
>>
>> I'll try to find out how buildman executes the make process.
>>
>> ```
>> $ ./tools/buildman/buildman -o /tmp/qemu-riscv32_spl/ -w -E -W -e -V -v --board qemu-riscv32_spl
>> $ cat /tmp/qemu-riscv32_spl/log | grep u-boot.map
>> riscv64-linux-ld.bfd -m elf32lriscv --gc-sections -static -pie -Bstatic --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o arch/riscv/lib/built-in.o board/emulation/common/built-in.o board/emulation/qemu-riscv/built-in.o boot/built-in.o cmd/built-in.o common/built-in.o disk/built-in.o drivers/built-in.o drivers/usb/cdns3/built-in.o drivers/usb/common/built-in.o drivers/usb/dwc3/built-in.o drivers/usb/emul/built-in.o drivers/usb/eth/built-in.o drivers/usb/host/built-in.o drivers/usb/mtu3/built-in.o drivers/usb/musb-new/built-in.o drivers/usb/musb/built-in.o drivers/usb/phy/built-in.o drivers/usb/ulpi/built-in.o env/built-in.o fs/built-in.o lib/built-in.o net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map; true
>> ```
>>
>
> This patch looks valid, it's the CI process which causes the error to occur.
>
> The command "dirname $(riscv64-linux-gcc -march=rv32imac -mabi=ilp32 -print-libgcc-filename)" gives
> different result from "dirname $(riscv64-linux-gcc -march=rv32imac_zicsr_zifencei -mabi=ilp32 -print-libgcc-filename)".
>
> The former gives "/opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0/lib32/ilp32"
> while the latter gives "/opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0".
>
> The symbol __ashldi3 and __lshrdi3 only exist in libgcc.a under /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0/lib32/ilp32,
> so linker search path have to be set to it to link successfully.
>
> The multilib setting does not cover zifencei and zicsr, so we get a default multilib path from -print-libgcc-file-name, and thus run into error like this.
>
> We should be able to fix this by using 32 bit toolchain in the CI process for 32 bit platform/configuration.
>
> Any thoughts or comments ?
In our Docker container command
tools/buildman/buildman -o build -w -E -W -e --board qemu-riscv32_spl
leads to build/toolchain with content
gcc /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc
path /opt/gcc-11.1.0-nolibc/riscv64-linux/bin
cross riscv64-linux-
arch riscv64
When compiling qemu-riscv32_defconfig with Alexandre's patch and
export
CROSS_COMPILE=/opt/gcc-11.1.0-nolibc/riscv64-linux/bin/riscv64-linux-
I see undefined reference to `__ashldi3'.
With
export
CROSS_COMPILE=/opt/gcc-11.1.0-nolibc/riscv32-linux/bin/riscv32-linux-
it works correctly.
So first thing to do is to fix the Docker container to install the
riscv32 toolchain.
Next we have to tell binman to use (installing the tool chain is not
enough).
Best regards
Heinrich
More information about the U-Boot
mailing list