[PATCH] riscv: Fix build against binutils 2.38

Thu Mar 3 12:06:09 CET 2022

Hi Alex,
On Tue, Mar 01, 2022 at 03:21:56AM +0000, Leo Liang wrote:
> Hi Alex,
> On Mon, Feb 21, 2022 at 05:42:41PM +0100, Alexandre Ghiti wrote:
> > On Sat, Feb 19, 2022 at 9:52 AM Leo Liang <ycliang at andestech.com> wrote:
> > >
> > > Hi Alex,
> > > On Thu, Feb 17, 2022 at 11:28:46AM +0100, Alexandre Ghiti wrote:
> > > > Hi Leo,
> > > >
> > > > On Thu, Feb 17, 2022 at 10:25 AM Leo Liang <ycliang at andestech.com> wrote:
> > > > >
> > > > > Hi Alexandre,
> > > > > On Fri, Jan 28, 2022 at 02:47:13PM +0100, Alexandre Ghiti wrote:
> > > > > > The following description is copied from the equivalent patch for the
> > > > > > Linux Kernel proposed by Aurelien Jarno:
> > > > > >
> > > > > > From version 2.38, binutils default to ISA spec version 20191213. This
> > > > > > means that the csr read/write (csrr*/csrw*) instructions and fence.i
> > > > > > instruction has separated from the `I` extension, become two standalone
> > > > > > extensions: Zicsr and Zifencei. As the kernel uses those instruction,
> > > > > > this causes the following build failure:
> > > > > >
> > > > > > arch/riscv/cpu/mtrap.S: Assembler messages:
> > > > > > arch/riscv/cpu/mtrap.S:65: Error: unrecognized opcode `csrr a0,scause'
> > > > > > arch/riscv/cpu/mtrap.S:66: Error: unrecognized opcode `csrr a1,sepc'
> > > > > > arch/riscv/cpu/mtrap.S:67: Error: unrecognized opcode `csrr a2,stval'
> > > > > > arch/riscv/cpu/mtrap.S:70: Error: unrecognized opcode `csrw sepc,a0'
> > > > > >
> > > > > > Signed-off-by: Alexandre Ghiti <alexandre.ghiti at canonical.com>
> > > > > > ---
> > > > > >  arch/riscv/Makefile | 11 ++++++++++-
> > > > > >  1 file changed, 10 insertions(+), 1 deletion(-)
> > > > >
> > > > > This patch seems to fail CI somehow.
> > > > > (https://source.denx.de/u-boot/custodians/u-boot-riscv/-/pipelines/11004)
> > > > >
> > > > > Could you take a look at it ?
> > > >
> > > > I have just tried on master (commit ab8903a24db1) and it failed for
> > > > the same reason, so this is not related to this patch. Nevertheless,
> > > > I'll try to bisect the problem :)
> > > >
> > > > Thanks,
> > > >
> > > > Alex
> > > >
> > >
> > > Thanks for putting the effort into it!
> > >
> > > AFAIK, this patch does nothing related to the error message "undefined reference to `__ashldi3'" from the failed CI.
> > >
> > > Nonetheless, I have tried a few times myself,
> > > and found that CI could pass with ab8903a24db1 but cannot pass with this patch on my side.
> > > https://source.denx.de/u-boot/custodians/u-boot-riscv/-/commits/staging
> > > https://source.denx.de/u-boot/custodians/u-boot-riscv/-/pipelines
> > >
> > 
> > To me it is an issue with the toolchain: libgcc is missing those
> > symbols. If I use an Ubuntu toolchain, it fails no matter which commit
> > I am on (I tested as far as v2021.10). But if I use a toolchain from
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/,
> > it works fine.
> > 
> > What I don't understand is how you manage to have different build
> > results with the same docker image: can you confirm that you use the
> > same toolchains in the following builds?
> >
> > https://source.denx.de/u-boot/custodians/u-boot-riscv/-/jobs/393701
> > https://source.denx.de/u-boot/custodians/u-boot-riscv/-/jobs/393783
> > 
> 
> Sorry for the late reply.
> I have checked the toolchain version of these two builds,
> they are using the same toolchain[1] from Tom's docker image on docker hub[2].
> 
> Also the fail is reproducible using this docker image with the following commands:
> 
> leo at host sudo docker run -it --name leo-test trini/u-boot-gitlab-ci-runner:focal-20220113-03Feb2022 /bin/bash
> uboot at 356268d27bf0:~$ git clone https://source.denx.de/u-boot/custodians/u-boot-riscv.git && cd u-boot-riscv
> uboot at 356268d27bf0:~/u-boot-riscv$ git checkout staging
> uboot at 356268d27bf0:~/u-boot-riscv$ export PATH=/opt/gcc-11.1.0-nolibc/riscv64-linux/bin:$PATH
> uboot at 356268d27bf0:~/u-boot-riscv$ export CROSS_COMPILE=riscv64-linux-
> uboot at 356268d27bf0:~/u-boot-riscv$ make qemu-riscv32_spl_defconfig
> uboot at 356268d27bf0:~/u-boot-riscv$ make -j`nproc` V=1
> 
> ...
> 
> riscv64-linux-ld.bfd  -m elf32lriscv --gc-sections -static -pie  -Bstatic  --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive  arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o  arch/riscv/lib/built-in.o  board/emulation/common/built-in.o  board/emulation/qemu-riscv/built-in.o  boot/built-in.o  cmd/built-in.o  common/built-in.o  disk/built-in.o  drivers/built-in.o  drivers/usb/cdns3/built-in.o  drivers/usb/common/built-in.o  drivers/usb/dwc3/built-in.o  drivers/usb/emul/built-in.o  drivers/usb/eth/built-in.o  drivers/usb/host/built-in.o  drivers/usb/mtu3/built-in.o  drivers/usb/musb-new/built-in.o  drivers/usb/musb/built-in.o  drivers/usb/phy/built-in.o  drivers/usb/ulpi/built-in.o  env/built-in.o  fs/built-in.o  lib/built-in.o  net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map;  true
> riscv64-linux-ld.bfd: drivers/virtio/virtio-uclass.o: in function `virtio_uclass_child_pre_probe':
> /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:339: undefined reference to `__lshrdi3'
> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:311: undefined reference to `__ashldi3'
> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/drivers/virtio/virtio-uclass.c:310: undefined reference to `__ashldi3'
> riscv64-linux-ld.bfd: drivers/nvme/nvme.o: in function `nvme_blk_rw':
> /home/uboot/u-boot-riscv/drivers/nvme/nvme.c:776: undefined reference to `__lshrdi3'
> riscv64-linux-ld.bfd: fs/ext4/ext4_common.o: in function `ext4fs_get_extent_block':
> /home/uboot/u-boot-riscv/fs/ext4/ext4_common.c:1560: undefined reference to `__ashldi3'
> riscv64-linux-ld.bfd: fs/ext4/dev.o: in function `ext4fs_set_blk_dev':
> /home/uboot/u-boot-riscv/fs/ext4/dev.c:46: undefined reference to `__lshrdi3'
> riscv64-linux-ld.bfd: lib/display_options.o: in function `print_size':
> /home/uboot/u-boot-riscv/lib/display_options.c:103: undefined reference to `__lshrdi3'
> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:102: undefined reference to `__ashldi3'
> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:95: undefined reference to `__ashldi3'
> riscv64-linux-ld.bfd: /home/uboot/u-boot-riscv/lib/display_options.c:124: undefined reference to `__lshrdi3'
> riscv64-linux-ld.bfd: lib/vsprintf.o: in function `number':
> /home/uboot/u-boot-riscv/lib/vsprintf.c:213: undefined reference to `__lshrdi3'
> make: *** [Makefile:1800: u-boot] Error 1
> 
> Furthermore, Using the same container and perform the identical build commands listed above on master branch produce no error.
> 
> What seems a bit odd is that when testing qemu-riscv32_spl,
> buildman still uses 64 bit toolchain for the build.
> I am not sure what the effect of "riscv64-linux-ld.bfd -m elf32lriscv" is.
> 
> Do you have any thoughts ?
> 
> [1] https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/11.1.0/x86_64-gcc-11.1.0-nolibc-riscv64-linux.tar.xz
> [2] https://hub.docker.com/layers/trini/u-boot-gitlab-ci-runner/focal-20220113-03Feb2022/images/sha256-b1cddec69526015c550ac8e528114c976ccd1a0e75676328c4f81b834e06b2b3?context=explore
> 
> Best regards,
> Leo
> 

I re-ran again with buildman's verbose option (./tools/buildman/buildman -o /tmp/qemu-riscv32_spl/ -w -E -W -e -V -v --board qemu-riscv32_spl) 
and found that the failed command is as below:

riscv64-linux-ld.bfd  -m elf32lriscv --gc-sections -static -pie  -Bstatic  --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive  arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o  arch/riscv/lib/built-in.o  board/emulation/common/built-in.o  board/emulation/qemu-riscv/built-in.o  boot/built-in.o  cmd/built-in.o  common/built-in.o  disk/built-in.o  drivers/built-in.o  drivers/usb/cdns3/built-in.o  drivers/usb/common/built-in.o  drivers/usb/dwc3/built-in.o  drivers/usb/emul/built-in.o  drivers/usb/eth/built-in.o  drivers/usb/host/built-in.o  drivers/usb/mtu3/built-in.o  drivers/usb/musb-new/built-in.o  drivers/usb/musb/built-in.o  drivers/usb/phy/built-in.o  drivers/usb/ulpi/built-in.o  env/built-in.o  fs/built-in.o  lib/built-in.o  net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map;  true
^^^^^^^^^^^^^^^^^^^^^^

The reason for this to fail is due to the wrong search path for libgcc.
The above command uses "-L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0", 
where it should be "-L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0/lib32/ilp32[d]".

This search path is generated from Makefile:865 `PLATFORM_LIBGCC := -L $(shell dirname `$(CC) $(c_flags) -print-libgcc-file-name`) -lgcc`.

I'll try to find out how buildman executes the make process.

```
$ ./tools/buildman/buildman -o /tmp/qemu-riscv32_spl/ -w -E -W -e -V -v --board qemu-riscv32_spl
$ cat /tmp/qemu-riscv32_spl/log | grep u-boot.map
riscv64-linux-ld.bfd  -m elf32lriscv --gc-sections -static -pie  -Bstatic  --no-dynamic-linker -z notext --build-id=none -Ttext 0x81200000 -o u-boot -T u-boot.lds arch/riscv/cpu/start.o --whole-archive  arch/riscv/cpu/built-in.o arch/riscv/cpu/generic/built-in.o  arch/riscv/lib/built-in.o  board/emulation/common/built-in.o  board/emulation/qemu-riscv/built-in.o  boot/built-in.o  cmd/built-in.o  common/built-in.o  disk/built-in.o  drivers/built-in.o  drivers/usb/cdns3/built-in.o  drivers/usb/common/built-in.o  drivers/usb/dwc3/built-in.o  drivers/usb/emul/built-in.o  drivers/usb/eth/built-in.o  drivers/usb/host/built-in.o  drivers/usb/mtu3/built-in.o  drivers/usb/musb-new/built-in.o  drivers/usb/musb/built-in.o  drivers/usb/phy/built-in.o  drivers/usb/ulpi/built-in.o  env/built-in.o  fs/built-in.o  lib/built-in.o  net/built-in.o --no-whole-archive -L /opt/gcc-11.1.0-nolibc/riscv64-linux/bin/../lib/gcc/riscv64-linux/11.1.0 -lgcc -Map u-boot.map;  true
```

Best regards,
Leo

> 
> > The only difference I see here is the gitlab-runner version (line 1).
> > 
> > Thanks,
> > 
> > Alex
> > 
> > > Best regards,
> > > Leo
> > >
> > > > >
> > > > > Best regards,
> > > > > Leo