[V3,1/1] riscv: Increase the upper bound of NR_CPUS to 4095

[Heinrich Schuchardt]

On 2/13/23 09:43, Leo Yu-Chi Liang wrote:
> From: Xiang W <wxjstz at 126.com>
> 
> The maximum hart number is 32 and is determined by
> the type of gd->arch.available_harts. However, various
> RISC-V specifications allow the hart number to be greater
> than 32. We can eliminate this limitation through bitmaps.
> 
> The patch modifies the upper bound of the hart number to 4095,
> which is also the maximum hart number RISC-V Advanced Core Local
> Interruptor Specification gives.
> 
> Some defconfigs, (e.g. qemu-riscv32_smode_defconfig,
> qemu-riscv64_smode_defconfig, openpiton_riscv64_defconfig, ...)
> do not define CONFIG_NR_CPUS, and may result in compile error,
> so define CONFIG_NR_CPUS to be 1 if CONFIG_NR_CPUS is not defined.
> 
> Tested on sifive unmatched.
> 
> Signed-off-by: Xiang W <wxjstz at 126.com>
> Signed-off-by: Leo Yu-Chi Liang <ycliang at andestech.com>
> ---
> Changes v2 -> v3:
> - Fix the calculation of the GD_AVAILABLE_HARTS start address
> - Fix compilation error for defconfigs that do not define CONFIG_NR_CPUS
> ---
>   arch/riscv/Kconfig                   |  4 ++--
>   arch/riscv/cpu/start.S               | 20 +++++++++++++++-----
>   arch/riscv/include/asm/global_data.h |  8 +++++++-
>   arch/riscv/lib/smp.c                 |  2 +-
>   4 files changed, 25 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index ebc4bef220..063734cbb9 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -232,8 +232,8 @@ config SPL_SMP
>   	  all, single processor machines.
>   
>   config NR_CPUS
> -	int "Maximum number of CPUs (2-32)"
> -	range 2 32
> +	int "Maximum number of CPUs (2-4095)"
> +	range 2 4095

Looking at patch
776e8aca0bad ("riscv: alloc space exhausted")
SYS_MALLOC_F_LEN needs to be increased if the number of CPUs is increased.

You could add an architecture specific rule to config.mk to adjust 
SYS_MALLOC_F_LEN.

>   	depends on SMP || SPL_SMP
>   	default 8
>   	help
> diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
> index 4687bca3c9..ae19c2b43d 100644
> --- a/arch/riscv/cpu/start.S
> +++ b/arch/riscv/cpu/start.S
> @@ -172,11 +172,21 @@ wait_for_gd_init:
>   	bnez	t1, 1b
>   
>   	/* register available harts in the available_harts mask */
> -	li	t1, 1
> -	sll	t1, t1, tp
> -	LREG	t2, GD_AVAILABLE_HARTS(gp)
> -	or	t2, t2, t1
> -	SREG	t2, GD_AVAILABLE_HARTS(gp)
> +	li	t1, GD_AVAILABLE_HARTS
> +	add	t1, t1, gp
> +#if defined(CONFIG_ARCH_RV64I)
> +	srli	t2, tp, 6
> +	slli	t2, t2, 3
> +#elif defined(CONFIG_ARCH_RV32I)
> +	srli	t2, tp, 5
> +	slli	t2, t2, 2
> +#endif

If QEMU provides > XLEN harts, this will result in a buffer overrun. You 
must check against the size of available_harts.

> +	add	t1, t1, t2
> +	LREG	t2, 0(t1)
> +	li	t3, 1
> +	sll	t3, t3, tp
> +	or	t2, t2, t3
> +	SREG	t2, 0(t1)
>   
>   	amoswap.w.rl zero, zero, 0(t0)
>   #endif
> diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h
> index 6fdc86dd8b..7e37e90db0 100644
> --- a/arch/riscv/include/asm/global_data.h
> +++ b/arch/riscv/include/asm/global_data.h
> @@ -10,9 +10,15 @@
>   #ifndef	__ASM_GBL_DATA_H
>   #define __ASM_GBL_DATA_H
>   
> +#include <config.h>
>   #include <asm/smp.h>
>   #include <asm/u-boot.h>
>   #include <compiler.h>
> +#include <linux/bitops.h>
> +
> +#ifndef CONFIG_NR_CPUS
> +#define CONFIG_NR_CPUS 1

CONFIG_ constants should not be defined in C code. Please, move this to 
Kconfig.

The chosen value will create problems for QEMU. The fallback value 
should be XLEN (BITS_PER_LONG) at least.

Best regards

Heinrich

> +#endif
>   
>   /* Architecture-specific global data */
>   struct arch_global_data {
> @@ -29,7 +35,7 @@ struct arch_global_data {
>   #endif
>   #if !CONFIG_IS_ENABLED(XIP)
>   #ifdef CONFIG_AVAILABLE_HARTS
> -	ulong available_harts;
> +	ulong available_harts[BITS_TO_LONGS(CONFIG_NR_CPUS)];
>   #endif
>   #endif
>   };
> diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c
> index 4f073a016f..511e3ed98d 100644
> --- a/arch/riscv/lib/smp.c
> +++ b/arch/riscv/lib/smp.c
> @@ -48,7 +48,7 @@ static int send_ipi_many(struct ipi_data *ipi, int wait)
>   #if !CONFIG_IS_ENABLED(XIP)
>   #ifdef CONFIG_AVAILABLE_HARTS
>   		/* skip if hart is not available */
> -		if (!(gd->arch.available_harts & (1 << reg)))
> +		if (!test_bit(reg, gd->arch.available_harts))
>   			continue;
>   #endif
>   #endif