[U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts

[Anup Patel]

On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer
<lukas.auer at aisec.fraunhofer.de> wrote:
>
> Harts on RISC-V boot independently and U-Boot is responsible for
> managing them. Functions are called on other harts with
> smp_call_function(), which sends inter-processor interrupts (IPIs) to
> all other harts. Functions are specified with their address and two
> function arguments (argument 2 and 3). The first function argument is
> always the hart ID of the hart calling the function. On the other harts,
> the IPI interrupt handler handle_ipi() must be called on software
> interrupts to handle the request and call the specified function.
>
> Functions are stored in the ipi_data data structure. Every hart has its
> own data structure in global data. While this is not required at the
> moment (all harts are expected to boot Linux), this does allow future
> expansion, where other harts may be used for monitoring or other tasks.
>
> Signed-off-by: Lukas Auer <lukas.auer at aisec.fraunhofer.de>
> ---
>
>  arch/riscv/Kconfig                   |  19 +++++
>  arch/riscv/include/asm/global_data.h |   5 ++
>  arch/riscv/include/asm/smp.h         |  53 +++++++++++++
>  arch/riscv/lib/Makefile              |   1 +
>  arch/riscv/lib/smp.c                 | 110 +++++++++++++++++++++++++++
>  5 files changed, 188 insertions(+)
>  create mode 100644 arch/riscv/include/asm/smp.h
>  create mode 100644 arch/riscv/lib/smp.c
>
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index c45e4d73a8..c0842178dd 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -116,4 +116,23 @@ config RISCV_RDTIME
>  config SYS_MALLOC_F_LEN
>         default 0x1000
>
> +config SMP
> +       bool "Symmetric Multi-Processing"
> +       help
> +         This enables support for systems with more than one CPU. If
> +         you say N here, U-Boot will run on single and multiprocessor
> +         machines, but will use only one CPU of a multiprocessor
> +         machine. If you say Y here, U-Boot will run on many, but not
> +         all, single processor machines.
> +
> +config NR_CPUS
> +       int "Maximum number of CPUs (2-32)"
> +       range 2 32
> +       depends on SMP
> +       default "8"
> +       help
> +         On multiprocessor machines, U-Boot sets up a stack for each CPU.
> +         Stack memory is pre-allocated. U-Boot must therefore know the
> +         maximum number of CPUs that may be present.
> +
>  endmenu
> diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h
> index a3a342c6e1..23a5f35af5 100644
> --- a/arch/riscv/include/asm/global_data.h
> +++ b/arch/riscv/include/asm/global_data.h
> @@ -10,12 +10,17 @@
>  #ifndef        __ASM_GBL_DATA_H
>  #define __ASM_GBL_DATA_H
>
> +#include <asm/smp.h>
> +
>  /* Architecture-specific global data */
>  struct arch_global_data {
>         long boot_hart;         /* boot hart id */
>  #ifdef CONFIG_SIFIVE_CLINT
>         void __iomem *clint;    /* clint base address */
>  #endif
> +#ifdef CONFIG_SMP
> +       struct ipi_data ipi[CONFIG_NR_CPUS];

The data passed by "main HART" via global data to other HARTs
is not visible reliably and randomly few HARTs fail to enter Linux
kernel on my board. I am suspecting per-HART "ipi" data in global
data not being cache-line aligned as the cause of behavior but there
could be other issue too.

I have a hack which works reliable for me. As-per this hack, we add
"mdelay(10)" just before calling riscv_send_ipi() in send_ipi_many().
This hack helped me conclude that there is some sync issue in per-HART
"ipi" data.

The above issue is not seen on QEMU so we are fine there.

I would suggest to make per-HART "ipi" data cache-line aligned
(just like Linux kernel).

Regards,
Anup