U-boot may fail to boot because of stale d-cache entry

[Jeungwoo Yoo]

During initialization, U-Boot may rely on clean caches for the code and data in the memory area where its binary is located. Cleaning that cache area could be a responsibility of the loader of U-Boot (e.g., BL2 from ARM Trusted Firmware). Otherwise, it may be hard for U-Boot to avoid using corrupted data. The situation may be different for memory areas outside of the U-Boot binary. That's probably why we can see the call to "invalidate_dcache_all" in the function "dcache_enable" in "u-boot/arch/arm/cpu/armv8/cache_v8.c". However, this function invalidates all cache, and U-Boot initializes its data in the binary before invoking "dcache_enable". This makes it inconsistent with any of the 3 following possibilities to make assumptions about the cache state:

  1.  It is assumed that all data cache may be dirty when U-Boot receives control. In this case, all data cache should be invalidated before any data is initialized. Otherwise, the stale cache entries may overwrite the initialized data in physical memory at any time, even before the data cache is enabled [3]. Besides, data cache hits are also permitted by the architecture when the cache is disabled [2].
  2.  It is assumed that the responsibility of the loader of U-Boot is to clean the cache for the area of the U-Boot binary, but the other memory areas may correspond to dirty caches. This would match the assumptions made by the Linux kernel (see "Caches, MMUs" in "Documentation/arm64/booting.txt" in Linux). In this case, U-Boot should only invalidate caches for the memory areas outside the U-Boot binary, and it should do so before initializing these memory areas.
  3.  It is assumed that the responsibility of the previously running firmware is to clean the complete data cache before passing control to U-Boot. In this case, it is unclear why U-Boot invalidates the data cache itself.

Because the current implementation in U-Boot is not consistent with any of the above assumptions, we come across memory corruption in our use cases. Specifically, U-Boot initializes the Global Data structure (pointed to by the variable 'gd') as zero before using it. The member variable of the Global Data structure, 'gd->arch.tlb_fillptr', is set after page table installation and is used to determine whether the page tables are set up already or not [1]. If the variable 'gd->arch.tlb_fillptr' value is zero, u-boot will set up a page table. Otherwise, u-boot will skip setting up the page table to avoid duplicated page table settings. However, we experience a stale D-cache entry where the variable 'gd->arch.tlb_fillptr' is located. In our use case, this stale entry overwrites the variable in physical memory when U-Boot invalidates the data cache because we run U-Boot in EL1 with stage 2 translation enabled, which promotes the cache invalidation into clean and invalidate [4]. In our virtualized use case, the pointer corruption is guaranteed this way, but in non-virtualized use cases, such corruption is also possible because the cache is not guaranteed to stay dirty [3].


I would like to propose U-Boot to follow the same set of assumptions as in Linux. That is, to assume that the cache is only clean for the memory area where the U-Boot binary is loaded (see 2. above).


[1] Based on latest master branch when writing this report: commit hash: e7fb67df319cec410c20906bbf33936a6f7479b2

(File path: u-boot/arch/arm/cpu/armv8/cache_v8.c)

/* to activate the MMU we need to set up virtual memory */
__weak void mmu_setup(void)
{
    int el;
    /* Set up page tables only once */
    if (!gd->arch.tlb_fillptr)
        setup_all_pgtables();
    el = current_el();
    set_ttbr_tcr_mair(el, gd->arch.tlb_addr, get_tcr(el, NULL, NULL),
              MEMORY_ATTRIBUTES);
    /* enable the mmu */
    set_sctlr(get_sctlr() | CR_M);
}


[2] ARM DDI 0487D.a ID103018; D4.4.5 Behavior of caches at reset

If an implementation permits cache hits when the Cacheability control fields force all memory locations to be treated as Non-cacheable then the cache initialization routine must:

— Provide a mechanism to ensure the correct initialization of the caches.

— Be documented clearly as part of the documentation of the device.

In particular, if an implementation permits cache hits when the Cacheability controls force all memory locations to be treated as Non-cacheable, and the cache contents are not invalidated at reset, the initialization routine must avoid any possibility of running from an uninitialized cache. It is acceptable for an initialization routine to require a fixed instruction sequence to be placed in a restricted range of memory.


[3] ARM DDI 0487D.a ID103018; D4.4.1 General behavior of the caches

An unlocked entry in a cache might not remain in that cache. The architecture does not guarantee that an unlocked cache entry remains in the cache or remains incoherent with the rest of memory. Software must not assume that an unlocked item that remains in the cache remains dirty.


[4] ARM DDI 0487D.a ID103018; D4.4 Cache support; The data cache maintenance instruction (DC)

When executed at EL1, a DC ISW instruction performs a clean and invalidate, meaning it performs the same maintenance as a DC CISW instruction, if all of the following apply:

  *   EL2 is implemented and enabled in the current Security state.
  *   Either:
     *   The value of HCR_EL2.SWIO is 1, forcing a cache clean to perform a clean and invalidate.
     *   The value of HCR_EL2.VM is 1, meaning EL1&0 stage two address translation is enabled.

Best Regards,
Jeungwoo Yoo

Senior Software Engineer

OpenSynergy GmbH
Rotherstr. 20, 10245 Berlin
Telefon: +49 (30) 60 98 54 0 - 37
Fax: +49 (30) 60 98 54 0 - 99

E-MAIL: jeungwoo.yoo at opensynergy.com
www.opensynergy.com

Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B
Geschäftsführer/Managing Director: Regis Adjamah


Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/>