[U-Boot] [Resend RFC PATCH 1/2] armv8: Fix dcache disable function

[Stephen Warren]

On 10/28/2016 12:17 PM, york sun wrote:
> On 10/28/2016 10:57 AM, Stephen Warren wrote:
>> On 10/28/2016 11:38 AM, york sun wrote:
>>> On 10/26/2016 02:02 PM, york.sun at nxp.com wrote:
>>>>
>>>> I came back from my testing and I have more questions than answers.
>>>>
>>>> For _this_ patch, I proposed to flush cache before disabling them,
>>>> noting once the dcache is disabled, the staled data in dirty cache is
>>>> not visible to the core. My argument was if we flush L1/L2, they could
>>>> end up in L3 (I don't know for sure). If I want to skip flushing L3, I
>>>> have to fix it.
>>>>
>>>> During this discussion, I thought I made a mistake by flushing L1/L2 by
>>>> way/set first, then flushing by VA. Actually I didn't. I flushed by VA
>>>> first.
>>>>
>>>> With my today's test, the baseline (working in the sense of booting
>>>> Linux) is
>>>>
>>>> PATCH 1/2 armv8: Fix dcache disable function
>>>> PATCH 2/2 armv8: Fix flush_dcache_all function
>>>>
>>>> With these two patches, I flush the stack up to top of U-Boot by VA,
>>>> followed by flush by set/way. L3 is not flushed. Then d-cache is
>>>> disabled. I know this is not a real "flush all" procedure. With this
>>>> modified procedure, I can continue to boot Linux.
>>>>
>>>> If I revert patch 1, i.e. to disable dcache before flushing, I can see
>>>> the data is not visible from the core (debug with JTAG tool). My hope
>>>> was the staled data should be flushed to main memory if flushed by VA.
>>>> That's not the case. The main memory doesn't have the correct data. So
>>>> my new question is, why flushing by VA doesn't flush the data to main
>>>> memory? Do I need to flush the cache while cache is enabled?
>>>>
>>>
>>> Guys,
>>>
>>> I think I found the root cause of my data loss.
>>>
>>> Current code disables D-cache and MMU before flushing the cache. I think
>>> the problem is turning off MMU. MMU should stay on when we flush
>>> D-cache. We can turn it off after the flushing. Once I make this change,
>>> I can see the correct data in memory after flushing (by VA).
>>>
>>> Do you agree we should leave MMU on during flushing?
>>
>> If you're "flushing" by VA, then I'm not surprised since the MMU is what
>> defines the VA->PA mapping, and perhaps you have some physically tagged
>> caches.
>>
>> However, I believe U-Boot mainline currently "flushes" by set/way, which
>> I wouldn't expect MMU status to influence at all.
>
> Flushing by set/way (only) is what I am trying to change. It would be
> better if we don't have to flush L3. Do you agree?

It depends on whether the L3 is before or after the Point of Coherency. 
If it's before, then it needs to be cleaned. If it's after, then I 
believe it's irrelevant and can be skipped. I don't believe there's any 
other factor that will allow/prevent you from skipping operations on 
your L3; there's no wiggle-room or leeway.

Related, consider the following from the Linux kernel's 
Documentation/arm64/booting.txt:

> - Caches, MMUs
>   The MMU must be off.
>   Instruction cache may be on or off.
>   The address range corresponding to the loaded kernel image must be
>   cleaned to the PoC.

(That only applies to the kernel image specifically, but doing the same 
for the entire cache content seems reasonable, perhaps even required for 
other reasons?)