[U-Boot] [Resend RFC PATCH 1/2] armv8: Fix dcache disable function

[Mark Rutland]

On Fri, Oct 21, 2016 at 07:31:52PM +0000, york sun wrote:
> On 10/20/2016 01:34 PM, Stephen Warren wrote:
> > On 10/19/2016 11:06 PM, york sun wrote:
> >> I understand the data in dirty cache is not lost when the dcache is
> >> disabled. It is just not accessible. In my case, after flushing L1/L2 by
> >> way/set, the data is probably in L3 cache. Without flushing L3, I have
> >> stalled data (including the stack) in main memory.
> >
> > I assume "stale" not "stalled".
> >
> > Yes, I agree: If you have only flushed L1/L2 by set/way (or attempted to
> > do everything, but the L3 cache doesn't implement by set/way
> > operations), then L3 can indeed still contain dirty data, and hence main
> > memory can be stale.
> >
> >> My previous test was trying to prove I can skip flushing L3 if I flush
> >> the necessary VA.
> >
> > It depends whether your L3 cache is before/after the Level of
> > Unification and/or the Level of Coherency for your HW, and whether you
> > flush by VA to the PoU or PoC.
> >
> > Looking at your "[PATCH 2/2] armv8: Fix flush_dcache_all function", it
> > uses __asm_flush_dcache_range() which cleans and invalidates to the
> > point of coherency (it invokes the dc civac instruction).
> >
> >  > Now I when recall it carefully, I think I made a
> >> mistake by flushing by VA _after_ flushing the cache by way/set. I might
> >> have a positive result if I flushed the cache by VA first. I will repeat
> >> this test when I get back to prove this theory.
> >
> > I'll assume you L3 cache is before the Point of Coherency. If so, then
> > performing a clean/invalidate by VA to the PoC will affect all of L1,
> > L2, and L3 caches. As such, you need only perform the c/i by VA, and you
> > can entirely skip the c/i by set/way; I believe it would be redundant,
> > assuming that the by VA operations cover all relevant VAs.
> 
> I believe the PoC and PoU is before L3 for me. 

If you are using CCN, then the PoC is beyond the L3.

Were the PoC before the L3, there would be no requirement to perform
maintenance on the L3. The PoC is the point at which *all* accesses
(cacheable or otherwise) see the same data.

Per the ARM ARM (for ARMv8-A), maintenance to the PoC *must* affect
system caches (including CCN).

> I can clean/invalidate by VA, it may not cover all the cache lines. So
> by set/way is still needed.

The problem is figuring out which VA ranges require maintenance.

Do we not have an idea of the set of memory banks present in the SoC?
Like the memblock array in Linux?

> > b) How can we implement the by VA code in a way that doesn't touch DRAM?
> >
> > Implementing by-set/way is fairly constrained in that all the
> > configuration data is in a few registers, and the algorithm just .
> > requires a few nested loops.
> >
> > A generic by VA implementation seems like it would require walking
> > U-Boot's struct mm_region *mem_map data structure (or the CPU's
> > translation tables) in RAM. Perhaps that's OK since it's read-only...

So long as you clean the structure by VA to the PoC first, you can
safely access it with non-cacheable accesses.

> I agree in general about your points, but it may not be always practical 
> to flush all by VA. U-Boot may map huge amount of memory. Walking 
> through MMU table and flush all will be too much. 

I would recommend that you benchmark that; from my own experiments, so
long as you only perform maintenance on the portions of the PA space you
care about (and amortize barriers), this can take surprisingly little
time.

> Without flushing all memory, we really cannot say we flushed all
> dcache. On the other side, for U-Boot itself to operate, we don't
> really have to flush all. I guess the key is if we need to flush all.
> For Linux to boot, we don't.

This depends; so long as you've *only* used Normal, Inner-Shareable,
Inner Write-Back, Outer Write-Back, you could omit some maintenance, but
you still need to clean the Linux image to the PoC.

Any memory mapped with other attributes *must* be invalidated (and
perhaps clean+invalidated) from the caches.

> We can flush some memory (including U-Boot stack), turn off the MMU.
> As soon as kernel boots up, it enables dcache and everything is back.

If you have used memory attributes inconsistent with Linux, things will
end badly from here, due to potential loss of coherency resulting from
mismatched memory attributes.

Thanks,
Mark.