[U-Boot] [RFC][PATCH] arm: Fix flush_dcache_range() on arm926

[Marek Vasut]

Hi Albert,

> Hi Marek,
> 
> On Wed, 10 Jul 2013 02:30:29 +0200, Marek Vasut <marex at denx.de> wrote:
> > The flush_dcache_range() on arm926 did not work as expected on i.MX28.
> > 
> > This can be observed during the operation of the FEC ethernet driver
> > where the driver did occasionally fail with timeout trying to transmit
> > a frame. The FEC ethernet driver uses DMA for transmitting the frame in
> > the following fashion:
> > 
> > 0) Set bits in DMA descriptor
> > 1) Write DMA descriptor into aligned DRAM address
> > 2) Flush D-Cache over the descriptor
> > 3) Start DMA
> > 4) Invalidate D-Cache over the descriptor
> > 5) Test if certain bits in DMA descriptor are unset
> > 
> > Very not so often it happened that the bits in the DMA descriptor were
> > still set even after hardware register -- which is not cached --
> > indicated otherwise.
> > 
> > Here I will theoreticise, Albert, can you please correct me if I'm wrong?
> > 
> > This leads me to believe the DMA descriptor was still in the cacheline
> > after being flushed in step 2) and during the DMA gets evicted into DRAM
> > therefore corrupting the result of readback in 5) . By reading the ARM926
> > datasheet DDI0198E_arm926ejs_r0p5_trm.pdf page 50 table 2-17, it is not
> > clear whether cacheline that is Valid+Clean will be invalidated in the
> > D-Cache using the "mcr p15, 0, <Rx>, c7, c14, 1" instruction or whether
> > only Valid+Dirty lines are cleaned+invalidated. The other thing that is
> > unclear to me is whether a cacheline that is Valid+Clear is written back
> > into DRAM when it is evicted from cache.
> 
> The way I see table 2-17, the "if valid and dirty" condition only
> applies to the "write" part of the description, not to the "marked not
> valid" one; this reading makes the most sense and is the most consistent
> with the cache functioning as a whole.

So basically all Valid+Dirty get cleaned .
Nothing is done on Valid+Clean .
Finally, as they are now all Valid+Clean , they all get Invalidated .

This is how it'd make the most sense, but I'm afraid this is not what I observe.

> As for a valid and clean (rather than "clear", I assume) line being
> written if evicted, I think it is not. Eviction is not flushing: the
> line is kicked out, plain and brutal. And even if we're talking about
> flushing, if the line is clean then it is coherent with the RAM area
> it caches, and writing it back there is unneeded.

Yes, this _does_ make full sense to me.

> > Interestingly enough, running invalidate_dcache_range() after doing the
> > flush_dcache_range() over the descriptor solved the problem and I see no
> > occasional timeout anymore. This confirms my opinion that the descriptor
> > might remain in the cache and be written back during the DMA operation.
> 
> Can you point me to the source code location(s) where the sequence
> above is implemented?

See:

 738         flush_dcache_range(addr, addr + size);

in drivers/net/fec_mxc.c . If I put invalidate_dcache_range(addr, addr + size); 
, all seems to work well. Note it also happens with multiple compilers, Debian 
gcc 4.8.0, ELDK 5.2 and ELDK 5.3 .


Best regards,
Marek Vasut