[U-Boot] i.MX51: FEC: Cache coherency problem?

[Aneesh V]

Hi David,

On Wednesday 20 July 2011 05:01 PM, David Jander wrote:
>
> Hi Aneesh,
>
> On Wed, 20 Jul 2011 15:59:42 +0530
> Aneesh V<aneesh at ti.com>  wrote:
>> On Wednesday 20 July 2011 02:51 PM, David Jander wrote:
>>> On Wed, 20 Jul 2011 10:56:07 +0200
>>
>> [snip ..]
>>
>>>>> Any ideas?
>>>>
>>>> Yes, one: I had issues with the Marvell Ethernet adapter, which has DMA
>>>> as well, not because of cache (it was not active at the time) but
>>>> because of instruction reordering done by the compiler when optimizing,
>>>> which resulted in out-of-order accesses to the descritpors and DMA
>>>> registered, so that the DMA start was written before the descriptors
>>>> were set up. The (proper and clean) solution was to introduce memory
>>>> barriers at strategic points of the driver to ensure that specific DMA
>>>> starts were done only after all writes to the descriptors (and possibly
>>>> buffers).
>>>
>>> Hmmm. I know that issue, but AFAICS, the driver already uses readX() and
>>> writeX() macros when accessing register and DMA memory. Those macros have
>>> compiler barriers included.... and armv7 is still in-order execution ;-)
>>
>> 1. armv7 is not necessarily in-order. Cortex-A8 is in-order while
>> Cortex-A9 is out-of-order.
>> 2. I am not sure if in-order execution guarantees memory ordering. My
>> understanding is that it doesn't.
>> 3. In u-boot's implementation of MMU for ARM, the register
>> space(everything other than SDRAM) is 'strongly-ordered' memory.
>> Strongly ordered accesses will be correctly ordered w.r.to all other
>> accesses in program order.
>
> Wow, I hadn't noticed dcache_enable() also enables the MMU. Cool. One step
> closer to splitting the mapping between a cached and uncached, strongly-ordered
> region of RAM.

Yes. MMU is enabled with an identity mapping. Please note that ARMv6
onwards D-cache can not be enabled when MMU is disabled. I-cache works
without MMU enabled.

>
>> 4. Compiler barriers prevent the compiler from doing certain
>> optimizations, but doesn't help in these situations.
>
> Thanks for this excellent summary!
>
>> In short, I think you don't need to worry about ordering, but not
>> because of compiler barriers or armv7 being in-order.
>
> Well, I do need to worry about the buffer-descriptors and buffers then, since
> they are in SDRAM, right?

Yes, you need to worry about buffers and descriptors. Here are the
rules-of-thumb to take care of them:

1. Flush before initiating memory to peripheral DMA(this applies to
descriptors too)
2. Invalidate after peripheral to memory DMA (before the CPU reads
the buffer)
3. Make sure that any buffer that you have to invalidate is cache-line
aligned at the beginning and at the end.

br,
Aneesh