[PATCH] nvme: Fix cache alignment

[Andre Przywara]

On Sun, 7 Feb 2021 14:13:37 -0500
Tom Rini <trini at konsulko.com> wrote:

Hi Tom, Marek,

> On Sun, Feb 07, 2021 at 07:20:14PM +0100, Marek Vasut wrote:
> > On 2/4/21 5:57 PM, Tom Rini wrote:
> > [...]
> >   
> > > > > > > > > > +static void nvme_flush_dcache_range(void *start, unsigned long size)
> > > > > > > > > > +{
> > > > > > > > > > +       unsigned long s, e;
> > > > > > > > > > +       nvme_align_dcache_range(start, size, &s, &e);
> > > > > > > > > > +       flush_dcache_range(s, e);  
> > > > > > > > 
> > > > > > > > There is no good reason for alignment restrictions when it comes to
> > > > > > > > clean (& invalidate), so there is no need for this wrapper.  
> > > > > > > 
> > > > > > > Is that on ARM64-specific or is that applicable in general ? The driver
> > > > > > > is expected to work on any CPU.  
> > > > > > 
> > > > > > Cache clean (actually: cache clean&invalidate) is what happens on evictions
> > > > > > all of the time, at the cache controller's discretion. So there is no
> > > > > > real harm in that operation per se. When an eviction happens on a
> > > > > > *clean* cache line, this is basically just an invalidate, which is also not
> > > > > > harmful.
> > > > > > 
> > > > > > There are harmful cases when buffers sharing a cache line are both "just invalidated"
> > > > > > and "cleaned" at different points in time.  
> > > > > 
> > > > > Is that on ARM64-specific or is that applicable in general ? (the above
> > > > > does not answer that question)  
> > > > 
> > > > I would say that's a property of *every* write-back cache
> > > > implementation:
> > > > https://en.wikipedia.org/wiki/Cache_(computing)#/media/File:Write-back_with_write-allocation.svg  
> > > 
> > > I've been reading and digesting the thread as it goes, and the only
> > > thing I do want to chime in on here right now is that yes, U-Boot
> > > does and will continue to support every CPU that someone wants to run it
> > > on, and one of the takeaways I see from this thread is we need some
> > > better documented abstractions around cache, as it's very tricky to get
> > > right all the time.  
> > 
> > Documenting the u-boot cache function behavior precisely is fine by me, but
> > that is somewhat separate topic from this bugfix.
> > 
> > So I will ask a simple question -- is there anything blocking this bugfix
> > from being applied ?  
> 
> While I can fix the typo, I'm waiting for an Ack/Reviewed-by from Bin
> (as he's spent the most time on this driver of late) and I'd really like
> one from Andre, or at least him agreeing this patch is a step in the
> right direction.

As I said: I don't see how this patch changes anything on arm64, which
the commit messages claims to be the reason for this post.
If someone please can confirm, but invalidate_dcache_range() always
works on arm64, in fact does the very rounding already that this patch
introduces here. So what is the actual fix here?

Plus, I consider this blanket rounding more harmful than helpful, since
it actually just silences the check_cache_range() check.

If we include armv7 here, which in fact would ignore(!) an unaligned
invalidate, this is my analysis (just for invalidate):
nvme_read_completion_status():		NEEDS ALIGNMENT
	size smaller than cache line, cqes[1] base address unaligned.
	fix needed, rounding *could* be a temporary fix with comments
	as of why this is legitimate in this case.
	Better alternative sketched in a previous email.
nvme_identify():			OK
	struct nvme_id_ctrl is 4K, if I counted correctly, so is fine.
	buffer comes the callers, the in-file users pass an aligned
	address, the only external user I see is in nvme_show.c, which
	also explicitly aligns the buffer.
nvme_blk_rw():				OK
	total_len seems to be a multiple of block size, so is hopefully
	already cache line aligned.
	buffer comes from the upper layers, I guess it's page
	aligned already (at least 512b sector aligned)?

I turned my sketch for the cqes[] fix into a proper patch and will send
this in a minute.

Cheers,
Andre