Account for ashift when gathering buffers to be written to l2arc device

[behlendorf]

If we don't account for that, then we might end up overwriting disk
area of buffers that have not been evicted yet, because l2arc_evict
operates in terms of disk addresses.

The discrepancy between the write size calculation and the actual
increment to l2ad_hand was introduced in commit 3a17a7a.

The change that introduced l2ad_hand alignment was almost correct
as the write size was accumulated as a sum of rounded buffer sizes.
See commit illumos/illumos-gate@e14bb32.

Also, we now consistently use asize / a_sz for the allocated size and
psize / p_sz for the physical size. The latter accounts for a
possible size reduction because of the compression, whereas the
former accounts for a possible subsequent size expansion because of
the alignment requirements.

The code still assumes that either underlying storage subsystems or
hardware is able to do read-modify-write when an L2ARC buffer size is
not a multiple of a disk's block size. This is true for 4KB sector disks
that provide 512B sector emulation, but may not be true in general.
In other words, we currently do not have any code to make sure that
an L2ARC buffer, whether compressed or not, which is used for physical
I/O has a suitable size.

Note that currently the cache device utilization is calculated based
on the physical size, not the allocated size. The same applies to
l2_asize kstat. That is wrong, but this commit does not fix that.
The accounting problem was introduced partially in commit 3a17a7a
and partially in 3038a2b (accounting became consistent but in favour
of the wrong size).

Porting Notes:

Reworked to be C90 compatible and the 'write_psize' variable was
removed because it is now unused.

References:
https://reviews.csiden.org/r/229/
https://reviews.freebsd.org/D2764

Ported-by: kernelOfTruth kerneloftruth@gmail.com
Signed-off-by: Brian Behlendorf behlendorf1@llnl.gov
Issue #3400
Issue #3433
Issue #3451

[kernelOfTruth]

#3114 might also be related - and/or rather to Illumos 5701 (#3498)

[behlendorf]

Merged as:

ef56b07 Account for ashift when gathering buffers to be written to l2arc device

[avg-I]

@behlendorf @kernelOfTruth thank you and all who helped with testing, etc, very much!
BTW, I have some follow-on work: https://reviews.freebsd.org/D2789
But it's not yet rebased on the latest ARC code.

[kernelOfTruth]

referencing #3492

@avg-I that also sounds very useful - Thanks a lot !

Last time I took a look at it - it was (still) too much over my head in terms of knowledge and comfort level to be merged against all those ARC changes

Also I've right now thesis work going on - so for the moment I'm leaving it for someone more knowledegable