zfs_arc_max what is the default, what happens when there is memory pressure

[byteharmony]

I had a machine configured for about 19.5 GB of ram when it only had about 16 GB of ram.

ZFS 8G for zfs_arc_max
Linux (I estimate it would use 1GB (which is high I know)
KVM VM: 10.5

It was running with RAM used, buffers at 100MB and 700 MB of swap. One VM crashed daily, the other 2 were stable. The strange thing was when I shutdown all the VMs I expected to see 8GB of ram in use, I had 4???

This is parameter and kernel / zfs updates are about the only reason for a reboot most of the time.

Any ideas if this is normal behavior?

BK

[behlendorf]

The default zfs_arc_max value is 1/2 of all available ram. Under memory pressure from applications that cache space will be released for the apps much like the page cache.

[byteharmony]

The default settings in RHEL / CENTOS fight each other signficantly resulting in memory wars.

Players: ksmd, swappiness, buffer and arc

KSM default values pound the box, when it runs it frees big chunks of ram, but perhaps the ARC is doing it because it also senses the pressure??
swappiness is 60 (pushes lots to swap SLOOOOOWWW)
buffer will it purge when ram pressure hits?
arc not sure how much and what the configs do.

Namely, if i set zfs_arc_max (override the 1/2 default) to a lower value will the system still release ram from the cache?

Is this memory war really what's best for these systems? I can certainly find ways to micro manage the configs for our implementations but should it be that hard?

My best guess right now which has shown good results is drop swappiness down to 0, reduce arc cash and let Linux ballance buffer after the KSM push. Perhaps this is as good as it gets. I did update a few systems to metadata for the arc to see how performance / load vary.

If this info isn't helpful please let me know.

Thanks,
BK

[ryao]

ZFS will still respond to memory pressure if you set zfs_arc_max.

The default on Solaris is 3/4 of system memory. It used to be the same on ZFSOnLinux as well, but it was changed due to memory management issues. Some of them have been solved, but some will remain until page cache unification is done.

As a side note, my original belief that fragmentation was the cause of these problems was incorrect. The upperbound on fragmentation in the SLAB allocator is lower than it is in the HOARD allocator. With that said, Linux seems to be happiest with zfs_arc_max set to 1/2 or less.

[behlendorf]

Closing issue since this is largely just a matter of documentation..