Async zfs_purgedir

[behlendorf]

Commit b00131d accidentally introduced a significant performance regression when unlinking a large number of files with xattrs. While the xattr directory cleanup needs to be done asynchronously to avoid deadlock, performing it as part of zfs_unlinked_drain() is undesirable. It becomes a problem because each unlink must traverse the entire unlinked set attached to the super block. When a large number of unlinks occur the list can be very long and the zfs_iput_taskq threads will be observed to consume a lot of cpu. It would be preferable to pass just the super block with a reference count and the exact inode number to have it's xattr directory purged to the taskq.

[gunnarbeutner]

Hm yeah, I knew there would be some performance drawback when I wrote that patch. It seemed like the right thing at the time though - i.e. stability over performance. :)

Hopefully I'll get some time during my USA vacation in December to work on some of the remaining issues.

[behlendorf]

I completely agree, that's why I merged it. :) I just wanted to file this so we remember to revisit the issue. This particular problem just caused me a lot of trouble when I was benchmarking the SA based xattrs and needed to run comparison tests against directory based xattrs. Removing 100,000 files with directory xattrs took a long time.

[chrisrd]

To avoid soft lockups when deleting large numbers of files with dir base xattrs, would it help to delete all the xattrs prior to deleting the files?

[behlendorf]

Yes, as a workaround it would probably help although I don't have any exact numbers for you.

[behlendorf]

Reopening issue, see commit 7973e46 and issue #1176 for details.

[kfreedman]

I have been running into this problem a lot lately. After having read about issues including xattr's I turned xattrs off.
I'm running wordpress/MariaDB installation. /tmp was a zfs filesystem and I notice a lot of temp tables created on disk generating a lot of activity on /tmp . nothing like the 'huge rsync' or other types of load reported by others.
turned off xattrs didn't seem to help.
made /tmp a tmpfs .. site got a smidge faster but still running into the zfs_iput_taskq 100% problem.

the traffic really isn't so incredible and the writes should be minimal relative to reads that my symptoms are different enough from what other people are finding, so I'm at a loss.

here's my versions of stuff:

# zfs get xattr data
NAME  PROPERTY  VALUE  SOURCE
data  xattr     off    local

Linux 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
zfs-dkms-0.6.2-1.el6.noarch
dkms-2.2.0.3-14.zfs1.el6.noarch
zfs-0.6.2-1.el6.x86_64
[root@wphost2 ~]# rpm -qa |grep -i spl
spl-dkms-0.6.2-1.el6.noarch
spl-0.6.2-1.el6.x86_64

the packages are installed using yum from:
http://archive.zfsonlinux.org/epel/6/x86_64/