Recurring ZFS deadlock; zfs_iput_taskq stuck at 100% for minutes #3687
Comments
|
a #perf top could show the kernel tasks involved during the lock. Can you post some more information about the system? |
|
The deadlock is recurring but can take weeks to occur. Unfortunately, I cannot make it happen at will to provide additional information. The motherboard is a Supermicro H8DG6 with dual AMD 6328: |
|
I'm experiencing this issue again right now. The server is beginning to grind to a halt and there is max CPU usage in zfs_iput_taskq/, kswapd0, kswapd1, kswapd2, kswapd3, and kthreadd. My Skype is Currently dumping perf |
|
I've made a pastbin of a http://pastebin.com/raw.php?i=vFpBFUFz Here is a log from |
|
the pastebin link has a redirection loop: <-- works @zhro can you post arcstats ? next time this occurs ? [hopefully not] cat /proc/spl/kstat/zfs/arcstats also try limiting the size of ARC to 1/4 or 1/3 of total memory stack issues ? referencing a few issues: |
|
Current The results of |
|
What non-default settings do you use for spl or the zfs kernel modules ? /sys/module/zfs/parameters/ |
|
Running: Motherboard/CPU: System memory: Note that there are two pools here: Pool Pool ZFS module parameters: |
|
That's a lot of swap usage (might be due to high swappiness setting, but still ...), what Java implementation do you use ? FYI: is most of the shared memory exhausted ? |
|
Currently running: The system will slows down, eventually grinding to a halt (dropping network connections), then will eventually perk back up and start running again. Every time this happens I am consistently, though with a small sample size so far, seeing ~100% CPU usage on zfs_iput_taskq, kswap, and kthread. In this case, |
|
I have an update to report which may or may not be related to this issue. At the time of this slowdown, an erroneous bash function was being executed: And at the teminal:
I am not an expert on the inner workings of ZFS but according to what I recorded, my virtual memory was being maxed out which caused zfs_iput_taskq, kswapd, and kthread become unresponsive despite there being physical memory available. Does this information shed any light onto this issue or is it a false lead? Despite having 196GB of memory is having a small swap of 4GB a problem? |
|
@zhro Having a swap or not depends on the workload you have. Allowing programs to swap brings a minimum of 100x slowdown. In your case: disable it! |
|
is swap on zvol? |
|
No. Swap is on the primary disk which contains only / and swap. Everything outside of the standard distro files are run from the /data pool. The primary disk is a 120GB SSD with nothing but the core OS and swap for IOPS. I will also use it as a large tmp or cache but this has never coincided with performance drops as mention in the first post. |
|
Experienced the problem today. Swap is currently disabled and memory is available so we can rule out swapping. Current Current |
|
Following up on this, I'm of the assumption that this is all due slab and disk cache allocation by ZFS. After a fresh reboot today and doing nothing but hammering zfs pools resulted in the following: The memory in use is all buff/cache and slab/dentry/inode cache. Here is an output of memory for the user processes: Results from evaluating slabtop and /proc/meminfo: I can confirm that dropping all caches frees this memory:
Result from I was under the assumption that this kind of cache would be freely available to the system if it was under memory pressure. But upon closer inspection of If memory allocated by zfs is located in SUnreclaim, is it still released when the system is under pressure? What makes it store so much in SUnreclaim and so little in SReclaimable? Manually dropping the caches takes brutally long and uses a significant amount of CPU for the Note that I found a post suggesting that it could take hours even on a lower memory system (16GB), which would definitely give the appearance of a deadlock: It seems that Thecus's solution was to run a script which would automatically drop all caches once memory pressure exceeded 50% of the available memory. My research also suggests that Thecus uses Linux with ZFS which suggests zfsonlinux; my point being that the server fault question linked is most likely in reference to ZFS cache allocation and deallocation (arc_reclaim). But regarding Thecus's script method, isn't zfs_arc_max already configured by default to be 1/2 the system's memory (#1134)? One might argue that the current behavior is still OK if most of the system tasks are relegated to ZFS operations. But as I've experienced when the pressure hits, on systems with large amounts of memory, the enormous time is takes to free the cache can cause massive swapping, process termination, write failure, and network timeouts. Thoughts? |
|
This issue may also be related: #3157 |
|
@zhro my issue is too general and there were no deadlocks so I doubt it could apply in this case |
and others
I have a recurring problem with ZFS on my server where the system crawls and/or deadlocks when attempting to interact with ZFS volumes. For example, I cannot read or execute scripts from these disks but directory navigation and file listing works (albeit slowly).
Sometimes the lockups get so bad that even the ssh sessions start timing out. It's not limited to the network interface as I sometimes can't even login from while at the box with a monitor and keyboard.
Java programs start throwing exceptions when trying to write to the ZFS arrays:
During these instances, according to TOP, zfs_iput_taskq and kswapd0 get stuck at 100%. txg_sync also blocks for longer than 120 seconds and is killed forcibly by the system.
These recurring deadlocks are causing both program crashes and downtime.
zpool statusreports everything OK:I'm currently using zfs 0.6.4.2-1 on CentOS 7.
Any thoughts?
Is issue #457 related?
The text was updated successfully, but these errors were encountered: