ZFS start to be slow after some work #723
Comments
|
I started reboot, which took too long and looks like with problems. During boot I got following message: |
|
By any chance, did "some work" include deleting a lot of files with xattrs, and the filesystem is xattr=dir (default)? If so, the slowness and the long time to unmount may be due to #457. |
|
we often mv big files inside zfs. We remove some big files but not often. We often make hard links and remove them later. |
|
One thing which usually kill zfs is a 20 threads reading the same file. This is a new issue. I didn't have it two weeks ago. |
|
Actually I am facing this problem now. I am copying 1.29T file to my zfs. The starting speed is around 130MB/s(nice enought), but after around 10G data, the speed comes down to around 6MB/s. It is a fresh block, no delete at all. But at the time, tsc_sync is only around 0.7% CPU times, There is no thread consuming significant amount of CPU. Every thread is around 1% CPU time. It is really hard to tell where is the problem. |
|
Getting this also, here's my setup: ext4 (noatime,data=writeback,barrier=0,nobh,commit=60) on a zvol. May be relevant: I run the yocto bitbake command as: When I cancel the build, zfs eventually falls back onto it's feet after a few minutes. config:
NAME STATE READ WRITE CKSUM
a ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRA589982 ONLINE 0 0 0
ata-Hitachi_HDS721010CLA330_JP2911N0324E1V ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-WDC_WD10EZEX-00RKKA0_WD-WMC1S0215048 ONLINE 0 0 0
ata-Hitachi_HDS721010CLA330_JP2911N03214KV ONLINE 0 0 0
cache
ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRA589587 ONLINE 0 0 0
my package info: http://pastebin.ca/2488158 |
|
... if I make the yocto build straight into a regular zfs dataset, all is well. So this really looks like it's a zvol thing. |
behlendorf
added
Difficulty - Medium
Type: Performance
|
I face with zfs_iput_taskq use 100% of one core within last 6.5 hours. I have some interesting munin graphs. zfs_iput_taskq high CPU usage started about 18 PM. There is no kernel stack trace currently. I think version is between 2104. sept. and 2014. dec. (version 1.2). |
|
@github-ivan thanks for drawing my attention to this old issue. The root cause of this was addressed in the 0.6.4.1 tag which is available through the ppa. I'd encourage you to give the latest version another shot and please report and stability issues you encounter so they can be addressed. |
|
@behlendorf thanks for your kind reply. I don't plan to upgrade, because I have really bad experiences about 0.6.4. I tried it in two test systems and two backup systems. |
The main sign of this problem is activity zfs_iput_taskq.
4085 root 0 -20 0 0 0 R 98 0.0 38:56.32 zfs_iput_taskq/
all writes start to be extremely slow:
time echo "test" >> test.txt
real 0m12.392s
user 0m0.004s
sys 0m0.000s
Unmounting system takes hours....
and standard product info:
~$ zpool status
pool: tank
state: ONLINE
scan: scrub repaired 0 in 53h10m with 0 errors on Fri Apr 6 18:49:24 2012
config:
errors: No known data errors
$ dpkg -s zfs-dkms zfsutils libuutil1 libzfs1 libzpool1
Package: zfs-dkms
Status: install ok installed
Priority: extra
Section: kernel
Installed-Size: 9500
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.62-0ubuntu1~precise1
Replaces: lzfs, lzfs-dkms
Provides: lustre-backend-fs, lzfs, lzfs-dkms
Depends: dkms (>> 2.1.1.2-5ubuntu1), spl-dkms (>= 0.6.0.62)
Conflicts: lzfs, lzfs-dkms
Description: Native ZFS filesystem kernel modules for Linux
An advanced integrated volume manager and filesystem that is designed for
performance and data integrity. Snapshots, clones, checksums, deduplication,
compression, and RAID redundancy are built-in features..
.
Includes the SPA, DMU, ZVOL, and ZPL components of ZFS.
Package: zfsutils
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 702
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.62-0ubuntu1~precise1
Replaces: zfs
Depends: libc6 (>= 2.8), libnvpair1, libselinux1 (>= 1.32), libuuid1 (>= 2.16), libuutil1, libzfs1, libzpool1
Recommends: zfs-dkms
Suggests: nfs-kernel-server, zfs-initramfs
Conflicts: zfs, zfs-fuse
Conffiles:
/etc/default/zfs a36d4561e19974d80a7b6cec75639c24
/etc/bash_completion.d/zfs 3e1c4a29c4f7d590e6a3041f2c61d6ff
/etc/init.d/zfs-share 003fc628cf32324eefe6ad6239a0ed80
/etc/init.d/zfs-mount cb9c3d88deeef1356198e8c5ca913ed4
/etc/zfs/zdev.conf b006284e64b215ca619aeb56d2df9bf5
Description: Native ZFS management utilities for Linux
This package provides the zpool and zfs commands that are used to
manage ZFS filesystems.
Package: libuutil1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 148
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.62-0ubuntu1~precise1
Replaces: libuutil0
Depends: libc6 (>= 2.14), libuuid1 (>= 2.16), zlib1g (>= 1:1.1.4)
Description: Solaris userland utility library for Linux
This library provides a variety of glue functions for ZFS on Linux:
possible to run Solaris user code in a Linux environment with relatively
minimal modification.
Package: libzfs1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 308
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.62-0ubuntu1~precise1
Replaces: libzfs0
Depends: libc6 (>= 2.8), libnvpair1, libzpool1
Description: Native ZFS filesystem library for Linux
The zfs management library.
Package: libzpool1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 1120
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.62-0ubuntu1~precise1
Replaces: libzpool0
Depends: libc6 (>= 2.14), libuutil1, zlib1g (>= 1:1.1.4)
Description: Native ZFS pool library for Linux
The zpool management library.
The text was updated successfully, but these errors were encountered: