poor performance

[ghost]

on zfs-fuse

# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME                                                    STATE     READ WRITE CKSUM
        tank                                                    ONLINE       0     0     0
          mirror-0                                              ONLINE       0     0     0
            disk/by-id/ata-ST3160815AS_6RA4G3G0                 ONLINE       0     0     0
            disk/by-id/ata-WDC_WD1600JD-00HBB0_WD-WCAL73743208  ONLINE       0     0     0

errors: No known data errors
# zfs create tank/xxx
# dd if=/dev/zero of=/tank/xxx/test.io bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 0.728856 s, 112 MB/s
# zpool destroy tank/xxx
# zpool export tank
# aptitude install ubuntu-zfs
The following NEW packages will be installed:
  libefi1{a} libnvpair1{a} libshare1{a} libspl1{a} libunicode1{a} libuutil1{a} libzavl1{a} libzfs1{a} libzpool1{a}
  ubuntu-zfs zfsutils{b}
0 packages upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/1,075 kB of archives. After unpacking 3,355 kB will be used.
The following packages have unmet dependencies:
  zfsutils: Conflicts: zfs-fuse but 0.6.9-1build1 is installed.
The following actions will resolve these dependencies:

     Remove the following packages:
1)     zfs-fuse
YES
...
# reboot
#  zpool import tank
#zpool status tank
  pool: tank
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

errors: No known data errors
# zpool create tank/xxx
# dd if=/dev/zero of=/tank/xxx/test.io bs=8k count=10000
#10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 35.5567 s, 2.3 MB/s

So i have 112 Mb/s (zfs-fuse) vs 2,3 Mb/s (ubuntu-zfs)

while runs dd top, iostat shows around zero values

[behlendorf]

Those results are particularly bad even for 8k block sizes I'd expect closer to 60-80MB/s. Try a large block size and see if it helps. We still have some performance work to do for small block sizes.

[ghost]

zfs-fuse:

dd if=/dev/zero of=/tank/xxx/test.io bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.48609 s, 124 MB/s

ubuntu-zfs:

dd if=/dev/zero of=/tank/xxx/test.io bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 114.533 s, 9.2 MB/s

Ok, i try recreate pool under ubuntu-zfs

# zpool offline tank sdc
# zpool detach tank sdc
# zpool create -f test sdc
# zpool status test
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          sdc       ONLINE       0     0     0

errors: No known data errors
# zfs create test/xxx
# dd if=/dev/zero of=/test/xxx/test.io bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 53.5897 s, 19.6 MB/s

[behlendorf]

There must be something significantly different about your setup/hardware than my test systems. Here are my results from a similar setup running RHEL6.1 on some pretty old hardware with slow SATA drives in a JBOD. Also your zfs-fuse results look surprisingly good, you might try increasing the file size and adding conv=fsync to your test to ensure you're not just getting cache performance. I do see much better results than this when using faster hardware.

# sudo zpool create -f tank sdc
# sudo zpool status tank
  pool: tank
 state: ONLINE
 scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      sdc       ONLINE       0     0     0

errors: No known data errors
# sudo zfs create tank/test
# sudo dd if=/dev/zero of=/tank/test/file bs=1024k count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 20.7703 s, 51.7 MB/s

Here's a little table of results comparing dd against the raw devices to a file in zfs on my system. It certainly could be better but it is reasonable. For example, there's something funny going on for block sizes between 8-32k. We're just starting to get serious about looking at performance issues and we expect that there is a fair bit of low hanging fruit.

Block Size (KB)  Blk Dev (MB/s)  ZFS (MB/s)
1                16.9            16.5
2                25.7            22.3
4                52.4            33.4
8                52.3            22.5
16               54.8            26.6
32               54.1            25.2
64               54.4            52.8
128              55.1            53.6
256              54.9            55.5
512              56.3            55.4
1024             56.8            52.1

Commands:
for (( i=1; i<=1024; i_=2 )); do dd if=/dev/zero of=/dev/sdc bs=${i}k count=4096 conv=fsync; done
for (( i=1; i<=1024; i_=2 )); do dd if=/dev/zero of=/tank/test/file bs=${i}k count=4096 conv=fsync; rm /tank/test/file; done

[ghost]

behlendorf, probably, i had bad results because i was try to use 32 bit OS.
Fresh install ferdora 16 32 bit was the same, but zfs on fedora 16 (x64) shows performance near to raw device.
Thanks.