[Performance] Extreme performance penalty, holdups and write amplification when writing to ZVOLs

[Binarus]

System information

Type | Version/Name
Linux | Debian
Distribution Name | debian
Distribution Version | buster (10.7)
Linux Kernel | 4.19.0-12-amd64
Architecture | amd64
ZFS Version | OpenZFS 2.0.0
SPL Version | (SPL integrated in OpenZFS 2.0.0)

Describe the problem you're observing

Setup

• Supermicro X10DRU-i+
• LSI 9361-8i connected to a 6 / 12 Gbps SATA/SAS backplane
• 2 x Seagate ST4000NM000A (denoted sda and sdb), connected to the backplane
• 1 x Seagate ST4000NM0035 (denoted sdc), connected to the backplane
• 128 GB RAM (ECC, of course)

sda and sdb make up a mirrored ZFS VDEV. The O/S boots from this VDEV. There is only one pool, called rpool. rpool does not contain any other VDEVs besides that mirror. The root file system is mounted to rpool/system.

There is no swap file on that system (yet).

rpool has been created using the following command:

zpool create -o ashift=12 -o altroot=/mnt -O acltype=posixacl -O canmount=off -O checksum=on -O compression=off -O mountpoint=none -O sharesmb=off -O sharenfs=off -O xattr=sa rpool mirror /dev/disk/by-id/ata-ST4000...-part1 /dev/disk/by-id/ata-ST4000...-part1

That is, the pool and the VDEV have ashift=12.

rpool/system has been created using the following command:

zfs create -o aclinherit=passthrough -o acltype=posixacl -o atime=on -o canmount=on -o checksum=on -o compression=off -o mountpoint=/ -o overlay=off -o primarycache=all -o redundant_metadata=all -o relatime=off -o secondarycache=none -o setuid=on -o sharesmb=off -o sharenfs=off -o logbias=latency -o snapdev=hidden -o snapdir=hidden -o sync=standard -o xattr=sa -o casesensitivity=sensitive -o normalization=none -o utf8only=off rpool/system

We further have created a ZVOL using the following command:

zfs create -b 4096 -o checksum=on -o compression=off -o primarycache=metadata -o redundant_metadata=all -o secondarycache=none -o logbias=latency -o snapdev=hidden -o sync=standard -V 100G rpool/zvol-test

That zvol is mounted on /blob.

sdc contains a partition with a normal ext4 file system which is mounted on /mnt. That file system just contains several dozens of ISO files (average size about 6 GB).

On that machine, nothing else runs than the standard services the distribution installs. Notably, there is no VM running and nothing else which could produce substantial workload.

In this state, when starting watch -n 1 zpool iostat -ylv 1 1 and watching it for a while, there is indeed nearly no load on the ZFS disks. Once in several seconds or so some kilobytes hit the VDEV, which is expected.

Copying to the dataset (not the ZVOL): No problem

Now we open the iostats in one terminal window (watch -n 1 zpool iostat -ylv 1 1) and start to copy ISO files from sdc onto the ZFS dataset rpool/system in another terminal window (rsync --progress /mnt/*.iso ~/test, where ~/test is part of the root file system and thus is on rpool/system).

While the copy runs, rsync shows a few drops in bandwidth every now and then, but there are no noticeable holdups, and the drops in bandwidth are short. Likewise, the zpool iostats shows that the two disks in the VDEV are hit with data rates which could be expected. The changes in disk load reported by zpool iostat are surprisingly high, though (the load constantly jumps between something like 30 MB/s and 300 MB/s), but there are no real holdups either. In summary, the copy in average runs with over 100 MB/s and does not stall for a longer time.

We have interrupted that test after 30 GB or so because we didn't expect anything new from letting it run longer. However, we repeated it several times, each time copying other ISO files, and each time rebooting before. The behavior was the same each time.

Copying to the ZVOL: Problem

When we do exactly the same thing, but copy to the ZVOL instead of the dataset (rsync --progress /mnt/*.iso /blob), the situation changes. rsync initially shows the copy running with roughly about 190 MB/s for a few seconds, then it stalls. Thereafter it continues with copying for a few seconds at the rate denoted above, then stalls again after a few seconds, and so on.

The problem is that the holdups last for a long time where absolutely nothing happens, up to several minutes (!). However, zpool iostat shows that the two ZFS disks are under heavy load during this time, constantly (more or less) being hit with over 100 MB/s. Even when we interrupt copying by hitting Ctrl-c in the terminal window where rsync runs, this high load lasts for several minutes until everything returns to normal.

There must be extreme write amplification somewhere, the amplification factor being somewhere between 5 and 10. For example, if we copy 40 GB that way, this would normally take about 5 minutes. But actually it takes at least half an hour, although the ZFS disks are under heavy load all that time.

For that reason, ZVOLs are currently just not usable for us, which imposes a major problem. What could be going on there?

Our own thoughts and what we have tried already:

At first, we'd like to stress again that the ZVOL test did not happen within a VM. The problem is definitely not due to QEMU or (para)virtualization of data transfer.

Secondly, I am aware that it might not be the best idea to have ZFS running on disks which are attached to a RAID controller like the LSI 9361-8i, or to have it running on hybrid disks like the ones we have. However, we have configured that controller to JBOD mode, and the O/S sees the disks as individual ones as expected. But the ultimate key point regarding possible hardware problems is that copying large amounts of data to the ZFS dataset (rpool/system) works as expected. If the problems with the ZVOL would be due to hardware, we would have the same problems with the dataset; this is not the case, though.

Thirdly, the problem is not due to ZFS versions. Debian buster comes with ZoL 0.7.12, and we originally have noticed the problem there. We desperately need ZVOLs working, so we have installed OpenZFS 2.0.0 on that machine, which did change exactly nothing with respect to that problem.

As a further test, we created the ZVOL with volblocksize=512 and did the tests again. Again, nothing changed. We repeated the process with volblocksizes of 8192, 16384 and 128k. Again, no luck: Maybe it stalled a few seconds earlier or later, longer or shorter in each test compared to the others, but the general situation remained the same. Between the stalls, the copy ran a few seconds with expected speed, then it stalled for a lot of seconds, mostly even a few minutes while iostat was showing a constant data rate of roughly 100 MB/s for each disk, and so on. After interrupting the copy, both ZFS disks continued to be hit with a data rate of 100 MB/s or more for several minutes.

Then we tested the ZVOL with sync=disabled. That didn't change anything. The same goes for primarycache=all (instead of metadata) (but at least this was expected), and for logbias=latency (instead of throughput).

Next, we thought that it may have something to do with the physical sector size of the ZFS disks being 512 bytes, while the pool (and the VDEVs) had ashift=12. Therefore, we destroyed the pool, re-created it with ashift=9, re-created all file systems / datasets as described above, and did all tests again. Once again, this didn't change anything.

We then went back to the original pool with ashift=12 and used it for the further tests. At this point, we were out of ideas what to do next, so we read about the ZFS I/O scheduler and tested a large number of combinations of zfs_dirty_data_max, zfs_delay_scale, zfs_vdev_async_write_max_active, zfs_vdev_async_write_min_active, zfs_vdev_async_write_active_max_dirty_percent, and zfs_vdev_async_write_active_min_dirty_percent.

To our surprise, the last five of these did barely influence the behavior. However, the first one (zfs_dirty_data_max), which originally was set to 4 GB, changed the situation when we set it to a low value, e.g. 512 MB. The improvement was that there were less long-lasting holdups: there were even more holdups, but all of them were so short that it became acceptable. However, the average data rate did not increase, because now the transfer rate rsync reported was limited to about 30 MB/s, mostly hanging around 10 MB/s or 20 MB/s. There were no phases with high data rates any more.

So the copying was more "responsive" with low values of zfs_dirty_data_max, but that didn't help because the data rate per se was drastically limited. In summary, changing the I/O scheduler parameters which are explained in the document linked above did not lead to anywhere.

The last thing we were looking into was zfs_txg_timeout. Setting it to a lower value didn't improve the situation with copying to the ZVOL (but increased the load which hit the ZFS disks when the system was completely idle). Setting it to a higher value didn't improve copying either (but reduced the load on the ZFS disks when the system was idle).

Now we are completely out of ideas. We probably could look into other parameters of the ZFS module (/sys/module/zfs/parameters) or the disk drivers (/sys/block/sdx). But this would be just wild guessing and a waste of time. Therefore, we are hoping that somebody is willing to give us some hints.

What we did not try, and why not

zfs_arc_max is set to 4 GB on that system, and we did not test larger values for the following reasons:

1. That parameter is about reading, not writing, and the copy source is an ext4 partition of a physical disk, so no ZFS parameter would have any effect on the copy source.
2. We clearly have a problem with writing here, not with reading (remembering that copying to the normal dataset (not the ZVOL) works normally).
3. When we began working with ZFS some years ago, the first thing we had to solve was a system which started normally, but then became totally unresponsive and finally totally locked up within minutes. The cause of that problem was that ZFS was eating up all available RAM for its ARC cache until the machine crashed or hung. Since then, we always limit the ARC size (and never ever had any stability issues or crashes with ZFS again).
4. Our goal is to run a bunch of VMs with ZVOL storage (the tests described above are just, eehm, tests before we put even more effort into switching completely to ZFS). The number of VMs and the memory they will be given is precisely known. It would not make any sense to test larger ARC sizes, because the ARC size at the end of the day couldn't be much larger than 4 GB.

We did not try to use a secondary cache (L2ARC). Again, the copy source is not on ZFS, and therefore this wouldn't make any sense, and furthermore, we have a writing problem here, not a reading problem.

We did not try to use an SLOG. This would not make any sense, because one of our tests was to set sync=disabled on the copy destination ZVOL, and this did not change the slightest bit in the behavior observed. Therefore, we know that our problem is not due to sync writes, and thus, an SLOG wouldn't improve the behavior.

Describe how to reproduce the problem

Install a system similar to the one described above, issue the commands described above, and watch the long-lasting holdups in the terminal window where rsync runs and the heavy disk load zpool iostat shows in the other terminal window, leading to high disk wear and low bandwidth.

Since it is not easy to setup a system like ours, we are willing to give remote access to one of that systems if somebody would be interested in investigating the problem. In this case, please leave a comment, stating how we can get into contact.

Include any warning/errors/backtraces from the system logs

If somebody tells us what exactly is needed here, we'll immediately do it :-). We guess zpool iostat or other tools produce output which is more valuable than the log files, but neither being Linux nor ZFS experts, we are a bit lost here. Notably, we don't know how to operate dtrace or strace properly. If somebody tells us what to do, we'll try our best.

[IvanVolosyuk]

You might have to decide what you value more - consistency or throughput. Decreasing zfs write buffers (zfs_dirty_data_max) will give you more consistency - consistently bad write speed with lots of TXGs.

If you want throughput, you don't want to measure it using minimum write speed during the write operation or seconds without visible progress (effectively what you seem to be doing). Instead I would suggest to pick a smaller test set, but write it fully and measure the time it took for completion.

For throughput I would pick larger volblocksize - default 8k should work better, keep default zfs_dirty_data_max, enable compression - it will benefit slow disks, sync=disabled to avoid wasting disk time on zil, primarycache=all to cache ext4 metadata.

Because of the in memory write cache initial writes will look faster and then stall when in memory buffers hit the hard limit while disks are busy writing back the data to free space in ram for more dirty data. But if you measure the total copy operation time - this information is irrelevant for overall throughput number you get.

Also, there should be minimal write amplification when you copy large ISO files.

Personal notes: with qemu you can use raw files, which can give slightly better performance than zvol in some cases. I use:
-drive file=/somefile.img,format=raw,id=disk,if=none,cache=none,aio=threads,discard=unmap

[Binarus]

@IvanVolosyuk Thank you very much for your help.

Your first comment (which you have removed) proposed to use logbias=throughput instead of logbias=latency. We have to apologize that we forgot to mention that we already had tested this setting, too, but it didn't change anything. I'll add this in the description of the issue, i.e. to our first post.

With respect to your other proposals, I guess I'll have to explain some specialties regarding our setup and goals:

You might have to decide what you value more - consistency or throughput.

We understand that there always is a trade-off. But what we are talking about here is a decision between between holdups of several minutes and extreme disk wear caused by extreme write amplification on one side and a data rate of 5% to 10% percent of what the hardware is able to provide on the other side. Trade-offs of such magnitude are in no way acceptable.

If you want throughput, you don't want to measure it using minimum write speed during the write operation or seconds without visible progress (effectively what you seem to be doing).

The problem here is that the holdups are taking so long and are putting the disks under such heavy load that the system cannot be used reasonably during that time. In my first post, I have explained that we did the tests without any VMs running. This is true, because we wanted to rule out any issues related to QEMU, KVM and block device drivers.

However, of course, we also did additional tests with running VMs. It turned out that the bad situation, notably the holdups, could be provoked by just copying large files from the third disk to the VM ZVOL storage in a VM, and that these holdup made the other VMs freak out - they just couldn't write data to their (virtual) storage as needed because they hit timeouts after one or two minutes.

This basically means that you can have only one VM on your server (provided you want to be it on a ZVOL), unless you are willing to put your data at risk.

Instead I would suggest to pick a smaller test set, but write it fully and measure the time it took for completion.

The outcome would be interesting. However, it is not our use case; more precisely, we have important other use cases. One of them (and an important one) is to have multiple VMs running on ZVOLs, where at least one VM will be used to copy large amounts of data scattered across a few files from a third disk to virtual ZVOL storage. It is completely not acceptable to have the other VMs freak out then, putting their data at risk.

Therefore we first have to solve the problems described in the first post before we proceed.

I have mentioned VMs solely to explain our issue in greater depth, and why it definitely is a show stopper in our case. Still, I would like to keep VM related problems out of this discussion, because the issue exists without any VM running, and adding VMs surely won't improve the situation.

For throughput I would pick larger volblocksize

This is one of tests we did and which we have described. We have tested 512 bytes, 4096 bytes, 8192 bytes, 16384 bytes and 128k bytes. None of theses settings made the behavior change in any way.

keep default zfs_dirty_data_max

Changing it was only for testing. Of course, we started testing with default values.

enable compression

We have compression disabled because (at a later stage) the ZVOL data will be encrypted (from within the VMs). Given that, compression won't do any good.

sync=disabled

We already have tested this setting without noticing any change in behavior (described in the first post I guess).

primarycache=all to cache ext4 metadata

Could you please elaborate? How could a ZFS setting influence reading from the third disk, which is not on ZFS? Did we miss something?

Apart from that, we already had tested primarycache=all, but it didn't change anything either. IMHO, this is expected because primarycache relates to reading data, while we obviously have a problem with writing.

But if you measure the total copy operation time - this information is irrelevant for overall throughput number you get.

Agreed, but the holdups would make VMs freak out (and hence, put data at risk) if we had several VMs running, which will be the case later.

Plus, the extreme write amplification we obviously experience will destroy our disks in no time.

Also, there should be minimal write amplification when you copy large ISO files.

This is exactly what we were convinced of when we began the tests. Imagine out surprise ...

We are seriously thinking of making a video which shows the two terminal windows for 10 minutes or so. Perhaps somebody could make sense of it. We'll first have to look for appropriate screen recording software, though (must be able to record cygwin terminal windows under Windows 10 at a reasonable frame rate (e.g. 10 frames / sec)).

Cheers,

Binarus

[IvanVolosyuk]

With primarycache=all for zvol - the filesystem on zvol (ext4 I assumed) will have its metadata cached in ARC.

I reproduced similar behavior with the copy of your settings and with the set of changed I suggested I've got some improvements when copying large files to zvol/ext4.

I think what you missed in your tunning is that filesystem on zvol will have a lot of dirty data accumulated as you have a lot of RAM. It will try to write it back when Linux kernel decides that it has too much dirty pages. You can tune it down to see if it will help with the write consistency, e.g.:
echo $[128 * 1024 * 1024] >/proc/sys/vm/dirty_bytes
This will force the aggressive writeback in the filesystem in zvol.
Try this with the other suggestions I gave before. It made a big difference for writes consistency in my setup.

[devZer0]

this not not a really new to me, i have seen quite some reports on accessing zvols performs much worse compared to accessing ordinary files on zfs datasets - and it confirms my own negative experiences with zvols, which also includes lockups/stalls etc.

this is the reason why i completely avoid using zvols on proxmox for quite a while (which are still default there)

see https://bugzilla.proxmox.com/show_bug.cgi?id=1453 or #10095 for example

[dswartz]

I'm sadly familiar with awful zvol performance. On December 28, 2020, at 7:14 PM, devZer0 <notifications@github.com> wrote: this not not a really new to me, i have seen quite some reports on accessing zvols performs much worse compared to accessing ordinary files on zfs datasets - and it confirms my own negative experiences with zvols, which also includes lockups/stalls etc. this is the reason why i completely avoid using zvols on proxmox for quite a while (which are still default there) see https://bugzilla.proxmox.com/show_bug.cgi?id=1453 or #10095 for example — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#11407 (comment)", "url": "#11407 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[sempervictus]

ZVOLs are pretty pathologically broken by design. A block device is a range of raw bits, full stop. A zvol is an abstraction presenting a range but mapping them across noncontiguous space with deterministic logic involved in linearizing it. So architecturally, it inherently will be slower to resolve a request. The fact that request paths elongate with data written prior to the zvol amplifies and exacerbates the poor design.
Atop that, zvols are almost never considered when changes are introduced, with performance regression after regression going into zfs for years and maintainers never having time or interest in the feature. You can find prior issues on this where we've discussed how the entire pipeline isn't even optimized for ssd, much less nvme and the expected-by-design iowait hampers not only the dmu but multiplicatively the zvols atop it. I've got benchmarks from a few years back showing drastic reduction in throughput after a zvol is filled once (write a g to a g-sized disk zvol and then write to it again, will make you sad).
IMO, zvols need a rethink from the ground up to actually be as thin an abstraction atop the bit ranges handled by the DMU as possible with consistent throughput as a primary design goal and os-native block device interfaces (SG?) to avoid problems like your have today if you map SCST atop a zvol and beat it up with 100 random writers. Problem is, that's a lot of work for a shrinking number of consumers because businesses have moved away from using iscsi zvols for one (iscsi in general but zvols are now known as garbage for business use), and because other tech like ceph actually keeps up with hardware development and optimizes for modern storage busses and media (they dropped zfs as a backing store way back).
Unless the powers that be put their weight behind making zvols a primary member of the ecosystem again, we'll keep seeing issues like this every year. Thanks for filing this one, I'm frankly tired of begging for this to be resolved. Since @ryao left these discussions there's been no real improvement, and in the end the removal of the sg layer may have actually made performance worse (~0.6.4). I tried to force-sync the virtual devices way back to make writes more consistent from db workloads, but really they just need to behave like proper disks with the full range of scheduling parameters and commit/read semantic adjustments from the consumer side and a much thinner/faster underlying implementation. Anyone got a really good storage dev with time on their hands and a hefty budget to fund the work? Semper Victus would be open to joining others from the community to fund a bounty project to un... this mess, if maintainers agree that zvol performance will be a primary consideration when adopting new features or merging commits so the money and effort aren't wasted when draid hits a tag or whatever. If there are any takers, we'd even consider trading an engagement for a completed PR + $0.01 to bind a contract (anyone who contracts red or blue teams knows the cost associated) - feel free to reach out if this sounds appealing and we'll work out a scope.

[sempervictus]

@Binarus: for use under Qemu/libvirt, we've found that the most consistent throughput is achieved by directly mapping the ZVOL to the VM as a virtio-scsi device a la

      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap' detect_zeroes='off'/>
      <source dev='/dev/zvol/<pool_name>/<path_to_volume>' index='1'/>

This removes a lot of the intermediate buffers/copies, but still works without the ZVOL having proper SG interfaces. It allows the ZFS pipeline to deal with zeroes/compression natively, with AIO (io=native) between the visor and block device.
Still a hack, not a solution, and still experiences the write amp and other issues, but at least thins out the visor interaction so you're not seeing amplification of amplifications.
You may also want to revert 5731140 which i had intended to help with the performance degradation resulting from interactions with the linux-tier write merge and subsequent rewrites of the merged blocks in the ZVOL. If you have a lot of async writes, it should raise the peak performance, but beware of degradation over time and potentially deeper valleys.

[prgwiz]

The following settings helped us: options zfs zfs_arc_min=8589934592 options zfs zfs_arc_max=106300440567 options zfs zfs_prefetch_disable=1 options zfs zfs_nocacheflush=1 options zfs zfs_arc_meta_limit_percent=95

…

On Sun, Dec 27, 2020 at 4:08 AM Binarus ***@***.***> wrote: System information Type | Version/Name Linux | Debian Distribution Name | debian Distribution Version | buster (10.7) Linux Kernel | 4.19.0-12-amd64 Architecture | amd64 ZFS Version | OpenZFS 2.0.0 SPL Version | (SPL integrated in OpenZFS 2.0.0) Describe the problem you're observing Setup - Supermicro X10DRU-i+ - LSI 9361-8i connected to a 6 / 12 Gbps SATA/SAS backplane - 2 x Seagate ST4000NM000A (denoted sda and sdb), connected to the backplane - 1 x Seagate ST4000NM0035 (denoted sdc), connected to the backplane - 128 GB RAM (ECC, of course) sda and sdb make up a mirrored ZFS VDEV. The O/S boots from this VDEV. There is only one pool, called rpool. rpool does not contain any other VDEVs besides that mirror. The root file system is mounted to rpool/system . There is no swap file on that system (yet). rpool has been created using the following command: zpool create -o ashift=12 -o altroot=/mnt -O acltype=posixacl -O canmount=off -O checksum=on -O compression=off -O mountpoint=none -O sharesmb=off -O sharenfs=off -O xattr=sa rpool mirror /dev/disk/by-id/ata-ST4000...-part1 /dev/disk/by-id/ata-ST4000...-part1 That is, the pool and the VDEV have ashift=12. rpool/system has been created using the following command: zfs create -o aclinherit=passthrough -o acltype=posixacl -o atime=on -o canmount=on -o checksum=on -o compression=off -o mountpoint=/ -o overlay=off -o primarycache=all -o redundant_metadata=all -o relatime=off -o secondarycache=none -o setuid=on -o sharesmb=off -o sharenfs=off -o logbias=latency -o snapdev=hidden -o snapdir=hidden -o sync=standard -o xattr=sa -o casesensitivity=sensitive -o normalization=none -o utf8only=off rpool/system We further have created a ZVOL using the following command: zfs create -b 4096 -o checksum=on -o compression=off -o primarycache=metadata -o redundant_metadata=all -o secondarycache=none -o logbias=latency -o snapdev=hidden -o sync=standard -V 100G rpool/zvol-test That zvol is mounted on /blob. sdc contains a partition with a normal ext4 file system which is mounted on /mnt. That file system just contains several dozens of ISO files (average size about 6 GB). On that machine, nothing else runs than the standard services the distribution installs. Notably, there is no VM running and nothing else which could produce substantial workload. In this state, when starting watch -n 1 zpool iostat -ylv 1 1 and watching it for a while, there is indeed nearly no load on the ZFS disks. Once in several seconds or so some kilobytes hit the VDEV, which is expected. Copying to the dataset (not the ZVOL): No problem Now we open the iostats in one terminal window (watch -n 1 zpool iostat -ylv 1 1) and start to copy ISO files from sdc onto the ZFS dataset rpool/system in another terminal window (rsync --progress /mnt/*.iso ~/test, where ~/test is part of the root file system and thus is on rpool/system). While the copy runs, rsync shows a few drops in bandwidth every now and then, but there is no real stalling, and the drops are short. Likewise, the iostats show that the two disks in the VDEV are hit with data rates which could be expected. The changes in disk load reported by zpool iostat are surprisingly high, though (constantly jumping between something like 30 MB/s and 300 MB/s), but there are no real stalls either. In summary, the copy in average runs with over 100 MB/s and does not stall. We have interrupted that test after 30 GB or so because we didn't expect anything new from letting it run longer. However, we repeated it several times, each time copying another part of the ISO files, and each time rebooting before. The behavior was the same each time. Copying to the ZVOL: Problem When we do exactly the same thing, but copy to the ZVOL instead of the dataset (rsync --progress /mnt/*.iso /blob), the situation changes. rsync shows the copy running with roughly about 190 MB/s, then it stalls. Thereafter it continues with copying for a few seconds at the rate denoted above, and stalls again after a few seconds, and so on. The problem is that stalling lasts for a long time where absolutely nothing happens, up to several minutes (!). However, the zpool iostat shows that the two ZFS disks are under heavy load during these holdups, constantly (more or less) being hit with over 100 MB/s. Even when we interrupt copying by hitting Ctrl-c in the terminal window where rsync runs, this high load lasts for several minutes until everything returns to normal. There must be extreme write amplification somewhere. I estimate that roughly 10 times more data than the payload actually is hits the disks. For example, if I copy 40 GB that way, this would normally take about 5 minutes; it actually takes about an hour, though, while the disks are under heavy load. For that reason, ZVOLs are currently just not usable for us, which imposes a major problem. What could be going on there? My own thoughts and what we have tried already: At first, we'd like to stress again that the ZVOL test did not happen within a VM. The problem is definitely not due to QEMU or (para)virtualization of data transfer. Secondly, I am aware that it might not be the best idea to have ZFS running on disks which are attached to a RAID controller like the LSI 9361-8i, or to have it running on hybrid disks like the ones we have. However, we have configured that controller to JBOD mode, and the O/S sees the disks as individual ones as expected. But the ultimate key point regarding possible hardware problems is that copying large amounts of data to the ZFS dataset (rpool/system) works as expected. If the problems with the ZVOL would be due to hardware, we would have the same problems with the dataset; this is not the case, though. Thirdly, the problem is not due to ZFS versions. Debian buster comes with ZoL 0.7.12, and we originally have noticed the problem there. We desperately need ZVOLs working, so we have installed OpenZFS 2.0.0 on that machine, which did change exactly nothing with respect to that problem. As a further test, we created the ZVOL with volblocksize=512 and did the tests again. Again, nothing changed. We repeated the process with volblocksizes of 8192, 16384 and 128k. Again, no luck: Maybe it stalled a few seconds earlier or later, longer or shorter in each test compared to the others, but the general situation remained the same. Between the stalls, the copy ran a few seconds with expected speed, then it stalled for a lot of seconds, mostly even a few minutes while iostat was showing a constant data rate of roughly 100 MB/s for each disk, and so on. After interrupting the copy, both ZFS disks continued to be hit with a data rate of 100 MB/s or more for several minutes. Then we tested the ZVOL with sync=disabled. That didn't change anything. The same goes for primarycache=all (instead of metadata) (but at least this was expected). Next, we thought that it may have something to do with the physical sector size of the ZFS disks being 512 bytes, while the pool (and the VDEVs) had ashift=12. Therefore, we destroyed the pool, re-created it with ashift=9, re-created all file systems / datasets as described above, and did all tests again. Once again, this didn't change anything. We then went back to the original pool with ashift=12 and used it for the further tests. At this point, we were out of ideas what to do next, so we read about the ZFS I/O scheduler <https://gist.github.com/szaydel/6244302> and tested a large number of combinations of zfs_dirty_data_max, zfs_delay_scale, zfs_vdev_async_write_max_active, zfs_vdev_async_write_min_active, zfs_vdev_async_write_active_max_dirty_percent, and zfs_vdev_async_write_active_min_dirty_percent. To our surprise, the last five of these did barely influence the behavior. However, the first one (zfs_dirty_data_max), which originally was set to 4 GB, changed the situation when we set it to a low value, e.g. 512 MB. The improvement was that there were less long-lasting holdups: there were even more holdups, but all of them were so short that it became acceptable. However, the average data rate did not increase, because now the transfer rate rsync reported was limited to about 30 MB/s, mostly hanging around 10 MB/s or 20 MB/s. There were no phases with high data rates any more. So the copying was more "responsive" with low values of zfs_dirty_data_max, but that didn't help because the data rate per se was drastically limited. In summary, changing the I/O scheduler parameters which are explained in the document linked above did not lead to anywhere. The last thing we were looking into was zfs_txg_timeout. Setting it to a lower value didn't improve the situation with copying to the ZVOL (but increased the load which hit the ZFS disks when the system was completely idle). Setting it to a higher value didn't improve copying either (but reduced the load on the ZFS disks when the system was idle). Now we are completely out of ideas. We probably could look into other parameters of the ZFS module (/sys/module/zfs/parameters) or the disk drivers (/sys/block/sdx). But this would be just wild guessing and a waste of time. Therefore, we are hoping that somebody is willing to give us some hints. What we did not try, and why not zfs_arc_max is set to 4 GB on that system, and we did not test larger values for the following reasons: 1. That parameter is about reading, not writing, and the copy source is an ext4 partition of a physical disk, so no ZFS parameter would have any effect on the copy source. 2. We clearly have a problem with writing here, not with reading (remember that copying to the normal dataset (not the ZVOL) works normally). 3. When I began working with ZFS some years ago, the first thing I had to solve was a system which started normally, but then became totally unresponsive and finally totally locked up within minutes. The cause of that problem was that ZFS was eating up all available RAM for its ARC cache until the machine crashed or hung. Since then, I always limit the ARC size (and never ever had any stability issues or crashes with ZFS again). 4. Our goal is to run a bunch of VMs with ZVOL storage (the tests described above are just, eehm, tests before we put even more effort into switching completely to ZFS). The number of VMs and the memory they will be given is precisely known. It would not make any sense to test larger ARC sizes, because the ARC size at the end of the day couldn't be much larger than 4 GB. We did not try to use a secondary cache (L2ARC). Again, the copy source is not on ZFS, and therefore this wouldn't make any sense, and furthermore, we have a writing problem here, not a reading problem. We did not try to use an SLOG. This would not make any sense, because one of our tests was to set sync=disabled on the copy destination ZVOL, and this did not change the slightest bit in the behavior observed. Therefore, we know that our problem is not due to sync writes, and thus, an SLOG wouldn't improve the behavior. Describe how to reproduce the problem Install a system similar to the one described above, issue the commands described above, and watch the long-lasting stalls in the terminal windows where rsync runs and the weird behavior zpool iostat reveals in the other terminal window. Since it is not easy to setup a system like ours, we are willing to give remote access to one of that systems if somebody would be interested in investigating the problem. In this case, please leave a comment, stating how we can get into contact. Include any warning/errors/backtraces from the system logs If somebody tells us what exactly is needed here, we'll immediately do it :-). We guess zpool iostat or other tools produce output which is more valuable than the log files, but neither being Linux nor ZFS experts, we are a bit lost here. Notably, we don't know how to operate dtrace or strace properly. If somebody tells us what to do, we'll try our best. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11407>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTQJT6ODFQNY33S4ANCZRTSW32PRANCNFSM4VKUQMWQ> .