Frequent Checksum Errors During Scrub on ZFS Pool

[KoffeinKaio]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	12
Kernel Version	6.10.3-amd64
Architecture	amd64
OpenZFS Version	zfs-2.2.5-1

Describe the problem you're observing

chksum errors on scrub - in rare cases none, in most cases 3-4 on diffrent drives. Sometimes not even the same drives.
No smart errors logged.

Changed:
RAM to ECC RAM
new PSU
connected 3 drives of the pool to another PCIe Sata card
changed all cables to the drives

Problem war originally a Q&A discussion - but since im out of options on what could cause it - maybe a bug somewhere?
#16445

Describe how to reproduce the problem

run a scrub. zpool scrub Data

Include any warning/errors/backtraces from the system logs

root@pve:~# zpool status -v
  pool: Data
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub in progress since Thu Aug 15 17:15:59 2024
	35.6T / 84.1T scanned at 14.4G/s, 3.92T / 84.1T issued at 1.58G/s
	516K repaired, 4.66% done, 14:24:32 to go
config:

	NAME                                   STATE     READ WRITE CKSUM
	Data                                   ONLINE       0     0     0
	  raidz2-0                             ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT4WHGT  ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT5NTBG  ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT5MER8  ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT0SCR9  ONLINE       0     0     1  (repairing)
	    ata-ST18000NM003D-3DL103_ZVT07MLA  ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT9JNZR  ONLINE       0     0     0
	    ata-ST18000NM003D-3DL103_ZVT9C7NM  ONLINE       0     0     1  (repairing)
	    ata-ST18000NM003D-3DL103_ZVTAEL6D  ONLINE       0     0     1  (repairing)

I dont want to blindly change out the CPU, Board or buy another HBA Controller - any ideas?

[rincebrain]

You said in the discussion you removed vfio and PCIe passthrough and still got checksum errors - that's not super surprising, depending on the origin. The question becomes, I think, whether you still get more on a second scrub on the HV - since, if we hypothesize these are errors on-disk, then yeah, you're going to find them on a scrub after disconnecting, the question becomes whether you find more later.

I would also remark that, from memory, AMD doesn't explicitly promise all the ECC functionality works on consumer boards, that's more of a "we don't stop board vendors from enabling it after testing it, good luck", so it's not impossible that there's still ECC errors that aren't being reported properly, though I wouldn't put that as the most likely answer without more data.

My first question would be what the errors look like - e.g. is it always reporting 0s for the checksums or something, which the events in zpool events -v should know. (You might need a debug build for it to report that, I never remember.)

My first suggestion would be, don't run 6.10 or 6.9 - even 2.2.5 doesn't claim 6.10 support, so let's back off to something earlier that's in the actual supported range and see if this keeps happening. I'd suggest pre-6.4, but just trying to pick something that people have been using for a while without screams.

My second question would be, does this go away if you change the parity or checksum computation to use a different implementation?
(e.g. given:

# grep . /sys/module/{icp,zcommon,zfs}/parameters/{zfs_fletcher_4_impl,icp_aes_impl,icp_gcm_impl,zfs_vdev_raidz_impl} 2>/dev/null
/sys/module/icp/parameters/icp_aes_impl:cycle [fastest] generic x86_64 aesni
/sys/module/icp/parameters/icp_gcm_impl:cycle [fastest] avx generic pclmulqdq
/sys/module/zcommon/parameters/zfs_fletcher_4_impl:[fastest] scalar superscalar superscalar4 sse2 ssse3 avx2
/sys/module/zfs/parameters/zfs_vdev_raidz_impl:cycle [fastest] original scalar sse2 ssse3 avx2

picking something other than "fastest" for zfs_fletcher_4_impl, like superscalar4, which does not use any SIMD instructions, and seeing if the problem stops occurring.

I should note, by nature of doing this, it's going to be substantially slower at checksums, so your scrubs may take longer and much more CPU time, but, if we want to eliminate weird bugs like that, ...)

Intel had a fun bug in Sapphire Rapids chips recently where virtualization played badly with SIMD instructions, so I wouldn't be totally astonished if AMD turned up something strange in the same vein, though I don't know of one offhand.

My third question would be, what exactly is every physical connection step between the disks and any SATA/SAS/USB controllers involved? Because the last time I personally had a weird problem with checksum errors cropping up weirdly and going up over time, it was because something in my controller setup was masking a bunch of write errors, so they only showed up as errors when I tried to actually read them back later, and, naturally, they were not what I had expected. (Moving the disks to a different machine and controller temporarily made this very clear.) That was all a USB->SATA setup, so I don't think it's the specific case here, but just to point out, getting checksum errors back can be what happens if you failed to write something, somehow.

My naive guess, since it's not reporting any uncorrectable errors at the moment, is that somehow, occasionally a write isn't getting to disk on one or two disks, and isn't properly bubbling up an error somehow, but since it's P=2, you've got enough redundancy to recover from it when you notice it later.

[rincebrain]

So, if it were me, I'd make sure I had backups of everything on there (not because of some urgent risk I haven't mentioned, just on principle before experimenting), run two or three scrubs without changing anything else outside of the VM and see if the checksum error count keeps going up each time (the first time wouldn't be surprising, it's the second and third time that are more interesting).

Capture the errors from zpool events -v that crop up when it errors, share them (if there's any filenames in them feel free to delete them, they shouldn't matter). (Note that zpool events -v doesn't persist after reboot, so capture it then.)

If they didn't keep going up, we dig more into why the virtualization is making things odd; if they did, then we dig into why that, probably by me suggesting you do superscalar4 on fletcher_4_impl and doing a few scrubs again, which will likely take a bit.

(And if you have an older kernel than 6.9 to conveniently try, I'd try it after 2 scrubs outside of the VM but before changing any other settings - and then do a few scrubs again in that and see if it turns out differently.)

[rincebrain]

Oh right, I thought I vaguely remembered one.

And it's for Zen 2, isn't that fun. #14557

So maybe a newer microcode would be a good time for you. (You could also test by booting with noxsaves and changing nothing else, and seeing if the problem goes away...)

[KoffeinKaio]

thank you so much for this wall of help!

ill first do a few scrubs and see if the errors increase (still on the HV itself)

zpool events -v: (snip)

Aug 17 2024 11:26:59.489108558 ereport.fs.zfs.checksum
        class = "ereport.fs.zfs.checksum"
        ena = 0xa8643cdeea300401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x46e14f483fbb48c0
                vdev = 0x9c76b56715dbd3b
        (end detector)
        pool = "Data"
        pool_guid = 0x46e14f483fbb48c0
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x9c76b56715dbd3b
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-ST18000NM003D-3DL103_ZVT0SCR9-part1"
        vdev_devid = "ata-ST18000NM003D-3DL103_ZVT0SCR9-part1"
        vdev_ashift = 0x9
        vdev_complete_ts = 0x8a8643ae5e04
        vdev_delta_ts = 0x12b736
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x2
        vdev_delays = 0x0
        parent_guid = 0xf8dd7a25055d838
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x0
        zio_flags = 0x1000b0
        zio_stage = 0x400000
        zio_pipeline = 0x3e00000
        zio_delay = 0x0
        zio_timestamp = 0x0
        zio_delta = 0x0
        zio_priority = 0x4
        zio_offset = 0xbae41798000
        zio_size = 0x2b000
        zio_objset = 0x283
        zio_object = 0x11cdb
        zio_level = 0x0
        zio_blkid = 0xb59
        bad_ranges = 0x560 0x580 
        bad_ranges_min_gap = 0x8
        bad_range_sets = 0x4d 
        bad_range_clears = 0x3b 
        bad_set_bits = 0x92 0x10 0xc 0x25 0x81 0x80 0xaf 0x32 0xe6 0x84 0x21 0x8c 0x23 0x80 0xb9 0x50 0xc0 0x1 0x2 0x0 0xa0 0x3 0x8c 0xcb 0x18 0xc0 0x0 0x3 0x1c 0xb3 0x80 0x21 
        bad_cleared_bits = 0x0 0xc0 0x11 0x90 0x46 0x20 0x50 0x0 0x0 0x0 0xc 0x61 0x4 0xf 0x0 0xa4 0x5 0x72 0x95 0x19 0x44 0x80 0x60 0x0 0x1 0x9 0x28 0x10 0xa2 0xc 0x4c 0xa 
        time = 0x66c06ce3 0x1d27344e 
        eid = 0x17

Aug 17 2024 11:26:59.489108558 ereport.fs.zfs.checksum
        class = "ereport.fs.zfs.checksum"
        ena = 0xa8643ce597102401
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x46e14f483fbb48c0
                vdev = 0xba078544e9c69f80
        (end detector)
        pool = "Data"
        pool_guid = 0x46e14f483fbb48c0
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0xba078544e9c69f80
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/ata-ST18000NM003D-3DL103_ZVTAEL6D-part1"
        vdev_devid = "ata-ST18000NM003D-3DL103_ZVTAEL6D-part1"
        vdev_ashift = 0x9
        vdev_complete_ts = 0x8a8643add8d6
        vdev_delta_ts = 0x124079
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x4
        vdev_delays = 0x0
        parent_guid = 0xf8dd7a25055d838
        parent_type = "raidz"
        vdev_spare_paths = 
        vdev_spare_guids = 
        zio_err = 0x0
        zio_flags = 0x1000b0
        zio_stage = 0x400000
        zio_pipeline = 0x3e00000
        zio_delay = 0x0
        zio_timestamp = 0x0
        zio_delta = 0x0
        zio_priority = 0x4
        zio_offset = 0xbae417ed000
        zio_size = 0x2b000
        zio_objset = 0x283
        zio_object = 0x11cdb
        zio_level = 0x0
        zio_blkid = 0xb5a
        bad_ranges = 0x14560 0x14580 
        bad_ranges_min_gap = 0x8
        bad_range_sets = 0x4b 
        bad_range_clears = 0x39 
        bad_set_bits = 0x14 0xa0 0xe1 0x1 0x30 0x82 0x93 0x49 0xc0 0x20 0x34 0x82 0x41 0x34 0xd0 0x20 0x6d 0x69 0x40 0xa0 0x91 0x44 0x9 0x20 0x9b 0xa0 0x1 0x90 0x58 0x85 0x2 0x40 
        bad_cleared_bits = 0x1 0x14 0x12 0x8c 0x0 0x4 0x4 0x4 0x2c 0x89 0xc1 0x0 0x82 0x0 0x20 0x44 0x0 0x86 0x2 0x1c 0x42 0x0 0x90 0x13 0x24 0x5a 0x18 0x22 0x7 0x2 0x8 0x26 
        time = 0x66c06ce3 0x1d27344e 
        eid = 0x18

if you need more of the log i'll upload it

I'll see if i can get back to kernel 6.4 or if this will break my ARC gpu support.
Microcode update should have already happened, I'll check.

For whats involved between disks and system:

8 seagate x20 going directly to the 8 onboard SATA ports on the ASROCK Phantom Gaming Pro 4, tried diffrent Cables, all new ones

For power: currently one cable from the PSU to 5 disks and another cable to the others with a sata-power-split cable
The previous PSU had enough sata ports to not need a splitter but also errored

Anything i forgot?

PS

I would also remark that, from memory, AMD doesn't explicitly promise all the ECC functionality works on consumer boards, that's more of a "we don't stop board vendors from enabling it after testing it, good luck", so it's not impossible that there's still ECC errors that aren't being reported properly, though I wouldn't put that as the most likely answer without more data.

Oh nice to know. I'll prob. get server HW on the next iteration.

[KoffeinKaio]

multiple scrubs - errorcount keeps rising

will test micocode update and older kernel next

[rincebrain]

Did you try the noxsaves kernel parameter? That'd be my suggestion as most likely to help.

[KoffeinKaio]

Did you try the noxsaves kernel parameter? That'd be my suggestion as most likely to help.

I did just now and after an hour of scrub the first error appeared (I did a zpool clear before).
Should I do a second scrub after to see if they keep appearing?

[rincebrain]

I'd let the first one finish and then run a second one, to be sure - since the problem there is "it screwed up a calculation sometimes", if it is that problem, it's possible these are actual incorrect things written out, the first time.

The second pass is the more interesting question.

[KoffeinKaio]

Second one also errors

Fun thing: now on every disk for the first time

[amotin]

ZFS verifies checksums per block. If the block is not trivially small, it may be difficult to say where the error actually happened.

PS: Rather than running scrubs in a loop collecting errors I would run good long memory test.

[KoffeinKaio]

Did a 10h memory test with the old ram
New ram (unbuf ECC) is new

[KoffeinKaio]

did a bios and with it - microcode update - no changes.

Still working on getting my setup to run on 6.1 or 6.3 kernel.

Should I still change the algorythm zfs uses to calculate checksums? would i have to rewrite the data on the disks?

[AttilaFueloep]

Should I still change the algorythm zfs uses to calculate checksums?

Yes, that would be great to rule out any FPU/SIMD related problems. Assuming you're using fletcher4 checksums and no encryption

echo scalar > /sys/module/zfs/parameters/zfs_fletcher_4_impl
echo scalar > /sys/module/zfs/parameters/zfs_vdev_raidz_impl

should do the trick.

would i have to rewrite the data on the disks?

No, by doing the above you're just changing the implementation of the algorithm used. The algorithm itself is still defined by the datasets checksum property.

[MartenBE]

(unbuf ECC) is new

New memory can have bit errors, happened to me once and your symtoms sound familiar. Don't assume, verify with proof: memtest the new RAM.