Closed
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | arch |
| Distribution Version | n/a |
| Kernel Version | 5.19.13-arch1-1 |
| Architecture | x86_64 |
| OpenZFS Version | 2.1.6-1 |
Describe the problem you're observing
With an existing pool with ashift 12:
y@arnold % zpool get ashift tank
NAME PROPERTY VALUE SOURCE
tank ashift 12 local
I create a new vdev with cryptsetup luksFormat /dev/sdw, with cryptsetup version 2.6.1
y@arnold % cryptsetup -V
cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI
This results in a luks (luks2) device (used as a vdev) with a sector size of 4096. Note that /dev/sdw is the underlying device (a 14 TB hard drive), and /dev/mapper/z14.3 is the cryptsetup device using sdw.
y@arnold % sudo blockdev --report /dev/sdw
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 14000519643136 /dev/sdw
y@arnold % sudo blockdev --report /dev/mapper/z14.3
RO RA SSZ BSZ StartSec Size Device
rw 256 4096 4096 0 14000502865920 /dev/mapper/z14.3
y@arnold % sudo dmsetup table | grep z14.3
z14.3: 0 27344732160 crypt aes-xts-plain64 :64:logon:cryptsetup:e5b69a87-e02c-4b14-be58-6109488e172a-d0 0 65:96 32768 2 allow_discards sector_size:4096
y@arnold console % sudo cryptsetup luksDump /dev/sdw
LUKS header information
Version: 2
Epoch: 3
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: e5b69a87-e02c-4b14-be58-6109488e172a
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 4096 [bytes]
Keyslots:
0: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2id
Time cost: 4
Memory: 1048576
Threads: 4
Salt: xxxx
AF stripes: 4000
AF hash: sha256
Area offset:32768 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
Digests:
0: pbkdf2
Hash: sha256
Iterations: 67632
Salt: xxxx
Digest: xxxxx
y@arnold console %
I then add it to my existing pool with zpool replace tank z4.2 z14.3. Eventually (before the replace/resliver completes), zio reports errors and the vdev is considered failed
y@arnold ~ % zpool status tank | grep -E '(z14.3|NAME)'
NAME STATE READ WRITE CKSUM
z14.3 FAULTED 0 20 0 too many errors
y@arnold ~ % journalctl -f | grep zio
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3772528418816 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3706396516352 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3790784475136 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3980643770368 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3338247639040 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3602675863552 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3504724443136 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3495903948800 size=4096 flags=1808aa
Feb 25 14:52:19 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3628836184064 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447128170496 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447125979136 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447126523904 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447126519808 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447126515712 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447130460160 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447139495936 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=2447147290624 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3011713654784 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3011713769472 size=4096 flags=1808aa
Feb 25 14:56:17 arnold kernel: zio pool=tank vdev=/dev/mapper/z14.3 error=5 type=2 offset=3011714031616 size=4096 flags=1808aa
Next, after detaching the vdev from the pool and cryptsetup close, use cryptsetup luksFormat --type luks1 instead (To force the use of luks1 instead of a luks2 header).
This results in a vdev with 512 byte sectors:
y@arnold ~ % sudo cryptsetup luksDump /dev/sdw
LUKS header information for /dev/sdw
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: xxxx
MK salt: xxxx
MK iterations: 70468
UUID: e5b69a87-e02c-4b14-be58-6109488e172a
Key Slot 0: ENABLED
Iterations: 1034098
Salt: xxxx
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
y@arnold ~ % sudo dmsetup table | grep z14.3
z14.3: 0 27344760832 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 65:96 4096 1 allow_discards
y@arnold ~ % sudo blockdev --report /dev/mapper/z14.3
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 14000517545984 /dev/mapper/z14.3
With this vdev (luks1, 512B sectors), no zio errors are observed and the zpool replace completes successfully.