L2ARC failed to rebuild after reboot

[aaronjwood]

System information

Type	Version/Name
Distribution Name	Debian (Proxmox)
Distribution Version	10
Linux Kernel	5.4.103-1-pve
Architecture	x64
ZFS Version	2.0.3-pve2
SPL Version	2.0.3-pve2

Describe the problem you're observing

Had a system up and running for months. L2ARC was a few hundred GB. Rebooted the system and found that the L2ARC was empty.

Describe how to reproduce the problem

Not sure about this. I've rebooted in the past and had my L2ARC preserved. I haven't been able to find any logs around this either. If anyone has suggestions on what I can check I'd be happy to post it here. FWIW /sys/module/zfs/parameters/l2arc_rebuild_enabled is set to 1.

One difference from the last reboot I did where it worked: I had a kernel update this time. Is there some special handling that forces the L2ARC to not rebuild when a new kernel version is used or something?

[gamanakis]

Is there some special handling that forces the L2ARC to not rebuild when a new kernel version is used or something?

No, that is not the case. Curious though what caused this. If it happens again could you post a "cat /proc/spl/kstat/zfs/arcstats" after it fails to rebuild the L2ARC contents? That might give us a clue.

You can also look in zpool history to see if there is anything L2ARC related.

[aaronjwood]

Is there some special handling that forces the L2ARC to not rebuild when a new kernel version is used or something?

No, that is not the case. Curious though what caused this. If it happens again could you post a "cat /proc/spl/kstat/zfs/arcstats" after it fails to rebuild the L2ARC contents? That might give us a clue.

You can also look in zpool history to see if there is anything L2ARC related.

Yeah, will do. Haven't been able to reproduce it yet. I don't see anything relevant in zpool history unfortunately.

[aaronjwood]

Still haven't reproduced this yet but saw something that I thought I should share:

...
l2_rebuild_success              4    0
l2_rebuild_unsupported          4    0
l2_rebuild_io_errors            4    0
l2_rebuild_dh_errors            4    1
l2_rebuild_cksum_lb_errors      4    0
l2_rebuild_lowmem               4    0
l2_rebuild_size                 4    0
l2_rebuild_asize                4    0
l2_rebuild_bufs                 4    0
l2_rebuild_bufs_precached       4    0
l2_rebuild_log_blks             4    0
...

Note the l2_rebuild_dh_errors field. I'm not clear on what this field means or if it's relevant to the issue I hit.

[gamanakis]

This means the header of the L2ARC device (used for rebuilding its contents) was corrupted for some reason.

…

On Mon, Apr 26, 2021, 5:42 AM Aaron Wood ***@***.***> wrote: Still haven't reproduced this yet but saw something that I thought I should share: ... l2_rebuild_success 4 0 l2_rebuild_unsupported 4 0 l2_rebuild_io_errors 4 0 l2_rebuild_dh_errors 4 1 l2_rebuild_cksum_lb_errors 4 0 l2_rebuild_lowmem 4 0 l2_rebuild_size 4 0 l2_rebuild_asize 4 0 l2_rebuild_bufs 4 0 l2_rebuild_bufs_precached 4 0 l2_rebuild_log_blks 4 0 ... Note the l2_rebuild_dh_errors field. I'm not clear on what this field means or if it's relevant to the issue I hit. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11787 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2Y2IPKSKYGVFGQSPUBJ63TKTOLJANCNFSM4ZWCG4YA> .

[aaronjwood]

This would cause the behavior I saw, right? Is there anything else within ZFS that I can look at that might indicate why this header got corrupted?

[aaronjwood]

I just had this happen to me again last night due to a power outage. Once my server came back I saw my L2 ARC was empty again. FWIW l2_rebuild_dh_errors is now sitting at 3.

Anything else I can check here to get more information on why I keep failing to rebuild?

[gamanakis]

Try a "zdb -lll /dev/cache device" . It seems the header of the L2ARC device is corrupted and that is why ZFS cannot read it. Only way to restore normal operation would be removing and re-adding the cache device.

…

On Tue, May 4, 2021, 5:10 PM Aaron Wood ***@***.***> wrote: I just had this happen to me again last night due to a power outage. Once my server came back I saw my L2 ARC was empty again. FWIW l2_rebuild_dh_errors is now sitting at 3. Anything else I can check here to get more information on why I keep failing to rebuild? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11787 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2Y2IPHVHIXX66U3SVP7ILTMAE5PANCNFSM4ZWCG4YA> .

[aaronjwood]

zdb -lll /dev/nvme0n1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

I can try removing and adding it back. It's interesting you say that because I had done that (for another reason) after upgrading to ZFS 2.0.0. I'll do it again right now and see if I can reproduce the issue again.

[aaronjwood]

Even after readding the device:

zdb -lll /dev/nvme0n1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

Also tried with the ID which is how I actually add it to the pool:

zdb -lll /dev/disk/by-id/nvme-HP_SSD_EX950_2TB_HBSE59340600764
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

[gamanakis]

This tells us the whole label of the cache device is corrupted and not only the header that is required for persistent L2ARC. I have never seen this before, either in production or in local VM testing.

…

On Tue, May 4, 2021, 7:50 PM Aaron Wood ***@***.***> wrote: zdb -lll /dev/nvme0n1 failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3 I can try removing and adding it back. It's interesting you say that because I had done that (for another reason) after upgrading to ZFS 2.0.0. I'll do it again right now and see if I can reproduce the issue again. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11787 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2Y2IMYPRFQMKWTJSXCC73TMAXWFANCNFSM4ZWCG4YA> .

[gamanakis]

Are you sure there is no underlying hardware problem? You can try zeroing out manually the label and header of the removed device with "zpool labelclear" and then adding it again.

…

On Tue, May 4, 2021, 7:56 PM Georgios Amanakis ***@***.***> wrote: This tells us the whole label of the cache device is corrupted and not only the header that is required for persistent L2ARC. I have never seen this before, either in production or in local VM testing. On Tue, May 4, 2021, 7:50 PM Aaron Wood ***@***.***> wrote: > zdb -lll /dev/nvme0n1 > failed to unpack label 0 > failed to unpack label 1 > failed to unpack label 2 > failed to unpack label 3 > > I can try removing and adding it back. It's interesting you say that > because I had done that (for another reason) after upgrading to ZFS 2.0.0. > I'll do it again right now and see if I can reproduce the issue again. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#11787 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB2Y2IMYPRFQMKWTJSXCC73TMAXWFANCNFSM4ZWCG4YA> > . >

[aaronjwood]

Hah, wow! Guess I am lucky :) Some more info for context:

The drive seems to work fine as is, regardless of being used as an L2ARC or not. Here's more data about the drive:

=== START OF INFORMATION SECTION ===
Model Number:                       HP SSD EX950 2TB
Serial Number:                      HBSE59340600764
Firmware Version:                   SS0411B
PCI Vendor/Subsystem ID:            0x126f
IEEE OUI Identifier:                0x000000
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,073,818,112 [2.00 TB]
Namespace 1 Utilization:            27,793,907,712 [27.7 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Tue May  4 10:59:34 2021 PDT
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0
 1 +     4.60W       -        -    1  1  1  1        0       0
 2 +     3.80W       -        -    2  2  2  2        0       0
 3 -   0.0450W       -        -    3  3  3  3     2000    2000
 4 -   0.0040W       -        -    4  4  4  4    15000   15000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        30 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    2%
Data Units Read:                    100,762,616 [51.5 TB]
Data Units Written:                 81,736,109 [41.8 TB]
Host Read Commands:                 842,433,926
Host Write Commands:                1,132,098,062
Controller Busy Time:               21,608
Power Cycles:                       262
Power On Hours:                     7,994
Unsafe Shutdowns:                   103
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Disk /dev/nvme0n1: 1.8 TiB, 2000073818112 bytes, 3906394176 sectors
Disk model: HP SSD EX950 2TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 755545AB-39E5-8747-989A-8216460E3F6E

Device              Start        End    Sectors  Size Type
/dev/nvme0n1p1       2048 3906377727 3906375680  1.8T Solaris /usr & Apple ZFS
/dev/nvme0n1p9 3906377728 3906394111      16384    8M Solaris reserved 1

ZFS is the one managing the label, partition, etc. of the entire disk right?

I'll try your suggestion above and post back in a bit.

[aaronjwood]

zpool remove vault nvme-HP_SSD_EX950_2TB_HBSE59340600764
zpool labelclear nvme-HP_SSD_EX950_2TB_HBSE59340600764
zpool add vault cache nvme-HP_SSD_EX950_2TB_HBSE59340600764
zdb -lll /dev/disk/by-id/nvme-HP_SSD_EX950_2TB_HBSE59340600764
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

Is ZFS corrupting this?