What Causes XFS Metadata Corruption on a Talos Node?

[tz-torchai]

Under what circumstances would a node experience XFS metadata corruption like this? Could this be related to configuring Kubernetes storage on the disk with Talos installation?

07/04/2025 07:51:28 eth0: renamed from tmpb812a 

08/04/2025 02:15:59 eth0: renamed from tmpac93b 

08/04/2025 02:16:06 XFS (nvme0n1p6): Corruption warning: Metadata has LSN (1030439791:1836084325) ahead of current LSN (437:65648). Please unmount and run xfs_repair (>= v4.3) to resolve. 

08/04/2025 02:16:06 XFS (nvme0n1p6): Metadata CRC error detected at xfs_refcountbt_read_verify+0x16/0xb0, xfs_refcountbt block 0x30  

08/04/2025 02:16:06 XFS (nvme0n1p6): Unmount and run xfs_repair 

08/04/2025 02:16:06 XFS (nvme0n1p6): First 128 bytes of corrupted metadata buffer: 

08/04/2025 02:16:06 00000000: 78 30 30 30 30 30 35 65 62 33 61 32 65 33 36 39  x000005eb3a2e369 

08/04/2025 02:16:06 00000010: 38 2c 20 73 74 61 74 65 3d 6b 43 6f 6d 70 6c 65  8, state=kComple 

08/04/2025 02:16:06 00000020: 74 65 29 3a 20 4c 65 61 64 65 72 20 73 74 65 70  te): Leader step 

08/04/2025 02:16:06 00000030: 20 64 6f 77 6e 20 64 6f 6e 65 20 61 74 74 65 6d   down done attem 

08/04/2025 02:16:06 00000040: 70 74 3d 31 2c 20 6c 65 61 64 65 72 5f 75 75 69  pt=1, leader_uui 

08/04/2025 02:16:06 00000050: 64 3d 37 33 35 35 62 35 64 33 30 32 36 66 34 30  d=7355b5d3026f40 

08/04/2025 02:16:06 00000060: 64 62 62 33 33 64 39 61 65 64 35 36 30 32 33 32  dbb33d9aed560232 

08/04/2025 02:16:06 00000070: 38 36 2c 20 63 68 61 6e 67 65 5f 75 75 69 64 3d  86, change_uuid= 

08/04/2025 02:16:06 XFS (nvme0n1p6): metadata I/O error in "xfs_btree_read_buf_block+0xa1/0x120" at daddr 0x30 len 8 error 74 

08/04/2025 02:16:06 XFS (nvme0n1p6): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x159/0x2b0 (fs/xfs/xfs_trans_buf.c:296).  Shutting down filesystem. 

08/04/2025 02:16:06 XFS (nvme0n1p6): Please unmount the filesystem and rectify the problem(s) 

08/04/2025 02:16:20 [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeStatusController", "error": "stat /etc/kubernetes/kubeconfig-kubelet: input/output error"} 

... repeating
 

Talos is installed on nvme0n1.

[smira]

There are two common cases:

• bad hardware
• bug in xfs code

For the second one - none that we are aware of in latest versions of Talos.

As for the bad hardware - it certainly happens. Talos should automatically try xfs repair on next mount (e.g. reboot the machine).