Closed
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Void Linux |
| Distribution Version | rolling |
| Linux Kernel | 5.10.9 |
| Architecture | x86_64 |
| ZFS Version | ZFS: Loaded module v2.0.1-1 |
| SPL Version | ZFS: Loaded module v2.0.1-1 |
Describe the problem you're observing
In normal usage (watching a movie, browser and other applications in the background, including torrent client potentially reading from the same file) on a simple layout (single zpool, zpool with a single NVME drive), I got a panic in the ZFS kernel module (shown below).
Looks a bit like #2932, and for what it's worth I also have xattrs=sa and acltype=posixacl.
It seems to have mostly locked up the device, since after this I got multiple timeout warnings in dmesg (also shown below).
With my normal shell (fish) I was unable to even launch commands (it failed to lock its history file, for example), since it relies on filesystem access quite a bit, and apparently some specific accesses were failing. I eventually launched dash, but it got to a point where ls in a directory backed by ZFS, such as ~/ and / hung (though I could still interrupt it with Ctrl-C). Calling sync also hung. The interface for some applications simply locked up (thunderbird and qutebrowser), and others that were working still couldn't actually be quit.
As a last ditch, I tried SysRq. After sysrq S and U (sync and remount ro), I tried E and finally I (SIGTERM and SIGKILL all processes). Since I use runit, SIGTERM'ing all processes already tried to shutdown the system, but that simply hung completely. I had to resort to SysRq B (unconditional reboot) for the system to close up.
The dmesg was dumped into my EFI partition, which could be accessed without any issues at all.
Describe how to reproduce the problem
I don't know for sure what caused it. qbittorrent might have been accessing a file that mpv was accessing as well.
Include any warning/errors/backtraces from the system logs
Panic
[132515.566980] VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (36028797018963967 < 32768)
[132515.566988] PANIC at zio.c:314:zio_data_buf_alloc()
[132515.566990] Showing stack for process 10663
[132515.566993] CPU: 8 PID: 10663 Comm: qbittorrent Tainted: P S U W O 5.10.9_1 #1
[132515.566995] Hardware name: Dell Inc. XPS 15 9570/0D0T05, BIOS 1.15.0 12/25/2019
[132515.566995] Call Trace:
[132515.567007] dump_stack+0x6b/0x83
[132515.567014] spl_panic+0xd4/0xfc [spl]
[132515.567020] ? spl_kmem_cache_alloc+0x74/0x7e0 [spl]
[132515.567026] ? kmem_cache_alloc+0xed/0x1f0
[132515.567031] ? spl_kmem_cache_alloc+0x97/0x7e0 [spl]
[132515.567100] ? aggsum_add+0x153/0x170 [zfs]
[132515.567106] ? mutex_lock+0xe/0x30
[132515.567148] ? aggsum_add+0x153/0x170 [zfs]
[132515.567203] zio_data_buf_alloc+0x55/0x60 [zfs]
[132515.567249] abd_alloc_linear+0x8a/0xc0 [zfs]
[132515.567295] arc_hdr_alloc_abd+0xdf/0x1e0 [zfs]
[132515.567340] arc_hdr_alloc+0x104/0x170 [zfs]
[132515.567387] arc_alloc_buf+0x46/0x150 [zfs]
[132515.567434] dbuf_hold_copy.constprop.0+0x31/0xa0 [zfs]
[132515.567480] dbuf_hold_impl+0x476/0x650 [zfs]
[132515.567525] dbuf_hold_level+0x2b/0x60 [zfs]
[132515.567568] dmu_tx_check_ioerr+0x35/0xd0 [zfs]
[132515.567593] dmu_tx_count_write+0xe5/0x1a0 [zfs]
[132515.567617] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
[132515.567657] zfs_write+0x3ed/0xc60 [zfs]
[132515.567661] ? reweight_entity+0x11e/0x130
[132515.567664] ? ttwu_do_wakeup.constprop.0+0x12/0xd0
[132515.567684] zpl_iter_write+0x109/0x190 [zfs]
[132515.567687] do_iter_readv_writev+0x152/0x1b0
[132515.567690] do_iter_write+0x7c/0x1b0
[132515.567691] vfs_writev+0xa4/0x140
[132515.567694] ? nvme_process_cq+0x159/0x1f0
[132515.567696] ? __x64_sys_futex+0x13d/0x1c0
[132515.567698] __x64_sys_pwritev+0xad/0xf0
[132515.567701] do_syscall_64+0x33/0x40
[132515.567702] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[132515.567705] RIP: 0033:0x7ff818be5d53
[132515.567707] Code: c3 8b 07 85 c0 75 24 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> e9 07 10 ff ff 48 31 ed 48 89 e7 48 8d 35 5a 80 03 00 48 83 e4
[132515.567708] RSP: 002b:00007ff7e32ed5a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000128
[132515.567710] RAX: ffffffffffffffda RBX: 00007ff7e32efb20 RCX: 00007ff818be5d53
[132515.567711] RDX: 0000000000000010 RSI: 00007ff7e32ed610 RDI: 000000000000003a
[132515.567712] RBP: 00007ff7e32ed5e0 R08: 0000000000000000 R09: 0000000000000000
[132515.567713] R10: 00000000e67ab093 R11: 0000000000000246 R12: 0000000000000128
[132515.567713] R13: 000000000000003a R14: 00007ff7e32ed610 R15: 0000000000000000
Timeouts
[132683.240777] INFO: task z_wr_int:524 blocked for more than 122 seconds.
[132683.240786] Tainted: P S U W O 5.10.9_1 #1
[132683.240788] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132683.240791] task:z_wr_int state:D stack: 0 pid: 524 ppid: 2 flags:0x00004000
[132683.240796] Call Trace:
[132683.240807] __schedule+0x249/0x770
[132683.240812] schedule+0x46/0xb0
[132683.240817] schedule_preempt_disabled+0xa/0x10
[132683.240821] __mutex_lock.constprop.0+0x236/0x470
[132683.240888] ? dsl_dataset_block_born+0x23f/0x380 [zfs]
[132683.240937] dbuf_write_done+0x3c/0x200 [zfs]
[132683.240983] arc_write_done+0x8f/0x420 [zfs]
[132683.241043] zio_done+0x403/0x1120 [zfs]
[132683.241104] zio_execute+0x81/0x120 [zfs]
[132683.241112] taskq_thread+0x2da/0x520 [spl]
[132683.241119] ? wake_up_q+0xa0/0xa0
[132683.241178] ? zio_vdev_io_done+0x1f0/0x1f0 [zfs]
[132683.241184] ? taskq_thread_spawn+0x50/0x50 [spl]
[132683.241190] kthread+0x11b/0x140
[132683.241195] ? __kthread_bind_mask+0x60/0x60
[132683.241200] ret_from_fork+0x22/0x30
[132683.241213] INFO: task txg_sync:623 blocked for more than 122 seconds.
[132683.241217] Tainted: P S U W O 5.10.9_1 #1
[132683.241218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132683.241221] task:txg_sync state:D stack: 0 pid: 623 ppid: 2 flags:0x00004000
[132683.241224] Call Trace:
[132683.241229] __schedule+0x249/0x770
[132683.241233] schedule+0x46/0xb0
[132683.241238] schedule_timeout+0x8b/0x140
[132683.241243] ? __next_timer_interrupt+0x100/0x100
[132683.241247] io_schedule_timeout+0x4c/0x80
[132683.241252] __cv_timedwait_common+0x11e/0x160 [spl]
[132683.241256] ? add_wait_queue_exclusive+0x70/0x70
[132683.241262] __cv_timedwait_io+0x15/0x20 [spl]
[132683.241321] zio_wait+0x129/0x2b0 [zfs]
[132683.241387] dsl_pool_sync+0xfe/0x4e0 [zfs]
[132683.241457] spa_sync+0x56e/0xf90 [zfs]
[132683.241464] ? mutex_lock+0xe/0x30
[132683.241543] ? spa_txg_history_init_io+0x101/0x110 [zfs]
[132683.241630] txg_sync_thread+0x2d4/0x4a0 [zfs]
[132683.241704] ? txg_fini+0x250/0x250 [zfs]
[132683.241714] thread_generic_wrapper+0x6f/0x80 [spl]
[132683.241724] ? __thread_exit+0x20/0x20 [spl]
[132683.241734] kthread+0x11b/0x140
[132683.241742] ? __kthread_bind_mask+0x60/0x60
[132683.241749] ret_from_fork+0x22/0x30
[132683.241759] INFO: task vdev_autotrim:641 blocked for more than 122 seconds.
[132683.241767] Tainted: P S U W O 5.10.9_1 #1
[132683.241775] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132683.241783] task:vdev_autotrim state:D stack: 0 pid: 641 ppid: 2 flags:0x00004000
[132683.241793] Call Trace:
[132683.241802] __schedule+0x249/0x770
[132683.241810] ? __wake_up_common_lock+0x8a/0xc0
[132683.241817] schedule+0x46/0xb0
[132683.241825] io_schedule+0x42/0x70
[132683.241834] cv_wait_common+0xa1/0x120 [spl]
[132683.241837] ? add_wait_queue_exclusive+0x70/0x70
[132683.241906] txg_wait_synced_impl+0xc3/0x100 [zfs]
[132683.241975] txg_wait_synced+0xc/0x40 [zfs]
[132683.242047] metaslab_enable+0xbd/0x100 [zfs]
[132683.242133] vdev_autotrim_thread+0x553/0x790 [zfs]
[132683.242199] ? vdev_trim_l2arc_thread+0x350/0x350 [zfs]
[132683.242211] thread_generic_wrapper+0x6f/0x80 [spl]
[132683.242221] ? __thread_exit+0x20/0x20 [spl]
[132683.242230] kthread+0x11b/0x140
[132683.242237] ? __kthread_bind_mask+0x60/0x60
[132683.242241] ret_from_fork+0x22/0x30
[132683.242367] INFO: task qbittorrent:10663 blocked for more than 122 seconds.
[132683.242370] Tainted: P S U W O 5.10.9_1 #1
[132683.242372] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132683.242375] task:qbittorrent state:D stack: 0 pid:10663 ppid: 1 flags:0x00000000
[132683.242378] Call Trace:
[132683.242383] __schedule+0x249/0x770
[132683.242389] schedule+0x46/0xb0
[132683.242394] spl_panic+0xfa/0xfc [spl]
[132683.242402] ? spl_kmem_cache_alloc+0x74/0x7e0 [spl]
[132683.242407] ? kmem_cache_alloc+0xed/0x1f0
[132683.242415] ? spl_kmem_cache_alloc+0x97/0x7e0 [spl]
[132683.242492] ? aggsum_add+0x153/0x170 [zfs]
[132683.242513] ? mutex_lock+0xe/0x30
[132683.242558] ? aggsum_add+0x153/0x170 [zfs]
[132683.242621] zio_data_buf_alloc+0x55/0x60 [zfs]
[132683.242669] abd_alloc_linear+0x8a/0xc0 [zfs]
[132683.242717] arc_hdr_alloc_abd+0xdf/0x1e0 [zfs]
[132683.242765] arc_hdr_alloc+0x104/0x170 [zfs]
[132683.242813] arc_alloc_buf+0x46/0x150 [zfs]
[132683.242864] dbuf_hold_copy.constprop.0+0x31/0xa0 [zfs]
[132683.242915] dbuf_hold_impl+0x476/0x650 [zfs]
[132683.242963] dbuf_hold_level+0x2b/0x60 [zfs]
[132683.243017] dmu_tx_check_ioerr+0x35/0xd0 [zfs]
[132683.243072] dmu_tx_count_write+0xe5/0x1a0 [zfs]
[132683.243132] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
[132683.243193] zfs_write+0x3ed/0xc60 [zfs]
[132683.243200] ? reweight_entity+0x11e/0x130
[132683.243205] ? ttwu_do_wakeup.constprop.0+0x12/0xd0
[132683.243250] zpl_iter_write+0x109/0x190 [zfs]
[132683.243257] do_iter_readv_writev+0x152/0x1b0
[132683.243261] do_iter_write+0x7c/0x1b0
[132683.243266] vfs_writev+0xa4/0x140
[132683.243271] ? nvme_process_cq+0x159/0x1f0
[132683.243276] ? __x64_sys_futex+0x13d/0x1c0
[132683.243280] __x64_sys_pwritev+0xad/0xf0
[132683.243285] do_syscall_64+0x33/0x40
[132683.243288] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[132683.243293] RIP: 0033:0x7ff818be5d53
[132683.243295] RSP: 002b:00007ff7e32ed5a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000128
[132683.243299] RAX: ffffffffffffffda RBX: 00007ff7e32efb20 RCX: 00007ff818be5d53
[132683.243301] RDX: 0000000000000010 RSI: 00007ff7e32ed610 RDI: 000000000000003a
[132683.243303] RBP: 00007ff7e32ed5e0 R08: 0000000000000000 R09: 0000000000000000
[132683.243305] R10: 00000000e67ab093 R11: 0000000000000246 R12: 0000000000000128
[132683.243307] R13: 000000000000003a R14: 00007ff7e32ed610 R15: 0000000000000000
[132806.120104] INFO: task z_wr_int:524 blocked for more than 245 seconds.
[132806.120109] Tainted: P S U W O 5.10.9_1 #1
[132806.120111] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132806.120113] task:z_wr_int state:D stack: 0 pid: 524 ppid: 2 flags:0x00004000
[132806.120118] Call Trace:
[132806.120126] __schedule+0x249/0x770
[132806.120131] schedule+0x46/0xb0
[132806.120134] schedule_preempt_disabled+0xa/0x10
[132806.120137] __mutex_lock.constprop.0+0x236/0x470
[132806.120202] ? dsl_dataset_block_born+0x23f/0x380 [zfs]
[132806.120256] dbuf_write_done+0x3c/0x200 [zfs]
[132806.120314] arc_write_done+0x8f/0x420 [zfs]
[132806.120373] zio_done+0x403/0x1120 [zfs]
[132806.120444] zio_execute+0x81/0x120 [zfs]
[132806.120452] taskq_thread+0x2da/0x520 [spl]
[132806.120458] ? wake_up_q+0xa0/0xa0
[132806.120523] ? zio_vdev_io_done+0x1f0/0x1f0 [zfs]
[132806.120529] ? taskq_thread_spawn+0x50/0x50 [spl]
[132806.120533] kthread+0x11b/0x140
[132806.120536] ? __kthread_bind_mask+0x60/0x60
[132806.120539] ret_from_fork+0x22/0x30
[132806.120549] INFO: task txg_sync:623 blocked for more than 245 seconds.
[132806.120551] Tainted: P S U W O 5.10.9_1 #1
[132806.120553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132806.120554] task:txg_sync state:D stack: 0 pid: 623 ppid: 2 flags:0x00004000
[132806.120557] Call Trace:
[132806.120561] __schedule+0x249/0x770
[132806.120563] schedule+0x46/0xb0
[132806.120567] schedule_timeout+0x8b/0x140
[132806.120570] ? __next_timer_interrupt+0x100/0x100
[132806.120572] io_schedule_timeout+0x4c/0x80
[132806.120576] __cv_timedwait_common+0x11e/0x160 [spl]
[132806.120579] ? add_wait_queue_exclusive+0x70/0x70
[132806.120582] __cv_timedwait_io+0x15/0x20 [spl]
[132806.120619] zio_wait+0x129/0x2b0 [zfs]
[132806.120662] dsl_pool_sync+0xfe/0x4e0 [zfs]
[132806.120726] spa_sync+0x56e/0xf90 [zfs]
[132806.120731] ? mutex_lock+0xe/0x30
[132806.120796] ? spa_txg_history_init_io+0x101/0x110 [zfs]
[132806.120862] txg_sync_thread+0x2d4/0x4a0 [zfs]
[132806.120923] ? txg_fini+0x250/0x250 [zfs]
[132806.120928] thread_generic_wrapper+0x6f/0x80 [spl]
[132806.120932] ? __thread_exit+0x20/0x20 [spl]
[132806.120936] kthread+0x11b/0x140
[132806.120939] ? __kthread_bind_mask+0x60/0x60
[132806.120941] ret_from_fork+0x22/0x30
[132806.120945] INFO: task vdev_autotrim:641 blocked for more than 245 seconds.
[132806.120948] Tainted: P S U W O 5.10.9_1 #1
[132806.120949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132806.120950] task:vdev_autotrim state:D stack: 0 pid: 641 ppid: 2 flags:0x00004000
[132806.120953] Call Trace:
[132806.120956] __schedule+0x249/0x770
[132806.120958] ? __wake_up_common_lock+0x8a/0xc0
[132806.120961] schedule+0x46/0xb0
[132806.120963] io_schedule+0x42/0x70
[132806.120966] cv_wait_common+0xa1/0x120 [spl]
[132806.120969] ? add_wait_queue_exclusive+0x70/0x70
[132806.121015] txg_wait_synced_impl+0xc3/0x100 [zfs]
[132806.121071] txg_wait_synced+0xc/0x40 [zfs]
[132806.121135] metaslab_enable+0xbd/0x100 [zfs]
[132806.121204] vdev_autotrim_thread+0x553/0x790 [zfs]
[132806.121264] ? vdev_trim_l2arc_thread+0x350/0x350 [zfs]
[132806.121272] thread_generic_wrapper+0x6f/0x80 [spl]
[132806.121279] ? __thread_exit+0x20/0x20 [spl]
[132806.121286] kthread+0x11b/0x140
[132806.121292] ? __kthread_bind_mask+0x60/0x60
[132806.121297] ret_from_fork+0x22/0x30
[132806.121398] INFO: task qbittorrent:10663 blocked for more than 245 seconds.
[132806.121400] Tainted: P S U W O 5.10.9_1 #1
[132806.121402] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132806.121404] task:qbittorrent state:D stack: 0 pid:10663 ppid: 1 flags:0x00000000
[132806.121406] Call Trace:
[132806.121410] __schedule+0x249/0x770
[132806.121413] schedule+0x46/0xb0
[132806.121417] spl_panic+0xfa/0xfc [spl]
[132806.121423] ? spl_kmem_cache_alloc+0x74/0x7e0 [spl]
[132806.121426] ? kmem_cache_alloc+0xed/0x1f0
[132806.121430] ? spl_kmem_cache_alloc+0x97/0x7e0 [spl]
[132806.121459] ? aggsum_add+0x153/0x170 [zfs]
[132806.121462] ? mutex_lock+0xe/0x30
[132806.121508] ? aggsum_add+0x153/0x170 [zfs]
[132806.121568] zio_data_buf_alloc+0x55/0x60 [zfs]
[132806.121617] abd_alloc_linear+0x8a/0xc0 [zfs]
[132806.121671] arc_hdr_alloc_abd+0xdf/0x1e0 [zfs]
[132806.121720] arc_hdr_alloc+0x104/0x170 [zfs]
[132806.121752] arc_alloc_buf+0x46/0x150 [zfs]
[132806.121785] dbuf_hold_copy.constprop.0+0x31/0xa0 [zfs]
[132806.121818] dbuf_hold_impl+0x476/0x650 [zfs]
[132806.121851] dbuf_hold_level+0x2b/0x60 [zfs]
[132806.121907] dmu_tx_check_ioerr+0x35/0xd0 [zfs]
[132806.121961] dmu_tx_count_write+0xe5/0x1a0 [zfs]
[132806.122000] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
[132806.122057] zfs_write+0x3ed/0xc60 [zfs]
[132806.122075] ? reweight_entity+0x11e/0x130
[132806.122080] ? ttwu_do_wakeup.constprop.0+0x12/0xd0
[132806.122114] zpl_iter_write+0x109/0x190 [zfs]
[132806.122124] do_iter_readv_writev+0x152/0x1b0
[132806.122130] do_iter_write+0x7c/0x1b0
[132806.122135] vfs_writev+0xa4/0x140
[132806.122139] ? nvme_process_cq+0x159/0x1f0
[132806.122142] ? __x64_sys_futex+0x13d/0x1c0
[132806.122144] __x64_sys_pwritev+0xad/0xf0
[132806.122148] do_syscall_64+0x33/0x40
[132806.122150] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[132806.122153] RIP: 0033:0x7ff818be5d53
[132806.122155] RSP: 002b:00007ff7e32ed5a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000128
[132806.122163] RAX: ffffffffffffffda RBX: 00007ff7e32efb20 RCX: 00007ff818be5d53
[132806.122168] RDX: 0000000000000010 RSI: 00007ff7e32ed610 RDI: 000000000000003a
[132806.122170] RBP: 00007ff7e32ed5e0 R08: 0000000000000000 R09: 0000000000000000
[132806.122172] R10: 00000000e67ab093 R11: 0000000000000246 R12: 0000000000000128
[132806.122173] R13: 000000000000003a R14: 00007ff7e32ed610 R15: 0000000000000000
[132928.999727] INFO: task z_wr_int:524 blocked for more than 368 seconds.
[132928.999735] Tainted: P S U W O 5.10.9_1 #1
[132928.999738] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132928.999741] task:z_wr_int state:D stack: 0 pid: 524 ppid: 2 flags:0x00004000
[132928.999748] Call Trace:
[132928.999760] __schedule+0x249/0x770
[132928.999767] schedule+0x46/0xb0
[132928.999772] schedule_preempt_disabled+0xa/0x10
[132928.999777] __mutex_lock.constprop.0+0x236/0x470
[132928.999858] ? dsl_dataset_block_born+0x23f/0x380 [zfs]
[132928.999917] dbuf_write_done+0x3c/0x200 [zfs]
[132928.999973] arc_write_done+0x8f/0x420 [zfs]
[132929.000047] zio_done+0x403/0x1120 [zfs]
[132929.000121] zio_execute+0x81/0x120 [zfs]
[132929.000130] taskq_thread+0x2da/0x520 [spl]
[132929.000138] ? wake_up_q+0xa0/0xa0
[132929.000211] ? zio_vdev_io_done+0x1f0/0x1f0 [zfs]
[132929.000219] ? taskq_thread_spawn+0x50/0x50 [spl]
[132929.000225] kthread+0x11b/0x140
[132929.000231] ? __kthread_bind_mask+0x60/0x60
[132929.000237] ret_from_fork+0x22/0x30
[132929.000253] INFO: task vdev_autotrim:641 blocked for more than 368 seconds.
[132929.000257] Tainted: P S U W O 5.10.9_1 #1
[132929.000260] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[132929.000263] task:vdev_autotrim state:D stack: 0 pid: 641 ppid: 2 flags:0x00004000
[132929.000267] Call Trace:
[132929.000275] __schedule+0x249/0x770
[132929.000281] ? __wake_up_common_lock+0x8a/0xc0
[132929.000286] schedule+0x46/0xb0
[132929.000292] io_schedule+0x42/0x70
[132929.000301] cv_wait_common+0xa1/0x120 [spl]
[132929.000306] ? add_wait_queue_exclusive+0x70/0x70
[132929.000423] txg_wait_synced_impl+0xc3/0x100 [zfs]
[132929.000510] txg_wait_synced+0xc/0x40 [zfs]
[132929.000623] metaslab_enable+0xbd/0x100 [zfs]
[132929.000734] vdev_autotrim_thread+0x553/0x790 [zfs]
[132929.000837] ? vdev_trim_l2arc_thread+0x350/0x350 [zfs]
[132929.000856] thread_generic_wrapper+0x6f/0x80 [spl]
[132929.000868] ? __thread_exit+0x20/0x20 [spl]
[132929.000880] kthread+0x11b/0x140
[132929.000890] ? __kthread_bind_mask+0x60/0x60
[132929.000899] ret_from_fork+0x22/0x30