Messing with a messy NVMe

Today, I’ve been trying to resolve a strange error that just started showing up on one of my Linux devices – specifically, errors in the dmesg output on boot, during runtime, and at shutdown:

[17411.181979] nvme0n1: I/O Cmd(0x2) @ LBA 587247848, 8 blocks, I/O Error (sct 0x2 / sc 0x81)
[17411.181991] critical medium error, dev nvme0n1, sector 587247848 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[17774.406908] nvme_log_error: 12 callbacks suppressed
[17774.406924] nvme0n1: I/O Cmd(0x2) @ LBA 633393408, 8 blocks, I/O Error (sct 0x2 / sc 0x81)
[17774.406937] blk_print_req_error: 12 callbacks suppressed
[17774.406940] critical medium error, dev nvme0n1, sector 633393408 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[17774.590170] nvme0n1: I/O Cmd(0x2) @ LBA 633393440, 8 blocks, I/O Error (sct 0x2 / sc 0x81)
[...]
[21832.814513] nvme0n1: I/O Cmd(0x2) @ LBA 1145110912, 8 blocks, I/O Error (sct 0x2 / sc 0x81)
[21832.814544] critical medium error, dev nvme0n1, sector 1145110912 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0

Needless to say, this doesn’t look happy; but, fsck is not showing errors. This is a Debian install, and I chose to install the nvme-cli and also the smartmontools packages; util-linux and e2fsprogs were installed by default and contain fsck and also the badblocks tool; blktool and hdparm are potentially also useful. The Arch Wiki seems to be particularly helpful on the nvme tools.

I’m currently booted from an alternate source (an SD card) and attempting to run fsck -fcDy /dev/nvme0n1p1 in an effort to look for bad blocks. I’m still seeing these I/O error messages scroll up the dmesg -w output in a second terminal even though fsck is not obviously telling me that there are problems. Based on my research I’m starting to think that this might be a hardware issue with the drive. This is a bit of an unusual setup for me (ARM64, uBoot is in play with eMMC as well) but fortunately the system is recent, so a clean rebuild is not out of the question…

Fairly sure that I’ll end up getting another M.2 NVMe drive (at least they are relatively cheaper than they used to be), but writing this down for now as I continue to investigate. Fun times…