The disk that filled itself TL;DR : my homelab box hit 100 percent disk full out of nowhere. I deleted half the things I could find, df still said full, du said I had plenty of space. Turned out the disk was holding on to files I had already deleted, because a long-running process still had them open. lsof +L1 was the magic. A service restart was the fix. So there I was, on a perfectly normal evening, ssh'd into the homelab box because something had stopped responding. The first thing I check on any "why is this dying" run is df -h , almost as a reflex. Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p2 450G 448G 2G 100% / Enter fullscreen mode Exit fullscreen mode Cool. So that is why nothing is working. I have a deal with this box. It runs my self-hosted things, it does not ask for much, and once a quarter or so I prune some old container images and we move on. So I went straight to the usual cleanup playbook, mildly annoyed that I had let it fill up.…