It started at 1:49 AM. PagerDuty fired — payments-service entering CrashLoopBackOff, 3 replicas simultaneously. On-call engineer paged. I joined the incident bridge 4 minutes later. By 2:36 AM, we had the fix deployed. 47 minutes of debugging for a 2-line YAML change. This is the postmortem. Not of the incident itself — those exist internally — but of the investigation . Every wrong turn, every wasted minute, and the exact signals that eventually cracked it. ## The First 10 Minutes: The Obvious Wrong Answer When pods crash simultaneously right after a deployment, the deployment is guilty until proven innocent. That's the right instinct most of the time. So the first 10 minutes were spent here: kubectl rollout history deployment/payments-service -n production kubectl describe deployment/payments-service -n production Enter fullscreen mode Exit fullscreen mode The last deployment had gone out at 7:52 PM — over 6 hours earlier. The pods had been healthy for 6 hours since that deploy.…