Mutex deadlocks in production: the patterns I found in my codebase and how I diagnosed them It was 11:47 PM and the service wasn't responding. No panic. No error in the logs. Railway showed the container alive, memory stable, CPU at zero. Zero. That's what caught my attention: zero activity on a service that should've been processing queues. I opened a tokio-console session and there it was — four tasks suspended, all waiting on the same MutexGuard . None of them were ever going to move. That was the first one. Then came the second. Then the third. All three with the same face: total silence, container "healthy," and a chain of locks that was never going to resolve itself. My thesis is this: mutex deadlocks in async Rust aren't rare or mysterious. They're predictable. They follow patterns. And once you've seen one, you recognize them from a mile away. The problem is that most resources teach you what a deadlock is, not how you diagnose it when it's already in production and you can't just pull a backtrace.…