In a single-machine system, an operation happens once or it fails. In a distributed system, an operation can happen once, fail, succeed twice, or end in an ambiguous state that nobody can cleanly observe. The network is allowed to drop the request on the way in, drop the response on the way out, time out in the middle, or silently deliver the same message twice. Any service that expects perfect delivery semantics from a network it does not own is a service that will eventually corrupt its own data. The mitigation is retry. The client did not get a response, so the client tries again. This is correct behavior, and it is the only behavior that keeps distributed systems running. The problem is that if the original request actually succeeded on the server — if the ambiguity was a lost response, not a lost request — the retry creates a duplicate. A second payment. A second email. A second row in the ledger. Idempotency is what turns retry from a foot-gun into a safety mechanism.…