Menu

Post image 1
Post image 2
1 / 2
0

Postmortem: How a Terraform 1.10 Bug Deleted Our Entire Staging Environment 2 Days Before Launch

DEV Community·ANKUSH CHOUDHARY JOHAL·28 days ago
#ViaJyb1t
#code#terraform#postmortem#deleted#state#echo
Reading 0:00
15s threshold

At 14:17 UTC on October 15, 2024, a single terraform apply run on Terraform 1.10.0 destroyed 94% of our staging environment’s infrastructure 48 hours before our Series B launch demo. We lost 12 RDS instances, 47 ECS services, 3 Redis clusters, and 112 S3 buckets in 11 minutes flat. No, it wasn’t a misconfigured state file. It was a regression in Terraform core’s resource graph pruning logic. 🔴 Live Ecosystem Stats ⭐ hashicorp/terraform — 48,319 stars, 10,333 forks Data pulled live from GitHub and npm. 📡 Hacker News Top Stories Right Now How OpenAI delivers low-latency voice AI at scale (118 points) I am worried about Bun (307 points) Securing a DoD contractor: Finding a multi-tenant authorization vulnerability (135 points) Talking to strangers at the gym (954 points) Formatting a 25M-line codebase overnight (51 points) Key Insights Terraform 1.10.0’s resource graph pruning incorrectly marks 1 in 8 dependent resources for deletion when using nested modules with dynamic for_each loops.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More