Postmortem: How a Corrupted Node Modules Folder Caused 3-Hour Outage for Our CI Pipeline Published: October 26, 2024 | Author: DevOps Team | 5 min read Executive Summary On October 24, 2024, our team experienced a 3-hour, 12-minute outage of our primary CI/CD pipeline, impacting 47 active pull requests and delaying 3 production releases. The root cause was identified as a corrupted node_modules directory on our shared CI runner, triggered by an incomplete npm install process during a high-concurrency job spike. Timeline of Events All times are in UTC: 14:02 – CI pipeline begins failing for all new jobs with MODULE_NOT_FOUND errors for core dependencies like express and jest . 14:07 – On-call engineer acknowledges alert, starts investigating failed job logs. 14:15 – Initial hypothesis: Recent dependency update to @company/utils v2.1.0 broke compatibility. Rollback attempt fails. 14:28 – Engineer notices node_modules on the primary shared runner has 0-byte files for 12+ dependencies.…