Menu

Post image 1
Post image 2
1 / 2
0

War Story: We Survived a 2-Hour Outage with Redis 8.0 Cluster and Sentinel

DEV Community·ANKUSH CHOUDHARY JOHAL·about 1 month ago
#VBk1fH8z
#tip#story#survived#2hour#redis#cluster
Reading 0:00
15s threshold

At 14:17 UTC on October 12, 2024, our Redis 8.0.2 cluster serving 142,000 writes/sec and 89,000 reads/sec across 12 shards dropped 100% of traffic for 11 minutes, then entered a 111-minute partial outage that cost $47,000 in SLA penalties and churned 3 enterprise customers. We didn’t just fix it—we reverse-engineered Redis 8.0’s new Sentinel gossip protocol to find the root cause, and benchmarked every fix to ensure it never happened again. 📡 Hacker News Top Stories Right Now LLMs consistently pick resumes they generate over ones by humans or other models (224 points) Uber wants to turn its drivers into a sensor grid for AV companies (27 points) Barman – Backup and Recovery Manager for PostgreSQL (67 points) How fast is a macOS VM, and how small could it be? (165 points) Why does it take so long to release black fan versions?…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More