Menu

Post image 1
Post image 2
Post image 3
Post image 4
1 / 4
0

Drop Traits: The Day We Stopped Restarting Pods Every 8 Hours

DEV Community·speed engineer·29 days ago
#d2AaHtyd
#pattern#when#memory#file#drop#cleanup
Reading 0:00
15s threshold

Or: how we learned that “eventually” isn’t good enough when you’re bleeding file descriptors Drop Traits: The Day We Stopped Restarting Pods Every 8 Hours Or: how we learned that “eventually” isn’t good enough when you’re bleeding file descriptors Deterministic cleanup means knowing exactly when resources are freed — the difference between memory chaos and predictable system behavior in production environments. So our video transcoding service was… how do I put this delicately… a complete disaster. Not in the “everything’s on fire” way. More like the “slow leak that nobody wants to admit is a real problem” way. We were processing 2.4 million videos daily, which sounds impressive until you realize we had to restart every single pod every 8 hours or it would just… die. Memory would start at a reasonable 2GB per pod. Then climb. And climb. And by hour 7, we’d be sitting at 14GB and sweating, watching the graphs, waiting for the OOM killer to show up like an unwelcome dinner guest.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More