Menu

Post image 1
Post image 2
Post image 3
Post image 4
Post image 5
Post image 6
1 / 6
0

Stop Paying a Streaming Bus to Carry Bytes That Live for Ninety Seconds

DEV Community: java·Samar Prakash·2 days ago
#uYhrwThE
#dev#write#file#bytes#consumer#article
Reading 0:00
15s threshold

How a shared filesystem became the cheapest, fastest outbox I've ever built — and why FSx for OpenZFS is the version of that idea that finally scales I was staring at an AWS bill last quarter where a single Kinesis Data Streams line item was costing more than the entire S3 footprint sitting behind it. The events on that stream had a useful lifetime of about ninety seconds. They were written by one service, read by another, processed, and dropped. We were paying full streaming-bus price for bytes that barely outlived a TCP timeout. That bill is what got me thinking about transitional data as a category that deserves its own architecture, and about why every "use the right tool" instinct I had — Kinesis, Kafka, MSK — was the wrong tool for this particular shape of work. The right tool, it turns out, is a filesystem. Specifically, AWS FSx for OpenZFS, used as an outbox between producers and consumers, with only a tiny pointer message traveling through whatever messaging bus you already have.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More