Menu

Post image 1
Post image 2
1 / 2
0

How to Optimize BigQuery Costs (Real Techniques That Work)

DEV Community·Arunkumar Amaran·27 days ago
#2wsUPyqf
Reading 0:00
15s threshold

I want to start with a number: $140,000 . That's how much a single misconfigured scheduled query cost one of the teams I worked with over a three-month period. Not a rogue machine learning job. Not a massive ETL pipeline. A scheduled query. Running every 15 minutes. Scanning a 4TB table every time. For 90 days before anyone looked at the bill closely enough to notice. Nobody wrote that query with malicious intent. The engineer who wrote it was experienced. The table had been small when the query was first created. Then the table grew. The query didn't know that. The bill did. I've spent the better part of the last decade building and optimizing data platforms on GCP — including BigQuery at enterprise scale at Macy's Systems & Technology. This post is what I wish someone had handed me early on. Not the official Google documentation version of cost optimization. The actual production version, including the mistakes that end up in the bill.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More