IplanRio2026in production
GCP cost investigation and Billing Guard
Diagnosing a Cloud Storage cost anomaly and building an automatic containment mechanism for APIs with abnormal usage.
R$ 128k
in projected annual savings
- 2.5×
- gap between bytes read and bytes billed, explained
- 1/h
- cap on automatic shutoffs
The problem
After a Google pricing change, one project's Cloud Storage transfer fee went from cents to more than R$ 10,000 a month. None of the internal hypotheses matched the numbers.
The solution
I cross-referenced billing data with BigQuery job history across several projects and found the cause, dbt runs with 23 models in parallel reading external Parquet in a multi-region bucket from a single region. I also explained why billed bytes were 2.5 times what BigQuery reported, since the Storage Read API reads entire row groups. Google's product team confirmed the explanation.
Technical notes
Billing Guard, in Go on Cloud Run, receives cost anomalies through Pub/Sub. During business hours it notifies the team on Discord, and outside them it confirms live traffic in Cloud Monitoring before disabling the API.
Separate IAM roles. The service can only disable, and only a person can re-enable.
Deduplication and concurrency control in Firestore, a limit of one shutoff per hour, protected projects and an allowlist of services, from which GKE is excluded.
Outcome
The fix projects R$ 128.9k a year in savings, and Billing Guard covers every project in the organization.