Cloud costs

IplanRio2026in production

GCP cost investigation and Billing Guard

Diagnosing a Cloud Storage cost anomaly and building an automatic containment mechanism for APIs with abnormal usage.

R$ 128k

in projected annual savings

2.5×
gap between bytes read and bytes billed, explained
1/h
cap on automatic shutoffs
real data

From cents to more than R$ 10k a month

After a Google pricing change, one project's transfer fee shot up. I found the cause, and the fix projects R$ 128.9k a year in savings.

cents
Before the pricing change
more than R$ 10k a month
After the change

R$ 128.9k

a year in projected savings from the fix

Bars not to exact scale, for comparison only. Figures from the investigation report.

The problem

After a Google pricing change, one project's Cloud Storage transfer fee went from cents to more than R$ 10,000 a month. None of the internal hypotheses matched the numbers.

The solution

I cross-referenced billing data with BigQuery job history across several projects and found the cause, dbt runs with 23 models in parallel reading external Parquet in a multi-region bucket from a single region. I also explained why billed bytes were 2.5 times what BigQuery reported, since the Storage Read API reads entire row groups. Google's product team confirmed the explanation.

Technical notes

  1. Billing Guard, in Go on Cloud Run, receives cost anomalies through Pub/Sub. During business hours it notifies the team on Discord, and outside them it confirms live traffic in Cloud Monitoring before disabling the API.

  2. Separate IAM roles. The service can only disable, and only a person can re-enable.

  3. Deduplication and concurrency control in Firestore, a limit of one shutoff per hour, protected projects and an allowlist of services, from which GKE is excluded.

Outcome

The fix projects R$ 128.9k a year in savings, and Billing Guard covers every project in the organization.