Skip to content

perf(redis): Upstash bandwidth 30× (~300–500 GB/day vs 17 GB/day baseline) — #4499 removed the CDN shield from /api/bootstrap; every dashboard boot reads the full registry from Redis #5249

Description

@koala73

Symptom

Upstash Redis bandwidth exploded from the 4-month baseline of ≤500 GB/month (~17 GB/day) to ~300–520 GB/day (console: Wed ~520 GB, Thu ~500 GB, Fri ~490 GB; weekend dips ~180–250 GB). The monthly plan allowance was exceeded mid-month. Daily commands run 45–80M/day.

Attribution (measured, not inferred)

Two MONITOR taps on the production instance (390 s + 300 s windows, Sun Jul 12 ~20:45–21:00 +04), joined with per-key STRLEN:

  • GET response bytes extrapolate to ~306 GB/day at Sunday-evening rate (weekday rate ≈ chart's 450–520 GB/day). Request-side bytes are only ~9 GB/day — this is a read-egress problem.
  • ~97% of GET bytes come from the bootstrap key registry, read at a uniform ~0.6–0.9 GET/s per key across the whole inventory — i.e. full-registry reads per dashboard boot/refresh, at visitor rate (~1.2 full sweeps/sec):
GB/day (Sun rate) key size
88 wildfire:fires:v1 1.36 MB
36 conflict:ucdp-events:v1 662 KB
22 energy:pipelines:{oil,gas}:v1 216/189 KB
21 thermal:escalation:v1 396 KB
16 market:commodities-bootstrap:v1 205 KB
11 forecast:predictions:v2 170 KB
10 news:digest:v1:full:en 207 KB
~100 ~35 more registry keys, 13–110 KB each

Gateway telemetry (Axiom wm_api_usage) is flat across the window (0.8–1.3M req/day, no surge) — /api/bootstrap and /api/health are not instrumented there, which is why this was invisible in route dashboards.

Root cause

PR #4499 / commit 7a20a4d5f (merged Jun 28), "restrict bootstrap shared caching"successCacheHeaders() now returns Cache-Control: no-store for every successful /api/bootstrap response except the anonymous single-key ?keys=weatherAlerts probe (api/bootstrap.js:400-406).

Because the browser runtime sends bootstrap fetches with default credentials: 'same-origin' (src/services/bootstrap.ts:119) and every visitor carries an anonymous wm-session cookie, hasBootstrapCredentialCookie() disqualifies ~100% of dashboard boots from the public path. Result: the CDN shield (CDN-Cache-Control: public, s-maxage=600/7200) that used to absorb nearly all bootstrap traffic is gone — every boot/refresh now pipelines the full registry (~5–10 MB) straight out of Upstash.

Timeline fits exactly: merged Jun 28 → first full billing month is July → plan exceeded mid-July; weekday/weekend shape matches visitor traffic.

Note: per #4499's own non-goals, the payload is identical for all callers (shared production seed keys; only static transforms like wildfire compaction + enrichmentMeta strip). The hardening was defense-in-depth on headers, not a content-variance fix — so a cacheable anonymous read path is still safe by #4499's own analysis.

Secondary finding (commands, not bytes)

/api/health?compact=1 is polled by every client every 5 min (healthFreshness, src/config/variants/base.ts:46) and is no-store end-to-end since #5008 (Jul 7) reverted the #4908 CDN cache. Each call runs a ~350–400-command sweep (STRLEN × ~200 data keys + GET × ~165 seed-meta keys). Measured: 1.24 sweeps/sec (~107K/day ≈ ~40M Redis commands/day, ~5 GB/day). It dominates command volume but not bytes.

Fix directions

  1. Restore a CDN-cacheable anonymous bootstrap path (primary, returns egress to ~baseline):
  2. Pre-compact heavy dashboard keys at the seedercompactWildfireBootstrapPayload currently trims to 500 detections after reading the full 1.36 MB from Redis; publishing a dashboard-sized key would cut the whale key ~10× even before caching (same pattern candidates: conflict:ucdp-events:v1, thermal:escalation:v1).
  3. Memoize the /api/health verdict (e.g. 30–60 s server-side snapshot key) so the ~370-command sweep runs once per TTL instead of once per poller; keeps fix(health): no-store on the compact status response so uptime monitors never read a stale verdict #5008's no-store monitor semantics (verdict staleness bounded by TTL).

Measurement artifacts

MONITOR aggregations (command mix, per-key byte attribution, client IPs) captured in session scratchpad; can be re-run with the same tap script if needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority, fix soonbugSomething isn't workingperformancePerformance optimization

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions