You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
perf(redis): Upstash bandwidth 30× (~300–500 GB/day vs 17 GB/day baseline) — #4499 removed the CDN shield from /api/bootstrap; every dashboard boot reads the full registry from Redis #5249
Upstash Redis bandwidth exploded from the 4-month baseline of ≤500 GB/month (~17 GB/day) to ~300–520 GB/day (console: Wed ~520 GB, Thu ~500 GB, Fri ~490 GB; weekend dips ~180–250 GB). The monthly plan allowance was exceeded mid-month. Daily commands run 45–80M/day.
Attribution (measured, not inferred)
Two MONITOR taps on the production instance (390 s + 300 s windows, Sun Jul 12 ~20:45–21:00 +04), joined with per-key STRLEN:
GET response bytes extrapolate to ~306 GB/day at Sunday-evening rate (weekday rate ≈ chart's 450–520 GB/day). Request-side bytes are only ~9 GB/day — this is a read-egress problem.
~97% of GET bytes come from the bootstrap key registry, read at a uniform ~0.6–0.9 GET/s per key across the whole inventory — i.e. full-registry reads per dashboard boot/refresh, at visitor rate (~1.2 full sweeps/sec):
GB/day (Sun rate)
key
size
88
wildfire:fires:v1
1.36 MB
36
conflict:ucdp-events:v1
662 KB
22
energy:pipelines:{oil,gas}:v1
216/189 KB
21
thermal:escalation:v1
396 KB
16
market:commodities-bootstrap:v1
205 KB
11
forecast:predictions:v2
170 KB
10
news:digest:v1:full:en
207 KB
~100
~35 more registry keys, 13–110 KB each
—
Gateway telemetry (Axiom wm_api_usage) is flat across the window (0.8–1.3M req/day, no surge) — /api/bootstrap and /api/health are not instrumented there, which is why this was invisible in route dashboards.
Root cause
PR #4499 / commit 7a20a4d5f (merged Jun 28), "restrict bootstrap shared caching" — successCacheHeaders() now returns Cache-Control: no-store for every successful /api/bootstrap response except the anonymous single-key ?keys=weatherAlerts probe (api/bootstrap.js:400-406).
Because the browser runtime sends bootstrap fetches with default credentials: 'same-origin' (src/services/bootstrap.ts:119) and every visitor carries an anonymous wm-session cookie, hasBootstrapCredentialCookie() disqualifies ~100% of dashboard boots from the public path. Result: the CDN shield (CDN-Cache-Control: public, s-maxage=600/7200) that used to absorb nearly all bootstrap traffic is gone — every boot/refresh now pipelines the full registry (~5–10 MB) straight out of Upstash.
Timeline fits exactly: merged Jun 28 → first full billing month is July → plan exceeded mid-July; weekday/weekend shape matches visitor traffic.
Note: per #4499's own non-goals, the payload is identical for all callers (shared production seed keys; only static transforms like wildfire compaction + enrichmentMeta strip). The hardening was defense-in-depth on headers, not a content-variance fix — so a cacheable anonymous read path is still safe by #4499's own analysis.
Secondary finding (commands, not bytes)
/api/health?compact=1 is polled by every client every 5 min (healthFreshness, src/config/variants/base.ts:46) and is no-store end-to-end since #5008 (Jul 7) reverted the #4908 CDN cache. Each call runs a ~350–400-command sweep (STRLEN × ~200 data keys + GET × ~165 seed-meta keys). Measured: 1.24 sweeps/sec (~107K/day ≈ ~40M Redis commands/day, ~5 GB/day). It dominates command volume but not bytes.
Fix directions
Restore a CDN-cacheable anonymous bootstrap path (primary, returns egress to ~baseline):
Client: send tier bootstrap fetches with credentials: 'omit' (payload needs no cookies), and/or
Pre-compact heavy dashboard keys at the seeder — compactWildfireBootstrapPayload currently trims to 500 detections after reading the full 1.36 MB from Redis; publishing a dashboard-sized key would cut the whale key ~10× even before caching (same pattern candidates: conflict:ucdp-events:v1, thermal:escalation:v1).
MONITOR aggregations (command mix, per-key byte attribution, client IPs) captured in session scratchpad; can be re-run with the same tap script if needed.
Symptom
Upstash Redis bandwidth exploded from the 4-month baseline of ≤500 GB/month (~17 GB/day) to ~300–520 GB/day (console: Wed ~520 GB, Thu ~500 GB, Fri ~490 GB; weekend dips ~180–250 GB). The monthly plan allowance was exceeded mid-month. Daily commands run 45–80M/day.
Attribution (measured, not inferred)
Two
MONITORtaps on the production instance (390 s + 300 s windows, Sun Jul 12 ~20:45–21:00 +04), joined with per-keySTRLEN:wildfire:fires:v1conflict:ucdp-events:v1energy:pipelines:{oil,gas}:v1thermal:escalation:v1market:commodities-bootstrap:v1forecast:predictions:v2news:digest:v1:full:enGateway telemetry (Axiom
wm_api_usage) is flat across the window (0.8–1.3M req/day, no surge) —/api/bootstrapand/api/healthare not instrumented there, which is why this was invisible in route dashboards.Root cause
PR #4499 / commit
7a20a4d5f(merged Jun 28), "restrict bootstrap shared caching" —successCacheHeaders()now returnsCache-Control: no-storefor every successful/api/bootstrapresponse except the anonymous single-key?keys=weatherAlertsprobe (api/bootstrap.js:400-406).Because the browser runtime sends bootstrap fetches with default
credentials: 'same-origin'(src/services/bootstrap.ts:119) and every visitor carries an anonymouswm-sessioncookie,hasBootstrapCredentialCookie()disqualifies ~100% of dashboard boots from the public path. Result: the CDN shield (CDN-Cache-Control: public, s-maxage=600/7200) that used to absorb nearly all bootstrap traffic is gone — every boot/refresh now pipelines the full registry (~5–10 MB) straight out of Upstash.Timeline fits exactly: merged Jun 28 → first full billing month is July → plan exceeded mid-July; weekday/weekend shape matches visitor traffic.
Note: per #4499's own non-goals, the payload is identical for all callers (shared production seed keys; only static transforms like wildfire compaction +
enrichmentMetastrip). The hardening was defense-in-depth on headers, not a content-variance fix — so a cacheable anonymous read path is still safe by #4499's own analysis.Secondary finding (commands, not bytes)
/api/health?compact=1is polled by every client every 5 min (healthFreshness,src/config/variants/base.ts:46) and isno-storeend-to-end since #5008 (Jul 7) reverted the #4908 CDN cache. Each call runs a ~350–400-command sweep (STRLEN × ~200 data keys + GET × ~165 seed-meta keys). Measured: 1.24 sweeps/sec (~107K/day ≈ ~40M Redis commands/day, ~5 GB/day). It dominates command volume but not bytes.Fix directions
credentials: 'omit'(payload needs no cookies), and/orkeys=weatherAlertsto credential-less tier GETs (same public payload for everyone; keepno-storefor API-key-authenticated calls and all failures, preserving fix(api): restrict bootstrap shared caching to anonymous weather bootstrap #4495/fix(api): restrict bootstrap shared caching #4499 intent).compactWildfireBootstrapPayloadcurrently trims to 500 detections after reading the full 1.36 MB from Redis; publishing a dashboard-sized key would cut the whale key ~10× even before caching (same pattern candidates:conflict:ucdp-events:v1,thermal:escalation:v1)./api/healthverdict (e.g. 30–60 s server-side snapshot key) so the ~370-command sweep runs once per TTL instead of once per poller; keeps fix(health): no-store on the compact status response so uptime monitors never read a stale verdict #5008's no-store monitor semantics (verdict staleness bounded by TTL).Measurement artifacts
MONITOR aggregations (command mix, per-key byte attribution, client IPs) captured in session scratchpad; can be re-run with the same tap script if needed.