fix(router): harden CFCacheStore L1/L2 reads — latency budgets, debug sink, SWR retention/herd fixes - #558

…558)

Merge origin/main into feat/cache-tags-v2. main carried PR #558 (CFCacheStore
latency budgets, debug sink, SWR retention + herd fixes); this branch carries the
cache-tag invalidation + resilience layer. Both rewrote the same get/getItem/
kvGet*/promote* methods, so the merge is a semantic reconcile that keeps BOTH
sides' fail-safes:

- kvGetOrEvict is now budget-aware (readWithTimeout + {value,timedOut}) and
  string-tolerant, preserving the transient-vs-corrupt eviction distinction.
- Tag-marker reads (isGloballyInvalidated/fetchTagMarker) inherit all three
  latency budgets and FAIL OPEN on a KV timeout (a degraded namespace never
  wrongly invalidates a hit).
- Unified L1-error contract: a transient match error or a corrupt L1 body is
  reported via onError (cache-read / cache-corrupt) AND falls through to L2/KV;
  a corrupt L1 entry is evicted only when no good L2 copy exists, so the evict
  cannot race the KV->L1 promote (review finding 1).
- getResponse surfaces the captured match error as cache-read, matching
  get/getItem (review finding 2/3).
- markRevalidating / promotes / KV envelopes carry both the hard-expiry header
  and the tag headers.

Validated: typecheck, 3547 unit tests, lint, format, and cache-tag e2e in dev +
production for both cloudflare-basic (real CFCacheStore + miniflare KV) and the
memory-store e2e test-app (24 e2e tests). New coverage pins the resilient L2
fall-through + no-false-evict and the tag-marker fail-open; both review-fix
regression tests proven red-before-green.

Adversarial review of the #534<->#558 reconcile surfaced ten findings; this
folds the worthwhile ones in before the single CI run (cheaper than a second
follow-up round).

Read-your-own-writes / correctness:
- fetchTagMarker no longer clobbers a concurrent invalidateTags() memo write
  that lands mid-read, so a parallel-slot updateTag() stays read-your-own-writes.
- Per-store isolation of the tag-marker memo/inflight maps: two stores in one
  request (app store + a cache({ store }) override on a different KV binding)
  can no longer cross-pollute marker values.
- kvGetOrEvict guards a KV body that parses to null/a primitive BEFORE
  validate(), so 'null'/'42' are evicted as corrupt instead of throwing past the
  validator into a transient cache-read that never self-heals.
- Non-array tags fail safe (Array.isArray at the write guards and the central
  isGloballyInvalidated check) instead of throwing .map on every read.

Fail-safe reads/writes (no perf degradation):
- Corrupt-entry KV eviction is dispatched via waitUntil, never awaited on the
  read path (an unbounded kv.delete would re-add exactly the stall the read
  budgets exist to prevent).
- The transient L1 tag-marker match error is reported as cache-read like the
  data read paths, instead of being silently discarded at the destructuring.
- The L1 marker write-through failure on invalidate is surfaced
  (cache-invalidate) and the stale marker evicted, instead of swallowed while
  updateTag reports success.

DX / hygiene:
- tagInvalidationTtl below KV's 60s expirationTtl floor is raised (with a
  one-time warning) so a too-small value no longer makes EVERY invalidation
  throw; a non-finite tagCacheTtl is coerced to 0 (disabled) rather than
  silently enabling an invalid max-age=Infinity marker.
- Remove the now-dead parseOrEvict (replaced during the reconcile by
  readJsonWithTimeout/healCorruptL1 + kvGetOrEvict).

Observability + API completeness (review below-the-cut items):
- New `markerMs` debug field: the tagged-read marker-resolution tail (memo -> L1
  marker cache -> KV) is no longer invisible in CFCacheReadDebugEvent (measured
  only under debug). New `match-error` debug outcome so a match() rejection is
  distinct from a genuine l1-miss and agrees with the cache-read on onError.
- Export CacheItemOptions / CacheItemResult from ./cache so a custom
  SegmentCacheStore author can name every type its getItem/setItem methods use.

Docs/comments corrected: caching skill corrupt-entry self-heal table (serves
from L2, evicts only when no L2 copy), healCorruptL1 cross-request timeout note,
the tag-invalidated tier-divergence comment, and a tagCacheTtl same-colo race
caveat.

Tests: +15 (11 proven red-before-green); strengthened the corrupt-L1
heal-by-overwrite and fail-open assertions to be non-vacuous; the fail-open mock
keys on the exported TAG_MARKER_PREFIX so a prefix rename can't make it pass
vacuously; added real-timer teardown to both CF suites. Unit 3562 green;
cache-tag e2e green dev+production against the real CFCacheStore.

…534)

* feat(cache): tag invalidation — cacheTag / updateTag / revalidateTag

Adds runtime cache-tag invalidation across the in-memory and Cloudflare
cache stores.

Public API (server-only, from @rangojs/router):
- cacheTag(...tags): tag the current "use cache" entry at runtime.
- updateTag(...tags): awaitable, read-your-own-writes invalidation for
  Server Actions (resolves after in-process invalidation completes; on
  CFCacheStore, rejects if a durable KV marker write fails instead of
  reporting a false success).
- revalidateTag(...tags): fire-and-forget background invalidation for
  route handlers / webhooks. Like updateTag it hard-purges; the only
  difference is awaitability (it is NOT stale-while-revalidate).

Tags attach via three axes: runtime cacheTag() inside "use cache",
cache({ tags }) on the route DSL, and loader cache tags. A per-handler
explicit-store registry (LRU-bounded) lets updateTag/revalidateTag reach
every configured store, including per-scope cache({ store }) boundaries.

Stores:
- MemorySegmentCacheStore: forward + reverse tag index; eager delete on
  invalidate.
- CFCacheStore: per-tag KV markers compared against an entry's taggedAt;
  optional per-colo L1 marker cache (tagCacheTtl) and onRevalidateTag CDN
  purge hook; tags encodeURIComponent'd so commas / non-Latin1 stay
  header-safe; over-512-byte marker keys and failed KV writes surface as a
  rejected updateTag rather than silent stale-serve.

Document caching: createDocumentCacheMiddleware tags full-page entries
with the request's resolved tags, snapshotted after the render stream
drains so Suspense-streamed cacheTag()/"use cache" tags are captured, so
updateTag/revalidateTag invalidate cached pages.

Docs and skills updated. e2e covers cacheTag / cache(tags) / updateTag /
revalidateTag and document invalidation in dev and production on both the
memory and the CF + KV stores.

Known limitation: combining whole-page document caching with segment
cache({ tags }) DSL tags can miss the document-level tag (resolved in the
deferred cacheRoute waitUntil); the segment entry itself still invalidates.

* perf+docs(cache): parallelize L1 marker write-through; document revalidateTag/keyGenerator contracts

From a 4-dimension (correctness/performance/consistency/security) review of the
branch:

- perf: invalidateTags wrote the per-colo L1 tag markers sequentially (await per
  tag inside the loop, only when tagCacheTtl>0). The memo write stays synchronous
  (read-your-own-writes); the independent L1 Cache API writes now fan out via
  Promise.all instead of stalling on each other.
- docs (security): revalidateTag JSDoc now states that, being fire-and-forget, a
  failed durable marker write is NOT surfaced to the caller (logged only) - use
  await updateTag() when the invalidation must be confirmed/retried.
- docs (security): CFCacheStore.keyGenerator now documents the reserved __tag__/
  and __tagmarker__/ prefixes a returned key must not collide with (tag markers
  share the data KV namespace).

The review's other confirmed items were verified as deliberate, documented
trade-offs and intentionally not changed: read-side isGloballyInvalidated
fail-open (availability over a KV-outage stampede), unbounded tag cardinality
and the marker/data key-collision (documented; key-prefix hard-namespacing would
invalidate existing caches), and the segment-DSL document-tag race (documented
caveat). Consistency + most security probes verified clean (naming, normalization
symmetry, header encode/decode, no injection/poisoning).

* fix(cache): report cache errors via onError + self-heal corrupt entries

Audited every cache read/write/parse path (67 sites). The resilience contract is
now: report loudly via the router's onError hook, degrade (never throw), and
evict corrupt/partial entries.

- onError routing: a new reportCacheError(error, category) helper logs (loud,
  works in background tasks) AND routes through the request context's
  _reportBackgroundError -> router onError (deduped). Both stores used only
  console.error before, so a consumer's onError never saw cache read/write
  failures. All store + document-cache + cache-runtime/scope sites now route
  through it with a category (cache-read/-write/-delete/-corrupt/-invalidate).
  The non-blocking L1 write (waitUntil) previously swallowed its rejection
  entirely; it now reports too.

- Self-heal on corrupt/partial: CF reads distinguish a PARSE failure (truncated
  Cache API body, malformed KV envelope, bad base64) from a transient infra
  error. On corruption the faulty entry is EVICTED (cache.delete / kv.delete) and
  reported as cache-corrupt, so subsequent reads miss + re-render instead of
  re-failing until TTL; a transient infra error degrades to a miss WITHOUT
  deleting a still-good entry. New parseOrEvict / kvGetOrEvict helpers + envelope
  shape validation. cache-runtime + cache-scope report corrupt deserializes (and
  cache-scope evicts; use-cache items self-heal via the miss-path overwrite).

- Degrade hardening: memory putResponse's un-guarded arrayBuffer() now degrades
  to a no-op instead of throwing.

The deliberate exception is unchanged: updateTag -> invalidateTags still REJECTS
on a failed durable marker write (read-your-own-writes honesty), not swallowed.

Tests: infra read error -> reports cache-read + null; corrupt L1 entry -> evicted
+ cache-corrupt; corrupt KV envelope -> kv.delete + cache-corrupt; write error ->
cache-write.

* fix(cache): harden cache-error resilience (transient vs corrupt, observability)

Review-driven hardening of the cache-tag error path. Three substantive fixes
plus consistency/typing nits, all backed by regression tests.

Correctness / availability:
- kvGetOrEvict no longer deletes a still-good cross-colo KV entry on a transient
  outage. It read via kv.get(key, {type:"json"}), which fused the network read
  and the JSON parse, so a 5xx/429/network blip was treated as corruption and
  evicted the entry (self-inflicted cross-colo miss storm). Now reads as text:
  a transient error propagates to the caller's outer catch (reported cache-read,
  entry intact); only a JSON.parse/validate failure is true corruption -> evict.
- kvGetResponse: the full Response reconstruction (base64 + headers + status) is
  now inside one corrupt-guard, so malformed `hd`/out-of-range `st` self-heals
  too; eviction goes through evictKvKey (reports cache-delete instead of silently
  swallowing the delete failure).
- Reserved-namespace guard: a misconfigured store keyGenerator returning a key
  under a reserved tag-marker prefix - `__tag__/` (KV marker key) or
  `__tagmarker__/` (L1 Cache API marker request) - can no longer clobber (on set)
  or evict (on get/delete) a live tag marker. The segment op is a reported
  no-op/miss, naming which prefix collided.

Observability:
- revalidateTag now routes a failed background invalidation through reportingAsync
  (loud log + onError, category cache-invalidate) instead of swallowing it; the
  JSDoc claim is now true. Passes the captured ctx since the run is detached.
- The stale-while-revalidate ("use cache") failure goes through reportCacheError
  (loud) with the captured ctx, and `stale-revalidation` is now a typed category.
- reportCacheError/reportingAsync accept an explicit ctx for detached waitUntil
  tasks where the ALS context is gone, so onError still fires. The CF store's own
  background writes deliberately omit it (request-agnostic singleton -> console
  loud only); a comment documents the asymmetry.
- CacheErrorCategory is exported from `@rangojs/router/cache`, the
  _reportBackgroundError signature is typed to it, and onError's metadata.category
  is documented.
- cache-scope segment-write failure now uses reportCacheError, not console.error.

#2 (L1 document body served verbatim): documented as a deliberate asymmetry -
hashing the body would require buffering the whole document on every hit,
defeating streaming; integrity relies on the Cache API, and the KV tier (the real
partial-read vector) is structurally validated.

Tests: KV transient-read -> miss WITHOUT eviction (#1 guard) + L1 transient
non-eviction assertion; per-tag L1 marker write-through fan-out; reserved-prefix
get/set/delete for both __tag__/ and __tagmarker__/; revalidateTag failure ->
onError; non-CF (CacheScope) corrupt segment self-heal.

* fix(cache): address tag-invalidation review findings

- record cached entry tags into the request tag union on segment-cache
  hits, so a document re-cached over a segment hit stays tag-invalidatable;
  narrow the document-cache caveat to the first-write waitUntil race
- gate the invalidateTags memo/L1 marker write-through on KV (markers are
  never read without it) and update the stale constructor comment
- warn with the real cause when updateTag/revalidateTag run outside a
  request context (queue consumer / cron) instead of the store-config message
- reword revalidateTag as background hard-purge (not SWR) in consumer JSDoc,
  the export comment, and the unit test title
- annotate URL-param tag e2e fixtures as test-only (untrusted-input warning)
- unit tests pin hit-path tag recording, the no-KV no-op, and the
  no-context warnings; unique namespace for the no-KV warn fixture

* fix(cache): adapt cache-tag e2e to bare-T JSON responses (#539)

The cache-tag e2e helpers read (await res.json()).data.ts from path.json
routes, but #539 (bare-T JSON responses) removed the { data } envelope:
path.json now serializes the handler's return value verbatim. The tagged
routes return a bare { ts, id }, so .data was undefined and reading .ts
threw "Cannot read properties of undefined (reading 'ts')".

This broke the cache-tag invalidation suites in four places (router dev +
production, cloudflare-basic dev + production). On the rangojs-router shards
the failing test's 8s expect.poll retries also destabilized the shard,
cascading into is-action and cache-loader-null timeouts; fixing the root
read clears those too.

- e2e/cache.test.ts: pollTs reads .ts (dev + production helpers)
- cloudflare-basic/e2e/cache-tag.test.ts: ts() reads .ts
- urls/cache-tag.tsx: correct the stale { data } envelope comment
- docs/internal/feature-map.md: oxfmt table padding

Verified locally: router dev cache-tag (6 passed), router production
cache-tag + is-action + cache-loader-null (9 passed), unit (3501),
typecheck, lint, format.

* harden(cache): fold post-reconcile review findings into CFCacheStore

Adversarial review of the #534<->#558 reconcile surfaced ten findings; this
folds the worthwhile ones in before the single CI run (cheaper than a second
follow-up round).

Read-your-own-writes / correctness:
- fetchTagMarker no longer clobbers a concurrent invalidateTags() memo write
  that lands mid-read, so a parallel-slot updateTag() stays read-your-own-writes.
- Per-store isolation of the tag-marker memo/inflight maps: two stores in one
  request (app store + a cache({ store }) override on a different KV binding)
  can no longer cross-pollute marker values.
- kvGetOrEvict guards a KV body that parses to null/a primitive BEFORE
  validate(), so 'null'/'42' are evicted as corrupt instead of throwing past the
  validator into a transient cache-read that never self-heals.
- Non-array tags fail safe (Array.isArray at the write guards and the central
  isGloballyInvalidated check) instead of throwing .map on every read.

Fail-safe reads/writes (no perf degradation):
- Corrupt-entry KV eviction is dispatched via waitUntil, never awaited on the
  read path (an unbounded kv.delete would re-add exactly the stall the read
  budgets exist to prevent).
- The transient L1 tag-marker match error is reported as cache-read like the
  data read paths, instead of being silently discarded at the destructuring.
- The L1 marker write-through failure on invalidate is surfaced
  (cache-invalidate) and the stale marker evicted, instead of swallowed while
  updateTag reports success.

DX / hygiene:
- tagInvalidationTtl below KV's 60s expirationTtl floor is raised (with a
  one-time warning) so a too-small value no longer makes EVERY invalidation
  throw; a non-finite tagCacheTtl is coerced to 0 (disabled) rather than
  silently enabling an invalid max-age=Infinity marker.
- Remove the now-dead parseOrEvict (replaced during the reconcile by
  readJsonWithTimeout/healCorruptL1 + kvGetOrEvict).

Observability + API completeness (review below-the-cut items):
- New `markerMs` debug field: the tagged-read marker-resolution tail (memo -> L1
  marker cache -> KV) is no longer invisible in CFCacheReadDebugEvent (measured
  only under debug). New `match-error` debug outcome so a match() rejection is
  distinct from a genuine l1-miss and agrees with the cache-read on onError.
- Export CacheItemOptions / CacheItemResult from ./cache so a custom
  SegmentCacheStore author can name every type its getItem/setItem methods use.

Docs/comments corrected: caching skill corrupt-entry self-heal table (serves
from L2, evicts only when no L2 copy), healCorruptL1 cross-request timeout note,
the tag-invalidated tier-divergence comment, and a tagCacheTtl same-colo race
caveat.

Tests: +15 (11 proven red-before-green); strengthened the corrupt-L1
heal-by-overwrite and fail-open assertions to be non-vacuous; the fail-open mock
keys on the exported TAG_MARKER_PREFIX so a prefix rename can't make it pass
vacuously; added real-timer teardown to both CF suites. Unit 3562 green;
cache-tag e2e green dev+production against the real CFCacheStore.

* chore(cache): raise CFCacheStore default kvReadTimeoutMs 50 -> 170 [skip ci]

The L2/KV read budget defaulted to 50ms, right at the documented healthy read,
which false-misses legitimate latency tails (larger payloads, far-from-colo
regions) into a full render. Raise the default to 170ms: a few multiples above
the ~50ms healthy read for headroom, still well under a degraded namespace's
multi-second tail. Per-store override (kvReadTimeoutMs) and the <=0-disables
contract are unchanged; the tag-marker KV reads inherit the same budget.

Unit tests reference the KV_READ_TIMEOUT_MS constant (not a literal 50) so they
adapt; the explicit-override tests (200, 0) are independent. SKILL default table
and the option JSDoc updated to 170.

* docs(cache): document markerMs/match-error debug surface + tag-marker read-latency rationale [skip ci]

- skills/caching/SKILL.md: bring the debug outcome list current with the full
CFCacheReadDebugEvent union; document the markerMs marker-resolution diagnostic
and match-error (distinct from l1-miss).
- docs/internal/feature-map.md: add match-error + markerMs to the ./cache
debug-sink description.
- docs/design/cache-tags-flow.md: add a Tag-marker read latency section -
measured baseline (1-6ms warm / ~73ms cold), why an in-isolate marker cache is
rejected, and the tagCacheTtl -> marker-parallel ladder if latency must drop.

Docs-only; the markerMs field and match-error outcome shipped in bcb8477.

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

Brings in #560 (rango state cookie + invalidateClientCache/keepClientCache),
#534 (cache tag invalidation cacheTag/updateTag/revalidateTag), and the
CFCacheStore latency-budget/SWR hardening (#558).

Non-trivial conflict resolutions:
- docs/internal/feature-map.md & feature-file-map.md: unioned both sides'
  tables. Took main's updated `.`/`./cache` rows (new cache-tag + client-cache
  exports) plus its `Client cache invalidation` feature row, and kept this
  branch's `./testing/*` export rows and `Consumer testing primitives` row.
- tests/cloudflare-basic/src/urls.tsx: kept main's updateTag/revalidateTag
  imports but dropped `cookies` - this branch extracted the inline
  cookies().set("mw-overlay") into middleware/cookie-overlay.ts.
- src/testing/render-route.tsx: removed the test NavigationBridge's
  updateAppShell member; main removed updateAppShell from the NavigationBridge
  type (app-shell updates now flow only through appShellRef).

Verified green: typecheck (12/12), test:unit:all (3915 passed / 6 todo),
lint (oxlint 0/0), format (oxfmt), check:e2e-bucketing.