You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Enhancement][Cache] Coalesce concurrent cache misses and support bounded stale refresh #7932
Reduce duplicate upstream work when many cache-eligible requests miss or expire on the same key, then optionally serve a bounded stale value while one refresh runs.
Existing capabilities and boundary
Extend the existing CachePlugin and reuse its cache eligibility, key construction, response/header and storage semantics. #7839 concerns Admin HTTP configuration-sync cache rebuilds, not business-response caching, and is a separate optimization.
MVP and staged extension
Add opt-in per-gateway single-flight for eligible requests sharing the effective cache key. One bounded upstream fill can supply a reusable cached representation to a bounded number of waiters.
Add stale-while-revalidate only after the miss path is correct, with distinct freshness/stale deadlines and a finite refresh budget. Preserve the old value only within the configured stale policy.
Start with finite, bounded HTTP responses. Do not share a one-shot response publisher or a DataBuffer across subscribers. Retain all isolation dimensions required by existing cache/auth/tenant/Vary policy; single-flight must not broaden cache eligibility or merge different callers' private responses.
Acceptance criteria
A concurrent burst for one eligible cold key performs one fill within a gateway; distinct keys and security scopes stay independent.
Bound active fill keys, waiters per key, response size, fill time and refresh concurrency. Overload behavior is documented and cannot create an unbounded second queue.
Define leader/waiter cancellation, upstream errors, non-cacheable responses and eviction/invalidation during a fill. Failures release state and permit a subsequent fill.
Invalidation/configuration generations prevent an old fill from repopulating a newly invalidated entry.
Stale serving never exceeds its configured window; refresh failure, no stale value and concurrent invalidation are covered. No background retry storm.
Preserve status, headers, body bytes and reference-counted buffer ownership; test with the real reactive response path.
Document per-gateway scope: replicas can still perform separate fills. Distributed locking is not required for the first version.
Activity
hanzhe-one commentedon Oct 22, 2026
/assign