Skip to content

[Enhancement][Cache] Coalesce concurrent cache misses and support bounded stale refresh #7932

Closed

Description

@Aias00

Goal

Reduce duplicate upstream work when many cache-eligible requests miss or expire on the same key, then optionally serve a bounded stale value while one refresh runs.

Existing capabilities and boundary

Extend the existing CachePlugin and reuse its cache eligibility, key construction, response/header and storage semantics. #7839 concerns Admin HTTP configuration-sync cache rebuilds, not business-response caching, and is a separate optimization.

MVP and staged extension

  1. Add opt-in per-gateway single-flight for eligible requests sharing the effective cache key. One bounded upstream fill can supply a reusable cached representation to a bounded number of waiters.
  2. Add stale-while-revalidate only after the miss path is correct, with distinct freshness/stale deadlines and a finite refresh budget. Preserve the old value only within the configured stale policy.

Start with finite, bounded HTTP responses. Do not share a one-shot response publisher or a DataBuffer across subscribers. Retain all isolation dimensions required by existing cache/auth/tenant/Vary policy; single-flight must not broaden cache eligibility or merge different callers' private responses.

Acceptance criteria

  • A concurrent burst for one eligible cold key performs one fill within a gateway; distinct keys and security scopes stay independent.
  • Bound active fill keys, waiters per key, response size, fill time and refresh concurrency. Overload behavior is documented and cannot create an unbounded second queue.
  • Define leader/waiter cancellation, upstream errors, non-cacheable responses and eviction/invalidation during a fill. Failures release state and permit a subsequent fill.
  • Invalidation/configuration generations prevent an old fill from repopulating a newly invalidated entry.
  • Stale serving never exceeds its configured window; refresh failure, no stale value and concurrent invalidation are covered. No background retry storm.
  • Preserve status, headers, body bytes and reference-counted buffer ownership; test with the real reactive response path.
  • Document per-gateway scope: replicas can still perform separate fills. Distributed locking is not required for the first version.
  • Compare upstream call counts, gateway latency and retained memory under cold-key and hot-expiry bursts using [Task][Performance] Establish reproducible benchmarks and track gateway optimizations #7650; disabled mode remains compatible.

Related: #7839 (different cache layer), #7650.

Activity

  1. hanzhe-one commented on Oct 22, 2026

    @hanzhe-one
    Contributor

    /assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions