Skip to content

feat(cache): TTL-aware cache tracker for batch workloads#61

Merged
Siddhant-K-code merged 1 commit into
mainfrom
feat/49-ttl-aware-metrics
May 2, 2026
Merged

feat(cache): TTL-aware cache tracker for batch workloads#61
Siddhant-K-code merged 1 commit into
mainfrom
feat/49-ttl-aware-metrics

Conversation

@Siddhant-K-code

Copy link
Copy Markdown
Owner

Closes #49

What

Adds TTLTracker to pkg/cache. Monitors Anthropic's 5-minute prompt cache TTL per prefix hash, detects cold-start penalties, and tells batch jobs the latest safe time to send the next request without letting the cache expire.

Changes

pkg/cache/ttl.go (new)

  • AnthropicCacheTTL constant (5 minutes)
  • TTLTracker: tracks TTLEntry per prefix hash behind sync.Mutex
  • Touch(prefixHash): records request, returns wasAlive (warm hit vs cold start); refreshes ExpiresAt on every call
  • NextDeadline / TimeUntilExpiry: time remaining before cache expires
  • ScheduleDeadline(hash, safetyMargin): latest safe time for next batch request (ExpiresAt - margin)
  • ExpiredEntries(): all prefixes past their TTL deadline
  • Evict(hash): remove entry on boundary retreat
  • Stats(): aggregate AlivePrefixes, ExpiredPrefixes, TotalHits, TotalMisses

pkg/cache/ttl_test.go (new) — 9 tests

Usage

tracker := cache.NewTTLTracker(0) // 0 = AnthropicCacheTTL (5 min)

// After each request with a cache_control marker:
wasAlive := tracker.Touch(plan.PrefixHash)
if !wasAlive {
    metrics.RecordCacheBoundary(sessionID, 0, false, true) // boundary retreated
}

// Batch scheduling: sleep until just before cache expires
deadline := tracker.ScheduleDeadline(plan.PrefixHash, 30*time.Second)
time.Sleep(time.Until(deadline))

Add TTLTracker to pkg/cache. Monitors Anthropic's 5-minute prompt cache
TTL per prefix hash and detects cold-start penalties.

- Touch(prefixHash): records request, returns wasAlive (warm/cold)
- NextDeadline / TimeUntilExpiry: time remaining before cache expires
- ScheduleDeadline(hash, margin): latest safe time for next batch request
- ExpiredEntries: all prefixes past their TTL deadline
- Evict: remove entry on boundary retreat
- Stats: aggregate alive/expired counts and hit/miss totals

AnthropicCacheTTL constant (5m) exported for use by callers.

Co-authored-by: Ona <[email protected]>
@Siddhant-K-code Siddhant-K-code force-pushed the feat/49-ttl-aware-metrics branch from 2f2a58b to b73b59c Compare May 2, 2026 14:29
@Siddhant-K-code Siddhant-K-code merged commit b671cd6 into main May 2, 2026
2 checks passed
@Siddhant-K-code Siddhant-K-code deleted the feat/49-ttl-aware-metrics branch May 2, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] TTL-aware request scheduler for batch workloads

1 participant