Summary

Cache common patterns (system prompts, tool definitions) for sub-millisecond retrieval.

Motivation

Many context patterns repeat across sessions: system prompts, tool definitions, common code snippets. Caching these avoids redundant embedding and compression operations.

Components

• Pattern detection - Hash common prompt prefixes and detect repeated blocks
• In-memory LRU cache - Hot patterns with configurable size
• Optional Redis backend - For distributed deployments
• TTL-based expiration - Automatic cache invalidation

Technical Design

type Cache interface {
    Get(ctx context.Context, key string) ([]byte, bool)
    Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
    Delete(ctx context.Context, key string) error
    Stats() CacheStats
}

type CacheStats struct {
    Hits       int64
    Misses     int64
    Size       int64
    MaxSize    int64
    HitRate    float64
}

Files to Create

pkg/cache/
├── cache.go           # Cache interface
├── memory.go          # In-memory LRU implementation
├── redis.go           # Redis implementation (optional)
├── patterns.go        # Pattern detection and hashing
└── cache_test.go      # Tests

Acceptance Criteria

• Cache hit returns in < 1ms
• Memory-bounded with configurable size
• Cache invalidation on content change
• Metrics exposed via /metrics endpoint