|
| 1 | +--- |
| 2 | +name: content-hash-cache-pattern |
| 3 | +description: Cache expensive file processing results using SHA-256 content hashes — path-independent, auto-invalidating, with service layer separation. |
| 4 | +--- |
| 5 | + |
| 6 | +# Content-Hash File Cache Pattern |
| 7 | + |
| 8 | +Cache expensive file processing results (PDF parsing, text extraction, image analysis) using SHA-256 content hashes as cache keys. Unlike path-based caching, this approach survives file moves/renames and auto-invalidates when content changes. |
| 9 | + |
| 10 | +## When to Activate |
| 11 | + |
| 12 | +- Building file processing pipelines (PDF, images, text extraction) |
| 13 | +- Processing cost is high and same files are processed repeatedly |
| 14 | +- Need a `--cache/--no-cache` CLI option |
| 15 | +- Want to add caching to existing pure functions without modifying them |
| 16 | + |
| 17 | +## Core Pattern |
| 18 | + |
| 19 | +### 1. Content-Hash Based Cache Key |
| 20 | + |
| 21 | +Use file content (not path) as the cache key: |
| 22 | + |
| 23 | +```python |
| 24 | +import hashlib |
| 25 | +from pathlib import Path |
| 26 | + |
| 27 | +_HASH_CHUNK_SIZE = 65536 # 64KB chunks for large files |
| 28 | + |
| 29 | +def compute_file_hash(path: Path) -> str: |
| 30 | + """SHA-256 of file contents (chunked for large files).""" |
| 31 | + if not path.is_file(): |
| 32 | + raise FileNotFoundError(f"File not found: {path}") |
| 33 | + sha256 = hashlib.sha256() |
| 34 | + with open(path, "rb") as f: |
| 35 | + while True: |
| 36 | + chunk = f.read(_HASH_CHUNK_SIZE) |
| 37 | + if not chunk: |
| 38 | + break |
| 39 | + sha256.update(chunk) |
| 40 | + return sha256.hexdigest() |
| 41 | +``` |
| 42 | + |
| 43 | +**Why content hash?** File rename/move = cache hit. Content change = automatic invalidation. No index file needed. |
| 44 | + |
| 45 | +### 2. Frozen Dataclass for Cache Entry |
| 46 | + |
| 47 | +```python |
| 48 | +from dataclasses import dataclass |
| 49 | + |
| 50 | +@dataclass(frozen=True, slots=True) |
| 51 | +class CacheEntry: |
| 52 | + file_hash: str |
| 53 | + source_path: str |
| 54 | + document: ExtractedDocument # The cached result |
| 55 | +``` |
| 56 | + |
| 57 | +### 3. File-Based Cache Storage |
| 58 | + |
| 59 | +Each cache entry is stored as `{hash}.json` — O(1) lookup by hash, no index file required. |
| 60 | + |
| 61 | +```python |
| 62 | +import json |
| 63 | +from typing import Any |
| 64 | + |
| 65 | +def write_cache(cache_dir: Path, entry: CacheEntry) -> None: |
| 66 | + cache_dir.mkdir(parents=True, exist_ok=True) |
| 67 | + cache_file = cache_dir / f"{entry.file_hash}.json" |
| 68 | + data = serialize_entry(entry) |
| 69 | + cache_file.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8") |
| 70 | + |
| 71 | +def read_cache(cache_dir: Path, file_hash: str) -> CacheEntry | None: |
| 72 | + cache_file = cache_dir / f"{file_hash}.json" |
| 73 | + if not cache_file.is_file(): |
| 74 | + return None |
| 75 | + try: |
| 76 | + raw = cache_file.read_text(encoding="utf-8") |
| 77 | + data = json.loads(raw) |
| 78 | + return deserialize_entry(data) |
| 79 | + except (json.JSONDecodeError, ValueError, KeyError): |
| 80 | + return None # Treat corruption as cache miss |
| 81 | +``` |
| 82 | + |
| 83 | +### 4. Service Layer Wrapper (SRP) |
| 84 | + |
| 85 | +Keep the processing function pure. Add caching as a separate service layer. |
| 86 | + |
| 87 | +```python |
| 88 | +def extract_with_cache( |
| 89 | + file_path: Path, |
| 90 | + *, |
| 91 | + cache_enabled: bool = True, |
| 92 | + cache_dir: Path = Path(".cache"), |
| 93 | +) -> ExtractedDocument: |
| 94 | + """Service layer: cache check -> extraction -> cache write.""" |
| 95 | + if not cache_enabled: |
| 96 | + return extract_text(file_path) # Pure function, no cache knowledge |
| 97 | + |
| 98 | + file_hash = compute_file_hash(file_path) |
| 99 | + |
| 100 | + # Check cache |
| 101 | + cached = read_cache(cache_dir, file_hash) |
| 102 | + if cached is not None: |
| 103 | + logger.info("Cache hit: %s (hash=%s)", file_path.name, file_hash[:12]) |
| 104 | + return cached.document |
| 105 | + |
| 106 | + # Cache miss -> extract -> store |
| 107 | + logger.info("Cache miss: %s (hash=%s)", file_path.name, file_hash[:12]) |
| 108 | + doc = extract_text(file_path) |
| 109 | + entry = CacheEntry(file_hash=file_hash, source_path=str(file_path), document=doc) |
| 110 | + write_cache(cache_dir, entry) |
| 111 | + return doc |
| 112 | +``` |
| 113 | + |
| 114 | +## Key Design Decisions |
| 115 | + |
| 116 | +| Decision | Rationale | |
| 117 | +|----------|-----------| |
| 118 | +| SHA-256 content hash | Path-independent, auto-invalidates on content change | |
| 119 | +| `{hash}.json` file naming | O(1) lookup, no index file needed | |
| 120 | +| Service layer wrapper | SRP: extraction stays pure, cache is a separate concern | |
| 121 | +| Manual JSON serialization | Full control over frozen dataclass serialization | |
| 122 | +| Corruption returns `None` | Graceful degradation, re-processes on next run | |
| 123 | +| `cache_dir.mkdir(parents=True)` | Lazy directory creation on first write | |
| 124 | + |
| 125 | +## Best Practices |
| 126 | + |
| 127 | +- **Hash content, not paths** — paths change, content identity doesn't |
| 128 | +- **Chunk large files** when hashing — avoid loading entire files into memory |
| 129 | +- **Keep processing functions pure** — they should know nothing about caching |
| 130 | +- **Log cache hit/miss** with truncated hashes for debugging |
| 131 | +- **Handle corruption gracefully** — treat invalid cache entries as misses, never crash |
| 132 | + |
| 133 | +## Anti-Patterns to Avoid |
| 134 | + |
| 135 | +```python |
| 136 | +# BAD: Path-based caching (breaks on file move/rename) |
| 137 | +cache = {"/path/to/file.pdf": result} |
| 138 | + |
| 139 | +# BAD: Adding cache logic inside the processing function (SRP violation) |
| 140 | +def extract_text(path, *, cache_enabled=False, cache_dir=None): |
| 141 | + if cache_enabled: # Now this function has two responsibilities |
| 142 | + ... |
| 143 | + |
| 144 | +# BAD: Using dataclasses.asdict() with nested frozen dataclasses |
| 145 | +# (can cause issues with complex nested types) |
| 146 | +data = dataclasses.asdict(entry) # Use manual serialization instead |
| 147 | +``` |
| 148 | + |
| 149 | +## When to Use |
| 150 | + |
| 151 | +- File processing pipelines (PDF parsing, OCR, text extraction, image analysis) |
| 152 | +- CLI tools that benefit from `--cache/--no-cache` options |
| 153 | +- Batch processing where the same files appear across runs |
| 154 | +- Adding caching to existing pure functions without modifying them |
| 155 | + |
| 156 | +## When NOT to Use |
| 157 | + |
| 158 | +- Data that must always be fresh (real-time feeds) |
| 159 | +- Cache entries that would be extremely large (consider streaming instead) |
| 160 | +- Results that depend on parameters beyond file content (e.g., different extraction configs) |
0 commit comments