Skip to content

Commit 5951279

Browse files
authored
Merge pull request anomalyco#222 from shimo4228/feat/skills/content-hash-cache-pattern
feat(skills): add content-hash-cache-pattern skill
2 parents bb08422 + 3847cc0 commit 5951279

1 file changed

Lines changed: 160 additions & 0 deletions

File tree

  • skills/content-hash-cache-pattern
Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
---
2+
name: content-hash-cache-pattern
3+
description: Cache expensive file processing results using SHA-256 content hashes — path-independent, auto-invalidating, with service layer separation.
4+
---
5+
6+
# Content-Hash File Cache Pattern
7+
8+
Cache expensive file processing results (PDF parsing, text extraction, image analysis) using SHA-256 content hashes as cache keys. Unlike path-based caching, this approach survives file moves/renames and auto-invalidates when content changes.
9+
10+
## When to Activate
11+
12+
- Building file processing pipelines (PDF, images, text extraction)
13+
- Processing cost is high and same files are processed repeatedly
14+
- Need a `--cache/--no-cache` CLI option
15+
- Want to add caching to existing pure functions without modifying them
16+
17+
## Core Pattern
18+
19+
### 1. Content-Hash Based Cache Key
20+
21+
Use file content (not path) as the cache key:
22+
23+
```python
24+
import hashlib
25+
from pathlib import Path
26+
27+
_HASH_CHUNK_SIZE = 65536 # 64KB chunks for large files
28+
29+
def compute_file_hash(path: Path) -> str:
30+
"""SHA-256 of file contents (chunked for large files)."""
31+
if not path.is_file():
32+
raise FileNotFoundError(f"File not found: {path}")
33+
sha256 = hashlib.sha256()
34+
with open(path, "rb") as f:
35+
while True:
36+
chunk = f.read(_HASH_CHUNK_SIZE)
37+
if not chunk:
38+
break
39+
sha256.update(chunk)
40+
return sha256.hexdigest()
41+
```
42+
43+
**Why content hash?** File rename/move = cache hit. Content change = automatic invalidation. No index file needed.
44+
45+
### 2. Frozen Dataclass for Cache Entry
46+
47+
```python
48+
from dataclasses import dataclass
49+
50+
@dataclass(frozen=True, slots=True)
51+
class CacheEntry:
52+
file_hash: str
53+
source_path: str
54+
document: ExtractedDocument # The cached result
55+
```
56+
57+
### 3. File-Based Cache Storage
58+
59+
Each cache entry is stored as `{hash}.json` — O(1) lookup by hash, no index file required.
60+
61+
```python
62+
import json
63+
from typing import Any
64+
65+
def write_cache(cache_dir: Path, entry: CacheEntry) -> None:
66+
cache_dir.mkdir(parents=True, exist_ok=True)
67+
cache_file = cache_dir / f"{entry.file_hash}.json"
68+
data = serialize_entry(entry)
69+
cache_file.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
70+
71+
def read_cache(cache_dir: Path, file_hash: str) -> CacheEntry | None:
72+
cache_file = cache_dir / f"{file_hash}.json"
73+
if not cache_file.is_file():
74+
return None
75+
try:
76+
raw = cache_file.read_text(encoding="utf-8")
77+
data = json.loads(raw)
78+
return deserialize_entry(data)
79+
except (json.JSONDecodeError, ValueError, KeyError):
80+
return None # Treat corruption as cache miss
81+
```
82+
83+
### 4. Service Layer Wrapper (SRP)
84+
85+
Keep the processing function pure. Add caching as a separate service layer.
86+
87+
```python
88+
def extract_with_cache(
89+
file_path: Path,
90+
*,
91+
cache_enabled: bool = True,
92+
cache_dir: Path = Path(".cache"),
93+
) -> ExtractedDocument:
94+
"""Service layer: cache check -> extraction -> cache write."""
95+
if not cache_enabled:
96+
return extract_text(file_path) # Pure function, no cache knowledge
97+
98+
file_hash = compute_file_hash(file_path)
99+
100+
# Check cache
101+
cached = read_cache(cache_dir, file_hash)
102+
if cached is not None:
103+
logger.info("Cache hit: %s (hash=%s)", file_path.name, file_hash[:12])
104+
return cached.document
105+
106+
# Cache miss -> extract -> store
107+
logger.info("Cache miss: %s (hash=%s)", file_path.name, file_hash[:12])
108+
doc = extract_text(file_path)
109+
entry = CacheEntry(file_hash=file_hash, source_path=str(file_path), document=doc)
110+
write_cache(cache_dir, entry)
111+
return doc
112+
```
113+
114+
## Key Design Decisions
115+
116+
| Decision | Rationale |
117+
|----------|-----------|
118+
| SHA-256 content hash | Path-independent, auto-invalidates on content change |
119+
| `{hash}.json` file naming | O(1) lookup, no index file needed |
120+
| Service layer wrapper | SRP: extraction stays pure, cache is a separate concern |
121+
| Manual JSON serialization | Full control over frozen dataclass serialization |
122+
| Corruption returns `None` | Graceful degradation, re-processes on next run |
123+
| `cache_dir.mkdir(parents=True)` | Lazy directory creation on first write |
124+
125+
## Best Practices
126+
127+
- **Hash content, not paths** — paths change, content identity doesn't
128+
- **Chunk large files** when hashing — avoid loading entire files into memory
129+
- **Keep processing functions pure** — they should know nothing about caching
130+
- **Log cache hit/miss** with truncated hashes for debugging
131+
- **Handle corruption gracefully** — treat invalid cache entries as misses, never crash
132+
133+
## Anti-Patterns to Avoid
134+
135+
```python
136+
# BAD: Path-based caching (breaks on file move/rename)
137+
cache = {"/path/to/file.pdf": result}
138+
139+
# BAD: Adding cache logic inside the processing function (SRP violation)
140+
def extract_text(path, *, cache_enabled=False, cache_dir=None):
141+
if cache_enabled: # Now this function has two responsibilities
142+
...
143+
144+
# BAD: Using dataclasses.asdict() with nested frozen dataclasses
145+
# (can cause issues with complex nested types)
146+
data = dataclasses.asdict(entry) # Use manual serialization instead
147+
```
148+
149+
## When to Use
150+
151+
- File processing pipelines (PDF parsing, OCR, text extraction, image analysis)
152+
- CLI tools that benefit from `--cache/--no-cache` options
153+
- Batch processing where the same files appear across runs
154+
- Adding caching to existing pure functions without modifying them
155+
156+
## When NOT to Use
157+
158+
- Data that must always be fresh (real-time feeds)
159+
- Cache entries that would be extremely large (consider streaming instead)
160+
- Results that depend on parameters beyond file content (e.g., different extraction configs)

0 commit comments

Comments
 (0)