|
| 1 | +--- |
| 2 | +name: santa-method |
| 3 | +description: "Multi-agent adversarial verification with convergence loop. Two independent review agents must both pass before output ships." |
| 4 | +origin: "Ronald Skelton - Founder, RapportScore.ai" |
| 5 | +--- |
| 6 | + |
| 7 | +# Santa Method |
| 8 | + |
| 9 | +Multi-agent adversarial verification framework. Make a list, check it twice. If it's naughty, fix it until it's nice. |
| 10 | + |
| 11 | +The core insight: a single agent reviewing its own output shares the same biases, knowledge gaps, and systematic errors that produced the output. Two independent reviewers with no shared context break this failure mode. |
| 12 | + |
| 13 | +## When to Activate |
| 14 | + |
| 15 | +Invoke this skill when: |
| 16 | +- Output will be published, deployed, or consumed by end users |
| 17 | +- Compliance, regulatory, or brand constraints must be enforced |
| 18 | +- Code ships to production without human review |
| 19 | +- Content accuracy matters (technical docs, educational material, customer-facing copy) |
| 20 | +- Batch generation at scale where spot-checking misses systemic patterns |
| 21 | +- Hallucination risk is elevated (claims, statistics, API references, legal language) |
| 22 | + |
| 23 | +Do NOT use for internal drafts, exploratory research, or tasks with deterministic verification (use build/test/lint pipelines for those). |
| 24 | + |
| 25 | +## Architecture |
| 26 | + |
| 27 | +``` |
| 28 | +┌─────────────┐ |
| 29 | +│ GENERATOR │ Phase 1: Make a List |
| 30 | +│ (Agent A) │ Produce the deliverable |
| 31 | +└──────┬───────┘ |
| 32 | + │ output |
| 33 | + ▼ |
| 34 | +┌──────────────────────────────┐ |
| 35 | +│ DUAL INDEPENDENT REVIEW │ Phase 2: Check It Twice |
| 36 | +│ │ |
| 37 | +│ ┌───────────┐ ┌───────────┐ │ Two agents, same rubric, |
| 38 | +│ │ Reviewer B │ │ Reviewer C │ │ no shared context |
| 39 | +│ └─────┬─────┘ └─────┬─────┘ │ |
| 40 | +│ │ │ │ |
| 41 | +└────────┼──────────────┼────────┘ |
| 42 | + │ │ |
| 43 | + ▼ ▼ |
| 44 | +┌──────────────────────────────┐ |
| 45 | +│ VERDICT GATE │ Phase 3: Naughty or Nice |
| 46 | +│ │ |
| 47 | +│ B passes AND C passes → NICE │ Both must pass. |
| 48 | +│ Otherwise → NAUGHTY │ No exceptions. |
| 49 | +└──────┬──────────────┬─────────┘ |
| 50 | + │ │ |
| 51 | + NICE NAUGHTY |
| 52 | + │ │ |
| 53 | + ▼ ▼ |
| 54 | + [ SHIP ] ┌─────────────┐ |
| 55 | + │ FIX CYCLE │ Phase 4: Fix Until Nice |
| 56 | + │ │ |
| 57 | + │ iteration++ │ Collect all flags. |
| 58 | + │ if i > MAX: │ Fix all issues. |
| 59 | + │ escalate │ Re-run both reviewers. |
| 60 | + │ else: │ Loop until convergence. |
| 61 | + │ goto Ph.2 │ |
| 62 | + └──────────────┘ |
| 63 | +``` |
| 64 | + |
| 65 | +## Phase Details |
| 66 | + |
| 67 | +### Phase 1: Make a List (Generate) |
| 68 | + |
| 69 | +Execute the primary task. No changes to your normal generation workflow. Santa Method is a post-generation verification layer, not a generation strategy. |
| 70 | + |
| 71 | +```python |
| 72 | +# The generator runs as normal |
| 73 | +output = generate(task_spec) |
| 74 | +``` |
| 75 | + |
| 76 | +### Phase 2: Check It Twice (Independent Dual Review) |
| 77 | + |
| 78 | +Spawn two review agents in parallel. Critical invariants: |
| 79 | + |
| 80 | +1. **Context isolation** — neither reviewer sees the other's assessment |
| 81 | +2. **Identical rubric** — both receive the same evaluation criteria |
| 82 | +3. **Same inputs** — both receive the original spec AND the generated output |
| 83 | +4. **Structured output** — each returns a typed verdict, not prose |
| 84 | + |
| 85 | +```python |
| 86 | +REVIEWER_PROMPT = """ |
| 87 | +You are an independent quality reviewer. You have NOT seen any other review of this output. |
| 88 | +
|
| 89 | +## Task Specification |
| 90 | +{task_spec} |
| 91 | +
|
| 92 | +## Output Under Review |
| 93 | +{output} |
| 94 | +
|
| 95 | +## Evaluation Rubric |
| 96 | +{rubric} |
| 97 | +
|
| 98 | +## Instructions |
| 99 | +Evaluate the output against EACH rubric criterion. For each: |
| 100 | +- PASS: criterion fully met, no issues |
| 101 | +- FAIL: specific issue found (cite the exact problem) |
| 102 | +
|
| 103 | +Return your assessment as structured JSON: |
| 104 | +{ |
| 105 | + "verdict": "PASS" | "FAIL", |
| 106 | + "checks": [ |
| 107 | + {"criterion": "...", "result": "PASS|FAIL", "detail": "..."} |
| 108 | + ], |
| 109 | + "critical_issues": ["..."], // blockers that must be fixed |
| 110 | + "suggestions": ["..."] // non-blocking improvements |
| 111 | +} |
| 112 | +
|
| 113 | +Be rigorous. Your job is to find problems, not to approve. |
| 114 | +""" |
| 115 | +``` |
| 116 | + |
| 117 | +```python |
| 118 | +# Spawn reviewers in parallel (Claude Code subagents) |
| 119 | +review_b = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer B") |
| 120 | +review_c = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer C") |
| 121 | + |
| 122 | +# Both run concurrently — neither sees the other |
| 123 | +``` |
| 124 | + |
| 125 | +### Rubric Design |
| 126 | + |
| 127 | +The rubric is the most important input. Vague rubrics produce vague reviews. Every criterion must have an objective pass/fail condition. |
| 128 | + |
| 129 | +| Criterion | Pass Condition | Failure Signal | |
| 130 | +|-----------|---------------|----------------| |
| 131 | +| Factual accuracy | All claims verifiable against source material or common knowledge | Invented statistics, wrong version numbers, nonexistent APIs | |
| 132 | +| Hallucination-free | No fabricated entities, quotes, URLs, or references | Links to pages that don't exist, attributed quotes with no source | |
| 133 | +| Completeness | Every requirement in the spec is addressed | Missing sections, skipped edge cases, incomplete coverage | |
| 134 | +| Compliance | Passes all project-specific constraints | Banned terms used, tone violations, regulatory non-compliance | |
| 135 | +| Internal consistency | No contradictions within the output | Section A says X, section B says not-X | |
| 136 | +| Technical correctness | Code compiles/runs, algorithms are sound | Syntax errors, logic bugs, wrong complexity claims | |
| 137 | + |
| 138 | +#### Domain-Specific Rubric Extensions |
| 139 | + |
| 140 | +**Content/Marketing:** |
| 141 | +- Brand voice adherence |
| 142 | +- SEO requirements met (keyword density, meta tags, structure) |
| 143 | +- No competitor trademark misuse |
| 144 | +- CTA present and correctly linked |
| 145 | + |
| 146 | +**Code:** |
| 147 | +- Type safety (no `any` leaks, proper null handling) |
| 148 | +- Error handling coverage |
| 149 | +- Security (no secrets in code, input validation, injection prevention) |
| 150 | +- Test coverage for new paths |
| 151 | + |
| 152 | +**Compliance-Sensitive (regulated, legal, financial):** |
| 153 | +- No outcome guarantees or unsubstantiated claims |
| 154 | +- Required disclaimers present |
| 155 | +- Approved terminology only |
| 156 | +- Jurisdiction-appropriate language |
| 157 | + |
| 158 | +### Phase 3: Naughty or Nice (Verdict Gate) |
| 159 | + |
| 160 | +```python |
| 161 | +def santa_verdict(review_b, review_c): |
| 162 | + """Both reviewers must pass. No partial credit.""" |
| 163 | + if review_b.verdict == "PASS" and review_c.verdict == "PASS": |
| 164 | + return "NICE" # Ship it |
| 165 | + |
| 166 | + # Merge flags from both reviewers, deduplicate |
| 167 | + all_issues = dedupe(review_b.critical_issues + review_c.critical_issues) |
| 168 | + all_suggestions = dedupe(review_b.suggestions + review_c.suggestions) |
| 169 | + |
| 170 | + return "NAUGHTY", all_issues, all_suggestions |
| 171 | +``` |
| 172 | + |
| 173 | +Why both must pass: if only one reviewer catches an issue, that issue is real. The other reviewer's blind spot is exactly the failure mode Santa Method exists to eliminate. |
| 174 | + |
| 175 | +### Phase 4: Fix Until Nice (Convergence Loop) |
| 176 | + |
| 177 | +```python |
| 178 | +MAX_ITERATIONS = 3 |
| 179 | + |
| 180 | +for iteration in range(MAX_ITERATIONS): |
| 181 | + verdict, issues, suggestions = santa_verdict(review_b, review_c) |
| 182 | + |
| 183 | + if verdict == "NICE": |
| 184 | + log_santa_result(output, iteration, "passed") |
| 185 | + return ship(output) |
| 186 | + |
| 187 | + # Fix all critical issues (suggestions are optional) |
| 188 | + output = fix_agent.execute( |
| 189 | + output=output, |
| 190 | + issues=issues, |
| 191 | + instruction="Fix ONLY the flagged issues. Do not refactor or add unrequested changes." |
| 192 | + ) |
| 193 | + |
| 194 | + # Re-run BOTH reviewers on fixed output (fresh agents, no memory of previous round) |
| 195 | + review_b = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...)) |
| 196 | + review_c = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...)) |
| 197 | + |
| 198 | +# Exhausted iterations — escalate |
| 199 | +log_santa_result(output, MAX_ITERATIONS, "escalated") |
| 200 | +escalate_to_human(output, issues) |
| 201 | +``` |
| 202 | + |
| 203 | +Critical: each review round uses **fresh agents**. Reviewers must not carry memory from previous rounds, as prior context creates anchoring bias. |
| 204 | + |
| 205 | +## Implementation Patterns |
| 206 | + |
| 207 | +### Pattern A: Claude Code Subagents (Recommended) |
| 208 | + |
| 209 | +Subagents provide true context isolation. Each reviewer is a separate process with no shared state. |
| 210 | + |
| 211 | +```bash |
| 212 | +# In a Claude Code session, use the Agent tool to spawn reviewers |
| 213 | +# Both agents run in parallel for speed |
| 214 | +``` |
| 215 | + |
| 216 | +```python |
| 217 | +# Pseudocode for Agent tool invocation |
| 218 | +reviewer_b = Agent( |
| 219 | + description="Santa Review B", |
| 220 | + prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}" |
| 221 | +) |
| 222 | +reviewer_c = Agent( |
| 223 | + description="Santa Review C", |
| 224 | + prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}" |
| 225 | +) |
| 226 | +``` |
| 227 | + |
| 228 | +### Pattern B: Sequential Inline (Fallback) |
| 229 | + |
| 230 | +When subagents aren't available, simulate isolation with explicit context resets: |
| 231 | + |
| 232 | +1. Generate output |
| 233 | +2. New context: "You are Reviewer 1. Evaluate ONLY against this rubric. Find problems." |
| 234 | +3. Record findings verbatim |
| 235 | +4. Clear context completely |
| 236 | +5. New context: "You are Reviewer 2. Evaluate ONLY against this rubric. Find problems." |
| 237 | +6. Compare both reviews, fix, repeat |
| 238 | + |
| 239 | +The subagent pattern is strictly superior — inline simulation risks context bleed between reviewers. |
| 240 | + |
| 241 | +### Pattern C: Batch Sampling |
| 242 | + |
| 243 | +For large batches (100+ items), full Santa on every item is cost-prohibitive. Use stratified sampling: |
| 244 | + |
| 245 | +1. Run Santa on a random sample (10-15% of batch, minimum 5 items) |
| 246 | +2. Categorize failures by type (hallucination, compliance, completeness, etc.) |
| 247 | +3. If systematic patterns emerge, apply targeted fixes to the entire batch |
| 248 | +4. Re-sample and re-verify the fixed batch |
| 249 | +5. Continue until a clean sample passes |
| 250 | + |
| 251 | +```python |
| 252 | +import random |
| 253 | + |
| 254 | +def santa_batch(items, rubric, sample_rate=0.15): |
| 255 | + sample = random.sample(items, max(5, int(len(items) * sample_rate))) |
| 256 | + |
| 257 | + for item in sample: |
| 258 | + result = santa_full(item, rubric) |
| 259 | + if result.verdict == "NAUGHTY": |
| 260 | + pattern = classify_failure(result.issues) |
| 261 | + items = batch_fix(items, pattern) # Fix all items matching pattern |
| 262 | + return santa_batch(items, rubric) # Re-sample |
| 263 | + |
| 264 | + return items # Clean sample → ship batch |
| 265 | +``` |
| 266 | + |
| 267 | +## Failure Modes and Mitigations |
| 268 | + |
| 269 | +| Failure Mode | Symptom | Mitigation | |
| 270 | +|-------------|---------|------------| |
| 271 | +| Infinite loop | Reviewers keep finding new issues after fixes | Max iteration cap (3). Escalate. | |
| 272 | +| Rubber stamping | Both reviewers pass everything | Adversarial prompt: "Your job is to find problems, not approve." | |
| 273 | +| Subjective drift | Reviewers flag style preferences, not errors | Tight rubric with objective pass/fail criteria only | |
| 274 | +| Fix regression | Fixing issue A introduces issue B | Fresh reviewers each round catch regressions | |
| 275 | +| Reviewer agreement bias | Both reviewers miss the same thing | Mitigated by independence, not eliminated. For critical output, add a third reviewer or human spot-check. | |
| 276 | +| Cost explosion | Too many iterations on large outputs | Batch sampling pattern. Budget caps per verification cycle. | |
| 277 | + |
| 278 | +## Integration with Other Skills |
| 279 | + |
| 280 | +| Skill | Relationship | |
| 281 | +|-------|-------------| |
| 282 | +| Verification Loop | Use for deterministic checks (build, lint, test). Santa for semantic checks (accuracy, hallucinations). Run verification-loop first, Santa second. | |
| 283 | +| Eval Harness | Santa Method results feed eval metrics. Track pass@k across Santa runs to measure generator quality over time. | |
| 284 | +| Continuous Learning v2 | Santa findings become instincts. Repeated failures on the same criterion → learned behavior to avoid the pattern. | |
| 285 | +| Strategic Compact | Run Santa BEFORE compacting. Don't lose review context mid-verification. | |
| 286 | + |
| 287 | +## Metrics |
| 288 | + |
| 289 | +Track these to measure Santa Method effectiveness: |
| 290 | + |
| 291 | +- **First-pass rate**: % of outputs that pass Santa on round 1 (target: >70%) |
| 292 | +- **Mean iterations to convergence**: average rounds to NICE (target: <1.5) |
| 293 | +- **Issue taxonomy**: distribution of failure types (hallucination vs. completeness vs. compliance) |
| 294 | +- **Reviewer agreement**: % of issues flagged by both reviewers vs. only one (low agreement = rubric needs tightening) |
| 295 | +- **Escape rate**: issues found post-ship that Santa should have caught (target: 0) |
| 296 | + |
| 297 | +## Cost Analysis |
| 298 | + |
| 299 | +Santa Method costs approximately 2-3x the token cost of generation alone per verification cycle. For most high-stakes output, this is a bargain: |
| 300 | + |
| 301 | +``` |
| 302 | +Cost of Santa = (generation tokens) + 2×(review tokens per round) × (avg rounds) |
| 303 | +Cost of NOT Santa = (reputation damage) + (correction effort) + (trust erosion) |
| 304 | +``` |
| 305 | + |
| 306 | +For batch operations, the sampling pattern reduces cost to ~15-20% of full verification while catching >90% of systematic issues. |
0 commit comments