Skip to content

Commit 6f13b05

Browse files
feat(skills): add santa-method - multi-agent adversarial verification (anomalyco#760)
* feat(skills): add santa-method Multi-agent adversarial verification with convergence loop. Two independent review agents evaluate output against a shared rubric. Both must pass before shipping. Includes architecture diagram, implementation patterns (subagent, inline, batch sampling), domain-specific rubric extensions, failure mode mitigations, and integration guidance with existing ECC skills. * Enhance SKILL.md with detailed Santa Method documentation Expanded the SKILL.md documentation for the Santa Method, detailing architecture, phases, implementation patterns, failure modes, integration with other skills, metrics, and cost analysis.
1 parent 0e73375 commit 6f13b05

1 file changed

Lines changed: 306 additions & 0 deletions

File tree

‎skills/santa-method/SKILL.md‎

Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
---
2+
name: santa-method
3+
description: "Multi-agent adversarial verification with convergence loop. Two independent review agents must both pass before output ships."
4+
origin: "Ronald Skelton - Founder, RapportScore.ai"
5+
---
6+
7+
# Santa Method
8+
9+
Multi-agent adversarial verification framework. Make a list, check it twice. If it's naughty, fix it until it's nice.
10+
11+
The core insight: a single agent reviewing its own output shares the same biases, knowledge gaps, and systematic errors that produced the output. Two independent reviewers with no shared context break this failure mode.
12+
13+
## When to Activate
14+
15+
Invoke this skill when:
16+
- Output will be published, deployed, or consumed by end users
17+
- Compliance, regulatory, or brand constraints must be enforced
18+
- Code ships to production without human review
19+
- Content accuracy matters (technical docs, educational material, customer-facing copy)
20+
- Batch generation at scale where spot-checking misses systemic patterns
21+
- Hallucination risk is elevated (claims, statistics, API references, legal language)
22+
23+
Do NOT use for internal drafts, exploratory research, or tasks with deterministic verification (use build/test/lint pipelines for those).
24+
25+
## Architecture
26+
27+
```
28+
┌─────────────┐
29+
│ GENERATOR │ Phase 1: Make a List
30+
│ (Agent A) │ Produce the deliverable
31+
└──────┬───────┘
32+
│ output
33+
▼
34+
┌──────────────────────────────┐
35+
│ DUAL INDEPENDENT REVIEW │ Phase 2: Check It Twice
36+
│ │
37+
│ ┌───────────┐ ┌───────────┐ │ Two agents, same rubric,
38+
│ │ Reviewer B │ │ Reviewer C │ │ no shared context
39+
│ └─────┬─────┘ └─────┬─────┘ │
40+
│ │ │ │
41+
└────────┼──────────────┼────────┘
42+
│ │
43+
▼ ▼
44+
┌──────────────────────────────┐
45+
│ VERDICT GATE │ Phase 3: Naughty or Nice
46+
│ │
47+
│ B passes AND C passes → NICE │ Both must pass.
48+
│ Otherwise → NAUGHTY │ No exceptions.
49+
└──────┬──────────────┬─────────┘
50+
│ │
51+
NICE NAUGHTY
52+
│ │
53+
▼ ▼
54+
[ SHIP ] ┌─────────────┐
55+
│ FIX CYCLE │ Phase 4: Fix Until Nice
56+
│ │
57+
│ iteration++ │ Collect all flags.
58+
│ if i > MAX: │ Fix all issues.
59+
│ escalate │ Re-run both reviewers.
60+
│ else: │ Loop until convergence.
61+
│ goto Ph.2 │
62+
└──────────────┘
63+
```
64+
65+
## Phase Details
66+
67+
### Phase 1: Make a List (Generate)
68+
69+
Execute the primary task. No changes to your normal generation workflow. Santa Method is a post-generation verification layer, not a generation strategy.
70+
71+
```python
72+
# The generator runs as normal
73+
output = generate(task_spec)
74+
```
75+
76+
### Phase 2: Check It Twice (Independent Dual Review)
77+
78+
Spawn two review agents in parallel. Critical invariants:
79+
80+
1. **Context isolation** — neither reviewer sees the other's assessment
81+
2. **Identical rubric** — both receive the same evaluation criteria
82+
3. **Same inputs** — both receive the original spec AND the generated output
83+
4. **Structured output** — each returns a typed verdict, not prose
84+
85+
```python
86+
REVIEWER_PROMPT = """
87+
You are an independent quality reviewer. You have NOT seen any other review of this output.
88+
89+
## Task Specification
90+
{task_spec}
91+
92+
## Output Under Review
93+
{output}
94+
95+
## Evaluation Rubric
96+
{rubric}
97+
98+
## Instructions
99+
Evaluate the output against EACH rubric criterion. For each:
100+
- PASS: criterion fully met, no issues
101+
- FAIL: specific issue found (cite the exact problem)
102+
103+
Return your assessment as structured JSON:
104+
{
105+
"verdict": "PASS" | "FAIL",
106+
"checks": [
107+
{"criterion": "...", "result": "PASS|FAIL", "detail": "..."}
108+
],
109+
"critical_issues": ["..."], // blockers that must be fixed
110+
"suggestions": ["..."] // non-blocking improvements
111+
}
112+
113+
Be rigorous. Your job is to find problems, not to approve.
114+
"""
115+
```
116+
117+
```python
118+
# Spawn reviewers in parallel (Claude Code subagents)
119+
review_b = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer B")
120+
review_c = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer C")
121+
122+
# Both run concurrently — neither sees the other
123+
```
124+
125+
### Rubric Design
126+
127+
The rubric is the most important input. Vague rubrics produce vague reviews. Every criterion must have an objective pass/fail condition.
128+
129+
| Criterion | Pass Condition | Failure Signal |
130+
|-----------|---------------|----------------|
131+
| Factual accuracy | All claims verifiable against source material or common knowledge | Invented statistics, wrong version numbers, nonexistent APIs |
132+
| Hallucination-free | No fabricated entities, quotes, URLs, or references | Links to pages that don't exist, attributed quotes with no source |
133+
| Completeness | Every requirement in the spec is addressed | Missing sections, skipped edge cases, incomplete coverage |
134+
| Compliance | Passes all project-specific constraints | Banned terms used, tone violations, regulatory non-compliance |
135+
| Internal consistency | No contradictions within the output | Section A says X, section B says not-X |
136+
| Technical correctness | Code compiles/runs, algorithms are sound | Syntax errors, logic bugs, wrong complexity claims |
137+
138+
#### Domain-Specific Rubric Extensions
139+
140+
**Content/Marketing:**
141+
- Brand voice adherence
142+
- SEO requirements met (keyword density, meta tags, structure)
143+
- No competitor trademark misuse
144+
- CTA present and correctly linked
145+
146+
**Code:**
147+
- Type safety (no `any` leaks, proper null handling)
148+
- Error handling coverage
149+
- Security (no secrets in code, input validation, injection prevention)
150+
- Test coverage for new paths
151+
152+
**Compliance-Sensitive (regulated, legal, financial):**
153+
- No outcome guarantees or unsubstantiated claims
154+
- Required disclaimers present
155+
- Approved terminology only
156+
- Jurisdiction-appropriate language
157+
158+
### Phase 3: Naughty or Nice (Verdict Gate)
159+
160+
```python
161+
def santa_verdict(review_b, review_c):
162+
"""Both reviewers must pass. No partial credit."""
163+
if review_b.verdict == "PASS" and review_c.verdict == "PASS":
164+
return "NICE" # Ship it
165+
166+
# Merge flags from both reviewers, deduplicate
167+
all_issues = dedupe(review_b.critical_issues + review_c.critical_issues)
168+
all_suggestions = dedupe(review_b.suggestions + review_c.suggestions)
169+
170+
return "NAUGHTY", all_issues, all_suggestions
171+
```
172+
173+
Why both must pass: if only one reviewer catches an issue, that issue is real. The other reviewer's blind spot is exactly the failure mode Santa Method exists to eliminate.
174+
175+
### Phase 4: Fix Until Nice (Convergence Loop)
176+
177+
```python
178+
MAX_ITERATIONS = 3
179+
180+
for iteration in range(MAX_ITERATIONS):
181+
verdict, issues, suggestions = santa_verdict(review_b, review_c)
182+
183+
if verdict == "NICE":
184+
log_santa_result(output, iteration, "passed")
185+
return ship(output)
186+
187+
# Fix all critical issues (suggestions are optional)
188+
output = fix_agent.execute(
189+
output=output,
190+
issues=issues,
191+
instruction="Fix ONLY the flagged issues. Do not refactor or add unrequested changes."
192+
)
193+
194+
# Re-run BOTH reviewers on fixed output (fresh agents, no memory of previous round)
195+
review_b = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...))
196+
review_c = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...))
197+
198+
# Exhausted iterations — escalate
199+
log_santa_result(output, MAX_ITERATIONS, "escalated")
200+
escalate_to_human(output, issues)
201+
```
202+
203+
Critical: each review round uses **fresh agents**. Reviewers must not carry memory from previous rounds, as prior context creates anchoring bias.
204+
205+
## Implementation Patterns
206+
207+
### Pattern A: Claude Code Subagents (Recommended)
208+
209+
Subagents provide true context isolation. Each reviewer is a separate process with no shared state.
210+
211+
```bash
212+
# In a Claude Code session, use the Agent tool to spawn reviewers
213+
# Both agents run in parallel for speed
214+
```
215+
216+
```python
217+
# Pseudocode for Agent tool invocation
218+
reviewer_b = Agent(
219+
description="Santa Review B",
220+
prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}"
221+
)
222+
reviewer_c = Agent(
223+
description="Santa Review C",
224+
prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}"
225+
)
226+
```
227+
228+
### Pattern B: Sequential Inline (Fallback)
229+
230+
When subagents aren't available, simulate isolation with explicit context resets:
231+
232+
1. Generate output
233+
2. New context: "You are Reviewer 1. Evaluate ONLY against this rubric. Find problems."
234+
3. Record findings verbatim
235+
4. Clear context completely
236+
5. New context: "You are Reviewer 2. Evaluate ONLY against this rubric. Find problems."
237+
6. Compare both reviews, fix, repeat
238+
239+
The subagent pattern is strictly superior — inline simulation risks context bleed between reviewers.
240+
241+
### Pattern C: Batch Sampling
242+
243+
For large batches (100+ items), full Santa on every item is cost-prohibitive. Use stratified sampling:
244+
245+
1. Run Santa on a random sample (10-15% of batch, minimum 5 items)
246+
2. Categorize failures by type (hallucination, compliance, completeness, etc.)
247+
3. If systematic patterns emerge, apply targeted fixes to the entire batch
248+
4. Re-sample and re-verify the fixed batch
249+
5. Continue until a clean sample passes
250+
251+
```python
252+
import random
253+
254+
def santa_batch(items, rubric, sample_rate=0.15):
255+
sample = random.sample(items, max(5, int(len(items) * sample_rate)))
256+
257+
for item in sample:
258+
result = santa_full(item, rubric)
259+
if result.verdict == "NAUGHTY":
260+
pattern = classify_failure(result.issues)
261+
items = batch_fix(items, pattern) # Fix all items matching pattern
262+
return santa_batch(items, rubric) # Re-sample
263+
264+
return items # Clean sample → ship batch
265+
```
266+
267+
## Failure Modes and Mitigations
268+
269+
| Failure Mode | Symptom | Mitigation |
270+
|-------------|---------|------------|
271+
| Infinite loop | Reviewers keep finding new issues after fixes | Max iteration cap (3). Escalate. |
272+
| Rubber stamping | Both reviewers pass everything | Adversarial prompt: "Your job is to find problems, not approve." |
273+
| Subjective drift | Reviewers flag style preferences, not errors | Tight rubric with objective pass/fail criteria only |
274+
| Fix regression | Fixing issue A introduces issue B | Fresh reviewers each round catch regressions |
275+
| Reviewer agreement bias | Both reviewers miss the same thing | Mitigated by independence, not eliminated. For critical output, add a third reviewer or human spot-check. |
276+
| Cost explosion | Too many iterations on large outputs | Batch sampling pattern. Budget caps per verification cycle. |
277+
278+
## Integration with Other Skills
279+
280+
| Skill | Relationship |
281+
|-------|-------------|
282+
| Verification Loop | Use for deterministic checks (build, lint, test). Santa for semantic checks (accuracy, hallucinations). Run verification-loop first, Santa second. |
283+
| Eval Harness | Santa Method results feed eval metrics. Track pass@k across Santa runs to measure generator quality over time. |
284+
| Continuous Learning v2 | Santa findings become instincts. Repeated failures on the same criterion → learned behavior to avoid the pattern. |
285+
| Strategic Compact | Run Santa BEFORE compacting. Don't lose review context mid-verification. |
286+
287+
## Metrics
288+
289+
Track these to measure Santa Method effectiveness:
290+
291+
- **First-pass rate**: % of outputs that pass Santa on round 1 (target: >70%)
292+
- **Mean iterations to convergence**: average rounds to NICE (target: <1.5)
293+
- **Issue taxonomy**: distribution of failure types (hallucination vs. completeness vs. compliance)
294+
- **Reviewer agreement**: % of issues flagged by both reviewers vs. only one (low agreement = rubric needs tightening)
295+
- **Escape rate**: issues found post-ship that Santa should have caught (target: 0)
296+
297+
## Cost Analysis
298+
299+
Santa Method costs approximately 2-3x the token cost of generation alone per verification cycle. For most high-stakes output, this is a bargain:
300+
301+
```
302+
Cost of Santa = (generation tokens) + 2×(review tokens per round) × (avg rounds)
303+
Cost of NOT Santa = (reputation damage) + (correction effort) + (trust erosion)
304+
```
305+
306+
For batch operations, the sampling pattern reduces cost to ~15-20% of full verification while catching >90% of systematic issues.

0 commit comments

Comments
 (0)