Skip to content

Commit e4d3f05

Browse files
korjavinclaude
authored andcommitted
docs: add whitespace padding detection (P9) implementation plan
Plan for issue NVIDIA#20 — detect large whitespace padding used to hide prompt-injection instructions from review. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Korzhavin Ivan <[email protected]>
1 parent 1dbfef0 commit e4d3f05

1 file changed

Lines changed: 188 additions & 0 deletions

File tree

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Detect Whitespace Padding Injection (P9)
2+
3+
Implements GitHub issue #20 — "Detect large whitespace padding used to hide prompt-injection instructions from review".
4+
5+
## Overview
6+
7+
- Attackers can pad a skill file (e.g. `SKILL.md`) with a large block of whitespace — dozens of blank lines, or a long horizontal run of spaces — so that injected instructions sit below or to the right of anything a human reviewer sees, while the agent reads the whole file and acts on them. The text-file equivalent of white-on-white text in a PDF.
8+
- Existing patterns miss this: `P2` (Hidden Instructions) keys off zero-width chars/comments/base64, `TP2` (Unicode Deception) off homoglyphs/RTL, and `MP2` (Context Window Stuffing) off character *repetition* — its regex anchors on `\S`, so runs of blank lines slip through.
9+
- This plan adds a new combined rule **P9 "Whitespace Padding"** (category: Prompt Injection) covering three signals, plus the same detection over MCP manifest description fields.
10+
- **Critical requirement from the issue:** "whitespace" must mean Unicode whitespace (categories `Zs`, `Zl`, `Zp`, control chars `\t \n \r \v \f`, and the zero-width family U+200B/U+200C/U+200D/U+2060/U+FEFF) — not just ASCII space/tab. The zero-width set must be **one shared definition with P2**, not a drifting copy.
11+
12+
### Decisions made during planning
13+
14+
| Question (from issue #20) | Decision |
15+
|---|---|
16+
| Placement | `P`-series, inside `static_patterns_prompt_injection.py` next to P2 |
17+
| Rule id | **P9** — one combined id for all three signals (P6–P8 are already taken by System Prompt Leakage); per-signal confidence carries the weighting |
18+
| Severity | Vertical/horizontal: MEDIUM (HIGH when non-blank content follows a very large vertical gap); ratio: LOW |
19+
| MCP manifest fields | **In scope** — wire the shared detector into `mcp_tool_poisoning.py` description checks, reporting the same P9 id |
20+
| Testing approach | Regular (code first, then tests, within each task) |
21+
22+
## Context (from discovery)
23+
24+
- Files/components involved:
25+
- `src/skillspector/nodes/analyzers/static_patterns_prompt_injection.py` — P1–P4 live here; P9 joins them (module is already registered as analyzer node `static_patterns_prompt_injection`, so **no `ANALYZER_NODE_IDS`/`ANALYZER_NODES` changes needed**)
26+
- `src/skillspector/nodes/analyzers/mcp_tool_poisoning.py` — `_extract_metadata_texts()` (line ~78) yields `(text, source_field, is_identifier)` tuples; `node()` (line ~807) dispatches to `_check_tp1/2/3/4`; has its own `_ZERO_WIDTH_RE` (line ~134) that should converge on the shared definition
27+
- `src/skillspector/nodes/analyzers/pattern_defaults.py` — needs P9 entries in `DEFAULT_EXPLANATIONS`, `RULE_ID_TO_CATEGORY`, `PATTERN_NAMES`, `DEFAULT_REMEDIATIONS`
28+
- `src/skillspector/nodes/analyzers/common.py` — shared helpers (`get_line_number`, `get_context`)
29+
- `README.md` — Prompt Injection pattern table (5 → 6 patterns)
30+
- Related patterns found: pattern modules expose `analyze(content, file_path, file_type) -> list[AnalyzerFinding]`; findings carry `rule_id`, `message`, `severity`, `location`, `confidence`, `tags`, `context`, `matched_text` (`models.py:46-62`). `static_runner.run_static_patterns` already skips eval datasets and files > 1 MB.
31+
- Dependencies identified: none new — stdlib `unicodedata` for category classification.
32+
33+
## Development Approach
34+
35+
- **Testing approach**: Regular (implement, then tests, within each task)
36+
- Complete each task fully before moving to the next
37+
- Make small, focused changes
38+
- **CRITICAL: every task MUST include new/updated tests** for code changes in that task
39+
- Tests are not optional — they are a required part of the checklist
40+
- Unit tests for new and modified functions; success and error/edge scenarios
41+
- **CRITICAL: all tests must pass before starting next task** — `make test-unit`
42+
- **CRITICAL: update this plan file when scope changes during implementation**
43+
- Maintain backward compatibility — P2/TP1 behavior must not change except via the shared zero-width definition (which is character-for-character identical to today's sets)
44+
45+
## Testing Strategy
46+
47+
- **Unit tests**: required for every task (see above). Pattern tests live in `tests/nodes/analyzers/test_static_patterns.py` (analyzer-level) and `tests/unit/test_patterns_new.py`; MCP tests in `tests/test_mcp_tool_poisoning.py`.
48+
- No UI, no e2e suite — `make test-unit` is the gate; `make test` (incl. integration) at the end.
49+
50+
## Progress Tracking
51+
52+
- Mark completed items with `[x]` immediately when done
53+
- Add newly discovered tasks with ➕ prefix
54+
- Document issues/blockers with ⚠️ prefix
55+
- Update plan if implementation deviates from original scope
56+
57+
## Solution Overview
58+
59+
A new pure-function helper module `whitespace_padding.py` owns the Unicode whitespace character sets and a `detect_whitespace_padding()` scanner that returns structured "padding run" records. Two consumers build findings from those records:
60+
61+
1. `static_patterns_prompt_injection.analyze()` — emits `AnalyzerFinding(rule_id="P9", ...)` for file bodies (all text files the runner feeds it).
62+
2. `mcp_tool_poisoning.node()` — a new `_check_p9_padding(text, source_field)` emits `Finding(rule_id="P9", ...)` for tool/parameter description fields (horizontal + ratio signals only; vertical gaps are meaningless in single-field descriptions at manifest granularity but blank-line runs inside a description still count as a contiguous-block signal).
63+
64+
`P2_PATTERNS`' zero-width character class is rebuilt from the shared `ZERO_WIDTH_CHARS` constant so the two patterns cannot drift (issue requirement). `mcp_tool_poisoning._ZERO_WIDTH_RE` likewise.
65+
66+
### The three signals (rule id P9 for all)
67+
68+
| # | Signal | Trigger | Severity | Confidence |
69+
|---|---|---|---|---|
70+
| 1 | Vertical blank-line run | ≥ 20 consecutive blank/whitespace-only lines | MEDIUM; **HIGH** when non-blank content follows a gap ≥ 40 lines | 0.8 when non-blank content follows the gap; 0.6 when the gap trails the file |
71+
| 2 | Horizontal whitespace run | ≥ 80 consecutive whitespace chars within a line (incl. leading indentation); fires on the run itself regardless of what follows on the line | MEDIUM | 0.7 |
72+
| 3 | Oversized whitespace ratio | a single contiguous whitespace block > 2 KB, **or** whitespace > 90% of a file that is > 4 KB | LOW | 0.4 |
73+
74+
Thresholds are module-level named constants (`VERTICAL_BLANK_LINES = 20`, `VERTICAL_HIGH_SEVERITY_LINES = 40`, `HORIZONTAL_RUN_CHARS = 80`, `BLOCK_BYTE_BUDGET = 2048`, `RATIO_THRESHOLD = 0.90`, `RATIO_MIN_FILE_BYTES = 4096`) so tuning is a one-line change.
75+
76+
### Whitespace classification (shared definitions)
77+
78+
```python
79+
# whitespace_padding.py
80+
ZERO_WIDTH_CHARS = frozenset("​‌‍⁠") # shared with P2 and mcp_tool_poisoning
81+
82+
def is_padding_char(ch: str) -> bool:
83+
# True for: ASCII controls \t \n \r \v \f; Unicode categories Zs, Zl, Zp
84+
# (covers U+00A0, U+2028, U+2029, U+3000, etc.); and ZERO_WIDTH_CHARS.
85+
```
86+
87+
Covers every evasion candidate enumerated in the issue: U+00A0, U+2028, U+2029, U+000C, U+000B, U+3000, U+200B/C/D, U+2060, U+FEFF.
88+
89+
### Detector output
90+
91+
```python
92+
@dataclass
93+
class PaddingRun:
94+
kind: str # "vertical" | "horizontal" | "block" | "ratio"
95+
start_offset: int # char offset where the run starts
96+
start_line: int # 1-based
97+
length: int # chars (or lines for "vertical")
98+
followed_by_content: bool
99+
summary: str # visible-ized snippet, e.g. "U+00A0 x82" or "\\n x82"
100+
```
101+
102+
`summarize_run()` renders the run as counts of `U+XXXX xN` segments (collapsing mixed runs to the top few char codes) so the reviewer can *see* what was hidden — per the issue's reporting requirement.
103+
104+
### False-positive guards
105+
106+
- **Fenced code blocks**: the horizontal signal skips runs whose line falls inside a Markdown ``` fence region (line-based fence toggle scan; only applied when `file_type == "markdown"`). Vertical and ratio signals are unaffected (20 blank lines inside a fence is still suspicious).
107+
- **Generated/vendored files**: skip detection entirely for filenames matching `*.min.js`, `*.min.css`, `*.lock`, `package-lock.json`, `yarn.lock`, `*.svg`, `*.map`.
108+
- **Binary-ish content**: skip when content contains U+FFFD (the `errors="replace"` marker from `build_context`) — the repo has no other binary classification.
109+
- **Ratio signal stays LOW/0.4** so it informs rather than dominates the score.
110+
- Eval-dataset prose and > 1 MB files are already skipped upstream by `static_runner`.
111+
112+
## Technical Details
113+
114+
- **Processing flow (file bodies):** `static_runner` → `static_patterns_prompt_injection.analyze()` → after the P4 loop, call `detect_whitespace_padding(content, file_path=..., file_type=...)` → map each `PaddingRun` to an `AnalyzerFinding` with `rule_id="P9"`, `message="Whitespace Padding"`, `tags=[PatternCategory.PROMPT_INJECTION.value]`, `matched_text=run.summary`, `context=get_context(content, run.start_offset)`. P9 runs for **all** file types fed by the runner (unlike P2's markdown/other restriction), minus the vendored/binary guards above — padding in a `.py` or `.txt` body is the same attack.
115+
- **Processing flow (MCP manifests):** in `mcp_tool_poisoning.node()`, alongside the `_check_tp1/_check_tp2` dispatch over `_extract_metadata_texts()`, call `_check_p9_padding(text, source_field)` for non-identifier fields. Manifest fields use the same `HORIZONTAL_RUN_CHARS` and `BLOCK_BYTE_BUDGET` thresholds for the first cut; the per-file ratio signal is skipped (fields are too short for a 4 KB floor to ever apply). Findings use the module's existing `Finding` construction style with `rule_id="P9"` and the MCP `_FRAMEWORK_TAGS`.
116+
- **Dedup within a file:** signals 1/2 report each distinct run; signal 3 ("block"/"ratio") reports at most one finding per file. A vertical run that also exceeds the 2 KB block budget reports only the vertical finding (higher-signal id wins; suppress the block record when its span equals a vertical run's span).
117+
- **Line/offset reporting:** finding points at the line where the padding **starts** (`get_line_number(content, run.start_offset)`); for a newline-free horizontal run the same line number plus the summary's char codes locate it.
118+
- **Shared zero-width definition:** `P2_PATTERNS[2]` regex is built as `"[" + "".join(ZERO_WIDTH_CHARS) + "]"` (import from `whitespace_padding`); `mcp_tool_poisoning._ZERO_WIDTH_RE` rebuilt from the same constant (note: its current set lacks U+2060/U+FEFF — converging is a strict coverage improvement, and U+2060 is also independently flagged by TP2's invisible-chars check, which is fine: different rule, different meaning).
119+
120+
## What Goes Where
121+
122+
- **Implementation Steps** (`[ ]` checkboxes): code, tests, README/docs — all in this repo.
123+
- **Post-Completion** (no checkboxes): threshold tuning against real-world skill corpora, issue/PR follow-ups.
124+
125+
## Implementation Steps
126+
127+
### Task 1: Whitespace padding detector helper module
128+
129+
**Files:**
130+
- Create: `src/skillspector/nodes/analyzers/whitespace_padding.py`
131+
- Create: `tests/nodes/analyzers/test_whitespace_padding.py`
132+
133+
- [ ] create `whitespace_padding.py` with `ZERO_WIDTH_CHARS`, `is_padding_char()` (unicodedata category `Z*` + controls + zero-width), threshold constants, `PaddingRun` dataclass, and `summarize_run()` visible-izer (`U+00A0 x82` / `\n x82` rendering)
134+
- [ ] implement `detect_whitespace_padding(content, *, file_type="other") -> list[PaddingRun]` covering all three signals: vertical blank-line runs (with `followed_by_content`), horizontal in-line runs, contiguous block > 2 KB and > 90%-of-file ratio
135+
- [ ] implement false-positive guards: Markdown fence-region skip for the horizontal signal, U+FFFD (binary-ish) bail-out, vertical-run/block dedup
136+
- [ ] write tests: each signal fires at its threshold and not below it (19 blank lines no, 20 yes; 79 ws chars no, 80 yes; 2 KB block boundary)
137+
- [ ] write tests: Unicode evasion cases — padding made of U+00A0, U+2028/U+2029, U+000B/U+000C, U+3000, and zero-width chars all detected; `summarize_run` renders `U+00A0 x82`-style output
138+
- [ ] write tests: guards — horizontal run inside a ``` fence not reported for markdown, reported for non-markdown; content with U+FFFD returns no runs; `followed_by_content` true/false distinguished
139+
- [ ] run `make test-unit` — must pass before task 2
140+
141+
### Task 2: P9 findings in the prompt-injection analyzer + shared zero-width definition
142+
143+
**Files:**
144+
- Modify: `src/skillspector/nodes/analyzers/static_patterns_prompt_injection.py`
145+
- Modify: `src/skillspector/nodes/analyzers/pattern_defaults.py`
146+
- Modify: `tests/nodes/analyzers/test_static_patterns.py`
147+
148+
- [ ] in `static_patterns_prompt_injection.py`: import the detector, add a P9 block in `analyze()` mapping `PaddingRun` → `AnalyzerFinding` (severity/confidence per the signal table above; `matched_text=run.summary`); add the vendored-filename skip; update module docstring "(P1–P4)" → "(P1–P4, P9)"
149+
- [ ] rebuild `P2_PATTERNS`' zero-width regex from the shared `ZERO_WIDTH_CHARS` constant (no behavior change — same five chars)
150+
- [ ] add P9 to `pattern_defaults.py`: `DEFAULT_EXPLANATIONS`, `RULE_ID_TO_CATEGORY` (→ `PatternCategory.PROMPT_INJECTION`), `PATTERN_NAMES` ("Whitespace Padding"), `DEFAULT_REMEDIATIONS`
151+
- [ ] write tests in `test_static_patterns.py`: a SKILL.md body with 80 blank lines then an injected instruction yields a P9 finding with HIGH severity, correct `start_line` (start of the gap), and a visible-ized `matched_text`; trailing-gap variant yields MEDIUM/0.6; horizontal and ratio variants yield their severities; `*.min.js` path yields no P9
152+
- [ ] write test: existing P2 zero-width detection still fires identically after the shared-constant refactor
153+
- [ ] run `make test-unit` — must pass before task 3
154+
155+
### Task 3: P9 over MCP manifest description fields
156+
157+
**Files:**
158+
- Modify: `src/skillspector/nodes/analyzers/mcp_tool_poisoning.py`
159+
- Modify: `tests/test_mcp_tool_poisoning.py`
160+
161+
- [ ] add `_check_p9_padding(text, source_field) -> list[Finding]` using the shared detector (horizontal + contiguous-block signals; skip per-file ratio); wire it into `node()`'s metadata-text loop for non-identifier fields
162+
- [ ] rebuild `_ZERO_WIDTH_RE` from the shared `ZERO_WIDTH_CHARS` constant (adds U+2060/U+FEFF coverage to TP1's hidden-text check — strict improvement; note it in the docstring)
163+
- [ ] write tests: a tool description padded with 100 spaces before an instruction yields a P9 finding naming the source field; a normal multi-sentence description yields none; identifier fields are not scanned
164+
- [ ] write test: TP1 zero-width behavior unchanged for the original three chars, and now also fires on U+2060/U+FEFF
165+
- [ ] run `make test-unit` — must pass before task 4
166+
167+
### Task 4: Verify acceptance criteria
168+
169+
- [ ] verify all issue #20 requirements: three signals implemented, Unicode-category-based classification, shared zero-width definition with P2, visible-ized snippets, line/offset reporting, fenced-code + vendored-file + ratio-confidence FP guards, MCP manifest coverage
170+
- [ ] adversarial self-check: craft a SKILL.md using each padding char from the issue's evasion list (U+00A0, U+2028, U+2029, U+000C, U+000B, U+3000, zero-width family) and confirm P9 fires on every one via the test suite
171+
- [ ] run full suite: `make test` (unit + integration)
172+
- [ ] run a real scan over a fixture skill (`uv run skillspector scan <fixture>` or project equivalent) and eyeball the P9 finding rendering in the report output
173+
174+
### Task 5: [Final] Update documentation
175+
176+
- [ ] add P9 row to the README Prompt Injection table and bump its "(5 patterns)" count and the total pattern count
177+
- [ ] update CLAUDE.md if a new pattern-authoring convention emerged (shared detector helper pattern)
178+
- [ ] comment on issue #20 summarizing the implemented signals/thresholds and the P9 id choice (P6–P8 were taken)
179+
- [ ] move this plan to `docs/plans/completed/`
180+
181+
## Post-Completion
182+
183+
**Manual verification:**
184+
- Tune thresholds against a corpus of real, benign skills (templates with spacer lines, ASCII-art-heavy READMEs) before relying on the ratio signal at anything above LOW confidence; issue #20 explicitly expects tuning.
185+
- Security review of the regex/scan performance on pathological inputs (the detector is a linear scan, no backtracking regex, but confirm on a 1 MB all-whitespace file).
186+
187+
**External follow-ups:**
188+
- Open PR against `NVIDIA/SkillSpector` referencing issue #20; the open questions answered here (single P9 id, MEDIUM/MEDIUM/LOW severities, MCP manifests in scope) should be re-stated in the PR description for maintainer sign-off, since the issue author offered to align on signals/thresholds before a PR.

0 commit comments

Comments
 (0)