Skip to content

Latest commit

 

History

History
44 lines (31 loc) · 3.78 KB

0x10-C02-Input-Validation.md

File metadata and controls

44 lines (31 loc) · 3.78 KB

C2 Input Validation

Control Objective

This chapter addresses validation of all inputs as a first-line defense against prompt injection, one of the most damaging attacks on AI systems.


C2.1 Prompt Injection Defenses

Prompt injection is one of the top risks for AI systems, and defending against it requires a combination of pattern filters, data classifiers, and instruction hierarchy enforcement.

# Description Level
2.1.1 Verify that input normalization is applied before tokenization or embedding. 1
2.1.2 Verify that encoding and representation smuggling in inputs is detected and mitigated. Approved mitigations include canonicalization, strict schema validation, policy-based rejection, or explicit marking. 1
2.1.3 Verify that all inputs that could steer model behavior are treated as untrusted and screened by a prompt injection detection ruleset or classifier, with flagged inputs blocked. 1
2.1.4 Verify that input length controls prevent content from exceeding the context window. The controls must reject inputs that exceed token limits rather than truncating them. 1
2.1.5 Verify that the system implements a character set restriction for all inputs. The restriction must use an allow-list approach derived from the languages and use cases the system supports, permitting only characters that are explicitly required. 1
2.1.6 Verify that the system enforces an instruction hierarchy in which system and developer messages override user instructions and other untrusted inputs, even after user instructions have been processed. Untrusted content cannot acquire higher instruction priority through retrieval, tool processing, or persistence across conversation turns. 2
2.1.7 Verify that reserved special tokens are encoded as literal characters and cannot be injected into the model context. 2
2.1.8 Verify that the system can detect many-shot jailbreaking patterns. 3

C2.2 Content & Policy Screening

Syntactically valid prompts may still request disallowed content such as policy-violating instructions, harmful material, or restricted information. Input-side content screening prevents such prompts from reaching the model.

# Description Level
2.2.1 Verify that every prompt is scored by a content classifier for violence, self-harm, hate, and sexual content against configurable thresholds. Prompts that exceed those thresholds are rejected or sanitized before reaching the model context. 1
2.2.2 Verify that prompt content classification is evaluated for unsupported languages. 1
2.2.3 Verify that non-text inputs (image/video/audio) are checked for adversarial perturbations, steganographic payloads, hidden or embedded content, or known attack patterns. 2
2.2.4 Verify that coordinated attacks spanning multiple input types (e.g., steganographic payloads in images combined with prompt injection in text) are detected and blocked. 3

References