This chapter addresses constraining, validating, and monitoring model outputs so that unsafe, malformed, or high-risk responses cannot reach users or downstream systems.
Model outputs must be structured and validated to reduce downstream injection risk.
| # | Description | Level |
|---|---|---|
| 7.1.1 | Verify that the application validates all model outputs against a defined schema and rejects any output that does not match. | 1 |
| 7.1.2 | Verify that model-generated output is bounded by length limits and termination controls. | 1 |
| 7.1.3 | Verify that, when an application uses an AI response to decide whether to perform an action, it does not perform that action if the AI refuses the request, reports a failure, or returns a response the application cannot correctly read or understand. | 2 |
Potentially inaccurate or fabricated content must be detected so unreliable outputs do not reach users or downstream systems.
| # | Description | Level |
|---|---|---|
| 7.2.1 | Verify that the system assesses the reliability of generated answers using a confidence estimation method. | 2 |
| 7.2.2 | Verify that the application automatically blocks answers or switches to a fallback message if the confidence score drops below a defined threshold. | 2 |
| 7.2.3 | Verify that for responses classified as high-risk by policy, the system performs an additional verification step. | 3 |
Technical controls must detect and remove unsafe content before it is shown to the user.
| # | Description | Level |
|---|---|---|
| 7.3.1 | Verify that automated classifiers scan every response and block content that matches defined harmful content categories. | 1 |
| 7.3.2 | Verify that output filters detect and block responses that disclose system prompt content or backend data. | 2 |
| 7.3.3 | Verify that model-generated output is prevented from triggering outbound requests. | 2 |
| 7.3.4 | Verify that model outputs are checked for hidden, encoded, or misleading content created through homoglyphs, formatting, metadata, or structured fields. | 3 |
RAG-grounded outputs must be traceable to their source documents, with cited claims verifiably supported by retrieved content.
| # | Description | Level |
|---|---|---|
| 7.4.1 | Verify that responses generated using retrieval-augmented generation (RAG) include attribution to the source documents. | 1 |
| 7.4.2 | Verify that RAG attributions are derived from retrieval metadata and are not generated by the model, so provenance cannot be fabricated. | 1 |
| 7.4.3 | Verify that claims in a RAG response can be traced to the retrieved chunk. | 2 |
| 7.4.4 | Verify that generated media is watermarked to prove it was AI-generated. | 3 |