Skip to content

Latest commit

 

History

History
65 lines (44 loc) · 4.12 KB

0x10-C07-Model-Behavior.md

File metadata and controls

65 lines (44 loc) · 4.12 KB

C7 Model Behavior, Output Control & Safety Assurance

Control Objective

This chapter addresses constraining, validating, and monitoring model outputs so that unsafe, malformed, or high-risk responses cannot reach users or downstream systems.


C7.1 Output Format Enforcement

Model outputs must be structured and validated to reduce downstream injection risk.

# Description Level
7.1.1 Verify that the application validates all model outputs against a defined schema and rejects any output that does not match. 1
7.1.2 Verify that model-generated output is bounded by length limits and termination controls. 1
7.1.3 Verify that, when an application uses an AI response to decide whether to perform an action, it does not perform that action if the AI refuses the request, reports a failure, or returns a response the application cannot correctly read or understand. 2

C7.2 Hallucination Detection & Mitigation

Potentially inaccurate or fabricated content must be detected so unreliable outputs do not reach users or downstream systems.

# Description Level
7.2.1 Verify that the system assesses the reliability of generated answers using a confidence estimation method. 2
7.2.2 Verify that the application automatically blocks answers or switches to a fallback message if the confidence score drops below a defined threshold. 2
7.2.3 Verify that for responses classified as high-risk by policy, the system performs an additional verification step. 3

C7.3 Output Safety

Technical controls must detect and remove unsafe content before it is shown to the user.

# Description Level
7.3.1 Verify that automated classifiers scan every response and block content that matches defined harmful content categories. 1
7.3.2 Verify that output filters detect and block responses that disclose system prompt content or backend data. 2
7.3.3 Verify that model-generated output is prevented from triggering outbound requests. 2
7.3.4 Verify that model outputs are checked for hidden, encoded, or misleading content created through homoglyphs, formatting, metadata, or structured fields. 3

C7.4 Source Attribution & Citation Integrity

RAG-grounded outputs must be traceable to their source documents, with cited claims verifiably supported by retrieved content.

# Description Level
7.4.1 Verify that responses generated using retrieval-augmented generation (RAG) include attribution to the source documents. 1
7.4.2 Verify that RAG attributions are derived from retrieval metadata and are not generated by the model, so provenance cannot be fabricated. 1
7.4.3 Verify that claims in a RAG response can be traced to the retrieved chunk. 2
7.4.4 Verify that generated media is watermarked to prove it was AI-generated. 3

References