This chapter addresses keeping AI systems reliable and abuse-resistant when facing evasion, inference, extraction, or poisoning attacks.
Model resilience to manipulated inputs designed to cause misclassification or policy bypass must be increased, primarily through adversarial testing and robustness benchmarking.
| # | Description | Level |
|---|---|---|
| 11.1.1 | Verify that the model has undergone alignment and safety training or fine-tuning to prevent the model from generating disallowed content categories. | 1 |
| 11.1.2 | Verify that a version-controlled alignment test suite is run on every model update or release. | 1 |
| 11.1.3 | Verify that models are evaluated against known adversarial attack techniques relevant to their modality. | 1 |
| 11.1.4 | Verify that models are hardened against adversarial inputs. | 2 |
| 11.1.5 | Verify that an automated evaluator measures harmful-content rate and flags regressions beyond a defined threshold. | 3 |
| 11.1.6 | Verify that adversarial evaluation results are reported alongside results for the same tasks, scored the same way, with no attack applied. | 2 |
The ability to determine whether a specific record was in the training data must be limited, and reconstruction of private training data or sensitive attributes from model outputs prevented.
| # | Description | Level |
|---|---|---|
| 11.2.1 | Verify that model-inferred sensitive attributes are not directly returned in outputs. | 1 |
| 11.2.2 | Verify that inference endpoints enforce per-principal and global rate limits sized to the extraction threat model, and not solely as a generic API throttle. | 1 |
| 11.2.3 | Verify that model outputs are calibrated to reduce overconfident predictions. | 2 |
| 11.2.4 | Verify that training on sensitive datasets employs differentially-private optimization. | 2 |
| 11.2.5 | Verify that membership-inference attack simulations demonstrate that attack accuracy does not exceed random guessing on evaluated data. | 3 |
Unauthorized model cloning through API abuse must be detected and deterred using rate limiting, query-pattern analysis, and watermarking.
| # | Description | Level |
|---|---|---|
| 11.3.1 | Verify that query-pattern analysis feeds an extraction-attempt detector. | 1 |
| 11.3.2 | Verify that raw model outputs are not directly exposed beyond the application backend, and that externally visible responses are calibrated to the extraction risk level. | 2 |
| 11.3.3 | Verify that model watermarking or fingerprinting techniques are applied so that unauthorized copies can be identified. | 3 |
| 11.3.4 | Verify that detection of suspected extraction triggers response measures. | 3 |
| 11.3.5 | Verify that inference endpoints constrain sampling and decoding parameters supplied by untrusted callers (e.g., temperature, top-k, top-p, logit bias, candidate/n count) to server-side approved ranges and defaults, rejecting out-of-policy values. | 2 |
Manipulated, backdoored, or adversarial data entering the model context at inference time via external sources must be identified and neutralized.
| # | Description | Level |
|---|---|---|
| 11.4.1 | Verify that inputs from external or untrusted sources pass through anomaly detection before model inference. | 2 |
| 11.4.2 | Verify that inputs flagged as anomalous trigger gating actions. | 2 |
| 11.4.3 | Verify that the safety violation feedback pipeline includes poisoning detection and human review gates to prevent adversarial manipulation of the improvement mechanism. | 3 |