[Task] Add unified AI and MCP invocation auditing #7950

Background

Parent #7949. This task belongs to the unified governance layer. Metrics and traces do not replace a correlated, privacy-preserving decision and terminal audit record, especially for requests rejected before execution.

Existing capabilities and reuse

Reuse the logging SPI/collectors, trusted identity, immutable request context and terminal observations from the completed identity, context and observability work (#7811, #7827, #7860, #7921). Reuse existing queue/drop instrumentation (#7687), not another logging delivery subsystem. At the research baseline those later context/observability PRs remained open; integrate against their final merged contracts.

Implementation scope

• Define a versioned audit record containing trusted tenant/subject, internal API-key identifier when available, target/model/tool, configuration and policy versions, decision, bounded reason code, terminal status and correlation identifiers.
• Emit one terminal record per logical invocation, including authentication, authorization and admission rejection before upstream execution. Correlate separate upstream-attempt records without counting them as additional logical calls.
• Use bounded nonblocking delivery through existing logging extension points. Specify queue capacity, overflow and drop visibility, sink-error handling, shutdown behavior and the actual delivery guarantee.
• Omit prompts, arguments, response bodies and credentials by default. Do not use W3C trace IDs as identity or raw API keys as identifiers.

Non-goals

A new authorization server, a billing or usage ledger (owned by #7934), guaranteed lossless remote delivery, body logging, or replacing existing metrics and tracing.

Acceptance criteria

• Success, failure, timeout, cancellation and pre-execution rejection have correctly correlated, tenant-isolated records with one logical terminal event.
• Retry and fallback attempts retain separate identities; late or duplicate terminal callbacks do not duplicate logical completion.
• Queue saturation and sink failures are observable and never block gateway event loops or leak sensitive data.
• Record schema, retention and delivery limitations, configuration and reason-code semantics are documented.

Tests

Unit tests for record mapping, redaction and terminal idempotence; concurrent Reactor-context isolation tests; a real logging-sink fixture covering slow or failing sinks, queue overflow and shutdown; regression tests confirming unchanged response status/body and existing logging metrics.

Dependencies

Completed identity, context and observability contracts; #7860 and #7921 integration availability. #7955 and #7956 consume this record and decision vocabulary; they must not introduce competing audit records.

Operational fields and publication/evaluation consumers (2026-10-30)

#8002 remains OPEN. Existing terminal hooks alone are not a complete merged audit sink; reuse their contract and the existing logging delivery ownership.

Map #8125 fields: actual retry_count; transition fallback_from/fallback_to and bounded fallback_reason; initial timeout_budget_ms, per-decision remaining_budget_ms; semantic final_status separate from assigned HTTP status. Preserve logical/attempt identities, trusted target references and supplied policy/price provenance. A suppressed fallback is a decision, not an executed attempt. Document unknown runtime fields rather than fabricating values.

When #8051 is enabled, carry actual prompt_id/prompt_version and supplied dependency/eval references. Node-local configuration generation is not a Prompt Registry or safety-policy revision. #8257 repair is separately correlated billable work, with #7955 measurements and #7934 cost ownership.

#8256 owns separately authorized bounded content export/feedback. Preserve body-free-by-default auditing; enabling audit does not permit prompt/response collection, shadow traffic or an online judge. Include #8121's 3-second scenario, early rejection and committed-stream timeout in mapping/delivery tests.

Coordinate the public requestId mapping with #8145; preserve internal invocation IDs separately when needed for joining existing events.

Trace/profile correlation and sensitive fields (2026-10-30)

Coordinate #7566's AI/RAG profile with this body-free-by-default audit contract and #8256's separately authorized evaluation sampling. Where trusted/known, correlate request/trace, tenant/app/pseudonymous user, actual Prompt Registry revision, provider/model/version/router policy, #7955 usage, tool outcome and the #8125 error/retry/fallback fields. Unknown model/app/policy revisions remain unknown; a trace ID is not identity and a node-local generation is not a published revision.

Retrieval queries, document identifiers and scores can be sensitive. Raw user IDs, prompts, retrieved text, tool arguments/outputs, credentials and free-form feedback are not enabled by default. If argument/user pseudonyms are approved, use bounded canonicalization and tenant-scoped keyed hashing with documented key rotation and authorization; a plain hash is not anonymization. Existing access controls, sampling, finite retention/deletion and nonblocking bounded delivery apply to exports and joins.

Late eval_score/feedback belong to a separate provenance-bearing record joined by an authorized sample/execution reference. Do not rewrite original terminal facts or an already ended span, and do not count feedback as another logical model invocation.

Five-layer evidence rather than interface-only success (2026-10-30)

#8055 is the canonical quality/performance/cost/reliability/safety metric dictionary. This audit task carries correlated decision/execution facts and provenance to the relevant owners, rather than making every audit record a computed quality score or adding another metrics backend.

Keep HTTP status, transport completion, semantic terminal status, tool business outcome, inspection coverage and late evaluation distinct. An HTTP-200 wrong answer, business-failed tool or interrupted stream cannot be silently recorded as task success. A successful fallback retains its primary-provider failure and all #7955/#7934 attempt usage/cost lineage. Missing detector/evaluation/price data stays unknown, not zero or pass.

Use authenticated tenant/app/feature mappings only when available; document missing dimensions and sampling/retention. Reuse bounded reason vocabularies for timeouts, actual retries/fallback, enforced versus shadow admission and safety decisions. Sensitive or high-cardinality join data belongs in authorized records/samples, not unrestricted metric labels. Quality/feedback remains #8256 evidence, cost is #7934, safety is the actual detector/enforcement owner, and #7566 supplies timing rather than verdicts.