[Task][AI] Add opt-in evaluation sample export and feedback correlation #8256

Goal

Provide bounded, explicitly authorized evaluation sample export and feedback joins for an operator-owned LLM workload. Parent: #8055. Reviewed master 8046953ca already has logical/attempt observation, logging sampling and body-free event contracts; those are not an evaluation dataset or feedback lifecycle.

This task supplies data contracts and one external-runner/export integration. It does not implement a universal evaluation service or run an LLM judge in the gateway request path.

First delivery

• Sampling is opt-in per authorized tenant/route with bounded rates, sample size, retention and access controls. Define whether successful and failed/rejected/cancelled work is eligible. Existing logging/audit configuration never implicitly enables prompt/response collection.
• Reuse logical request and actual attempt identities, outcomes, normalized usage and available prompt/model/policy versions. Record unknown/unsupported/truncated data explicitly; node-local configuration generation is not a prompt, dataset or policy version.
• Export only allowlisted/redacted content with declared ownership and capacity limits through existing compatible async delivery facilities. State overflow/loss, sink failure, retry/idempotency, expiry and deletion behavior; sample export cannot delay or alter the primary response.
• Begin with bounded non-streaming supported text/JSON. Streaming content capture requires a separate reviewed policy; metadata-only recording must not claim a complete answer dataset or retain unbounded SSE bodies.
• Assign opaque sample references and authorize feedback joins to that tenant/sample. Support a small typed label schema, provenance and duplicate/update semantics; do not accept a forged request ID as authority or rewrite original execution facts.
• Export a versioned manifest usable by an operator-owned offline runner. [Feature][AI] Add versioned prompt templates with validation, rollout and rollback #8051 consumes eval report/dataset references for prompt publication; [Task][AI] Define evaluation gates for cost and quality aware model routing #8052 owns cost/quality-routing evaluation gates.

Acceptance

• One documented workload exports versioned samples and joins authenticated feedback without crossing tenant/retention boundaries.
• Existing request bodies, status and streaming/non-streaming behavior remain unchanged; no additional upstream/model/tool call occurs merely to sample or collect feedback.
• Overload, unavailable sink, malformed feedback, duplicated events, retries/fallback, cancellation, missing usage and opt-out preserve business response and bounded state.
• Fixtures prove redaction/allowlisting, authorization, version/unknown semantics, truncation, deletion and idempotency under concurrency.
• Measure gateway latency/allocation/retained-memory overhead and demonstrate that slow export or feedback cannot block event loops.

Non-goals and reuse

#7950/#8002 own body-free invocation audit, #7934 pricing/ledger, #7955 usage, #7933 online HTTP shadow execution and #8050 semantic-cache evaluation. This task does not mirror production prompts, create a second logging/trace transport, silently reuse production secrets, automatically train models or deploy/promote policies from feedback.

AI/RAG trace profile, feedback and timing (2026-10-30)

Reuse #7566's actual-stage profile and #8145/#8125 correlation. A sample may reference trusted tenant/app/pseudonymous user, actual prompt/model/router-policy revisions, per-attempt token observations, declared TTFT/TPOT/latency measurements, bounded retrieval references, tool status and error/retry/fallback facts. Publish a source/availability matrix; unknown versions and insufficient token timing remain unknown, not fabricated values.

By default export metadata only. retrieval_query, doc IDs/scores, prompt/completion content and free-form feedback require independent allowlisting, redaction, sampling, tenant-authorized access and finite retention/deletion. User/argument hashing needs scoped keying and documented rotation; hashes and document IDs are not automatically non-sensitive. Prefer protected sample references over raw content attributes in traces/baggage.

eval.sample enqueue/export is asynchronous and cannot extend request latency. Keep bounded state and span links/execution IDs when export or feedback arrives after response completion; do not keep ServerWebExchange alive or mutate an ended request span. Scores/feedback carry eval-set/judge/provenance versions and join through an authorized opaque sample ID, preserving original execution facts and distinguishing missing, pending and completed evaluation. Test forged joins, delayed feedback, truncation, expiry/deletion, sink failures and no-extra-model-call behavior.

Quality metrics, feedback denominators and evidence (2026-10-30)

Consume the five-layer dictionary in #8055. Quality outputs cover task success, answer correctness, groundedness/faithfulness, retrieval hit rate, tool business success, format pass, human approval and thumbs up/down. Publish source, rubric/evaluator/dataset versions, eligible/evaluated populations, sample count and late-label window semantics for every reported rate. Missing evaluation/gold/feedback is unknown or not-applicable, never an invented pass or negative rating.

Task success needs an operator-defined business outcome. Correctness/faithfulness require labelled evidence, an identified evaluator or human judgement; HTTP/RPC success, a citation or nonempty retrieval alone are insufficient. Define retrieval hit@k (at least one gold evidence item) separately from recall@k (fraction of gold retrieved), with versioned corpus/index/gold set and k. Keep transport-completed and business-successful tools distinct; blocked/unstarted calls are not successful executions.

Format pass applies only where a format/schema contract is enabled; preserve initial versus post-repair outcomes and #8257 schema revision. Human approval uses reviewed submissions as its declared denominator; thumbs up/down need authenticated provenance, duplicate/update policy and counts of feedback/evaluated coverage, not an assumption that silence is thumbs-down.

Tie quality to actual prompt/model/router/safety and retrieval/index references when supplied, including cache-served cohorts. Scores and feedback can arrive after the trace ends and join through the existing authorized sample identity. Do not add an online judge, implicit body collection or automatic rollout/training here. Test HTTP-200 wrong answers, absent gold/labels, partial/truncated samples, tool domain errors, repaired formats, late/duplicate feedback, cohort sampling bias and expired/deleted joins.