[Task][AI] Define evaluation gates for cost and quality aware model routing #8052

Feature Request

Weighted and conditional routing should precede automatic selection. A router cannot promise lower cost while preserving quality without an agreed workload and evaluation method.

Proposed scope

• Identify one operator-owned workload, candidate models, privacy/residency constraints and quality/latency/cost objectives.
• Define offline evaluation and shadow comparison using versioned datasets; count router calls, retries and duplicated provider usage.
• Propose opt-in routing policy, explicit capability filters, deterministic fallback and rollback based on measured evidence.

Acceptance criteria

• Publish reproducible results against static routing and concrete quality/regression thresholds.
• Document data handling, bounded router latency and behavior when selection is unavailable.
• Obtain design agreement before implementation; absent a qualifying workload, retain this as discovery.

Dependencies and boundaries

Depends on #8046 and #7955/#7934. Do not create another usage ledger or silently send production prompts to a classifier.

Additional context

Proposed priority: P2 discovery (for discussion, not a maintainer commitment). Workspace evidence must be reconciled with merged master and open PR ownership before implementation.

References: Higress AI plugins, agentgateway documentation. These are design references, not claims of ShenYu feature parity.

Evaluation ownership and decision gates (2026-10-30)

Keep this as evidence-gated routing discovery. First define one operator-owned workload, frozen dataset/split and reproducible static-routing baseline. #8046 routing and #7955/#7934 measurement provenance remain prerequisites.

Publish task quality/format-compliance thresholds with success/rejection rate, latency and estimated cost. Include router/repair calls, all retries/fallback, missing usage and safety exclusions. State sample size, variability, equivalent comparison conditions and promotion/rollback gates; fast rejection or worse answers are not equivalent-workload savings.

#8256 owns opt-in sample export/feedback; #8051 owns prompt publication gates. Neither permits implicit production prompt mirroring or body logging. Measure #8257 repair as additional latency/cost and quality behavior. Without an approved workload and qualifying results, retain discovery status and static routing; do not expand into a universal eval/workflow service.

Five-layer comparison and rollout evidence (2026-10-30)

Use #8055's metric dictionary and #8256's qualified evidence for the chosen workload. Compare task success/correctness/faithfulness, gold-evidence retrieval, tool business outcomes, initial/final format compliance and available human feedback alongside latency, total attributable cost, reliability and inspected safety. Do not equate HTTP success with answer quality or assume a higher automatic judge score proves business success.

Predeclare dataset/split/evaluator versions, cohort and cache treatment, eligible/evaluated denominators, minimum sample counts/uncertainty and missing/delayed feedback handling. Report TTFT/TPOT measurement scope and P50/P95/P99 over compatible populations; retries/fallback/repair and router calls contribute latency and cost. A successful fallback still records primary-provider failure. Unknown usage/pricing or uninspected safety cannot improve the score by becoming zero.

Promotion needs workload-specific quality and safety gates plus declared latency/cost/reliability limits; insufficient evidence retains static routing/discovery status. Shadow evaluation cannot silently mirror production content or bypass existing resource/security boundaries. Reuse the same report identity and frozen inputs when #8051 uses offline-eval, canary, promotion and rollback gates.