You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(evals): add an offline TypeSafe/Jev routing replay harness #882
I plan to implement this and will link a draft pull request shortly.
Problem
The TypeSafe/Jev work in #723 and #762 includes useful benchmark and stability evidence, but the evidence is not stored in a versioned, repeatable form. Reviewers and operators cannot replay routing decisions, compare candidate-order behavior, or evaluate a threshold policy without making new provider calls and potentially repeating completion-model spend.
Proposed scope
Add a small offline-first evaluation harness that:
reads versioned, synthetic JSON fixtures;
records fixture provenance, candidate descriptions, candidate orders, raw probability distributions, expected target outcomes, and optional per-target cost;
deterministically recomputes aggregate probabilities, selected targets, confidence, and fallback decisions;
reports selection counts, order sensitivity, fallback counts, quality, cost, and regret against fixed-target baselines;
runs without credentials, network access, or provider SDK dependencies;
documents how to create sanitized fixtures without including credentials, private prompts, or provider-controlled error bodies.
The initial pull request will include representative synthetic fixtures, focused tests, and usage documentation.
@cpakkamisaac-sae Do you want to propose a method or design for this proposal as a follow up comment before the implementation. Would be good to discuss first
Activity
ayushag-nv commentedon Oct 30, 2026
@cpakkamisaac-sae Do you want to propose a method or design for this proposal as a follow up comment before the implementation. Would be good to discuss first