Skip to content

feat(evals): add an offline TypeSafe/Jev routing replay harness #882

Open

Description

@cpakkamisaac-sae

I plan to implement this and will link a draft pull request shortly.

Problem

The TypeSafe/Jev work in #723 and #762 includes useful benchmark and stability evidence, but the evidence is not stored in a versioned, repeatable form. Reviewers and operators cannot replay routing decisions, compare candidate-order behavior, or evaluate a threshold policy without making new provider calls and potentially repeating completion-model spend.

Proposed scope

Add a small offline-first evaluation harness that:

  • reads versioned, synthetic JSON fixtures;
  • records fixture provenance, candidate descriptions, candidate orders, raw probability distributions, expected target outcomes, and optional per-target cost;
  • deterministically recomputes aggregate probabilities, selected targets, confidence, and fallback decisions;
  • reports selection counts, order sensitivity, fallback counts, quality, cost, and regret against fixed-target baselines;
  • runs without credentials, network access, or provider SDK dependencies;
  • documents how to create sanitized fixtures without including credentials, private prompts, or provider-controlled error bodies.

The initial pull request will include representative synthetic fixtures, focused tests, and usage documentation.

Out of scope

  • Implementing or changing the runtime TypeSafe/Jev router in feat(routing): compare configured TypeSafe candidates #762.
  • Calling TypeSafe or completion-model providers from the default evaluator.
  • Claiming that the included synthetic fixtures establish production routing quality.
  • Committing real customer prompts, credentials, or private provider responses.

Acceptance criteria

  • The fixture format is explicitly versioned and rejects unsupported versions.
  • Replaying the same fixture produces byte-for-byte stable machine-readable output.
  • Candidate-order sensitivity and invalid probability data are covered by tests.
  • Quality and cost comparisons use declared fixture outcomes rather than live calls.
  • The evaluator is entirely offline by default.
  • Repository lint, type checking, tests, and formatting checks pass.

This provides the reproducible evaluation boundary suggested in #723 while remaining independent of the runtime implementation under review in #762.

Activity

  1. ayushag-nv commented on Oct 30, 2026

    @ayushag-nv
    Contributor

    @cpakkamisaac-sae Do you want to propose a method or design for this proposal as a follow up comment before the implementation. Would be good to discuss first

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions