Skip to content

feat: Map OpenAI Evals YAML format → waza graders #14

Description

@spboyer

Migrated from spboyer/waza#386

Summary

Map OpenAI Evals' modelgraded YAML schema to waza prompt grader config with concrete translation examples.

OpenAI Format (examples from their registry)

fact.yaml: prompt + choice_strings (ABCDE) + input_outputs
closedqa.yaml: prompt + eval_type (cot_classify) + choice_scores (Y:1.0, N:0.0)
battle.yaml: prompt + choice_scores + dual input_outputs for head-to-head comparison

Deliverable

Field-level mapping table and working translations of fact.yaml, closedqa.yaml, battle.yaml → waza eval.yaml.

Parent: #385

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions