Skip to content

feat: Go-module-style grader/eval references #15

Open
Feature
Open
feat: Go-module-style grader/eval references#15
Feature
Labels
design-completeDesign spike is complete; remaining work should move to implementation-scoped issuesenhancementNew feature or requestepic:evaluationE3: Evaluation Frameworkgo:needs-researchNeeds investigationgradersneeds-author-feedbackNeeds additional details from issue author to validate or implementquestionFurther information is requestedrelease:backlogNot yet targetedtype:spikeResearch/investigation — produces a plan, not code

Description

@spboyer
spboyer
opened on Feb 28, 2026

Migrated from spboyer/waza#387

Summary

Design how eval.yaml can reference remote graders via Go-module-style paths.

Concept

graders:
  - ref: github.com/waza-evals/[email protected]
  - ref: github.com/myorg/[email protected]
    config:
      threshold: 0.8

Design Questions

  • How does waza get resolve and cache remote graders?
  • Version pinning semantics (exact, range, latest)?
  • Offline/air-gapped fallback from local cache?
  • How to handle transitive dependencies (grader A depends on dataset B)?
  • Authentication for private repos?

Parent: #385

Metadata

Metadata

Assignees

No one assigned

    Labels

    design-completeDesign spike is complete; remaining work should move to implementation-scoped issuesenhancementNew feature or requestepic:evaluationE3: Evaluation Frameworkgo:needs-researchNeeds investigationgradersneeds-author-feedbackNeeds additional details from issue author to validate or implementquestionFurther information is requestedrelease:backlogNot yet targetedtype:spikeResearch/investigation — produces a plan, not code

    Type

    Feature

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions