Skip to content

Implement scripts/knowledge_gate.py and fail CI on KB benchmark regressions #32

Open

Description

@Brohammad

Summary

Onboarding already has scripts/eval_gate.py in CI. Knowledge-base benchmarks are documented with thresholds but the gate script is still marked future.

Context

  • docs/benchmarks/knowledge-base.md (citation p95 < 5 ms, search p95 < 100 ms, zero org-isolation failures)
  • scripts/eval_gate.py + existing eval-gate CI job (pattern to copy)
  • scripts/benchmark_knowledge_base.py

Acceptance criteria

  • python scripts/knowledge_gate.py exits 0 on current mock benchmarks
  • Parses JSON from the knowledge-base benchmark
  • CI job fails when thresholds regress (not continue-on-error forever)
  • Unit test for threshold parsing
  • Docs updated so “future” is removed

Distinct from #25 (nightly live-provider smoke).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions