Skip to content

Latest commit

 

History

History
99 lines (71 loc) · 4.52 KB

BENCHMARK.md

File metadata and controls

99 lines (71 loc) · 4.52 KB

Skill Benchmark: physicsnemo-shard-tensor

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

  • Skill: physicsnemo-shard-tensor
  • Evaluation date: 2026-08-05
  • Evaluator version: 1.0.0
  • Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)
  • Tasks: 4 evaluation tasks (3 positive, 1 negative)
  • Dataset digest: sha256:dd5c60518eac8911bdb11a1e2b2eb755fb9faa4bf701c996e6334702ab7a589c (skill-evaluator-dataset-snapshot/1)
  • Attempts per task: 1
  • Environment: k8s-sandbox
  • Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

  • is safe to use;
  • produces correct answers;
  • is discovered and activated when needed;
  • helps the agent complete the user's goal and expected workflow; and
  • avoids wasted skill and tool usage.

Results at a Glance

Measure Claude Code (Baseline → Skill Uplift) Codex (Baseline → Skill Uplift)
Overall 73% → 96% (+24 points) 63% → 94% (+31 points)
Security 100% → 100% (±0 points) 100% → 100% (±0 points)
Correctness 90% → 100% (+10 points) 65% → 100% (+35 points)
Discoverability 61% → 100% (+39 points) 62% → 95% (+33 points)
Effectiveness 67% → 92% (+24 points) 60% → 88% (+28 points)
Efficiency 47% → 91% (+44 points) 25% → 85% (+60 points)

How to read this table: baseline is the same task attempted without the target skill. Uplift is skill score - baseline score, shown in percentage points.

Example: 47% → 92% (+45 points) means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.

Tier Status

Tier Purpose Status Evidence
Tier 1 Static validation PASSED WITH OBSERVATIONS 1 validator(s); 2 finding(s)
Tier 2 Semantic deduplication NOT RUN No result was recorded
Tier 3 Live agent evaluation PASS 2 agent(s); 4 task(s)

Findings and Observations

Show detailed findings and successful checks
  • MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (skills/physicsnemo-shard-tensor/SKILL.md)
  • MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (skills/physicsnemo-shard-tensor/SKILL.md)

Scoring Methodology

Show dimension definitions, source signals, and thresholds
Dimension Question Scored signals
Security Is it safe to use? security (100%)
Correctness Is the answer correct? accuracy (100%)
Discoverability Was the right skill loaded when needed? skill_execution (100%)
Effectiveness Did the skill help complete the task? goal_accuracy (50%) + behavior_check (50%)
Efficiency Did it avoid wasted tool or skill usage? skill_efficiency (100%)
  • Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
  • Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
  • Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
  • The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
  • Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).
  • Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.

Signals present in this run:

  • security (Security): unsafe operations, secret leakage, and unauthorized access.
  • skill_execution (Skill Execution): whether the expected skill was found and executed.
  • skill_efficiency (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
  • accuracy (Accuracy): final-answer correctness against the reference answer.
  • goal_accuracy (Goal Accuracy): whether the user's goal was achieved.
  • behavior_check (Behavior Check): whether the expected workflow behavior was followed.

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.