Skip to content

eperf(skills): load conditional shared text only when it applies - #1996

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:perf/skill-conditional-blocks
Nov 4, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:perf/skill-conditional-blocks

Conversation

@potiuk

@potiuk potiuk commented Nov 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Moves text that only some runs need out of SKILL.md into sibling files, behind pointers that name when to read them, so a run stops paying for branches it never takes.
  • Clears all four skill-line-limit warnings, and takes the skills over the 5,000-token body target from 22 to 17 (estimates, characters ÷ 4).
  • Adds pointer-routing eval steps, because the existing suites cannot tell whether a pointer is followed.
Text moved Skills Reader that still needs it
tracker-backend detail 12 tracker.platform is local-files or jira
pre-pr-adversarial-review 8 a run that reaches its PR or review step
item-failure-budget rule 6 the budget is configured, an item fails, or a write fails
## Failure modes table 2 one of the triggers the pointer names

tracker-backend and item-failure-budget each split into a pointer block that stays in SKILL.md and a new -detail block in the sibling.
pre-pr-adversarial-review follows what issue-fix-workflow and audit-finding-fix already do.
The generator and validator did not need changing.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Documentation (docs/, README.md, CONTRIBUTING.md)

Test plan

  • prek run --all-files --skip workspace-pytest passes, including lychee
  • workspace-pytest passes for the changed files
  • Validator exits 0; skill-line-limit warnings 4 → 0
  • No surface_hash changed, so adopters are not told their configuration went stale
  • All 33 moved block copies match their source byte for byte
  • Affected eval suites: 45/45 before and 45/45 after (11 suites)
  • New pointer-routing steps: 12 cases, also run against throwaway copies with the pointers deleted

The 45/45 result is partial evidence: also_include pastes the sibling into the prompt, so those suites show that the moved rules still work, not that an agent reaches them.
pointer-routing covers that gap by giving the agent the whole body and no sibling text.

Suite Real skill Pointers deleted
security-issue-fix/pointer-routing 6/6 3/6 (all 3 cases that need a file fail)
issue-triage/pointer-routing 6/6 1/6 (all 5 cases that need a file fail)

The first version of this test passed 6/6 on security-issue-fix with both pointers deleted, because its output spec listed the filenames.
The spec now names no file, so the agent has to get each name from the skill text.
issue-triage case-2 failed once on the real skill with a pool of 12, most likely because that also triggers the N > 5 bulk-mode.md pointer (the runner does not show the extra file).
With a pool of 4 and the same expected answer it passed 3/3 on the real skill and failed on the copy without pointers.

RFC-AI-0004 compliance

  • Vendor neutrality — placeholders kept; the check-placeholders hook passes

No mutation, host access, outbound message or LLM routing changes.

Linked issues

None.

Notes for reviewers

  • The test records a stated decision. The agent says which files it would open; it does not run with file tools.
  • Coverage is partial. pointer-routing exercises two skills. The issue-stale-sweep failure-modes pointer has its own wording and is untested, and the pre-PR pointer was placed by hand in 8 skills but tested in one.
  • Single model. No cross-model pass and no adversarial review were run.
  • Saving varies by run. A run that reaches the PR step pays the same tokens plus one extra read, and security-issue-import-from-pr likely always does; its gain is getting under the line cap.
  • Failure-modes pointers name each trigger. A generic "when a step cannot complete" would have hidden rows such as "reproducer cannot run → continue".
  • tools/spec-loop/.last-sync is not bumped. The specs for these blocks are updated by hand here; it is 62 commits behind main for reasons unrelated to this PR.

🤖 Generated with Claude Code

Move text that only some runs need out of SKILL.md into sibling files,
behind pointers that name when to read them: the tracker-backend detail
(12 skills), the pre-PR adversarial review (8), the item failure budget
(6), and the failure-modes tables of issue-triage and issue-stale-sweep.

Shared text moves byte-for-byte, and no surface hash changes. All four
skills that were over the 500-line cap are now within it.

Add pointer-routing eval steps to issue-triage and security-issue-fix.
They show the agent the whole skill without the sibling text, because the
existing suites paste that text into the prompt and so cannot tell
whether a pointer is followed.

Generated-by: Claude Opus 5
@potiuk potiuk added the family:cross-family Sweeps over the labeler limit label Nov 4, 2026
@github-actions github-actions Bot added family:pr-management pr-management-* skills family:security security-* skills family:setup setup-* skills family:tools tools/* family:docs Docs, MISSION.md, READMEs family:ci .github workflows, prek, validators family:issue issue-* skills capability:triage Sweep + classify + propose disposition capability:review Deep per-item code review or contributor mentoring capability:fix Implement code change to resolve an issue capability:intake Import external signal + sync tracker state capability:resolve Close-out: invalidate, dedupe, CVE allocate, announcement capability:reassess Re-run historical issues against current code capability:stats Read-only dashboards, metrics, governance evidence capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:authoring Skills that author or maintain other skills: write-skill, optimize-skill family:mentoring mentoring skills substrate:framework-dev Tool substrate: build / validate / eval the framework itself family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json labels Nov 4, 2026
@potiuk
potiuk merged commit b1bc935 into apache:main Nov 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:authoring Skills that author or maintain other skills: write-skill, optimize-skill capability:fix Implement code change to resolve an issue capability:intake Import external signal + sync tracker state capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:reassess Re-run historical issues against current code capability:resolve Close-out: invalidate, dedupe, CVE allocate, announcement capability:review Deep per-item code review or contributor mentoring capability:stats Read-only dashboards, metrics, governance evidence capability:triage Sweep + classify + propose disposition family:ci .github workflows, prek, validators family:cross-family Sweeps over the labeler limit family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json family:docs Docs, MISSION.md, READMEs family:issue issue-* skills family:mentoring mentoring skills family:pr-management pr-management-* skills family:security security-* skills family:setup setup-* skills family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant