Skip to content

Set up the rest of the starting docs#3

Merged
olaservo merged 12 commits into
mainfrom
set-up-docs
Feb 4, 2026
Merged

Set up the rest of the starting docs#3
olaservo merged 12 commits into
mainfrom
set-up-docs

Conversation

@olaservo

@olaservo olaservo commented Feb 4, 2026

Copy link
Copy Markdown
Member

Transitioning content from google doc to this repo.

@olaservo
olaservo merged commit d54ff57 into main Feb 4, 2026
olaservo added a commit that referenced this pull request Feb 27, 2026
Add new references, findings, and discussion outcomes from the Feb 26
office hours (Discussion #2316):

- Add MCPB and agentskills#110 to related work
- Document push vs. pull update model comparison (FastMCP vs. PR #16)
- Add skills-as-resources direction under Approach 3 (client helper
  tools, three-part framing, hinting workaround, next steps)
- Add dependency declaration reference to Q4 and Use Case #3
- Add new question on skills/resources discovery via Server Cards

Co-Authored-By: Claude Opus 4.6 <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
…ol#2/modelcontextprotocol#3 write-up

Adds Scenario modelcontextprotocol#3 — `hf-train-with-monitoring` — testing whether the
host's <available_skills> catalog drives multi-skill activation when
one skill mentions another by topic without an explicit skill:// URI
reference. Also documents Scenario modelcontextprotocol#2 (hf-jobs dry-run) and modelcontextprotocol#3 in
docs/experimental-findings.md.

Plan evaluator changes
- expected_skill_uris (list) supersedes expected_skill_uri (singular,
  still accepted via backward compat)
- skill-read-before-plan passes only when EVERY listed skill's
  SKILL.md was read before the gate; banner note carries
  read=[...] missed=[...] for diagnostics
- raw payload gains skill_md_indices + missed_skills

Scenario validator
- Either expected_skill_uri or expected_skill_uris must be present
  (was: expected_skill_uri required as a base field)

New tests (45 total)
- Multi-skill happy path, partial activation (one skill missed),
  ordering (one skill after gate), backward-compat for legacy
  single-URI YAML

Findings (docs/experimental-findings.md)
- Scenario modelcontextprotocol#2 — four-client matrix on hf-mcp-server, all PASS;
  documents the prompt/skill conflict that surfaced in early runs
  (later fixed by reframing) and the goose gate_tools fix
- Scenario modelcontextprotocol#3 — cross-skill composition works for fast-agent / codex
  / goose; gemini-cli misses cross-skill activation across both flash
  and pro tiers, even when the model would otherwise cover the
  prescriptions via training-data synthesis. Suggests host-side
  ranking of <available_skills> rather than model capability is the
  constraint

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
…rotocol#3/modelcontextprotocol#4 as N=1 samples

Adds a one-line note under each scenario header so readers don't
mistake single-trial outcomes for stable behavioral claims. Re-running
flipped haiku from FAIL to PASS on the dispose-rule criterion in
Scenario modelcontextprotocol#4, which is exactly the noise the caveat warns about.

Pass-rate replication (N=3+ per cell) is the proper fix and pending
in a follow-up — N-trial wrapper in each client harness, then re-run
the matrix and rewrite the findings to report rates rather than
single-sample binaries.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
Drop the "Scenarios modelcontextprotocol#2/modelcontextprotocol#3/modelcontextprotocol#4 are N=1, replication pending" snapshot —
that fact rots as findings get re-run. Keep just the principle:
treat individual outcomes as samples, not stable behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 28, 2026
…extprotocol#3/modelcontextprotocol#4

Three new YAMLs that exercise the plan-kind evaluator path:

- hf-jobs-plan.yaml (modelcontextprotocol#2): probes structured plan output (PEP 723 UV
  script + HF_TOKEN secret forwarding + Trackio instrumentation) on
  hf-mcp-server with HF_JOBS_DRY_RUN=true, so submissions are
  intercepted server-side and graded from the call args
- hf-train-with-monitoring.yaml (modelcontextprotocol#3): cross-skill composition --
  agent must independently activate both huggingface-llm-trainer
  and huggingface-trackio from the catalog, with required phrases
  drawn from both skills' references
- transformers-js-demo.yaml (modelcontextprotocol#4): code-output domain (browser HTML/JS
  artifact). No tool execution; phrase-grep checks @huggingface/transformers
  package + pipeline API + dispose() rule

All three use gate_tools to anchor skill-read-before-plan against
meaningful actions instead of housekeeping calls (todo writes,
catalog enumeration), so agentic planners aren't penalized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant