Skip to content

Add Rado's GitHub handle to Facilitators table#4

Merged
olaservo merged 1 commit into
mainfrom
set-up-docs
Feb 4, 2026
Merged

Add Rado's GitHub handle to Facilitators table#4
olaservo merged 1 commit into
mainfrom
set-up-docs

Conversation

@olaservo

@olaservo olaservo commented Feb 4, 2026

Copy link
Copy Markdown
Member

No description provided.

@olaservo
olaservo merged commit e755249 into main Feb 4, 2026
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
…tocol#4 write-up

Adds Scenario modelcontextprotocol#4 — `transformers-js-demo` — testing skill loading on
a code-output skill. Agent reads transformers-js SKILL.md and writes
a self-contained browser HTML file; no tool execution, the artifact
is the agent's response. Three graded prescriptions from SKILL.md:
@huggingface/transformers package name, pipeline() API, dispose()
memory-management rule.

Plan evaluator
- DISCOVERY_TOOLS gains agent-internal planning calls (goose's
  todo__todo_write, gemini-cli's update_topic) so the
  skill-read-before-plan fallback gate doesn't fire on housekeeping
  before skill activation
- One new test for the discovery/neutral-tools fallback path

Findings (docs/experimental-findings.md)
- Scenario modelcontextprotocol#4 — only codex passes overall. Two distinct failure modes:
  gemini-cli skipped catalog activation entirely; claude variants
  (haiku via fast-agent, sonnet via goose) read the skill but missed
  the dispose() prescription despite the ⚠️ warning. Suggests
  in-body emphasis isn't sufficient on its own — open question on
  what makes SKILL.md prescriptions stick

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
…rotocol#3/modelcontextprotocol#4 as N=1 samples

Adds a one-line note under each scenario header so readers don't
mistake single-trial outcomes for stable behavioral claims. Re-running
flipped haiku from FAIL to PASS on the dispose-rule criterion in
Scenario modelcontextprotocol#4, which is exactly the noise the caveat warns about.

Pass-rate replication (N=3+ per cell) is the proper fix and pending
in a follow-up — N-trial wrapper in each client harness, then re-run
the matrix and rewrite the findings to report rates rather than
single-sample binaries.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 27, 2026
Drop the "Scenarios modelcontextprotocol#2/modelcontextprotocol#3/modelcontextprotocol#4 are N=1, replication pending" snapshot —
that fact rots as findings get re-run. Keep just the principle:
treat individual outcomes as samples, not stable behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo added a commit to olaservo/experimental-ext-skills that referenced this pull request Apr 28, 2026
…extprotocol#3/modelcontextprotocol#4

Three new YAMLs that exercise the plan-kind evaluator path:

- hf-jobs-plan.yaml (modelcontextprotocol#2): probes structured plan output (PEP 723 UV
  script + HF_TOKEN secret forwarding + Trackio instrumentation) on
  hf-mcp-server with HF_JOBS_DRY_RUN=true, so submissions are
  intercepted server-side and graded from the call args
- hf-train-with-monitoring.yaml (modelcontextprotocol#3): cross-skill composition --
  agent must independently activate both huggingface-llm-trainer
  and huggingface-trackio from the catalog, with required phrases
  drawn from both skills' references
- transformers-js-demo.yaml (modelcontextprotocol#4): code-output domain (browser HTML/JS
  artifact). No tool execution; phrase-grep checks @huggingface/transformers
  package + pipeline API + dispose() rule

All three use gate_tools to anchor skill-read-before-plan against
meaningful actions instead of housekeeping calls (todo writes,
catalog enumeration), so agentic planners aren't penalized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant