Add Rado's GitHub handle to Facilitators table#4
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 4.5 <[email protected]>
olaservo
added a commit
to olaservo/experimental-ext-skills
that referenced
this pull request
Apr 27, 2026
…tocol#4 write-up Adds Scenario modelcontextprotocol#4 — `transformers-js-demo` — testing skill loading on a code-output skill. Agent reads transformers-js SKILL.md and writes a self-contained browser HTML file; no tool execution, the artifact is the agent's response. Three graded prescriptions from SKILL.md: @huggingface/transformers package name, pipeline() API, dispose() memory-management rule. Plan evaluator - DISCOVERY_TOOLS gains agent-internal planning calls (goose's todo__todo_write, gemini-cli's update_topic) so the skill-read-before-plan fallback gate doesn't fire on housekeeping before skill activation - One new test for the discovery/neutral-tools fallback path Findings (docs/experimental-findings.md) - Scenario modelcontextprotocol#4 — only codex passes overall. Two distinct failure modes: gemini-cli skipped catalog activation entirely; claude variants (haiku via fast-agent, sonnet via goose) read the skill but missed the dispose() prescription despite the⚠️ warning. Suggests in-body emphasis isn't sufficient on its own — open question on what makes SKILL.md prescriptions stick Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo
added a commit
to olaservo/experimental-ext-skills
that referenced
this pull request
Apr 27, 2026
…rotocol#3/modelcontextprotocol#4 as N=1 samples Adds a one-line note under each scenario header so readers don't mistake single-trial outcomes for stable behavioral claims. Re-running flipped haiku from FAIL to PASS on the dispose-rule criterion in Scenario modelcontextprotocol#4, which is exactly the noise the caveat warns about. Pass-rate replication (N=3+ per cell) is the proper fix and pending in a follow-up — N-trial wrapper in each client harness, then re-run the matrix and rewrite the findings to report rates rather than single-sample binaries. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo
added a commit
to olaservo/experimental-ext-skills
that referenced
this pull request
Apr 27, 2026
Drop the "Scenarios modelcontextprotocol#2/modelcontextprotocol#3/modelcontextprotocol#4 are N=1, replication pending" snapshot — that fact rots as findings get re-run. Keep just the principle: treat individual outcomes as samples, not stable behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
olaservo
added a commit
to olaservo/experimental-ext-skills
that referenced
this pull request
Apr 28, 2026
…extprotocol#3/modelcontextprotocol#4 Three new YAMLs that exercise the plan-kind evaluator path: - hf-jobs-plan.yaml (modelcontextprotocol#2): probes structured plan output (PEP 723 UV script + HF_TOKEN secret forwarding + Trackio instrumentation) on hf-mcp-server with HF_JOBS_DRY_RUN=true, so submissions are intercepted server-side and graded from the call args - hf-train-with-monitoring.yaml (modelcontextprotocol#3): cross-skill composition -- agent must independently activate both huggingface-llm-trainer and huggingface-trackio from the catalog, with required phrases drawn from both skills' references - transformers-js-demo.yaml (modelcontextprotocol#4): code-output domain (browser HTML/JS artifact). No tool execution; phrase-grep checks @huggingface/transformers package + pipeline API + dispose() rule All three use gate_tools to anchor skill-read-before-plan against meaningful actions instead of housekeeping calls (todo writes, catalog enumeration), so agentic planners aren't penalized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.