Repository navigation
Support isolated Git worktree harness in run-evals skill - #3
Merged
reidbaker merged 1 commit intoAug 28, 2026
Merged
Conversation
- Pre-flight detection: Detect repository layout via git rev-parse to choose between native branch workspaces (Strategy A) and isolated git worktrees (Strategy B). - Subpackage & multi-workspace support: Create detached worktrees (<skill-name>-workspace) for evaluation execution when subagent Workspace: branch is unsupported. - Strict isolation: Prevent un-isolated execution in active working trees and mandate cleanup teardown. - Single source of truth: Consolidate execution documentation into run-evals/SKILL.md. - Evals & test suite: Add test cases and update gitignore patterns for *-workspace/. Co-authored-by: Reid Baker <[email protected]>
reidbaker
self-requested a review
August 25, 2026 20:43
auto-submit Bot
pushed a commit
to flutter/packages
that referenced
this pull request
Aug 27, 2026
…mit author check to pre-push-skill (#12624) This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
reidbaker
reviewed
Aug 28, 2026
| @@ -47,13 +47,36 @@ | |||
| "id": 4, | |||
| "prompt": "Please run the evals for the run-evals skill. Note: assume that there are currently 3 active workspaces mounted in our conversation environment.", | |||
| "expected_chat_output": [ | |||
Collaborator
There was a problem hiding this comment.
FWIW I also verified this by hand. It makes working in "workspaces" much nicer if you also need to manipulate evals. That is a common pattern when you are working on published skills that are in a host packages and you want to confirm they work in a consumer package.
reidbaker
approved these changes
Aug 28, 2026
victorsanni
pushed a commit
to victorsanni/packages
that referenced
this pull request
Sep 9, 2026
…mit author check to pre-push-skill (flutter#12624) This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
victorsanni
pushed a commit
to victorsanni/packages
that referenced
this pull request
Sep 9, 2026
…mit author check to pre-push-skill (flutter#12624) This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This work came out of using run-evals in the camera_android_camerax repo to eval the skills there. It seems jetski regressed in its ability to fork a repo when working in a sub folder.
This prs goal is to help the skill work in more workspace types by manually creating a worktree then doing eval work.
Agent authored pr description
Summary
Support Isolated Git Worktree harnesses (
<skill-name>-workspace) inrun-evalsas a fallback when nativeWorkspace: branchsubagents cannot be provisioned (e.g., in monorepo subpackages, multiple mounted workspaces, or unsupported VCS environments).Motivation and Context
When running skill evaluations from a subpackage directory or in an environment with multiple active workspaces, native
Workspace: branchfails with"branch workspace not supported for VCS type Unknown"because.gitresides in a parent directory.Falling back to un-isolated
Workspace: inheritdirectly in the active working tree is dangerous because setup scripts, index flags, and trap files pollute the developer's working tree. This PR introduces pre-flight Git detection and an isolated Git worktree harness (Strategy B) that creates a temporary detached checkout (<skill-name>-workspace), executes the eval within the target package path, and tears it down upon completion.What changed
run-evals/SKILL.md: Added pre-flight layout detection (git rev-parse --show-toplevelandgit rev-parse --show-prefix) and Strategy A (Workspace: branch) vs Strategy B (git worktreedetached into<skill-name>-workspace) selection, with strict guarantees against un-isolated execution and mandatory teardown.run-evals/resources/: Updatedwith_skill_execution_prompt.mdandbaseline_execution_prompt.mdto support both branch workspaces and worktree targets.run-evals/evals/evals.json: Added test cases for subpackages, multiple mounted workspaces, and VCS branch failure fallback.evals/README.md&.agents/skills/README.md: Consolidated execution documentation to referencerun-evals/SKILL.mdas the single source of truth to prevent documentation drift..gitignore: Added*-workspace/to ignore dynamic evaluation worktrees.Evaluation Verification
run-evals-workspace)run-evals-workspace/packages/skills_lint)definition-of-done-workspace)Testing Instructions
dart test: Run all 225 package tests across the suite.dart analyze: Verify zero analyzer issues.dart run bin/skills_lint.dart: Verify that all skills pass specification validation.