Skip to content

Commit 3e63635

Browse files
[camera_android_camerax] Add the ability to unit test skills. (flutter#12285)
Add the concept of evals for skills to camera_android_camerax this is part of project-oneshot. This is basically the equivalent of llm unit tests. The idea and harness comes from dart_skills_lint see flutter/agent-plugins#189 The infra for running evals can and should be replaced by a tool that is designed for this use case when it is available. In the meantime, the valuable parts are the evals.jsons files. They document the expectations for the skills under test in this repo. - @reidbaker --- Agent authored description Updates agent evals for check-readiness and run-evals, including fixing missing license blocks and enforcing relative paths. ## Evaluation Results: `cross_skill_rubric` ### Eval ID 1: Failure Scenario - **Status:** PASS 🟢 - **Scenario:** Execute 'bash evals/test_data/simulate_failure.sh'. - **Grading:** Correctly detected and explicitly flagged all expected failures (Scope Limitation, Pigeon Generation, and Lint Cheating). ### Eval ID 2: Success Scenario - **Status:** PASS 🟢 - **Scenario:** Execute 'bash evals/test_data/simulate_success.sh'. - **Grading:** Correctly identified that all code quality criteria passed successfully without any false positive failures.
1 parent c9d79e7 commit 3e63635

15 files changed

Lines changed: 383 additions & 0 deletions

File tree

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"repo_criteria": [],
3+
"evals": [
4+
{
5+
"id": 1,
6+
"prompt": "Run the check-readiness skill.",
7+
"expected_chat_output": [
8+
"The agent ran the check.dart script."
9+
],
10+
"expected_repo_state": [
11+
"No files were modified in the repository."
12+
],
13+
"agent_config": "reidbaker-agent"
14+
},
15+
{
16+
"id": 2,
17+
"prompt": "Are we ready to start new work?",
18+
"expected_chat_output": [
19+
"The agent ran the check.dart script."
20+
],
21+
"expected_repo_state": [
22+
"No files were modified in the repository."
23+
],
24+
"agent_config": "reidbaker-agent"
25+
}
26+
]
27+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../../../../third_party/skill-repos/agent-plugins/.agents/skills/run-evals

‎packages/camera/camera_android_camerax/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,4 @@
2323
- Maintain high test coverage using [dart-add-unit-test](.agents/skills/dart-add-unit-test/SKILL.md)
2424
and [dart-collect-coverage](.agents/skills/dart-collect-coverage/SKILL.md).
2525
- Avoid duplicating constant strings; reuse existing ones from adjacent code.
26+
- **CRITICAL**: When spawning subagents, NEVER provide absolute file paths in prompts. ALWAYS use relative paths. Passing absolute paths breaks `Workspace: branch` isolation and causes state bleed into the active workspace.
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"evals": [
3+
{
4+
"id": "scope_limitation",
5+
"expected_repo_state": [
6+
"All code modifications, additions, and deletions are strictly confined to the 'packages/camera/camera_android_camerax' package directory.",
7+
"No changes are made to sibling plugins."
8+
]
9+
},
10+
{
11+
"id": "pigeon_generation",
12+
"expected_repo_state": [
13+
"If the file 'pigeons/camerax_library.dart' is modified, the Pigeon code generator must be executed to regenerate the corresponding platform output files."
14+
]
15+
},
16+
{
17+
"id": "lint_cheating",
18+
"expected_repo_state": [
19+
"No lint configurations (e.g., analysis_options.yaml or its rules) are removed or disabled.",
20+
"No file-level Dart ignores (e.g., // ignore_for_file:) are added to any Dart files."
21+
]
22+
}
23+
]
24+
}
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"repo_criteria": [
3+
"evals/cross_skill_rubric.json"
4+
],
5+
"evals": [
6+
{
7+
"id": 1,
8+
"prompt": "First, execute 'bash evals/test_data/simulate_failure.sh'. Then, evaluate the current repository's uncommitted changes using cross_skill_rubric.json. Output your findings into a markdown artifact file. DO NOT FIX OR COMMIT ANY OF THE ISSUES YOU FIND. ONLY EVALUATE THE STATE.",
9+
"expected_chat_output": [
10+
"Any natural language output summarizing the completed work is acceptable."
11+
],
12+
"expected_repo_state": [
13+
"The evaluation artifact explicitly flags a failure for scope limitation.",
14+
"The evaluation artifact explicitly flags a failure for pigeon generation.",
15+
"The evaluation artifact explicitly flags a failure for lint cheating."
16+
],
17+
"agent_config": "reidbaker-agent"
18+
},
19+
{
20+
"id": 2,
21+
"prompt": "First, execute 'bash evals/test_data/simulate_success.sh'. Then, evaluate the current repository's uncommitted changes using cross_skill_rubric.json. Output your findings into a markdown artifact file. DO NOT FIX OR COMMIT ANY OF THE ISSUES YOU FIND. ONLY EVALUATE THE STATE.",
22+
"expected_chat_output": [
23+
"Any natural language output summarizing the completed work is acceptable."
24+
],
25+
"expected_repo_state": [
26+
"The evaluation artifact explicitly flags that all code quality criteria passed successfully without any failures."
27+
],
28+
"agent_config": "reidbaker-agent"
29+
}
30+
]
31+
}
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
# Copyright 2013 The Flutter Authors
3+
# Use of this source code is governed by a BSD-style license that can be
4+
# found in the LICENSE file.
5+
6+
# Simulates a failure of the cross_skill_rubric.json by modifying files incorrectly.
7+
8+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
9+
PACKAGE_ROOT="$(cd "$SCRIPT_DIR/../../" && pwd)"
10+
11+
# 1. Modify a sibling package (Violates scope_limitation)
12+
# We append a comment to a file in the sibling directory.
13+
# README.md is highly unlikely to be moved or deleted.
14+
echo "<!-- dummy eval modification -->" >> "$PACKAGE_ROOT/../camera_android/README.md"
15+
16+
# 2. Modify pigeon file without running pigeon generator (Violates pigeon_generation)
17+
# We append a comment to the pigeon file.
18+
echo "// dummy eval modification" >> "$PACKAGE_ROOT/pigeons/camerax_library.dart"
19+
20+
# 3. Add file-level dart ignore & remove lint config (Violates lint_cheating)
21+
echo "// ignore_for_file: unused_local_variable" >> "$PACKAGE_ROOT/lib/camera_android_camerax.dart"
22+
rm -f "$PACKAGE_ROOT/analysis_options.yaml"
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/bash
2+
# Copyright 2013 The Flutter Authors
3+
# Use of this source code is governed by a BSD-style license that can be
4+
# found in the LICENSE file.
5+
6+
# Simulates a successful adherence to cross_skill_rubric.json
7+
8+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
9+
PACKAGE_ROOT="$(cd "$SCRIPT_DIR/../../" && pwd)"
10+
11+
# Modify a safe file inside the package.
12+
echo "<!-- dummy eval modification -->" >> "$PACKAGE_ROOT/README.md"
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
// Copyright 2013 The Flutter Authors
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:convert';
6+
import 'dart:io';
7+
8+
import 'package:flutter_test/flutter_test.dart';
9+
import 'package:path/path.dart' as p;
10+
11+
void main() {
12+
group('Evals structure consistency', () {
13+
test('all evals.json files across skills share consistent expected keys', () {
14+
final String packageRoot = Directory.current.path;
15+
16+
final List<File> evalsFiles = [
17+
..._findEvalsFiles(Directory(p.join(packageRoot, '.agents', 'skills'))),
18+
..._findEvalsFiles(Directory(p.join(packageRoot, 'evals'))),
19+
]..sort((a, b) => a.path.compareTo(b.path));
20+
21+
expect(
22+
evalsFiles,
23+
isNotEmpty,
24+
reason: 'Should find at least one evals.json file in .agents/skills or evals.',
25+
);
26+
27+
_verifyStructuralConsistency(evalsFiles, 'evals');
28+
});
29+
30+
test('all rubric JSON files in evals/ share consistent structure and keys', () {
31+
final String packageRoot = Directory.current.path;
32+
33+
final rubricsDir = Directory(p.join(packageRoot, 'evals'));
34+
if (!rubricsDir.existsSync()) {
35+
return;
36+
}
37+
38+
final List<File> rubricFiles =
39+
rubricsDir
40+
.listSync()
41+
.whereType<File>()
42+
.where((File f) => f.path.endsWith('.json') && !f.path.endsWith('_evals.json'))
43+
.toList()
44+
..sort((a, b) => a.path.compareTo(b.path));
45+
46+
if (rubricFiles.isEmpty) {
47+
return;
48+
}
49+
50+
_verifyStructuralConsistency(rubricFiles, 'evals');
51+
});
52+
});
53+
}
54+
55+
void _verifyStructuralConsistency(List<File> files, String itemsKey) {
56+
Set<String>? expectedRootKeys;
57+
String? expectedRootKeysFilePath;
58+
Set<String>? expectedItemKeys;
59+
String? expectedItemFilePath;
60+
61+
for (final file in files) {
62+
final Object? decoded = jsonDecode(file.readAsStringSync());
63+
final Map<String, dynamic> decodedMap = switch (decoded) {
64+
final Map<String, dynamic> map => map,
65+
_ => fail('${file.path} must be a JSON map.'),
66+
};
67+
final Set<String> rootKeys = decodedMap.keys.toSet();
68+
if (expectedRootKeys == null) {
69+
expectedRootKeys = rootKeys;
70+
expectedRootKeysFilePath = file.path;
71+
} else {
72+
expect(
73+
rootKeys,
74+
equals(expectedRootKeys),
75+
reason:
76+
'${file.path} root keys do not match consistency pattern. '
77+
'Expected keys to match the first processed file ($expectedRootKeysFilePath).',
78+
);
79+
}
80+
81+
final Object? itemsRaw = decodedMap[itemsKey];
82+
final List<dynamic> itemsList = switch (itemsRaw) {
83+
final List<dynamic> list => list,
84+
_ => fail('$itemsKey key in ${file.path} must be a List.'),
85+
};
86+
for (final Object? item in itemsList) {
87+
final Map<String, dynamic> itemMap = switch (item) {
88+
final Map<String, dynamic> map => map,
89+
_ => fail('Item in $itemsKey list in ${file.path} must be a JSON map.'),
90+
};
91+
final Set<String> itemKeys = itemMap.keys.toSet();
92+
if (expectedItemKeys == null) {
93+
expectedItemKeys = itemKeys;
94+
expectedItemFilePath = file.path;
95+
} else {
96+
expect(
97+
itemKeys,
98+
equals(expectedItemKeys),
99+
reason:
100+
'Item in ${file.path} keys do not match consistency pattern. '
101+
'Expected item keys to match the first processed file ($expectedItemFilePath).',
102+
);
103+
}
104+
}
105+
}
106+
}
107+
108+
List<File> _findEvalsFiles(Directory baseDir) {
109+
if (!baseDir.existsSync()) {
110+
return [];
111+
}
112+
return baseDir.listSync(recursive: true).whereType<File>().where((File f) {
113+
final String name = p.basename(f.path);
114+
return name == 'evals.json' || name.endsWith('_evals.json');
115+
}).toList();
116+
}
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: run-evals
3+
description: Run evaluations for one, multiple, or all skills using the agent orchestration framework. Make sure to use this skill whenever the user asks to run evals, test a skill's performance, run benchmarks, or compare baseline versus with-skill execution.
4+
metadata:
5+
internal: true
6+
---
7+
8+
# Run Skill Evals
9+
10+
1. **Read Framework**: Read `<target-package-root>/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals (where `<target-package-root>` is the directory containing the `.agents` or `skills` folder).
11+
2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `<target-package-root>/evals/`.
12+
3. **Determine Agent Configuration**: Check the `agent_config` field in the target target JSON file to determine the environment/harness to spawn. If `agent_config` is `"bare-agent"`, spawn a subagent with the `bare-agent` profile. If it is a specific contributor profile (e.g. `"reidbaker-agent"`), use that profile to provide the necessary contributor context.
13+
4. **Orchestrate**: By default, run an Integration Test by spawning a single **With-Skill** subagent using `Workspace: branch` and the identified `agent_config`.
14+
- Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in `<path-to-skill>`, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `<path-to-skill>` with `"none (cross-skill meta-eval)"`. Also, replace `<target-package-root>` with the actual directory path in both templates.
15+
- **Only if the user explicitly requests a comparison or benchmark**, also spawn a **Baseline** subagent. See `resources/baseline_execution_prompt.md` for the template.
16+
Instruct the subagent(s) to return their `git diff` and verification outputs (`dart pub get`, `dart format`, `dart analyze`, `dart test`) without committing. Ensure you instruct them to run these commands exclusively from within the `<target-package-root>` directory to avoid analyzing unrelated packages.
17+
**CRITICAL**: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace.
18+
**WORKSPACE LIMITATION WARNING**: If the user has multiple active workspaces mounted, the `Workspace: branch` feature will fail. In this situation, you MUST warn the user that running concurrent evaluations in `Workspace: inherit` mode will cause git state bleed and cross-eval pollution (e.g., changes made by a failure scenario will be visible to a success scenario running simultaneously in the same shared directory). Instruct the user to fix this by closing all workspaces except the primary package workspace, and then re-run the evaluations. Do NOT silently fallback to `Workspace: inherit` for concurrent tasks.
19+
5. **Grade**: Parse the combined rubric (resolving `repo_criteria` + `evals.json` expectations). Use the grading instructions in `resources/agent_judge_prompt.md`. When an expectation fails, you MUST explicitly list both the expectation and what was actually found that caused the failure.
20+
6. **Artifact**: Grade the outputs and generate a Markdown artifact (e.g., `<skill>_eval_results.md`) containing the metadata, pass/fail rationale, and raw diffs/stdout.
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
{
2+
"repo_criteria": [
3+
],
4+
"evals": [
5+
{
6+
"id": 1,
7+
"prompt": "Run the evals for the definition-of-done skill.",
8+
"expected_chat_output": [
9+
"Any natural language output summarizing the completed work is acceptable."
10+
],
11+
"expected_repo_state": [
12+
"The agent successfully triggered subagents to run the test cases.",
13+
"There is an artifact with the results.",
14+
"No files are modified in the parent workspace after the eval is done running (all modifications must be confined to the isolated subagent worktree).",
15+
"The evaluation artifact includes evals sourced from code_quality_rubric.json for a skill that produces code."
16+
],
17+
"agent_config": "reidbaker-agent"
18+
},
19+
{
20+
"id": 2,
21+
"prompt": "Can you test the definition-of-done skill and see if it passes its rubric?",
22+
"expected_chat_output": [
23+
"Any natural language output summarizing the completed work is acceptable."
24+
],
25+
"expected_repo_state": [
26+
"There is an artifact with the results.",
27+
"No files are modified in the parent workspace after the eval is done running (all modifications must be confined to the isolated subagent worktree).",
28+
"The evaluation artifact includes evals sourced from code_quality_rubric.json for a skill that produces code."
29+
],
30+
"agent_config": "reidbaker-agent"
31+
},
32+
{
33+
"id": 3,
34+
"prompt": "Run an A/B benchmark evaluation for the definition-of-done skill, comparing it explicitly against a baseline without the skill.",
35+
"expected_chat_output": [
36+
"Any natural language output summarizing the completed work is acceptable."
37+
],
38+
"expected_repo_state": [
39+
"The agent successfully triggered both baseline and with-skill subagents.",
40+
"There is an artifact with the results.",
41+
"The evaluation artifact contains explicit sections for both the 'With-Skill Agent' and the 'Baseline Agent'.",
42+
"No files are modified in the parent workspace after the eval is done running (all modifications must be confined to the isolated subagent worktree)."
43+
],
44+
"agent_config": "reidbaker-agent"
45+
},
46+
{
47+
"id": 4,
48+
"prompt": "Please run the evals for the run-evals skill. Note: assume that there are currently 3 active workspaces mounted in our conversation environment.",
49+
"expected_chat_output": [
50+
"The response explicitly warns the user that running concurrent evaluations in 'Workspace: inherit' mode will cause git state bleed and cross-eval pollution.",
51+
"The response explicitly instructs the user to close all workspaces except the primary package workspace.",
52+
"The response refuses to silently fallback to 'Workspace: inherit' for concurrent tasks."
53+
],
54+
"expected_repo_state": [
55+
"No evaluation artifact is generated.",
56+
"No subagents are spawned."
57+
],
58+
"agent_config": "reidbaker-agent"
59+
}
60+
]
61+
}

0 commit comments

Comments
 (0)