Skip to content

Commit 6bf42a8

Browse files
[camera_android_camerax] Enforce CHANGELOG backticks and add eval commit author check to pre-push-skill (flutter#12624)
This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
1 parent ccd22e1 commit 6bf42a8

11 files changed

Lines changed: 289 additions & 50 deletions

File tree

‎packages/camera/camera_android_camerax/.agents/skills/pre-push-skill/SKILL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,8 @@ If this command fails, the code WAS NOT ready to push.
105105
The required version bump and changelog entry must be made
106106
and committed before code can be pushed.
107107

108+
Additionally, ensure `CHANGELOG.md` formatting follows the [CHANGELOG style guide](https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changelog-style).
109+
108110
## 6. Check License Headers
109111

110112
All source files in this repository must include
@@ -167,7 +169,7 @@ communicate:
167169
[x] Check for Required Documentation
168170
[x] Check for Added Tests
169171

170-
If for some reason you had to skip a check or it partially failed but you still think the code is ready to push then call out the skipped work like this:
172+
If for some reason you had to skip a check or it partially failed but you still think the code is ready to push then call out the skipped work like this:
171173

172174
# YES, you are ready to push!
173175
Unit tests failing for <path to failing test> but failure appears unrelated to the work being pushed.

‎packages/camera/camera_android_camerax/.agents/skills/pre-push-skill/evals/evals.json‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@
2020
"prompt": "First, run 'dart run .agents/skills/pre-push-skill/evals/test_data/setup_success_native_test.dart'. Then, run the /pre-push skill. Do not attempt to fix any issues you find, only report the output of the skill.",
2121
"expected_chat_output": [
2222
"Agent explicitly states the code is ready to push.",
23-
"Agent confirms that native unit tests pass."
23+
"Agent confirms that native unit tests pass.",
24+
"Agent confirms that the version bump and CHANGELOG updates pass."
2425
],
2526
"expected_repo_state": [
2627
"No changes are pushed to the remote repository.",
@@ -54,6 +55,19 @@
5455
"The agent does not create any new commits or modify the working tree."
5556
],
5657
"agent_config": "reidbaker-agent"
58+
},
59+
{
60+
"id": 5,
61+
"prompt": "First, run 'dart run .agents/skills/pre-push-skill/evals/test_data/setup_user_facing_docs_change.dart'. Then, run the /pre-push skill. Do not attempt to fix any issues you find, only report the output of the skill.",
62+
"expected_chat_output": [
63+
"Agent explicitly states that the code is not ready to push.",
64+
"Agent points out that the CHANGELOG.md entry contains code references without backticks."
65+
],
66+
"expected_repo_state": [
67+
"No changes are pushed to the remote repository.",
68+
"The agent does not create any new commits or modify the working tree."
69+
],
70+
"agent_config": "reidbaker-agent"
5771
}
5872
]
5973
}

‎packages/camera/camera_android_camerax/.agents/skills/pre-push-skill/evals/test_data/setup_missing_native_test.dart‎

Lines changed: 4 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,10 @@
44

55
import 'dart:io';
66

7+
import '../../../../../evals/tool/test_utils.dart';
8+
79
void main() {
8-
final ProcessResult branchResult = Process.runSync('git', ['branch', '--show-current']);
9-
final String branch = branchResult.stdout.toString().trim();
10-
if (branch == 'main') {
11-
stdout.writeln('Error: Cannot run setup scripts on main branch.');
12-
exit(1);
13-
}
10+
ensureNotMainBranch();
1411

1512
final javaFile = File('android/src/main/java/io/flutter/plugins/camerax/DummyEvalFeature.java');
1613
javaFile.createSync(recursive: true);
@@ -26,14 +23,5 @@ public class DummyEvalFeature {
2623
}
2724
''');
2825

29-
Process.runSync('git', ['add', javaFile.path]);
30-
Process.runSync('git', [
31-
'-c',
32-
'user.name=Author',
33-
'-c',
34-
35-
'commit',
36-
'-m',
37-
'Add DummyEvalFeature.java without tests',
38-
]);
26+
commitFiles(<String>[javaFile.path], 'Add DummyEvalFeature.java without tests');
3927
}

‎packages/camera/camera_android_camerax/.agents/skills/pre-push-skill/evals/test_data/setup_mixed_missing_native_test.dart‎

Lines changed: 8 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,10 @@
44

55
import 'dart:io';
66

7+
import '../../../../../evals/tool/test_utils.dart';
8+
79
void main() {
8-
final ProcessResult branchResult = Process.runSync('git', ['branch', '--show-current']);
9-
final String branch = branchResult.stdout.toString().trim();
10-
if (branch == 'main') {
11-
stdout.writeln('Error: Cannot run setup scripts on main branch.');
12-
exit(1);
13-
}
10+
ensureNotMainBranch();
1411

1512
final javaFile = File('android/src/main/java/io/flutter/plugins/camerax/DummyEvalFeature.java');
1613
javaFile.createSync(recursive: true);
@@ -58,14 +55,9 @@ void main() {
5855
}
5956
''');
6057

61-
Process.runSync('git', ['add', javaFile.path, dartFile.path, dartTestFile.path]);
62-
Process.runSync('git', [
63-
'-c',
64-
'user.name=Author',
65-
'-c',
66-
67-
'commit',
68-
'-m',
69-
'Add DummyEvalFeature with Dart test but missing Java test',
70-
]);
58+
commitFiles(<String>[
59+
javaFile.path,
60+
dartFile.path,
61+
dartTestFile.path,
62+
], 'Add DummyEvalFeature with Dart test but missing Java test');
7163
}

‎packages/camera/camera_android_camerax/.agents/skills/pre-push-skill/evals/test_data/setup_success_native_test.dart‎

Lines changed: 11 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,10 @@
44

55
import 'dart:io';
66

7+
import '../../../../../evals/tool/test_utils.dart';
8+
79
void main() {
8-
final ProcessResult branchResult = Process.runSync('git', ['branch', '--show-current']);
9-
final String branch = branchResult.stdout.toString().trim();
10-
if (branch == 'main') {
11-
stdout.writeln('Error: Cannot run setup scripts on main branch.');
12-
exit(1);
13-
}
10+
ensureNotMainBranch();
1411

1512
final javaFile = File('android/src/main/java/io/flutter/plugins/camerax/DummyEvalFeature.java');
1613
javaFile.createSync(recursive: true);
@@ -50,14 +47,12 @@ public class DummyEvalFeatureTest {
5047
}
5148
''');
5249

53-
Process.runSync('git', ['add', javaFile.path, javaTestFile.path]);
54-
Process.runSync('git', [
55-
'-c',
56-
'user.name=Author',
57-
'-c',
58-
59-
'commit',
60-
'-m',
61-
'Add DummyEvalFeature.java and tests',
62-
]);
50+
updateReleaseInfo(changelog: 'Adds `DummyEvalFeature` and tests.');
51+
52+
commitFiles(<String>[
53+
javaFile.path,
54+
javaTestFile.path,
55+
'pubspec.yaml',
56+
'CHANGELOG.md',
57+
], 'Add DummyEvalFeature.java, tests, and changelog update');
6358
}
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Copyright 2013 The Flutter Authors
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:io';
6+
7+
import '../../../../../evals/tool/test_utils.dart';
8+
9+
void main() {
10+
ensureNotMainBranch();
11+
12+
// Update README.md with user-facing documentation
13+
final readmeFile = File('README.md');
14+
final String readmeContent = readmeFile.readAsStringSync();
15+
readmeFile.writeAsStringSync('''
16+
$readmeContent
17+
18+
## Mascot
19+
20+
Dash is a very cute mascot who loves having her photo taken using `takePicture`.
21+
''');
22+
23+
// Intentionally omits backticks around takePicture to test CHANGELOG style enforcement.
24+
updateReleaseInfo(changelog: 'Documents how Dash likes getting pictures taken with takePicture.');
25+
26+
commitFiles(<String>[
27+
readmeFile.path,
28+
'pubspec.yaml',
29+
'CHANGELOG.md',
30+
], 'Add Dash documentation and changelog entry without backticks');
31+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
.agents/
2+
evals/

‎packages/camera/camera_android_camerax/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,5 @@
2929
- **Testing Guidelines**: You MUST read and follow all rules in [TESTING.md](TESTING.md) BEFORE writing or modifying any tests. This is CRITICAL for preventing CI flakiness.
3030
- **Native Unit Tests**: When modifying `.java` or `.kt` files in `android/src/main/` (excluding Pigeon generated files like `.g.java` or `.g.kt`) with logic changes, you MUST add or update corresponding native unit test files in `android/src/test/`.
3131
- **CRITICAL**: When spawning subagents, NEVER provide absolute file paths in prompts. ALWAYS use relative paths. Passing absolute paths breaks `Workspace: branch` isolation and causes state bleed into the active workspace.
32+
- **Repository Guidelines**: Adhere to all repository-wide principles in the root [AGENTS.md](../../../AGENTS.md).
3233
- **Validation**: Never run `.ci/scripts/*` or `script/tool_runner.sh` globally to validate local changes. They are slow and modify the entire repository. Always use targeted skills (like `dart-run-static-analysis` or `pre-push-skill`) to run the repo tool scoped to this package.
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
// Copyright 2013 The Flutter Authors
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:io';
6+
7+
import 'package:args/args.dart';
8+
9+
import 'test_utils.dart';
10+
11+
const String _remoteFlag = 'remote';
12+
const String _baseBranchFlag = 'base-branch';
13+
const String _headFlag = 'head';
14+
const String _helpFlag = 'help';
15+
16+
void main(List<String> args) {
17+
final String defaultRemote = detectDefaultRemote();
18+
final parser = ArgParser()
19+
..addOption(
20+
_remoteFlag,
21+
defaultsTo: defaultRemote,
22+
help:
23+
'The git remote to compare against (defaults to detected flutter/packages remote or origin).',
24+
)
25+
..addOption(_baseBranchFlag, defaultsTo: 'main', help: 'The base branch to compare against.')
26+
..addOption(_headFlag, defaultsTo: 'HEAD', help: 'The head commit or reference to compare.')
27+
..addFlag(_helpFlag, abbr: 'h', negatable: false, help: 'Prints usage information.');
28+
29+
final ArgResults argResults;
30+
try {
31+
argResults = parser.parse(args);
32+
} on FormatException catch (e) {
33+
stderr.writeln(e.message);
34+
stderr.writeln(parser.usage);
35+
exitCode = 1;
36+
return;
37+
}
38+
39+
if (argResults.wasParsed(_helpFlag)) {
40+
stdout.writeln('Checks git commit history for evaluation test credentials.\n');
41+
stdout.writeln(parser.usage);
42+
return;
43+
}
44+
45+
final remote = argResults[_remoteFlag] as String;
46+
final baseBranch = argResults[_baseBranchFlag] as String;
47+
final head = argResults[_headFlag] as String;
48+
49+
// Fetch upstream to ensure base branch is available.
50+
final ProcessResult fetchResult = Process.runSync('git', <String>['fetch', remote, baseBranch]);
51+
if (fetchResult.exitCode != 0) {
52+
stderr.writeln('Warning: Failed to fetch $remote/$baseBranch: ${fetchResult.stderr}');
53+
}
54+
55+
final ProcessResult result = Process.runSync('git', <String>[
56+
'log',
57+
'$remote/$baseBranch..$head',
58+
'--format=%h | Author: %an <%ae> | Committer: %cn <%ce> | %s',
59+
]);
60+
61+
if (result.exitCode != 0) {
62+
stderr.writeln(
63+
'SCRIPT_ERROR: Failed to run git log for $remote/$baseBranch..$head:\n${result.stderr}',
64+
);
65+
exitCode = 2;
66+
return;
67+
}
68+
69+
final logOutput = result.stdout.toString();
70+
71+
final List<String> identities = logOutput
72+
.split('\n')
73+
.map((String line) => line.trim())
74+
.where((String line) => line.isNotEmpty)
75+
.toList();
76+
77+
final forbiddenIdentities = <String>[evalAuthorEmail, evalAuthorName];
78+
79+
for (final identity in identities) {
80+
for (final forbiddenIdentity in forbiddenIdentities) {
81+
if (identity.contains(forbiddenIdentity)) {
82+
stderr.writeln(
83+
'ERROR: Found commit(s) authored or committed by evaluation test credentials:\n'
84+
' $identity\n'
85+
'Evaluation test commits must not be pushed. Clean or rebase your branch before pushing.',
86+
);
87+
exitCode = 1;
88+
return;
89+
}
90+
}
91+
}
92+
93+
stdout.writeln('SUCCESS: No evaluation test commits detected.');
94+
}

0 commit comments

Comments
 (0)