[ Skill ] Adds a "Flutter PR Failure Collector" skill#185480
Conversation
This skill is meant to make it easier for agents to retrieve the set of build and test failures in checks on Flutter PRs without requiring it to interact directly with rendered webpages and interpret screenshots. It uses either the `gh` CLI tool or GitHub HTTP APIs to retrieve the state of a PR's checks. For each failing check, the agent is instructed to construct URLs to the raw LUCI log output based on heuristics created from https://flutter.googlesource.com/recipes, as well as the various configuration files in the flutter/flutter repository (.ci.yaml, engine/src/flutter/ci/builders/<config_name>.json, etc). Once the logs are collected, the agent scans each log thoroughly for failures and reports: - The name of the failing bot/check - Which platform the check targets - A summary of the failure - A reproduction command (if possible) This skill has been tested against PRs with failing Flutter Tool tests and failing engine builds / tests.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new skill documentation file, SKILL.md, which outlines the workflow for collecting and parsing Flutter PR failures from LUCI. The review feedback identifies a duplicate instruction with a formatting error and corrects the description of how LUCI log URLs are constructed for flutter_drone builders.
|
Example output from a PR with failing engine checks: Failures on PR #185377
Details from Raw Logs for
|
|
Example output for a PR with failing analysis checks and tools tests: Failures on PR #184685
Details from Raw LogsFor the checks with test failures, the identical failures were located in the same file:
For the checks with analysis failures:
Reproduction CommandTo reproduce the failures locally, navigate to the
|
reidbaker
left a comment
reidbaker
left a comment
There was a problem hiding this comment.
Dont forget to add yourself as CODEOWNER
| - **TIP**: You can often access the raw logs directly by appending `?format=raw` to the log URL. | ||
| Example: `https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/<Build ID>/+/u/<Step Name>/stdout?format=raw` | ||
| - **NOTE**: Step names can be very specific and hard to guess. | ||
| - For builders using the **`flutter_drone`** recipe (check `.ci.yaml`), the step name usually follows the pattern: `run test.dart for <shard> shard and subshard <subshard>`. In the raw log URL, spaces are replaced by underscores. If `subshard` is not specified, it defaults to `None`. Example: For `Linux analyze` (which has `shard: analyze` and no `subshard`), the URL step name becomes `run_test.dart_for_analyze_shard_and_subshard_None`. |
There was a problem hiding this comment.
@vashworth has a skill with a more prescriptive version of the drone checks
https://github.com/vashworth/flutter-skills/blob/f0144bcdf639eaee8bf6d34d4d199841874f0c17/personal-running-ci-test-locally/SKILL.md?plain=1#L11-L23
There was a problem hiding this comment.
I think this is strong evidence that going from github pr to logs is a "primitive" that should probably be its own entity. I say not entity not skill because we can probably author a script for it.
There was a problem hiding this comment.
That looks like it could be a useful input to this skill, but we'll still need these extra steps to be able to lookup the logs from LUCI.
There was a problem hiding this comment.
I think she's working on a draft to move it into the repo here #185084
There was a problem hiding this comment.
I think this is strong evidence that going from github pr to logs is a "primitive" that should probably be its own entity. I say not entity not skill because we can probably author a script for it.
I'm not sure this is something that we can create a useful script for, mostly because the LUCI APIs require authentication to use, so manually building the URL is the easiest way to retrieve the logs. When prompted to create a script for this, Gemini gave me this:
import 'dart:io';
void main(List<String> args) {
if (args.length < 2) {
print('Usage: dart build_luci_url.dart <Build_ID_or_Details_URL> <Step_Name>');
exit(1);
}
String buildIdentifier = args[0];
String stepName = args[1];
// Extract Build ID if a full URL was passed
String buildId = buildIdentifier;
if (buildIdentifier.startsWith('http')) {
final uri = Uri.parse(buildIdentifier);
buildId = uri.pathSegments.last;
}
// Transform step name: spaces to underscores
String formattedStepName = stepName.replaceAll(' ', '_');
// Construct the URL
String url = 'https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/$buildId/+/u/$formattedStepName/stdout?format=raw';
print(url);
}It's extremely trivial and most of the complicated work is still being done by the agent to determine the build ID and step name, so I'm not sure it's worth including.
| ## Prerequisites | ||
|
|
||
| - `gh` (GitHub CLI) must be installed and authenticated. If not in your PATH, check common locations like `/opt/homebrew/bin/gh` on macOS or `C:\Program Files\GitHub CLI\gh.exe` on Windows. | ||
| - Access to `curl` or similar tool to fetch raw logs from LUCI. |
There was a problem hiding this comment.
FWIW @vashworth has authored steps for how to have the github mcp server installed and I think that is probably the longer term solution. Non blocking for now but I think you will want to set that up.
reidbaker
left a comment
reidbaker
left a comment
There was a problem hiding this comment.
I think these are good enough to land and experiment with.
Piinks
added
c: contributor-productivity
flutter-dashboard
Bot
removed
the
autosubmit
4 participants
This skill is meant to make it easier for agents to retrieve the set of build and test failures in checks on Flutter PRs without requiring it to interact directly with rendered web pages and interpret screenshots.
It uses either the
ghCLI tool or GitHub HTTP APIs to retrieve the state of a PR's checks. For each failing check, the agent is instructed to construct URLs to the raw LUCI log output based on heuristics created from https://flutter.googlesource.com/recipes, as well as the various configuration files in the flutter/flutter repository (.ci.yaml, engine/src/flutter/ci/builders/<config_name>.json, etc).Once the logs are collected, the agent scans each log thoroughly for failures and reports:
This skill has been tested against PRs with failing Flutter Tool tests and failing engine builds / tests.