Add agentic workflow daily-test-improver#1139
Conversation
github-actions
Bot
added
github_actions
Coverage Report
File CoverageNo changed files found. |
There was a problem hiding this comment.
Pull request overview
This pull request adds a new agentic workflow called daily-test-improver that systematically identifies and implements test coverage improvements across the repository. The workflow operates in phases: Phase 1 researches the testing landscape and creates a coverage plan, while Phase 2 iteratively implements test improvements based on that plan.
Changes:
- Adds agentic workflow definition file (
.github/workflows/daily-test-improver.md) with instructions for an AI agent to analyze and improve test coverage - Adds generated workflow YAML (
.github/workflows/daily-test-improver.lock.yml) compiled from the definition using gh-aw v0.48.1 - Updates
.gitattributesto mark*.lock.ymlfiles as generated with merge strategyours
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
.github/workflows/daily-test-improver.md |
Defines the agentic workflow behavior with phase-based instructions for test coverage improvement |
.github/workflows/daily-test-improver.lock.yml |
Generated GitHub Actions workflow YAML with complete job definitions, MCP server configuration, and security controls |
.gitattributes |
Marks lock files as generated and sets merge strategy to prevent conflicts |
|
|
||
| 1. **Goal selection**. Build an understanding of what to work on and select an area of the test coverage plan to pursue | ||
|
|
||
| a. Consult your memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` (especially `build-notes.md`, `coverage-notes.md`, and `testing-notes.md`), and build and test the repository taking coverage. If coverage steps failed then create fix PR, update memory notes and exit. |
There was a problem hiding this comment.
The memory path specified here is /tmp/gh-aw/repo-memory-daily-test-improver/, but according to the repo-memory configuration in the frontmatter (lines 36-41) and the standard pattern shown in line 115 of the generated lock file, the correct path should be /tmp/gh-aw/repo-memory/daily-test-improver/ (note the hyphen placement). This inconsistency will cause the workflow to fail when trying to read or write memory files.
| a. Consult your memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` (especially `build-notes.md`, `coverage-notes.md`, and `testing-notes.md`), and build and test the repository taking coverage. If coverage steps failed then create fix PR, update memory notes and exit. | |
| a. Consult your memory notes in `/tmp/gh-aw/repo-memory/daily-test-improver/` (especially `build-notes.md`, `coverage-notes.md`, and `testing-notes.md`), and build and test the repository taking coverage. If coverage steps failed then create fix PR, update memory notes and exit. |
| gh aw logs daily-test-improver --repo ${{ github.repository }} | ||
|
|
||
| **Include a "What Happens Next" section at the end of the discussion that explains:** | ||
|
|
There was a problem hiding this comment.
The "What Happens Next" section header is defined but has no content following it. This section should explain to users what will happen after the discussion is created (e.g., "The workflow will wait for maintainer review and approval before proceeding to Phase 2" or similar). The incomplete section may confuse users about the workflow's next steps.
| - This workflow run will stop after creating this research and planning discussion; it will **not** immediately start adding or changing tests. | |
| - A maintainer (or another human reviewer) will read this discussion, review your coverage plan and commands, and may comment with questions or adjustments. | |
| - After the plan is approved or updated, the maintainer can re-run the `daily-test-improver` workflow (for example using `gh aw run daily-test-improver --repo ${{ github.repository }}`), at which point the workflow will continue with **Phase 2** to implement test improvements based on this plan. |
|
|
||
| 2. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Organize the steps in order as a series of YAML steps suitable for inclusion in a GitHub Action. | ||
|
|
||
| 3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. |
There was a problem hiding this comment.
Grammatical error: "If the a step needs updating" should be "If a step needs updating" (remove "the").
| 3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. | |
| 3. Try to run through the steps you worked out manually one by one. If a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. |
| - **Automatic Push**: Changes are automatically committed and pushed after the workflow completes | ||
| - **Merge Strategy**: In case of conflicts, your changes (current version) win | ||
| - **Persistence**: Files persist across workflow runs via git branch storage | ||
| - **Allowed File Types**: Only the following file extensions are allowed: ``. Files with other extensions will be rejected during validation. |
There was a problem hiding this comment.
The allowed file extensions list is empty (shown as empty backticks). According to the frontmatter configuration (line 39), the file-glob specifies ".md" and ".json" files, which suggests these extensions should be allowed. An empty allowed extensions list will cause validation to reject all files the agent tries to write to memory, breaking the workflow's ability to persist notes between runs.
| - **Allowed File Types**: Only the following file extensions are allowed: ``. Files with other extensions will be rejected during validation. | |
| - **Allowed File Types**: Only the following file extensions are allowed: `.md`, `.json`. Files with other extensions will be rejected during validation. |
| BRANCH_NAME: memory/daily-test-improver | ||
| MAX_FILE_SIZE: 10240 | ||
| MAX_FILE_COUNT: 4 | ||
| ALLOWED_EXTENSIONS: '[]' |
There was a problem hiding this comment.
The ALLOWED_EXTENSIONS is set to an empty array '[]', which will cause validation to reject all files. According to the frontmatter configuration in the .md file (line 39), the file-glob specifies ".md" and ".json" files. The ALLOWED_EXTENSIONS should include these extensions (e.g., '[".md", ".json"]') to allow the workflow to persist memory files between runs.
| ALLOWED_EXTENSIONS: '[]' | |
| ALLOWED_EXTENSIONS: '[".md", ".json"]' |
|
|
||
| 2. Have a careful think about the CI commands needed to build the repository, run tests, produce a combined coverage report and upload it as an artifact. Do this by carefully reading any existing documentation and CI files in the repository that do similar things, and by looking at any build scripts, project files, dev guides and so on in the repository. If multiple projects are present, perform build and coverage testing on as many as possible, and where possible merge the coverage reports into one combined report. Organize the steps in order as a series of YAML steps suitable for inclusion in a GitHub Action. | ||
|
|
||
| 3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. |
There was a problem hiding this comment.
Step 3 references "the branch you created in step 4", but step 4 (which handles creating memory notes) doesn't mention creating a branch. This creates a circular dependency where step 3 depends on step 4, but step 4 comes after step 3. Either the steps need to be reordered, or an explicit branch creation step should be added before step 3, or this reference should be removed if no branch is needed for the manual testing in step 3.
| 3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. | |
| 3. Try to run through the steps you worked out manually one by one. If a step needs updating, then update the steps in your notes from step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. |
|
|
||
| 3. Try to run through the steps you worked out manually one by one. If the a step needs updating, then update the branch you created in step 4. Continue through all the steps. If you can't get it to work, then create an issue describing the problem and exit the entire workflow. | ||
|
|
||
| 4. Keep memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` about how to do this, and what the commands are, so you can refer back to them in future runs. Store notes in structured files: |
There was a problem hiding this comment.
The memory path specified here is /tmp/gh-aw/repo-memory-daily-test-improver/, but according to the repo-memory configuration in the frontmatter (lines 36-41) and the standard pattern shown in line 115 of the generated lock file, the correct path should be /tmp/gh-aw/repo-memory/daily-test-improver/ (note the hyphen placement). This inconsistency will cause the workflow to fail when trying to read or write memory files.
| 4. Keep memory notes in `/tmp/gh-aw/repo-memory-daily-test-improver/` about how to do this, and what the commands are, so you can refer back to them in future runs. Store notes in structured files: | |
| 4. Keep memory notes in `/tmp/gh-aw/repo-memory/daily-test-improver/` about how to do this, and what the commands are, so you can refer back to them in future runs. Store notes in structured files: |
Add agentic workflow daily-test-improver