Skip to content

Commit 6ac8f35

Browse files
ainameclaude
andauthored
feat: upload validation evidence per file and link it from the proof comment (#35)
## Overview Validation evidence (screenshots, recordings, logs) is uploaded one non-archived artifact per file, and the proof comment links every file so it opens in the browser from the issue. - New `upload-artifacts` operation: uploads each file under `artifact_path` plus the result JSON via `@actions/artifact` `skipArchive`; emits `artifact_links` / `artifact_count`; never fails the step. - Proof comment: `Artifacts` row (file count, links to the run's artifact list), a download command under the overview table, and links wherever an evidence cell mentions an uploaded file. - `octestra.sh artifacts RUN_ID [DIR]`: bulk download through the API, because `gh run download` cannot fetch non-archived artifacts ([cli/cli#13012](cli/cli#13012), noted in `TODO.md`). - Contract: the example cites evidence by relative path, and `check-output.sh` rejects absolute paths and warns about missing files when given the artifact directory. - Template: the validation job runs `upload-artifacts` after the agent (`if: always()`) and passes `artifact_links` to finalize. <details> <summary>Example proof comment (rendered from this branch; links point at a made-up run)</summary> ## ✅ Passed validation proof The settings screen opens from the home screen and every acceptance criterion holds. | Target | Result | | --- | --- | | Pull request | #57 | | Validated commit | `0123456789ab` | | Outcome | ✅ Passed | | Known gaps | 1 | | Artifacts | [5 files](https://github.com/example-org/consumer/actions/runs/123456#artifacts) | Download all files from a checkout of this repository: ```sh .github/octestra/octestra.sh artifacts 123456 ``` ### Acceptance criteria | ID | Criterion | Result | Evidence | | --- | --- | --- | --- | | AC-1 | The home screen shows the album title | ✅ Passed | [screens/01-home.png](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1001) | | AC-2 | Tapping the gear opens Settings | ✅ Passed | [screens/02-settings.png](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1002); [journey.mp4](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1003) | ### Checks | Check | Type | Scope | Result | Evidence | | --- | --- | --- | --- | --- | | Build (Debug) | build | — | ✅ Passed | [logs/build.log](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1004) | | Home → Settings journey | agentic E2E | AC-1, AC-2 | ✅ Passed | [journey.mp4](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1003) | ### Evidence | Evidence | Type | Reference | | --- | --- | --- | | Home screen after launch | image | [screens/01-home.png](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1001) | | Settings screen | image | [screens/02-settings.png](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1002) | | Journey recording | video | [journey.mp4](https://github.com/example-org/consumer/actions/runs/123456/artifacts/1003) | ### Known gaps - Only the English locale was checked. <details> <summary>Additional details</summary> 1. Built the app and launched it on the simulator. 2. Opened Settings from the gear icon and recorded the journey. </details> <details> <summary>Technical metadata</summary> | Field | Value | | --- | --- | | Issue | #42 | | Pull request | #57 | | Subject SHA | `0123456789abcdef0123456789abcdef01234567` | | Workflow run | [View run](https://github.com/example-org/consumer/actions/runs/123456) | | Run attempt | 1 | | Actor | @Octestra[bot] | | Task owner | @task-owner | | Recorded at | 2026-09-29T12:00:00.000Z | </details> </details> ## Notes - Evidence linking matches relative paths, and file names when unique, anywhere in a cell: with a directory prefix, inside backticks, or glued to prose. `xhome.png` / `home.png.bak` stay untouched. Nothing uploaded → comment identical to today's. - Hidden entries are skipped as `actions/upload-artifact` does; symlinks to files are followed; 500 artifacts per job is GitHub's cap. - `octestra.sh artifacts` saves under `$TMPDIR` by default so the checkout stays clean; archived artifacts get `.zip`. ## Verification - `make all`: typecheck, vitest (116), ruby, installer/maintenance/workflow tests, `dist/index.js` rebuilt. - Probe run on a throwaway branch: four sample files uploaded non-archived from this branch, served inline with the right content type, and fetched back with `octestra.sh artifacts`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent ddd241a commit 6ac8f35

20 files changed

Lines changed: 3394 additions & 275 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ performs eligible `Todo` to `Ready` updates.
3737
action.yml composite action surface (inputs are the public API)
3838
src/
3939
index.ts operation dispatch; builds context per namespace
40-
shared/ config.ts, github-client.ts, prompt.ts, proof.ts, result.ts
40+
shared/ artifacts.ts, config.ts, github-client.ts, prompt.ts, proof.ts, result.ts
4141
lifecycle/operations.ts lifecycle/<verb> implementations
4242
loop/operations.ts loop/list-epics, prepare-triage and finalize-triage
4343
dist/index.js committed esbuild bundle — regenerate, never hand-edit

‎TODO.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,3 +79,16 @@ Revisit if the write endpoint gains ID addressing.
7979
`config.yml`; today it regenerates the file.
8080
- The installer overwrites workflows, prompts, and the agent skill on every run. Say so before doing
8181
it.
82+
83+
## 4. Replace `octestra.sh artifacts` with `gh run download` once it handles non-archived artifacts
84+
85+
Validation evidence is uploaded one file per artifact without archiving (`upload-artifacts`), so a
86+
link opens the file in the browser. `gh run download` (2.99) treats every artifact as a zip and
87+
fails on these with `zip: not a valid zip file`; the API's `/zip` endpoint returns the raw file with
88+
its `Content-Type` and an `inline` `Content-Disposition`, which is what the installed script uses.
89+
90+
Tracked upstream as [cli/cli#13012](https://github.com/cli/cli/issues/13012). When a `gh` release
91+
handles non-archived artifacts, `renderDownloadCommand` in `src/shared/proof.ts` can print
92+
`gh run download <run id>` instead, and `artifacts_command` in `octestra.sh` becomes a thin
93+
wrapper or goes away. Keep the API path until the fix is in the `gh` version consumers actually
94+
have; the maintenance script must keep working with the oldest `gh` that has `run download`.

‎action.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,12 @@ inputs:
4141
pull_number:
4242
description: Task pull request number
4343
required: false
44+
artifact_path:
45+
description: Directory whose files upload-artifacts publishes as one non-archived artifact each
46+
required: false
47+
artifact_links:
48+
description: Uploaded files to link from the proof comment, one per line as "<name> <url>"; upload-artifacts emits this format and the name is optional
49+
required: false
4450
next_status:
4551
description: Status used by the individual update-status operation
4652
required: false
@@ -97,6 +103,10 @@ outputs:
97103
description: Number of EPICs enabled for Todo triage
98104
ready_count:
99105
description: Number of Todo tasks moved to Ready by loop/finalize-triage
106+
artifact_links:
107+
description: One "<name> <url>" line per file that upload-artifacts published
108+
artifact_count:
109+
description: Number of files upload-artifacts published
100110

101111
runs:
102112
using: node24

‎dist/index.js‎

Lines changed: 353 additions & 187 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/integration.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,34 @@ The action runs in the same job, runner, and checked-out workspace as `lifecycle
171171
The validation agent uses the installed `/octestra-contracts` skill to write JSON to
172172
`inputs.result_path` and check its format.
173173

174+
### Evidence files
175+
176+
After the agent finishes, the workflow runs `operation: upload-artifacts` with `artifact_path` and
177+
`result_path`, even when the agent failed. It uploads every file under the directory, and the result
178+
JSON, as its own artifact without archiving, so the link to a screenshot or recording opens the file
179+
in the browser. The artifact name is the file's path relative to the directory with `/` replaced by
180+
`--`; the step's `artifact_links` output lists one `<relative path> <url>` line per file and
181+
`artifact_count` gives the number. Hidden files and directories are left out, as
182+
`actions/upload-artifact` leaves them out, and a symbolic link to a file is uploaded as that file.
183+
A file that fails to upload is logged and skipped, GitHub's limit of 500 artifacts per job caps what
184+
is uploaded, and the step never fails: the finalize step after it must still run.
185+
186+
The workflow passes `artifact_links` to `lifecycle/finalize-validation` (`report-proof` accepts the
187+
same input). The proof comment on the task issue then shows:
188+
189+
- an `Artifacts` row in the overview table, linking to the run's artifact list with the file count;
190+
- a command to download every file of the run,
191+
`.github/octestra/octestra.sh artifacts <run id>`, because `gh run download` cannot fetch
192+
non-archived artifacts ([cli/cli#13012](https://github.com/cli/cli/issues/13012)). It saves
193+
the files under the temporary directory unless a directory is given, so they do not appear as
194+
untracked files in the checkout;
195+
- links in the `Evidence` cells of the acceptance, checks, and evidence tables wherever the agent
196+
named an uploaded file by its relative path, or by its file name when that name is unique.
197+
198+
Leave `artifact_links` empty to post the comment without any of these. When the lines come from
199+
somewhere other than `upload-artifacts`, a line with only a URL is labelled `Artifact` and numbered
200+
when there are several, and a line without a URL is skipped.
201+
174202
### Use the agent debug flag
175203

176204
Set the repository Actions variable `OCTESTRA_AGENT_DEBUG` to `true` when a task, validation, or

0 commit comments

Comments
 (0)