devops: upload test results to Flakiness.io#7359
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughAdds flakiness reporting for Playwright visual-regression tests: installs the flakiness reporter, configures CI to emit flakiness artifacts per matrix entry, and adds a workflow that uploads those reports to Flakiness.io for fork PRs. ChangesFlakiness tracking integration for visual regression tests
Sequence DiagramsequenceDiagram
participant VRTests as Playwright VR Tests
participant CIJob as CI (vr_tests)
participant Artifact as GitHub Artifacts
participant UploadWF as Flakiness Upload Workflow
participant FlakinessIO as Flakiness.io
VRTests->>VRTests: Emit flakiness report (via `@flakiness/playwright`)
VRTests->>CIJob: Report files produced
CIJob->>Artifact: Upload flakiness-report-* artifacts (1d retention)
UploadWF->>Artifact: Download flakiness-report-* from triggering run
UploadWF->>FlakinessIO: Run `npx flakiness upload` with report.json files
FlakinessIO->>FlakinessIO: Store/track flakiness data
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PavelVanecek
left a comment
PavelVanecek
left a comment
There was a problem hiding this comment.
How do credentials work? Will this upload report even for PRs of first contributors?
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 480-483: Replace the mutable tag uses of
actions/upload-artifact@v4 with the corresponding full commit SHAs to hard-pin
the action; locate every occurrence of "uses: actions/upload-artifact@v4" in the
workflow and update each to the verified commit SHA for the specific v4 release
you want to lock to, ensuring consistency across all upload-artifact steps and
committing the updated workflow.
In @.github/workflows/flakiness-upload-fork-prs.yml:
- Around line 26-27: Replace the unpinned `npx flakiness upload` invocation with
a pinned CLI version (e.g. `npx flakiness@<VERSION> upload`) so the workflow
uses a deterministic, auditable flakiness CLI; update the command in the step
that currently runs `find . -path '*/flakiness-report-*/report.json' -exec npx
flakiness upload {} \;` to call `npx flakiness@<VERSION> upload` (or use an
equivalent pinned install like `npm exec --package=flakiness@<VERSION> --
flakiness upload`) and choose a specific version string to freeze in CI.
In `@test-vr/playwright-ct.config.ts`:
- Around line 50-53: The reporter configuration contains an invalid comma before
a ternary operator causing a parse error; locate the reporter array in
test-vr/playwright-ct.config.ts (the reporters/CI branch conditional block) and
remove the extra trailing comma after the CI branch array so the ternary
expression is syntactically valid (ensure the conditional uses arrayA : arrayB
with no stray comma before the colon and that the two branch arrays remain
intact).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c677ab21-ea91-4f93-bd43-1845608489a3
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (4)
.github/workflows/ci.yml.github/workflows/flakiness-upload-fork-prs.ymlpackage.jsontest-vr/playwright-ct.config.ts
| - name: Upload Flakiness report artifact (fork PRs only) | ||
| if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork | ||
| uses: actions/upload-artifact@v4 | ||
| with: |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the referenced section in the workflow
sed -n '450,520p' .github/workflows/ci.yml
# Find all uses of actions/upload-artifact in the repository
rg -n "uses:\s*actions/upload-artifact@" .github/workflows -S || trueRepository: recharts/recharts
Length of output: 3088
Pin actions/upload-artifact to a full commit SHA (workflow security)
Line 482 uses actions/upload-artifact@v4 (mutable tag). The same @v4 usage appears elsewhere in this workflow as well (lines 74, 178, 334, 380, 490, 533), so pin all upload-artifact steps to full commit SHAs.
Suggested fix
- uses: actions/upload-artifact@v4
+ uses: actions/upload-artifact@<full-length-commit-sha>🧰 Tools
🪛 zizmor (1.25.2)
[error] 482-482: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 480 - 483, Replace the mutable tag
uses of actions/upload-artifact@v4 with the corresponding full commit SHAs to
hard-pin the action; locate every occurrence of "uses:
actions/upload-artifact@v4" in the workflow and update each to the verified
commit SHA for the specific v4 release you want to lock to, ensuring consistency
across all upload-artifact steps and committing the updated workflow.
| - name: Upload to Flakiness.io | ||
| run: find . -path '*/flakiness-report-*/report.json' -exec npx flakiness upload {} \; No newline at end of file |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and display the workflow file around the referenced lines
FILE=".github/workflows/flakiness-upload-fork-prs.yml"
echo "=== File exists? ==="
ls -l "$FILE" || true
echo "=== Snippet around lines 15-60 ==="
nl -ba "$FILE" | sed -n '15,60p'
echo
echo "=== Search for flakiness usage in repo ==="
rg -n "flakiness" .github/workflows . -S --hidden || trueRepository: recharts/recharts
Length of output: 276
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/flakiness-upload-fork-prs.yml"
ls -l "$FILE" || true
echo "=== Snippet around lines 15-60 ==="
nl -ba "$FILE" | sed -n '15,60p'
echo
echo "=== Search for flakiness usage in repo ==="
rg -n "flakiness" .github/workflows . -S --hidden || trueRepository: recharts/recharts
Length of output: 254
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/flakiness-upload-fork-prs.yml"
ls -l "$FILE"
nl -ba "$FILE" | sed -n '1,120p'Repository: recharts/recharts
Length of output: 219
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/flakiness-upload-fork-prs.yml"
ls -l "$FILE"
nl -ba "$FILE" | sed -n '1,120p'Repository: recharts/recharts
Length of output: 219
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "npx flakiness" .github/workflows -S || true
rg -n "flakiness" package*.json -S || true
ls -laRepository: recharts/recharts
Length of output: 4208
🏁 Script executed:
ls .github/workflows | headRepository: recharts/recharts
Length of output: 210
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/flakiness-upload-fork-prs.yml"
echo "=== WORKFLOW FILE ==="
nl -ba "$FILE" | sed -n '1,120p'
echo
echo "=== SEARCH: npx flakiness ==="
rg -n "npx[[:space:]]+flakiness" .github/workflows -S || true
echo
echo "=== SEARCH: flakiness package.json ==="
rg -n "\"flakiness\"" package*.json -S || true
rg -n "flakiness" package*.json -S || trueRepository: recharts/recharts
Length of output: 144
🏁 Script executed:
pwd && lsRepository: recharts/recharts
Length of output: 508
Pin the Flakiness CLI version in CI.
The workflow .github/workflows/flakiness-upload-fork-prs.yml runs npx flakiness upload without a fixed version, so the CLI can change between runs (determinism + supply-chain risk).
Suggested fix
- run: find . -path '*/flakiness-report-*/report.json' -exec npx flakiness upload {} \;
+ run: find . -path '*/flakiness-report-*/report.json' -exec npx flakiness@<exact-version> upload {} \;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/flakiness-upload-fork-prs.yml around lines 26 - 27,
Replace the unpinned `npx flakiness upload` invocation with a pinned CLI version
(e.g. `npx flakiness@<VERSION> upload`) so the workflow uses a deterministic,
auditable flakiness CLI; update the command in the step that currently runs
`find . -path '*/flakiness-report-*/report.json' -exec npx flakiness upload {}
\;` to call `npx flakiness@<VERSION> upload` (or use an equivalent pinned
install like `npm exec --package=flakiness@<VERSION> -- flakiness upload`) and
choose a specific version string to freeze in CI.
Bundle ReportBundle size has no change ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7359 +/- ##
=======================================
Coverage 89.04% 89.04%
=======================================
Files 548 548
Lines 40546 40546
Branches 5570 5570
=======================================
Hits 36105 36105
Misses 4432 4432
Partials 9 9 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
@PavelVanecek there are 2 different paths.
Is there anything special about first contributors? |
It's rather "PRs originating from this repository" which just me and @ckifer have access and "PRs originating from forks" which is everyone else. You explained it all I think, thank you, nothing special about "first". |
2 participants
Fixes #7358
Summary by CodeRabbit