fix(ci): increase devstack wait timeout and add failure diagnostics#3156
Conversation
The full-stack-test workflow was timing out on generate-configs with WAIT_TIMEOUT=600 (10 min). CI runners are slow and services sometimes need longer to initialize. - Increase WAIT_TIMEOUT from 600 to 900 (15 min) - Add timeout-minutes: 30 to prevent runaway jobs (was using 6h default) - Add diagnostic step on failure: dump tilt resource status and recent logs - Run teardown with if: always() so cleanup happens on failure too Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Greptile SummaryThis PR improves the full-stack CI job by increasing
Confidence Score: 3/5The 30-minute job timeout is likely too tight given the 15-minute environment wait, and could cause the same class of intermittent CI failures this PR is trying to fix. A single P1 finding: the job-level timeout and WAIT_TIMEOUT interact in a way that may still produce spurious timeout failures. Score pulled down from the P1 ceiling of 4 because the fix directly targets CI reliability, and this interaction undermines that goal. .github/workflows/full-stack-test.yml — job timeout value needs to be revisited Important Files Changed
Reviews (1): Last reviewed commit: "fix(ci): increase devstack wait timeout ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 |
There was a problem hiding this comment.
timeout-minutes: 30 gives the entire job 30 minutes, but WAIT_TIMEOUT=900 (line 29) alone already reserves up to 15 minutes just for the environment to become ready. The remaining ~15 minutes must cover pip install (~1-2 min), the Kubernetes flow run (variable, often several minutes on a slow runner), the diagnostic step, and teardown — leaving very little headroom. A slow runner that hits near the 15-minute wait would almost certainly cause the job to be killed by the job-level timeout before the flow even finishes.
Consider raising this to at least 60 minutes to match the realistic worst-case path, or lower WAIT_TIMEOUT to free up more budget for the actual test run.
1 participant
Summary
WAIT_TIMEOUTfrom 600s to 900s for the full-stack Kubernetes test — CI runners are slow andgenerate-configsintermittently times outtimeout-minutes: 30to the job (was using the 6-hour GitHub default)if: always()so cleanup happens even on failureTest plan
testjob in "Test Metaflow with complete Kubernetes stack" no longer times out ongenerate-configs🤖 Generated with Claude Code