Mark no longer flakey tests as bringup:false #126281
Conversation
reidbaker
requested review from
camsim99,
gmackall,
godofredoc,
keyonghan,
sealesj and
yusuf-goog
.ci.yaml
Outdated
| timeout: 60 | ||
| scheduler: release | ||
| bringup: true | ||
| bringup: false |
There was a problem hiding this comment.
This target has been timing out frequently: https://ci.chromium.org/p/flutter/builders/prod/Linux%20flutter_packaging_test?limit=50, we do not want to simply flip it back.
There was a problem hiding this comment.
But those are infra failures not flakes. Should they be treated the same?
There was a problem hiding this comment.
Yes, we should treat them the same, whenever the test becomes flaky and caused tree red.
.ci.yaml
Outdated
| - name: Staging_build_linux analyze | ||
| presubmit: false | ||
| bringup: true | ||
| bringup: false |
There was a problem hiding this comment.
Instead, we may want to remove this target: #126278
There was a problem hiding this comment.
This test is fine. The one you link to is framework_tests_misc not analyze.
See https://ci.chromium.org/p/flutter/builders/staging/Staging_build_linux%20analyze?cursor=id%3E8782771663962116337&limit=200
There was a problem hiding this comment.
Thanks for the link. IIRC, these Staging_build targets were created for experiment purpose, not expected to affect the tree. @yusuf-goog Has that experiment done? If yes, we can remove these targets.
There was a problem hiding this comment.
There was a problem hiding this comment.
The Staging_build_linux analyze is same as Linux analyze. We do not want to run two same tests in prod CI.
Before confirmed by Yusuf whether we want to delete it, I would suggest keep the bringup: true to unblock here.
…iling in infra step
|
Holding autosubmit on @keyonghan's lgtm |
reidbaker
added
the
autosubmit
auto-submit
bot
removed
the
autosubmit
|
auto label is removed for flutter/flutter, pr: 126281, due to - The status or check suite Windows plugin_test has failed. Please fix the issues identified (or deflake) before re-applying this label. |
reidbaker
added
the
autosubmit
auto-submit
bot
removed
the
autosubmit
This is a result of auditing the ci.yaml tests marked as bringup true and determining which ones were still flakey. Each commit markes one tests as non flakey along with a link of proof of the lack of flakes. - Mark new_gallery_impeller__transition_perf as not flakey, flutter#124699 closed, not recent flakes - No failures in 500 runs for Staging_build_linux analyze https://ci.chromium.org/p/flutter/builders/staging/Staging_build_linux%20analyze\?cursor\=id%3E8782771663962116337\&limit\=200 - No flakes in 600 builds https://ci.chromium.org/p/flutter/builders/staging/Linux%20flutter_packaging_test\?cursor\=id%3E8787892377160059793\&limit\=200 - No flakes in 600 builds https://ci.chromium.org/p/flutter/builders/staging/Mac%20flutter_packaging_test\?cursor\=id%3E8787870631931126673\&limit\=200 - No flakes in 600 builds https://ci.chromium.org/p/flutter/builders/staging/Mac_arm64%20flutter_packaging_test\?cursor\=id%3E8787809799161254193\&limit\=200 flutter#124699 is the only closed bug that was linked for a test that is no longer flakey. Closed status helps the case that flakes are no longer an issue.
3 participants
This is a result of auditing the ci.yaml tests marked as bringup true and determining which ones were still flakey.
Each commit markes one tests as non flakey along with a link of proof of the lack of flakes.
#124699 is the only closed bug that was linked for a test that is no longer flakey. Closed status helps the case that flakes are no longer an issue.
Pre-launch Checklist
///).