Repository navigation
Combine benchmarks that build the same app with the same configuration to a single devicelab task #195654
Description
Example: These 10 Mac_ios devicelab tasks all build dev/benchmarks/macrobenchmarks with the same entrypoint and each draws for about 10 seconds. The benchmark step includes a profile build of the same app 10 times. So 10 Mac + iPhone bots run for ~100 seconds of actual drawing per commit.
Medians of the last 8 postsubmit runs:
| Task | Bot run | Setup | Benchmark step |
|---|---|---|---|
| dynamic_path_tessellation_perf_ios__timeline_summary | 9.7 min | 7.0 | 2.6 |
| static_path_tessellation_perf_ios__timeline_summary | 9.0 | 6.3 | 2.6 |
| dynamic_path_stroke_tessellation_perf_ios__timeline_summary | 9.1 | 6.9 | 2.2 |
| draw_points_perf_ios__timeline_summary | 9.0 | 6.7 | 2.2 |
| rrect_blur_perf_ios__timeline_summary | 8.3 | 6.3 | 2.0 |
| draw_vertices_perf_ios__timeline_summary | 6.9 | 4.9 | 2.0 |
| static_path_stroke_tessellation_perf_ios__timeline_summary | 7.5 | 5.7 | 1.8 |
| draw_arcs_all_stroke_styles_perf_ios__timeline_summary | 5.5 | 3.9 | 1.6 |
| draw_arcs_all_fill_styles_perf_ios__timeline_summary | 6.3 | 4.9 | 1.4 |
| draw_atlas_perf_ios__timeline_summary | 5.0 | 3.6 | 1.4 |
| Per commit | 76 min | 56 min | 22 min |
("Setup" is everything outside the task step like checkout, CIPD dependencies, device preparation, result upload, etc).
Proposal
Run all 10 in one devicelab task. Each benchmark keeps its own flutter drive, so the app is reinstalled and relaunched every time and no in-process state (pipeline cache, image or glyph caches, heap) carries over. Because the source and target are the same, drives after the first reuse the already-built .app and only pay install, launch, and the benchmark run itself. None of these are testing thermals, so it doesn't seem like a big deal to test them back-to-back because the re-install and re-launch still take some time between tests so that cools it down some.
It would still be possible to keep each benchmark's SkiaPerf history if metrics_center were tweaked to handle a passed in task name.
This would mean ~16 bot-minutes per commit instead of ~76, or ~100 Mac+iPhone (expensive/scarce CI bot combo) hours a week. Note the combined task would be the longest Mac_ios task (~16 min), but it's a postsubmit task so... meh?
However, a failure in any one benchmark fails the task and drops that commit's points for all 10.
Activity
jmagman commentedon Oct 30, 2026
Another combo: animated_advanced_blend, animated_blur_backdrop_filter, post_backdrop_filter, imagefiltered_transform_animation, animation_with_microtasks, simple_animation_perf_ios are also all the same config, target, etc
gaaclarke commentedon Oct 30, 2026
It seems like we could work around that with some plumbing. Is that something built into skiaperf that it will throw away results if a task fails? Otherwise just collect failures and only report them at the end after all bookkeeping is sent for passing tests?
jmagman commentedon Oct 30, 2026
No it's not a skiaperf problem, it's our own runner logic.
TaskResult.failuredropsdata/benchmarkScoreKeysandtoJsononly writes them on task success. The devicelab_drone recipe's upload step runs afterrun_test, which raises on a non-zero exit, so a failed task never uploads the data.Fixing it would be a
TaskResult.failurethat carries partial data, plus a recipe change to upload before failing the step, which seems a bit extreme.Marking the whole task as succeeding would allow the upload, but of course that means no one will know that the benchmark is broken and the tree won't close.
Also, a sucky thing I didn't consider before is the devicelab runner retries a failed task twice, so one flaky benchmark will rerun all ten. However these benchmarks don't tend to be that flaky except for infra issues?
Run 10 impeller iOS benchmarks as one devicelab task (flutter#195661)
Run 10 impeller iOS benchmarks as one devicelab task (flutter#195661)