Skip to content

Making flutter-engine-gold.skia.org blocking w/o causing tree pain #143617

@matanlurey

Description

@matanlurey

🚫 Blocking #143352.

In April 2023, we made the SkiaGold check non-blocking with the assumption the engine sheriff would review them.

In practice, that has not (IMO) worked out particularly well:

  1. The sheriff is poorly suited to make decisions other than "this needs further triage" or not
  2. Lack of a blocking tree means its easy to introduce unstable or even broken golden tests
  3. Finding the right person to review and act on untriaged goldens a time consuming and error-prone task.

We need to decide:

  • Do we want to block the tree on untriaged goldens
  • If yes, how do we deal with tree breakages due to untriaged goldens

The rest of this issue assumes "yes", but please chime in if you think the answer should be "no".

Questions to answer:

  • What if a team wants to "try" goldens on CI, without committing to them?
    • Do we have "bringup"-style goldens?
    • Do we consider either a dimension that means "can fail" or even a separate tree (flutter-engine-bringup.skiagold)?
  • What if a golden is flaky, i.e. a slightly different image is emitted and Skia gold can't stabilize?
  • What if the tree somehow makes it to a red state out of band (i.e. there is no obvious PR to revert)?
  • Should the sheriff be disabling failing/flaky golden tests in order to bring the tree green?

@zanderso @dnfield @gaaclarke @jmagman @reidbaker @johnmccutchan @jonahwilliams @Piinks @yjbanov @mdebbar

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: proposalA detailed proposal for a change to Flutterc: tech-debtTechnical debt, code quality, testing, etc.engineflutter/engine related. See also e: labels.fyi-androidFor the attention of Android platform teamfyi-iosFor the attention of iOS platform teamfyi-webFor the attention of Web platform teamteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions