Skip to content

Conversation

@Piinks
Copy link
Contributor

@Piinks Piinks commented Oct 23, 2019

Description

This will re-land the changes introduced in #40710, which were reverted after golden file tests started to time out when making requests to the details/ api.

What's Changed

Noted below, the case for querying the details/ api has been changed.

Before:

  • While iterating through expectations, the details/ were checked for each failure while looking for a positive

Now:

  • The details/ are only checked after no positive has been found, rather than in step with the initial pass through
  • This ensures that this call will only be made in cases where a golden file has changed, which is expected far less often than at every test execution as it was before.

The details/ api is also scheduled for performance improvements, tracking available here https://bugs.chromium.org/p/skia/issues/detail?id=9557 for Gold side changes.

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I signed the CLA.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I updated/added relevant documentation (doc comments with ///).
  • All existing and new tests are passing.
  • The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
  • I am willing to follow-up on review comments in a timely manner.

Breaking Change

Does your PR require Flutter developers to manually update their apps to accommodate your change?

  • Yes, this is a breaking change (Please read Handling breaking changes). Replace this with a link to the e-mail where you asked for input on this proposed change.
  • No, this is not a breaking change.

@Piinks Piinks added a: tests "flutter test", flutter_test, or one of our tests c: contributor-productivity Team-specific productivity, code health, technical debt. labels Oct 23, 2019
@fluttergithubbot fluttergithubbot added f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. c: API break Backwards-incompatible API changes labels Oct 23, 2019
@fluttergithubbot
Copy link
Contributor

It looks like this pull request includes a golden file change. Please make sure to follow Handling Breaking Changes. While there are exceptions to this rule, if this patch modifies an existing golden file, it is probably not an exception. Only new golden files are not considered breaking changes.

Writing a golden file test for package:flutter may also provide guidance for this change.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.


if (result.passed) {
return true;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior version called isValidDigestForExpectation here, to collect potential valid failures as the comparator searched for a positive.
In this case, it may needlessly make this call multiple times before finding a positive.

failureDiffs[expectation] = result;
}
failureDiffs.forEach((String expectation, ComparisonResult result) async {
if (await skiaClient.isValidDigestForExpectation(expectation, golden.path))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting to make this call until there is no positive found will save a lot of calls, as it will only reach this point when a change has been detected.

@Piinks Piinks removed f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. c: API break Backwards-incompatible API changes labels Oct 23, 2019
@Piinks Piinks requested a review from goderbauer October 23, 2019 20:24
@Piinks
Copy link
Contributor Author

Piinks commented Oct 23, 2019

cc/ @kjlubick

I changed the use case of the details/ api, which should greatly reduce the number of calls.

Copy link
Member

@goderbauer goderbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kjlubick
Copy link
Contributor

I hope to have the QPS limit on details/ live by EOD today. Will ping the bug when they are live.

@kjlubick
Copy link
Contributor

Change to gold is live to take more traffic on /json/details

@Piinks
Copy link
Contributor Author

Piinks commented Oct 25, 2019

Performed an all-fail load test this morning and experienced no issues. 🎉

I also found that our post-submit LUCI builds were falling back to the Local comparator, so I'm re-applying the Skip comparator for those environments. The Post-submit comparator is used on Cirrus, so having the Local comparator running on LUCI, making additional requests, is unnecessary.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

a: images Loading, displaying, rendering images a: quality A truly polished experience a: tests "flutter test", flutter_test, or one of our tests c: contributor-productivity Team-specific productivity, code health, technical debt. framework flutter/packages/flutter repository. See also f: labels. will affect goldens Changes to golden files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants