Skip to content

Conversation

@jmagman
Copy link
Member

@jmagman jmagman commented Feb 10, 2020

Description

Revisiting #49585 in light of revent Cirrus network unstability on mac VMs.

Related Issues

Reverts part of #37880.

Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I signed the [CLA].
  • I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
  • I read the [Tree Hygiene] wiki page, which explains my responsibilities.
  • I updated/added relevant documentation (doc comments with ///).
  • All existing and new tests are passing.
  • The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
  • I am willing to follow-up on review comments in a timely manner.

Breaking Change

  • No, no existing tests failed, so this is not a breaking change.
  • Yes, this is a breaking change. If not, delete the remainder of this section.

@jmagman jmagman added c: contributor-productivity Team-specific productivity, code health, technical debt. team-infra Owned by Infrastructure team labels Feb 10, 2020
@jmagman jmagman self-assigned this Feb 10, 2020
@jmagman
Copy link
Member Author

jmagman commented Feb 10, 2020

No more dart SDK, pub, or artifacts download during setup. Also reduces setup time from 3 minutes to 1:30 minutes.

.cirrus.yml Outdated
Comment on lines +590 to +593
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some way we can prevent this from breaking? Maybe a test that checks that this is the only file changed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this file invalidate the cirrus cache? Aren't we fingerprinting bin/internal/*.stamp?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh.. do we use this in the tool to say we don't trust the cache?

Copy link
Contributor

@christopherfujino christopherfujino Feb 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the shell wrappers don't check it. It looks to me like this is just used for how often we show the user about a newer upstream version.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than making it readonly, I can't think how. bin/cache is gitignored so there's no way to tell what's changed in the working directory. Creating a sub-repo in that directory in setup to then reset before upload would take minutes to init and commit because of the large binaries.

#44526 would make this better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this file invalidate the cirrus cache? Aren't we fingerprinting bin/internal/*.stamp?

The cache invalidation doesn't come from the fingerprint, it comes from Cirrus's hashing of the cache. From https://cirrus-ci.org/guide/writing-tasks/#cache-instruction

After the last script instruction for the task succeeds, Cirrus CI will calculate checksum of the cached folder (note that it's unrelated to fingerprint_script instruction) and re-upload the cache if it finds any changes. To avoid a time-costly re-upload, remove volatile files from the cache (for example, in the last script instruction of a task).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh, i see. thus any changes of the cache during execution will trigger a re-upload.

.cirrus.yml Outdated
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hixie @dnfield Do either of you know why the other shards have separate Cirrus caches for pkg and artifacts in the bin/cache but not that entire directory? I don't think the Linux or Windows caches are getting used since all the stamps are missing and it just re-downloads everything again.
Example:
https://cirrus-ci.com/task/6607166220009472?command=setup#L420

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's just how they were always set up - it does look like we're getting cache hits (https://cirrus-ci.com/task/6607166220009472?command=flutter_pkg#L16) but maybe failing to use them in setup...

It seems likely to me that this worked at some point and we fixed something in the tool that broke it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, even worse than ignoring the cache, we're downloading it, then the tool blows it away because the stamp files aren't there and re-downloads them. Should this PR include linux + windows?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to do Windows and Linux in a separate PR. I want to get the mac changes out the door ASAP with Cirrus's network flakiness.

Copy link
Contributor

@christopherfujino christopherfujino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dnfield
Copy link
Contributor

dnfield commented Feb 11, 2020

Talked offline - I think the only way to address my concern that makes sense is to split readable and writable cache as Jenn is suggesting. LGTM

@jmagman
Copy link
Member Author

jmagman commented Feb 11, 2020

Holding this until I get clarity from @fkorotkov about what seems to be cache corruption at https://cirrus-ci.com/task/6396945455382528?command=all_flutter#L17 that's causing cache misses.

@jmagman
Copy link
Member Author

jmagman commented Feb 11, 2020

Holding this until I get clarity from @fkorotkov about what seems to be cache corruption at https://cirrus-ci.com/task/6396945455382528?command=all_flutter#L17 that's causing cache misses.

cirruslabs/cirrus-ci-docs#567

@jmagman jmagman merged commit e002698 into flutter:master Feb 11, 2020
@jmagman jmagman deleted the cache-cirrus branch February 11, 2020 18:37
@jmagman jmagman restored the cache-cirrus branch February 12, 2020 20:42
@jmagman jmagman deleted the cache-cirrus branch February 12, 2020 20:45
jmagman added a commit to jmagman/flutter that referenced this pull request Feb 12, 2020
jmagman added a commit to jmagman/flutter that referenced this pull request Feb 26, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt. team-infra Owned by Infrastructure team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants