-
Notifications
You must be signed in to change notification settings - Fork 29.7k
Turn back on macOS shard Cirrus caching #50496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
dc32884 to
f600493
Compare
|
No more dart SDK, pub, or artifacts download during setup. Also reduces setup time from 3 minutes to 1:30 minutes. |
.cirrus.yml
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there some way we can prevent this from breaking? Maybe a test that checks that this is the only file changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would this file invalidate the cirrus cache? Aren't we fingerprinting bin/internal/*.stamp?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahhh.. do we use this in the tool to say we don't trust the cache?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know the shell wrappers don't check it. It looks to me like this is just used for how often we show the user about a newer upstream version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Other than making it readonly, I can't think how. bin/cache is gitignored so there's no way to tell what's changed in the working directory. Creating a sub-repo in that directory in setup to then reset before upload would take minutes to init and commit because of the large binaries.
#44526 would make this better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would this file invalidate the cirrus cache? Aren't we fingerprinting
bin/internal/*.stamp?
The cache invalidation doesn't come from the fingerprint, it comes from Cirrus's hashing of the cache. From https://cirrus-ci.org/guide/writing-tasks/#cache-instruction
After the last script instruction for the task succeeds, Cirrus CI will calculate checksum of the cached folder (note that it's unrelated to fingerprint_script instruction) and re-upload the cache if it finds any changes. To avoid a time-costly re-upload, remove volatile files from the cache (for example, in the last script instruction of a task).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahhh, i see. thus any changes of the cache during execution will trigger a re-upload.
.cirrus.yml
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Hixie @dnfield Do either of you know why the other shards have separate Cirrus caches for pkg and artifacts in the bin/cache but not that entire directory? I don't think the Linux or Windows caches are getting used since all the stamps are missing and it just re-downloads everything again.
Example:
https://cirrus-ci.com/task/6607166220009472?command=setup#L420
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's just how they were always set up - it does look like we're getting cache hits (https://cirrus-ci.com/task/6607166220009472?command=flutter_pkg#L16) but maybe failing to use them in setup...
It seems likely to me that this worked at some point and we fixed something in the tool that broke it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, even worse than ignoring the cache, we're downloading it, then the tool blows it away because the stamp files aren't there and re-downloads them. Should this PR include linux + windows?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was going to do Windows and Linux in a separate PR. I want to get the mac changes out the door ASAP with Cirrus's network flakiness.
christopherfujino
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
|
Talked offline - I think the only way to address my concern that makes sense is to split readable and writable cache as Jenn is suggesting. LGTM |
|
Holding this until I get clarity from @fkorotkov about what seems to be cache corruption at https://cirrus-ci.com/task/6396945455382528?command=all_flutter#L17 that's causing cache misses. |
74ec9b9 to
25879c4
Compare
925e2dd to
36ad793
Compare
36ad793 to
352b09d
Compare
|
This reverts commit e002698.
Description
Revisiting #49585 in light of revent Cirrus network unstability on mac VMs.
Related Issues
Reverts part of #37880.
Checklist
///).flutter analyze --flutter-repo) does not report any problems on my PR.Breaking Change