TST: Cache .hypothesis - #11161

I have no idea what's in this directory and if it's really useful to cache it. Maybe cc @Zac-HD if you can provide some insights ?

Here's the Hypothesis docs about the database. In short:

• The benefit of caching .hypothesis between CI jobs is not speed, but avoiding flakiness: if you have a rarely-failing test, the database will ensure that the failure is replayed in future runs until fixed. In turn that means that if a test fails, you try to fix the bug, and then the test passes, you can be confident that the bug was actually fixed instead of the test getting unlucky.
• You can share exactly the same cache between all jobs, python versions, operating systems, etc. Any invalid entries are just silently skipped.
• The directory is created in the current working directory by default, but this is configurable as shown by Numpy: https://github.com/numpy/numpy/blob/91d9bbebc159ada6cccd0e1fcec1926e26b19c6f/numpy/conftest.py#L17-L41

I can't answer questions about how GitHub Actions caching works, but that's the Hypothesis side. Always happy to help with follow-up questions etc. too 🙂

Thanks @Zac-HD , really useful !

I modified the code to make the .hypothesis cache to be toxenv specific. Do you happen to know how big each cache is expected to be? Actions has a hard limit of 5 GB. When we expand things that we cache and each run generates 5 GB of cache or more, that means caching is meaningless. Something to keep in mind.

CI failure is unrelated. This is what the log looks like on first run:

2020-12-15T18:49:15.1092310Z ##[group]Run actions/cache@v2
2020-12-15T18:49:15.1092751Z with:
2020-12-15T18:49:15.1093205Z   path: .tmp/py39-test/.hypothesis
2020-12-15T18:49:15.1093808Z   key: hypothesis-py39-test
2020-12-15T18:49:15.1094406Z   restore-keys: hypothesis-
...
2020-12-15T18:49:15.1097820Z ##[endgroup]
2020-12-15T18:49:15.3307896Z Cache not found for input keys: hypothesis-py39-test, hypothesis-

I guess we'll see what it looks like on subsequent pushes after a successful CI run. I expect something like "cache hit, restoring cache" or something. 🤞

At the end of a successful run, you will see this:

2020-12-15T20:24:37.8040280Z Cache saved successfully

But if I understand https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#example-of-search-priority correctly, when a new PR is opened and master branch already has a cache from the last run (say, from a merge commit of unrelated PR), the new PR will grab the cache for that toxenv from master. Is this desirable?

At the next run, you will see something like this:

2020-12-15T20:47:12.5792988Z Received 22 of 22 (100.0%), 0.0 MBs/sec
2020-12-15T20:47:12.5794005Z Cache Size: ~0 MB (22 B)
...
2020-12-15T20:47:12.5928181Z Cache restored from key: hypothesis-py39-test
...
2020-12-15T20:56:03.9312308Z Cache hit occurred on the primary key hypothesis-py39-test, not saving cache.

So, I guess it is not going to be a problem as far as cache size limit is concerned...

Is this desirable?

Yes! That's what we want ;)

Cache hit occurred on the primary key hypothesis-py39-test, not saving cache.

🚨🚨🚨 That's not desirable though - if the cached database is loaded and then updated, we'd want GitHub Actions to update the cache.

That could probably be forced by adding e.g. build number as a final key suffix? So there would always be a cache miss, and then load the most recent alternative... not exactly the intended use, but it would work.

[cache size questions]

TLDR: it's designed not to be too large, I'd be surprised if the cache ever got to 1GB.
If it did, you're safe to just delete the whole thing and start over at any time.

Details: the maximum possible size of each value is 8 KiB, but typical sizes are more like tens of bytes with a right-tail up to ~hundreds or ~1K at the largest realistic end. Each key (=test) can have in principle unlimited keys; but in practice it shouldn't go above tens unless you have failing examples - and latter passing runs discard now-passing entries. Running fuzzing mode can grow it further, but it's still reasonably bounded and you're not doing that in CI anyway.

(Unrelated: I think both CircleCI and astropy-bot hate me now. One runs when I said to skip, the other one just never runs when I open a PR.)

Here is what the logs look like now:

Successful run

Cache restored from key: hypothesis-py39-test-{{ github.run_id }}
...
Cache saved successfully

The -{{ github.run_id }} is from a commit earlier where I forgot the $, which has since been fixed. But that run was cached, so now it reminds me of my shame until the cache is automatically invalidated in the future, as there is no way to manually delete the cache. This message means that it must have it the hypothesis-${{ matrix.toxenv }}- restore key.

This also means that if someone rerun this workflow with the same Run ID, it will restore but not write the cache, is that okay, @Zac-HD ?

Failed run

Cache not found for input keys: hypothesis-py37-test-devdeps-426074051, hypothesis-py37-test-devdeps-, hypothesis-

This job has been failing for a while, so a cache was never created. You can see that the run ID is properly populated here, hiding my shame a little. Since it is still failing, a cache is also not written out.

This also means that if someone rerun this workflow with the same Run ID, it will restore but not write the cache, is that okay?

I think it's unlikely enough to ever be a problem that we should just ignore the possibility - we'd only care if the repeat run found a bug that the first run didn't. You could use eg unix timestamps instead of run ID, but I'd just leave it as is.

Okay, I'll leave it as-is then, because I am not sure how to do the timestamp thing in an OS-agnostic way and I don't feel like dabbling in Powershell for the Windows job. 😬 Thanks for the review and advice! 😸

It's been a while and we haven't seen any problems that indicates that we actually need this. Should we close without merge?

@saimn voted to close. Can always revisit if the need arises.

We haven't seen any problems that indicate we actually need this.

The 'problem' that caching .hypothesis/ prevents is that one CI run might find a rare bug, which is then not rediscovered by a subsequent run. If this kind of flakiness is concerning, that's exactly what the database is designed to prevent.

On the other hand, derandomize=True (which you set in conftest.py) also prevents this problem, by making runs deterministic - so you either always find or never find that hypothetical rare bug - and the print_blob=True setting means that you can always reproduce failures in CI anyway.

TLDR; closing this is fine 👍