Thanks @marload. This looks like a bug that should not just be ignored. np.test() can be run on installed versions of numpy, and it should not leave behind any stray files.

ping @Zac-HD

Should the same be applied to .pytest_cache which is already in .gitignore?

Should the same be applied to .pytest_cache which is already in .gitignore?

Yes, pytest should clean up after itself as well, and we should remove it from .gitignore to ensure we catch regressions. Unless there's some setting for enabling the cache, that we could use to disable by default and let devs enable for speedups if that helps (which is hard to imagine, test discovery is very fast compared to running the tests).

I would suggest adding .hypothesis/ to .gitignore, but also using a different settings profile when invoked via np.test().

Because this mode is really just checking for packaging or config related regressions rather than looking for bugs, it should probably set use the derandomize=True and database=None settings to ensure determinism and avoid persisting any data. That's presently configured here:

I'd be happy to write the PR for that, if someone can show me how to tell if we're running np.test() v.s. running the tests in development or CI.

how to tell if we're running np.test() v.s. running the tests in development or CI.

Calling np.test() uses the __call__ method of PytestTester from numpy/_pytesttester.py. That may be a place to set the profile. Do you know if pytest also uses a profile?

I would suggest adding .hypothesis/ to .gitignore

Another problem with this approach is that it doesn't catch the case where pytest is called from a subdirectory, e.g.

cd numpy/lib/tests
pytest test_function_base.py

This creates a .hypothesis folder in numpy/lib/tests. It'd be good if this use-case were handled as well.

Calling np.test() uses the __call__ method of PytestTester from numpy/_pytesttester.py. That may be a place to set the profile. Do you know if pytest also uses a profile?

Sounds good!

Assuming you mean "does invoking pytest test_foo.py in Numpy use a Hypothesis profile", the answer is yes - that's set up in the root conftest.py I linked to above.

I would suggest adding .hypothesis/ to .gitignore

Another problem with this approach is that it doesn't catch the case where pytest is called from a subdirectory, ... this creates a .hypothesis folder in numpy/lib/tests. It'd be good if this use-case were handled as well.

gitignoring .hypothesis/ does in fact ignore subdirectories as well; I think you're conflating it with /.hypothesis/ which would only ignore it in the root directory of the repo.

You're correct though that it would be better to force the database dir to always be at the repo root. I'll set that up too, so that the cache is shared regardless of the working directory from which pytest is invoked.

gitignoring .hypothesis/ does in fact ignore subdirectories as well; I think you're conflating it with /.hypothesis/ which would only ignore it in the root directory of the repo.

Understood - my point was that I prefer a solution that cleans up any .hypothesis/ dirs wherever they're created rather than ignoring them.

The PR I've opened ensures that we avoid generating a .hypothesis dir in the repo at all 🙂

Do you know if pytest also uses a profile?

Clarifying: do you know if there is a mechanism to convince pytest not to create a .cache directory at the $PWD? I think the answer is somewhere on the pytest-cache documentation page, which references "Deactivating a plugin by name" but I couldn't see how to do that programatically.

Clarifying: do you know if there is a mechanism to convince pytest not to create a .cache directory at the $PWD? I think the answer is somewhere on the pytest-cache documentation page, which references "Deactivating a plugin by name" but I couldn't see how to do that programatically.

I think adding pytest_args += ["-p", "no:cacheprovider"] to numpy._pytesttester.PytestTester.__call__ would do it.

Closing, replaced by gh-16879. It would be nice to turn the comment above into a PR.