Request: way to have development testing set up with pinned test dependencies

@taldcroft

I have been just suggesting that the environment installed by pip install -e .[test] include at least some pinning (in particular on pytest which seems notorious for breaking things in ways that are inscrutable) so that local developer testing does not break suddenly for mysterious reasons.

which is clearly better in a separate issue than in a closed one! So herewith.

Basically, the request is to have some standard way in which one can set up an environment in which one can edit code and quickly run the appropriate tests to see if changes worked, i.e., just to pytest <path-to-test>. Right now, the above pip install -e .[test] sometimes fails as test dependencies are upgraded, and it would be good to have a way to get an environment with a known-to-work combination.

From what @bsipocz wrote:

To reuse the env: tox creates a .tox directory for the virtualenvs inside. Run tox once to set up everything and then you can manually run the activate script to bring up the same environment and then iterate on the individual tests.

It seems this is already possible with tox, although at least for me it was not immediately clear how to actually do that (and be able to edit regular source code). Anyway, this "feature request" may simply be a docs update so that we all can share the best practice. For now, labeling it as such.

@bsipocz

From @bsipocz: the docs certainly can point out the very pinfile tox uses for CI.

I'd be curious to see where that pinfile lives. I'm also up for playing around with this and working on simple changes where possible within astropy to improve the developer experience.

I don't really use tox but mess with my main environment all the time, so some details may not work well. But this recipe I think will to some extent:

tox is setting up the env of dependencies, but not astropy itself. When in doubt, I always use the ci config files to see how to set stuff up, given most of those are always starting from scratch.

So, to set up the env run e.g. once:

run tox -e alldeps

feel free to cancel once the tests are starting to run. This puts the env and configs in .tox/alldeps. So next time run source .tox/alldeps/bin/activate. Once the env is active, use pip install -e . to build astropy in the editable mode (same as `python setup.py develop was before), and keep testing with pytest on the file/module etc level as you wish. Given that the envs and pin file (that lives otherwise here: https://raw.githubusercontent.com/bsipocz/ci-helpers/pytest_remove_5.4_pin/pip_pinnings.txt) can change, it's worth to recreate this tox environment regularly. E.g monthly/bi-monthly?

This is quite hacky, but I feel it's probably the most convenient way to keep up with the env without getting into the weeds of the tools we use for this.

Ah, that's pretty neat! Since I'm normally already in an astropy-dev virtual environment, this would change very little to my work-flow.

p.s. And if I understand correctly, even if I were to do pip install -e .[test] because that's what my fingers are used to, it would in fact be OK, since all the test dependencies have already been installed (and setup.cfg would never ask explicitly for versions inconsistent with those that are in tox.ini).

tox has a --develop which installs the project with pip install -e, saving the time of the sdist creation/installation. That can be a good compromise, once the tox env has been created running tox --develop should be almost as fast as running pytest directly.

@saimn

@saimn - that sounds good, but there is a problem (at least on my system):

tox -e test --develop
# works fine
# let's try again
tox -e test --develop
# ModuleNotFoundError: No module named 'extension_helpers'
# oops, looks like it is now using my system's pytest version...

@bsipocz

Just for completeness, I thought I'd better explicitly test the suggestion of @bsipocz:

tox -e test
source .tox/test/bin/activate
pip install -e .
pytest -P uncertainty
# Yay, works and is fast!

I actually think this is easily unhacky enough to be part of the documentation. Indeed, in a way it is something that helps me understand what tox actually does, which is good in itself. What do you think, @taldcroft?

(there are some blank territories of tox for me, too, e.g. it tends not picking up the right pyenv versions I normally use but somehow digged up the osx default ancient one).

Also, I would suggest to list the tox -e alldeps so you also get all our dependencies and not just the test ones, although scipy etc can be manually installed into the env later, too.

@saimn

@saimn - that sounds good, but there is a problem (at least on my system):

Hmm that's annoying, and I can reproduce. This is discussed here :
tox-dev/tox#809 (comment) (and tox-dev/tox#1100) and seems to due to the usage of the develop mode, which is not well supported yet by the PEP 517 / 518.

tox -e test
source .tox/test/bin/activate
pip install -e .
pytest -P uncertainty

Well this feels a little hacky to be an officially recommended workflow in the docs. The initial test will take a long time, so one would maybe modify to add -- -k dont-match-any-test. Then needing to re-install (since the --develop doesn't seem to work). Finally, this then precludes actually using that toxcommand later. Right now tox and pytest are both available options that are not mutually exclusive.

What I wonder is whether it is possible to put version limitations in setup.cfg. I did actually try putting pytest<5.4 but that didn't seem to work (I got some weird fail that I don't remember now). I read that pip version conflict resolution is going to get better but is still weak (doesn't globally resolve version requirements like conda).

It's certainly better than nothing, especially as a back-up solution for when pip install -e .[test] does not work...

But I looked a bit more at the tox manpage and found a --notest. That in itself indeed does not test, but also does not leave a working state.. However, what does seem to work is the following, which I think is quite nice (and only a few characters longer than what I would do by hand in my own venv):

tox --develop --notest -e test
source .tox/test/bin/activate
pytest -P uncertainty

(I checked that in this state one can edit tests in the regular astropy directories and they will get executed. I also checked that my own venv still gives an error with pip install -e .[test])

@mhvk

@mhvk - OK I tried that out and it seems to work and gets the hack-level down a bit. I would just recommend (following on from @bsipocz ):

tox --develop --notest -e test-alldeps
source .tox/test-alldeps/bin/activate
pip install sphinx-astropy  # Want to build docs as well

I also confirmed that this makes a virtualenv that gets picked up by my IDE (VS code) and works in that integrated environment. Presumably PyCharm will do the same and recognize the virtualenv as a possible run/debug environment.

So one outstanding item is GraphViz. pip install graphviz does install something, but I just checked that it is not enough to successfully build the docs without a blizzard of warnings. So the base environment from which the virtualenv is built needs a conda install or system install of the graphviz library. Ugh.

BTW, I agree that independent of instructions for pytest-style testing, that explaining a little about tox and how it just makes a virtualenv that you can take over, this would be very helpful.

PR including this in the docs in #10394...

These days the nice answer is

tox -e test --develop --notest
source .tox/test/bin/activate

Ideally, this should be documented somewhere...

These days the nice answer is
tox -e test --develop --notest
source .tox/test/bin/activate
Ideally, this should be documented somewhere...

Nice! Are you brave enough to revive #10394? 😄

@Cadair

Nice! Are you brave enough to revive #10394? 😄

Here you go - #15878 (I think this will be much less controversial, since @Cadair mentioned it as the right way go about it - #10394 (review))

@Cadair

Well, actually, @Cadair recommending it is not true at all! Sorry!

fixed by #15878

Navigation Menu

Repository navigation

Request: way to have development testing set up with pinned test dependencies #10303

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Request: way to have development testing set up with pinned test dependencies #10303

Description

Activity

taldcroft commented on May 4, 2020

bsipocz commented on May 4, 2020

mhvk commented on May 4, 2020

saimn commented on May 4, 2020

mhvk commented on May 5, 2020

mhvk commented on May 5, 2020

bsipocz commented on May 5, 2020

bsipocz commented on May 5, 2020

saimn commented on May 5, 2020

taldcroft commented on May 5, 2020

mhvk commented on May 5, 2020

taldcroft commented on May 5, 2020

taldcroft commented on May 5, 2020

mhvk commented on May 25, 2020

mhvk commented on Jan 11, 2024

taldcroft commented on Jan 12, 2024

mhvk commented on Jan 12, 2024

mhvk commented on Jan 12, 2024

mhvk commented on Nov 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions