Describe how to do development within a tox testing environment by mhvk · Pull Request #10394 · astropy/astropy

mhvk · 2020-05-25T01:05:13Z

As promised in #10303, once we found out how to easily set up an environment for testing, I would add it to the documentation. This is meant to do that. In the process, I also saw an annoying note about py.test instead of pytest so I went through an removed all out-of-date references.

I'm not 100% sure about whether there shouldn't be references to this elsewhere...

fixes #10303

saimn · 2020-05-26T22:08:38Z

@pllim - tox works in conda, what error do you get ?

pllim · 2020-05-28T21:31:23Z

Huh... I try the tox stuff in my conda env again and things work. The only difference is I have recently blew that old env away and created a new one. conda can be such a mystery sometimes. Anyway, I'll hide my comment above. Sorry for the noise!

pllim

Nice clean-up. Thanks!

pllim

Need to know how to deactivate tox env.

Cadair · 2020-05-29T11:00:56Z

+development and then use direct ``pytest`` commands within that environment.
+For instance::
+
+    tox -e test-alldeps --develop --notest


Is this really something that we should be recommending? It seems like a very weird way to spin up a virtualenv and people might end up here in all manner of weird ways and not really understand what they are doing, (i.e. virtualenv inside conda etc).

The justification from the sentence below is:

Now, one can run any of the pytest commands below to only run tests that are relevant to the development one is doing.

You can already customise the pytest behaviour by passing extra args to the tox command, and you can skip the (slow) sdist creation and build with the --develop flag. I really think this should at least come with a large health warning on the top. I see the utility if you really have to debug a failure in one specific tox env and you really want to get in there with a python interpreter, but I really struggle to see the justification for doing this to run anything but the most spectacularly complex pytest commands.

@Cadair - this came from real life problems from experienced developers (@mhvk and myself) having trouble just being able to run tests in an agile manner. What we need is a way to be able to have a one-time setup that reproduces accurately the travis test environment and then be able to re-run tests (over and over again, sometimes) in a few seconds, ala:

pytest astropy/time --pdb -k test_darn_its_still_failing

In discussions referenced in #10303, this was the best we came up with. If you have a better suggestion, please do tell!! That would be great. You can see already below that I also am not quite happy with the conda/virtualenv mixture that is likely coming out of this.

And note there are still outstanding problems with pytest as far as I can tell. If I just blindly install pytest, that gives me 5.4.0 and the dreaded error in test collection:

E pytest.PytestDeprecationWarning: direct construction of DocTestModulePlus has been deprecated, please use DocTestModulePlus.from_parent

Maybe I'm doing something wrong, but if I am having problems that means others may be in the same boat.

The point being that pinning certain packages (which happens in the tox setup) can be important.

@taldcroft - what version of pytest-doctestplus are you using? The issues should be fixed with v0.7.0

I was using 0.5. On conda (astropy / conda-forge) the latest available seems to be 0.6.1.

@Cadair - yes, if there is a good way just with tox, that would be preferable! The one advantage of the present suggestion is that it shows, indirectly, a bit how tox actually works.

The suggested workaround on the tox issue is to put the build deps from pyproject.toml into the deps= list in tox.ini which does seem to work, although isn't an ideal solution imo.

There is a tox rewrite underway, I just asked upstream if the bug will be properly addressed in the new version 🤞🏽

@Cadair, @astrojuanlu - this PR got stuck because my recommendation for fast testing with tox --develop --no-test and then enabling the virtual environment was clearly not ideal, but on the other hand there didn't seem to be a good alternative. I see that @Cadair's tox issue tox-dev/tox#1595 has been closed because this is fixed in "the rewrite branch", though I think that implies tox 4, which is not out yet. It certainly still fails with ~current tox.

Since it now has been more than a year with no other effective solution, my sense is to get this PR in and revisit when there is a better solution. But I'm also happy to just let it auto-close - after all, my own work flow is now fine...

My feeling is that tox 4 is "almost there", but that can still mean a few months. This is a net improvement on the current docs, so I'm 👍🏽 on getting it in.

taldcroft · 2020-05-29T11:05:56Z

+
+Now, one can run any of the ``pytest`` commands below to only run tests that
+are relevant to the development one is doing.  If the development includes
+documentation, one could install the documentation dependencies in the same


There is one catch, if I have things right, namely the graphviz dependency which must come either via conda or via a separate system-wide install. So for this pip-centric workflow you cannot imply all the docs dependencies are part of the "environment".

Now on Mac, doing a system install of graphviz requires either homebrew or macports. I don't use either of those. So then for me that means making a base environment with conda and then a tox-generated virtualenv under that. This leads me to have a command line prompt like (base) (test-alldeps) which is a bit unseemly to me.

So despite having agreed the tox-virtualenv might be a good way forward, I have stopped using it in favor of just making a conda environment that has the right deps using #10343.

I have removed the documentation part, as this could just make things harder than easier.

taldcroft · 2020-05-29T13:15:57Z

+---------------------------------------------
+
+As noted, running ``tox`` can be quite slow, yet it has the advantage over the
+other methods that it puts together a consistent set of dependencies.  One can


In general use "you" not "one".
https://docs.astropy.org/en/stable/development/style-guide.html#a-note-about-style-and-tone

mhvk · 2020-10-23T19:37:09Z

I updated this small PR to describe how to do development within a tox environment, since my sense would be that it is better to have something reasonable rather than nothing. But fine also to close or postpone.

bsipocz · 2020-10-26T23:52:46Z

Could a tox enthusiast say a final say on this? maybe @Cadair?

pllim · 2021-04-28T20:43:34Z

Yes, would be nice if @Cadair or @saimn can have a final look at this, since it is all about tox.

saimn · 2021-04-28T20:55:36Z

tox4 should fix that (tox-dev/tox#1945) so maybe we wait a little more ?

github-actions · 2021-09-04T05:37:07Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 10 hours. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

pllim · 2021-09-04T17:06:54Z

this pull request hasn't had any new commits for approximately 10 hours.

LoL... looks like bot still a little buggy. I'll investigate. 😬

Update: Hopefully I fixed the bug. Sorry about that!

github-actions · 2022-02-02T05:34:12Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

astrojuanlu · 2022-02-02T06:49:18Z

tox4 will release a beta soonish tox-dev/tox#2298 I'm already using the latest alpha for all my projects and it does solve a number of issues. Maybe this can hold off a couple if months?

pllim · 2022-02-02T14:07:15Z

Sure, I removed the Close? label but it doesn't really reset the clock unless @mhvk rebase.

mhvk · 2022-02-02T15:04:56Z

Yes, absolutely fine to wait a bit longer - would be nice to have a more elegant solution!

github-actions · 2022-02-04T05:34:14Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 5 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

github-actions · 2022-03-07T05:34:25Z

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

astrojuanlu · 2022-03-07T05:52:39Z

FYI: https://github.com/tox-dev/tox/releases/tag/4.0.0b1

taldcroft · 2022-03-29T20:46:12Z

I needed this documentation today, too bad it is only available in a closed PR...

Use case - tricky test failure that only shows up for numpy 1.18 (pydev-test-oldestdeps), so I want to use the VS code debugger an editable install to see what's going on. Any advice @Cadair on the "right" way to do this efficiently?

taldcroft · 2022-03-29T20:57:45Z

You can already customise the pytest behaviour by passing extra args to the tox command, and you can skip the (slow) sdist creation and build with the --develop flag

Am I doing something wrong?

(astropy) ➜  astropy git:(table-aggregate-mixins) ✗ tox -e pydev-test-oldestdeps --develop -- -k mixins_unsupport

Traceback (most recent call last):
  File "setup.py", line 66, in <module>
    from extension_helpers import get_extensions  # noqa
ModuleNotFoundError: No module named 'extension_helpers'
ERROR: invocation failed (exit code 1)
_________________________________________________ summary _________________________________________________
ERROR:   pydev-test-oldestdeps: InvocationError for command /Users/aldcroft/git/astropy/.tox/pydev-test-oldestdeps/bin/python setup.py --name (exited with code 1)

pllim · 2022-03-29T21:06:00Z

The bot will mark this as stale again unless you rebase, FYI.

Cadair · 2022-03-30T10:06:32Z

Am I doing something wrong?

No, this is the bug that caused this PR to stall out. Tox doesn't support doing this with build deps. tox-dev/tox#1595

I am happy to put the sequence of commands in this diff in the documentation, but I think the surrounding justification is misleading. We should be recommending people do this is the case where they really need an environment controlled by tox to debug an issue with CI etc, and link to the tox bug / explain why you can't do --develop with tox.

On a related note, the infrastructure team should probably start porting our plugins to tox 4...

taldcroft · 2022-03-30T10:57:39Z

@Cadair - ah, thanks for the explanation. So maybe following the transition to tox4 we can revisit this...

Cadair · 2022-12-07T21:24:30Z

Tox 4 has been released today (and broke all our CI), when we have adapted our config for tox 4 etc we should be able to revisit this.

mhvk added testing Docs labels May 25, 2020

mhvk added this to the v4.2 milestone May 25, 2020

mhvk requested review from astrofrog and taldcroft May 25, 2020 01:05

mhvk mentioned this pull request May 25, 2020

Request: way to have development testing set up with pinned test dependencies #10303

Closed

mhvk added the no-changelog-entry-needed label May 25, 2020

astrofrog requested a review from Cadair May 26, 2020 14:33

This comment has been minimized.

Sign in to view

pllim approved these changes May 28, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

pllim reviewed May 28, 2020

View reviewed changes

Comment thread docs/development/testguide.rst Resolved

pllim requested changes May 28, 2020

View reviewed changes

Cadair reviewed May 29, 2020

View reviewed changes

taldcroft reviewed May 29, 2020

View reviewed changes

mhvk force-pushed the docs-dev-test-with-tox branch from 07f5dd9 to cf422d6 Compare October 23, 2020 19:34

bsipocz requested a review from Cadair October 26, 2020 23:52

bsipocz modified the milestones: v4.2, v4.2.1 Oct 27, 2020

mhvk mentioned this pull request Jan 6, 2021

Documentation for trying the development version of Astropy ignores virtual environments #11207

Open

pllim modified the milestones: v4.2.1, v4.2.2 Apr 8, 2021

mhvk force-pushed the docs-dev-test-with-tox branch from cf422d6 to a7ce964 Compare September 3, 2021 19:22

pllim removed the Close? Tell stale bot that this issue/PR is stale label Sep 4, 2021

astrofrog removed this from the v5.0 milestone Nov 1, 2021

github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Feb 1, 2022

pllim removed the Close? Tell stale bot that this issue/PR is stale label Feb 2, 2022

github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Feb 3, 2022

github-actions Bot added the closed-by-bot Closed by stale bot label Mar 7, 2022

github-actions Bot closed this Mar 7, 2022

taldcroft reopened this Mar 29, 2022

pllim removed Close? Tell stale bot that this issue/PR is stale closed-by-bot Closed by stale bot labels Mar 29, 2022

github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Mar 30, 2022

taldcroft closed this Mar 30, 2022

mhvk deleted the docs-dev-test-with-tox branch August 26, 2023 10:12

mhvk mentioned this pull request Jan 12, 2024

DOC: describe how to create a test environment with tox #15878

Merged

1 task

Conversation

mhvk commented May 25, 2020

Uh oh!

This comment has been minimized.

saimn commented May 26, 2020

Uh oh!

This comment has been minimized.

pllim commented May 28, 2020

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taldcroft May 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk commented Oct 23, 2020

Uh oh!

bsipocz commented Oct 26, 2020

Uh oh!

pllim commented Apr 28, 2021

Uh oh!

saimn commented Apr 28, 2021

Uh oh!

github-actions Bot commented Sep 4, 2021

Uh oh!

pllim commented Sep 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Feb 2, 2022

Uh oh!

astrojuanlu commented Feb 2, 2022

Uh oh!

pllim commented Feb 2, 2022

Uh oh!

mhvk commented Feb 2, 2022

Uh oh!

github-actions Bot commented Feb 4, 2022

Uh oh!

github-actions Bot commented Mar 7, 2022

Uh oh!

astrojuanlu commented Mar 7, 2022

Uh oh!

taldcroft commented Mar 29, 2022

Uh oh!

taldcroft commented Mar 29, 2022

Uh oh!

pllim commented Mar 29, 2022

Uh oh!

Cadair commented Mar 30, 2022

Uh oh!

taldcroft May 29, 2020 •

edited

Loading

pllim commented Sep 4, 2021 •

edited

Loading