Skip to content

5596 simplify conda env#5597

Merged
wyli merged 13 commits intodevfrom
fixes-conda-tests
Nov 29, 2022
Merged

5596 simplify conda env#5597
wyli merged 13 commits intodevfrom
fixes-conda-tests

Conversation

@wyli
Copy link
Copy Markdown
Contributor

@wyli wyli commented Nov 28, 2022

Fixes #5596
Fixes #5088

Description

simplify the conda env

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

wyli added 4 commits November 27, 2022 11:05
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
@wyli wyli requested review from Nic-Ma, ericspod and rijobro November 28, 2022 11:22
@wyli wyli changed the title simplify conda env 5596 simplify conda env Nov 28, 2022
wyli added 2 commits November 28, 2022 11:51
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
@ericspod
Copy link
Copy Markdown
Member

ericspod commented Nov 28, 2022

I've noticed that mlflow is super slow at times to install using the existing environment, and einops and transformers are really slow as well. I hadn't figured out why but removing these made installing everything else reasonable. I had issues on Windows that the environment would also default to CPU Pytorch for some reason. On the Pytorch site the conda command now includes pytorch-cuda as a package with a specification for a version of CUDA so that possibly should be added here as well.

A bare-minimum environment that appears to work is:

name: monai
channels:
  - pytorch
  - defaults
  - nvidia
dependencies:
  - numpy>=1.17
  - pytorch>=1.6
  - pytorch-cuda=11.6

@ericspod
Copy link
Copy Markdown
Member

I also created a runtime version of the environment with every package for development removed, this was a little slimmer for just deploying MONAI somewhere, but its contents will change now with the new way Pytorch is doing its packages.

@wyli
Copy link
Copy Markdown
Contributor Author

wyli commented Nov 28, 2022

I just tested this works fine:

name: monai
channels:
  - pytorch
  - defaults
  - conda-forge
dependencies:
  - numpy>=1.17
  - pytorch>=1.8
  - pip
  - pip:
    - -r requirements-dev.txt

https://github.com/Project-MONAI/MONAI/actions/runs/3564985942

assuming pytorch is installed following the 'conda' option here https://pytorch.org/get-started/locally/#start-locally

Signed-off-by: Wenqi Li <[email protected]>
@ericspod
Copy link
Copy Markdown
Member

In the logs I see that the Windows build chose the cpuonly package by default. This is specified in conda.yml but I've seen this happening as a default thing when installing in Windows and I've always had to setup environments manually there.

@wyli
Copy link
Copy Markdown
Contributor Author

wyli commented Nov 28, 2022

In the logs I see that the Windows build chose the cpuonly package by default. This is specified in conda.yml but I've seen this happening as a default thing when installing in Windows and I've always had to setup environments manually there.

I think manually installing pytorch is the recommended way described here https://pytorch.org/get-started/locally/#start-locally let me try to include the cuda requirement for the ci..

wyli added 3 commits November 28, 2022 14:33
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
@wyli
Copy link
Copy Markdown
Contributor Author

wyli commented Nov 28, 2022

I've added the cuda dependency following your minimum environment example @ericspod, it works fine with ubuntu and windows: https://github.com/Project-MONAI/MONAI/actions/runs/3566365419/jobs/5992720823
but is not compatible with macos, so I drop the macOS test.

Apart from the downloading took some time, it looks good to me. Any further suggestions? @ericspod @rijobro. Dropping these pip package requirements in the yaml will make it faster,

  - pip:
    - -r requirements-dev.txt

I can do that, just wanted to get your opinion as it'll be very different from the current behaviour of conda update -f environment-dev.yml.

@ericspod
Copy link
Copy Markdown
Member

I've added the cuda dependency following your minimum environment example @ericspod, it works fine with ubuntu and windows: https://github.com/Project-MONAI/MONAI/actions/runs/3566365419/jobs/5992720823 but is not compatible with macos, so I drop the macOS test.

Apart from the downloading took some time, it looks good to me. Any further suggestions? @ericspod @rijobro. Dropping these pip package requirements in the yaml will make it faster,

I think this looks good but I want to test this at home with my home Windows box I know I've had issues with. I was wondering if we wanted to do multiple versions of the environment file to have everything, only packages needed for deploy, and a minimal package.

@ericspod
Copy link
Copy Markdown
Member

I've run with the environment file on my Windows box and it's installed things as I expected with the GPU version of Pytorch.

@wyli wyli force-pushed the fixes-conda-tests branch from 89babe7 to e33d38c Compare November 29, 2022 12:35
@wyli
Copy link
Copy Markdown
Contributor Author

wyli commented Nov 29, 2022

/build

@wyli wyli enabled auto-merge (squash) November 29, 2022 12:36
@wyli wyli merged commit d0db5fd into dev Nov 29, 2022
@wyli wyli deleted the fixes-conda-tests branch November 29, 2022 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

regular conda tests don't work Conda Environment Config Very Slow, Need Lightweight Versions

2 participants