Update virtualenv + OSX + conda guide

@efiring

I was working on a pull request (in version 1.5.1+1707.g506df7b.dirty) on OSX (10.9.5) with Python 2.7 in a Conda environment. Using the current virtualenv FAQ (http://matplotlib.org/devdocs/faq/virtualenv_faq.html, specifically the frameworkpython script solution), I was not able to load matplotlib.

In the discussion of the pull request, @efiring suggested a procedure (#6358 (comment)) that did work after adding one final step. Inside the Conda enviroment you want to develop matplotlib in, the procedure is as follows (mostly copied from @efiring):

Use conda to make a test environment with the desired python version, and use conda to install matplotlib in it.
Use conda to uninstall matplotlib.
In your github clone of matplotlib, make a feature branch, and make the desired changes to it.
In the root of that clone directory run pip install ..
Install the Framework version of python with conda install python.app.
Open a new terminal and activate the conda environment (or run hash -r (bash) / rehash (zsh) in the old terminal to make sure your shell finds the conda version of pythonw instead of the OSX version).
Use pythonw instead of python from now on.

Can you put that text into a PR into the documentation?

Actually, two more steps are necessary:

Edit $CONDA_ENV_PATH/bin/activate; at the bottom of the file add:

export OLD_PYTHONPATH=$PYTHONPATH
export OLD_DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH
export PYTHONPATH=$CONDA_ENV_PATH/lib/python2.7/site-packages
export DYLD_FALLBACK_LIBRARY_PATH=$CONDA_ENV_PATH/lib

Edit $CONDA_ENV_PATH/bin/deactivate; at the top of the file add:

export PYTHONPATH=$OLD_PYTHONPATH
export DYLD_FALLBACK_LIBRARY_PATH=$OLD_DYLD_FALLBACK_LIBRARY_PATH
unset OLD_PYTHONPATH
unset OLD_DYLD_FALLBACK_LIBRARY_PATH

I'll write it in the docs some time soon.

Conda supports putting activation scrips in a activate.d and deactivate.d directory http://conda.pydata.org/docs/using/envs.html#saved-environment-variables I suggest doing that instead of modifying the activate and deactivate directly

Sounds good, will do.

You should not be setting PYTHONPATH that way; there is no need to put site-packages on it because that location is compiled in. If you need to disable your PYTHONPATH that is your own special case, not a general condition for using conda. The same applies to DYLD_FALLBACK_LIBRARY_PATH. I'm sure it doesn't need to be set as you have it for the conda environment, and normally it would not even exist. If it is in your environment then at most you need to unset it when entering the conda environment.

It's not in my regular enviroment. I didn't come up with this myself, this solution is advertized in README.osx:

Note that your environment is somewhat important. Some conda users have
found that, to run the tests, their PYTHONPATH must include
/path/to/anaconda/.../site-packages and their DYLD_FALLBACK_LIBRARY_PATH
must include /path/to/anaconda/lib.

When I first tried your suggestion (steps 1-4 above), I had these variables set and it worked. I then thought that perhaps I could now turn them off again, but when I did, first the freetype lib could no longer be found (which was fixed by setting local_freetype to True in setup.cfg) and when that was fixed libpng started complaining:

/Users/patrick/anaconda/envs/mpl_dev/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in <module>()
     17 
     18 import matplotlib
---> 19 from matplotlib.backends import _macosx
     20 
     21 from .backend_agg import RendererAgg, FigureCanvasAgg

ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev/lib/python2.7/site-packages/matplotlib/backends/_macosx.so, 2): Symbol not found: __cg_png_create_info_struct
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Users/patrick/anaconda/envs/mpl_dev/lib/libpng16.16.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO

I hope we can get this straight and understand what is going on here. I am still pretty sure that having a PYTHONPATH set should not be needed, and I doubt that the DYLD variable is really needed in general either. It would be good to get this sorted out in README.osx as well.
Have you tried my original instructions (that is, with a fresh conda environment) with neither PYTHONPATH nor DYLD_FALLBACK_LIBRARY_PATH set? My guess is that the problem you encountered was a result of having started with those variables set. Otherwise I don't understand why you ran into the ImportError, and I don't. I do have local_freetype set to True in setup.cfg; this is necessary to make the image comparisons with text pass.

Ok, I did the following (in a terminal without DYLD vars, but with a PYTHONPATH with only paths to some custom modules, no critical stuff):

git clone [email protected]:matplotlib/matplotlib.git matplotlib_2
conda create --name mpl_dev2 python
source activate mpl_dev2
conda install matplotlib
conda uninstall matplotlib
conda install python.app
cd matplotlib_2
rehash
pip install .

Then in python:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 17, in <module>
    import matplotlib.font_manager as font_manager
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/font_manager.py", line 58, in <module>
    from matplotlib import ft2font
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so, 2): Library not loaded: libfreetype.6.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so
  Reason: image not found

In pythonw:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 17, in <module>
    import matplotlib.font_manager as font_manager
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/font_manager.py", line 58, in <module>
    from matplotlib import ft2font
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so, 2): Library not loaded: libfreetype.6.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so
  Reason: image not found

Second try, in the same env (in the matplotlib_2 dir):

make clean
pip uninstall matplotlib
sed -e 's/#local_freetype = False/local_freetype = True/' setup.cfg.template > setup.cfg
pip install .

Then in python:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 20, in <module>
    from matplotlib.mathtext import MathTextParser
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/mathtext.py", line 62, in <module>
    import matplotlib._png as _png
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/_png.so, 2): Library not loaded: libpng16.16.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/_png.so
  Reason: image not found

And in pythonw the same.

Third try: without my custom PYTHONPATH, new conda env, inside the matplotlib_2 dir

make clean
unset PYTHONPATH
conda create --name mpl_dev3 python
source activate mpl_dev3
conda install matplotlib
conda uninstall matplotlib
conda install python.app
rehash
pip install .  # still with local_freetype = True set in setup.cfg

Then python(w) says, again:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/textpath.py", line 20, in <module>
    from matplotlib.mathtext import MathTextParser
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/mathtext.py", line 62, in <module>
    import matplotlib._png as _png
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so, 2): Library not loaded: libpng16.16.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so
  Reason: image not found

Finally, inside mpl_dev3:

export DYLD_FALLBACK_LIBRARY_PATH=/Users/patrick/anaconda/envs/mpl_dev3/lib
pythonw

>>> import matplotlib.pyplot as plt
>>>

So, indeed, PYTHONPATH does not seem necessary, but DYLD_FALLBACK_LIBRARY_PATH, I can't seem to go without...

@efiring

@efiring Is it possible that you have some custom built libs that your install finds, e.g. from Macports / Homebrew? I have Macports installed and I see now that I also have libpng there. I tried upgrading libpng, but to no avail.

Related: #3416.

$ otool -L /Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so 
/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so:
    libpng16.16.dylib (compatibility version 34.0.0, current version 34.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 2577.0.0)
$ pkg-config libpng --libs
-L/opt/local/lib -lpng16

Seems like it is indeed building against the Macports install, i.e. if it uses pkg-config to find libpng...

As mentioned in #11006 (comment), things seem to have gotten a bit simpler on my most recent try (other machine by now, so clean environment compared to the one used for the above discussion of this issue).

For reference, here's what I need now to get it correctly installed from scratch (inside a clean matplotlib clone):

conda create -n mpl_dev matplotlib clang clangxx libpng freetype pkg-config python.app
conda activate mpl_dev
conda uninstall matplotlib
export CC=clang
export CXX=clang++
python -mpip install -e .

And then run with pythonw to use the macosx backend (or switch to Agg or TkAgg in matplotlibrc).

Having this week made my first matplotlib branch, I must admit I found the developers' installation instructions slightly terrifying! I would have been a lot more comfortable with something conda based, as that's what I'm used to.

I've no idea if this would be realistic for matplotlib, but in iris we have a yml file that defines its dependencies and everything you could possibly need during testing (sample data, sphinx, flake8, black, etc.) This makes creating a conda development environment ridiculously easy.

There's a draft for a conda yml at #17096.

Still needs clarification on the handling of some optional dependencies (maybe even only check if they are used in the tests). I'm happy to get feedback if the yml works.

The Conda yml file is now on main and is nicely documented here:
https://matplotlib.org/stable/devel/development_setup.html#create-a-dedicated-environment

So I wonder if this issue could be closed.

Yes, I think this can be closed. Thanks for the notification!

Navigation Menu

Repository navigation

Update virtualenv + OSX + conda guide #6418

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Update virtualenv + OSX + conda guide #6418

Description

Activity

tacaswell commented on May 13, 2016

egpbos commented on May 15, 2016

jenshnielsen commented on May 15, 2016

egpbos commented on May 15, 2016

efiring commented on May 15, 2016

egpbos commented on May 15, 2016

efiring commented on May 16, 2016

egpbos commented on May 16, 2016

egpbos commented on May 16, 2016

egpbos commented on May 16, 2016

egpbos commented on May 16, 2016

egpbos commented on May 16, 2016

egpbos commented on May 16, 2016

7 remaining items

egpbos commented on Apr 9, 2018

rcomer commented on Jan 1, 2021

timhoffm commented on Jan 1, 2021

rcomer commented on Oct 22, 2022

timhoffm commented on Oct 22, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions