Error: cannot import name 'soft_unicode' from 'markupsafe' (v4.3)

Description

Using pip install astropy==4.3 (or having astropy==4.3 in the requirements.txt file for a package dependent on astropy) errors when "Getting requirements to build wheel".

Expected behavior

Successful installation of astropy v4.3

Actual behavior

Installation fails.

The error thrown is

ImportError: cannot import name 'soft_unicode' from 'markupsafe'

The root cause is described here.

Steps to Reproduce

pip install astropy==4.3

(Note 4.2.1) suffers same affliction, as do others, presumably.
5.0.4 installs fine, of course.

System Details

import platform; print(platform.platform())
Windows-10-10.0.22000-SP0
import sys; print("Python", sys.version)
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
import numpy; print("Numpy", numpy.version)
Numpy 1.22.3
import erfa; print("pyerfa", erfa.version)
pyerfa 2.0.0.1
import astropy; print("astropy", astropy.version)
astropy 5.0.4
import scipy; print("Scipy", scipy.version)
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'scipy'
import matplotlib; print("Matplotlib", matplotlib.version)
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'matplotlib'

@embray

Welcome to Astropy 👋 and thank you for your first issue!

A project member will respond to you as soon as possible; in the meantime, please double-check the guidelines for submitting issues and make sure you've provided the requested details.

GitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the Astropy Discourse user forum and close this issue.

If you feel that this issue has not been responded to in a timely manner, please leave a comment mentioning our software support engineer @embray, or send a message directly to the development mailing list. If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail [email protected].

This has been fixed in 5.x series. There is currently no plan to ship another 4.x. Is there a reason why you cannot upgrade astropy?

Is there a reason why you cannot upgrade astropy?
Well, that is a much larger discussion. Upgrading astropy (and adjusting cascading and overlapping/conflicting dependencies,) will indeed address the symptom and solve my own immediate problem (but not the root cause, nor the problem that will be encountered by anyone else deploying or building apps with pinned dependencies on astropy<5).

For example, the app I'm installing, from a third party, runs on Python 3.7.4, which is seemingly not supported by the current astropy, per python_requires = >=3.8 in your setup.cfg. It also has a pinned dependency on pyerfa==1.7.3, which conflicts with astropy>=5's requirement of pyerfa>=2.0.

This instance is a case study in the whole paradigm of conflicting dependencies, virtual environments, and pinning dependencies to ensure reliable deployment.

I assumed, based on your maintenance of the tagging and availability of prior versions that you would want to know that those prior versions won't build out-of-the-box. You could certainly argue that the bug belongs to jinja2, for not pinning the markupsafe dependency in their build, but they aren't going to fix it, and the failure cascades up to everyone who depends on them, making them all unreliable as a consequence.

The fix is to simply pin markupsafe in the requires collection in pyproject.toml to a version prior to markupsafe's breaking change, exactly as you already have done in version 5, in this patch. I.e., add the markupsafe version spec as shown below to the pyproject.toml file in versions 4.x

[build-system]
requires = ["setuptools",
            "setuptools_scm>=6.2",
            "wheel",
            "cython==0.29.22",
            "markupsafe==2.0.1",  # <-------- add this to pin markupsafe to < deprecation of soft_unicode
            "jinja2==2.10.3",
            "oldest-supported-numpy",
            "extension-helpers"]
build-backend = 'setuptools.build_meta'
[...]

It breaks nothing and restores your prior version to a buildable and working state, thereby unbreaking whatever depends on it that got broken in February by markupsafe v2.1.

(As an aside, I think you don't need to pin markupsafe to the old version in astropy>=5 because you also set the jinja2 version to a newer one that can use the newer markupsafe in that same patch referenced above. In other words, that patch performed two actions, either of which would work alone. But I digress.)

I certainly sympathize with your situation. Our release cycles are described in APE 2 and v5.0.x is currently the LTS, which means v4.x is no longer supported. Also, our CPython support is described in APE 18.

However, given that Python 3.7 is not reaching EOL until mid 2023, I can see why people are still sticking with Python 3.7. I will ping the @astropy/astropy-project-release-team here since ultimately they are the ones who decide whether to release another v4.x with the proposed patch. Thanks for your patience!

This has some tricky consequences. I think that anything that depends on Astropy currently cannot build wheels from source for Python 3.7.

This is fine if there's an existing binary Astropy wheel... But Astropy back in v4 only made x86_64 and i686 Linux wheels (see https://pypi.org/project/astropy/4.3.1/#files). There's no aarch64 Linux wheels available, so when on an ARM platform one must build from source.

If you run Docker on Apple Silicon (the newish M1 and M2 chips), then you are running on an ARM platform. So, for me, I'm just trying to get an ARM wheel for Healpy, which has an Astropy dependency, and is trying to maintain Python 3.7 support. But that's not possible right now - Healpy (and, I suppose, Astropy) can work on Apple Silicon, or Python 3.7, but not both.

@spenczar

@spenczar what about only supporting healpy on Python 3.7 for platforms for which there are astropy wheels? I very much doubt there will be many e.g. Apple Silicon Python 3.7 users in practice, right? (is there even an Apple Silicon Python 3.7 installer?)

@astrofrog

@astrofrog The issue isn't really about natively working on native Apple Silicon directly, for me. I'm trying to collaborate with others who are working on older code. They can use an old Ubuntu Docker container image, and install Python 3.7 in it, and Astropy will work happily for them because binary wheels exist for manylinux x86. But I can't even do that; Docker still needs to obey the instruction set of the host.

So I'm not looking for Apple Silicon support, exactly; I'm looking for manylinux_aarch64 support.

There's an escape hatch, which is that I can emulate x86, but it runs at a very slow speed.

Do the Debian builds help here? It covers almost every architecture and astropy binaries have been provided in Debian since about the beginning.

Note that I don't think beyond our current LTS support, we can do much about this from the astropy side (unless new people are willing to make the considerable effort).

@mhvk

@mhvk Debian doesn't provide astropy 4.3.X: Bullseys is 4.2; upcoming Bookworm 5.2.1.

I have basically the same problem, as in one of our environments the Python version is fixed to 3.7 (despite EOL this year). Even a requirements.txt

MarkupSafe ==2.0.1
jinja2     ==2.10.3
astropy    ==4.3.1

doesn't work, as jinja2 pulls the most actual version of markupsafe.

I have pinged @astropy/astropy-project-release-team to see if they are willing to do a special one-off 4.3.x release just to fix this problem. 🤞

Shouldn't using pip without build-system isolation be the fix here? Instead of astropy making a release for a no-longer supported version for a problem in jinja2.

So this should work:

# install astropy build deps into env first:
$ pip install setuptools setuptools_scm wheel cython==0.29.22 jinja2==2.10.3 numpy markupsafe==2.0.1 extension-helpers
$ pip install --no-build-isolation astropy==4.3.1

I just tested the above on an M1 mac-mini and it works.

e5b-admin@e5b-kaffeetante ~ % pyenv install 3.7.14
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.14.tar.xz...
-> https://www.python.org/ftp/python/3.7.14/Python-3.7.14.tar.xz
Installing Python-3.7.14...
patching file Doc/library/ctypes.rst
patching file Lib/test/test_unicode.py
patching file Modules/_ctypes/_ctypes.c
patching file Modules/_ctypes/callproc.c
patching file Modules/_ctypes/ctypes.h
patching file setup.py
patching file 'Misc/NEWS.d/next/Core and Builtins/2020-06-30-04-44-29.bpo-41100.PJwA6F.rst'
patching file Modules/_decimal/libmpdec/mpdecimal.h
e5b-admin@e5b-kaffeetante ~ % pyenv shell 3.7.14
e5b-admin@e5b-kaffeetante ~ % python --version
Python 3.7.14
e5b-admin@e5b-kaffeetante ~ % python -m venv test_astropy
e5b-admin@e5b-kaffeetante ~ % source test_astropy/bin/activate

(test_astropy) e5b-admin@e5b-kaffeetante ~ % pip install setuptools setuptools_scm wheel cython==0.29.22 jinja2==2.10.3 numpy markupsafe==2.0.1 extension-helpers
Requirement already satisfied: setuptools in ./test_astropy/lib/python3.7/site-packages (47.1.0)
Collecting setuptools_scm
  Using cached setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Collecting cython==0.29.22
  Using cached Cython-0.29.22-py2.py3-none-any.whl (980 kB)
Collecting jinja2==2.10.3
  Using cached Jinja2-2.10.3-py2.py3-none-any.whl (125 kB)
Collecting numpy
  Downloading numpy-1.21.6.zip (10.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.3/10.3 MB 28.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting markupsafe==2.0.1
  Downloading MarkupSafe-2.0.1.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Collecting extension-helpers
  Downloading extension_helpers-1.0.0-py3-none-any.whl (20 kB)
Collecting importlib-metadata
  Downloading importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting tomli>=1.0.0
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting packaging>=20.0
  Downloading packaging-23.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 KB 7.6 MB/s eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting zipp>=0.5
  Downloading zipp-3.11.0-py3-none-any.whl (6.6 kB)
Using legacy 'setup.py install' for markupsafe, since package 'wheel' is not installed.
Building wheels for collected packages: numpy
  Building wheel for numpy (pyproject.toml) ... done
  Created wheel for numpy: filename=numpy-1.21.6-cp37-cp37m-macosx_12_0_arm64.whl size=4534276 sha256=0c499f8f959ee1eeb89ac7e2bbd5f23dbf84ebedda97e5ea0a3615e8b935924b
  Stored in directory: /Users/e5b-admin/Library/Caches/pip/wheels/4e/7e/9e/0fde042ccff2493994076dac9c3fbd78feb444c3bd94eb386a
Successfully built numpy
Installing collected packages: zipp, wheel, typing-extensions, tomli, packaging, numpy, markupsafe, extension-helpers, cython, jinja2, importlib-metadata, setuptools_scm
  Running setup.py install for markupsafe ... done
Successfully installed cython-0.29.22 extension-helpers-1.0.0 importlib-metadata-6.0.0 jinja2-2.10.3 markupsafe-2.0.1 numpy-1.21.6 packaging-23.0 setuptools_scm-7.1.0 tomli-2.0.1 typing-extensions-4.4.0 wheel-0.38.4 zipp-3.11.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/Users/e5b-admin/test_astropy/bin/python -m pip install --upgrade pip' command.


(test_astropy) e5b-admin@e5b-kaffeetante ~ % pip install --no-build-isolation astropy==4.3.1
Collecting astropy==4.3.1
  Downloading astropy-4.3.1.tar.gz (7.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 36.4 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: importlib-metadata in ./test_astropy/lib/python3.7/site-packages (from astropy==4.3.1) (6.0.0)
Collecting pyerfa>=1.7.3
  Downloading pyerfa-2.0.0.1.tar.gz (808 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 808.5/808.5 KB 20.6 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.17 in ./test_astropy/lib/python3.7/site-packages (from astropy==4.3.1) (1.21.6)
Requirement already satisfied: typing-extensions>=3.6.4 in ./test_astropy/lib/python3.7/site-packages (from importlib-metadata->astropy==4.3.1) (4.4.0)
Requirement already satisfied: zipp>=0.5 in ./test_astropy/lib/python3.7/site-packages (from importlib-metadata->astropy==4.3.1) (3.11.0)
Building wheels for collected packages: astropy, pyerfa
  Building wheel for astropy (pyproject.toml) ... done
  Created wheel for astropy: filename=astropy-4.3.1-cp37-cp37m-macosx_12_0_arm64.whl size=6488853 sha256=86f585c114c421e82f690e099d1c24d04a6153520cc53346a11dbb8e03eec8e5
  Stored in directory: /Users/e5b-admin/Library/Caches/pip/wheels/a5/12/55/66d114465eb0bba7a670cb4bc7ea57216a58bd1d97c4681f72
  Building wheel for pyerfa (pyproject.toml) ... done
  Created wheel for pyerfa: filename=pyerfa-2.0.0.1-cp37-cp37m-macosx_12_0_arm64.whl size=336958 sha256=ac69207c9d85439067056caa5f34a7526fdcbdc63a49ab9a79d35182ec7f970c
  Stored in directory: /Users/e5b-admin/Library/Caches/pip/wheels/b4/86/e6/b8f25d23ad84b512982075450f5a69caa3c9453aa155bde6e1
Successfully built astropy pyerfa
Installing collected packages: pyerfa, astropy
Successfully installed astropy-4.3.1 pyerfa-2.0.0.1
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/Users/e5b-admin/test_astropy/bin/python -m pip install --upgrade pip' command.

@maxnoe

@maxnoe The difference is that for Mac there is a matching wheel, while the ones for Linux/x86_64 are not used here. I don't know about the name convention for "manylinux" wheels: When I try to install astropy 4.2.0 (on Debian Buster; Python 3.7), it installs

astropy-4.2-cp37-cp37m-manylinux1_x86_64.whl

$ pip3 install "astropy==4.2.0"
Collecting astropy==4.2.0
  Downloading https://files.pythonhosted.org/packages/dc/16/7724205e25a8fdd2bc5ccb79c9240947f5cba9dc07e58c30e6df85fd82c0/astropy-4.2-cp37-cp37m-manylinux1_x86_64.whl (9.7MB)

However, for astropy 4.3.1, the name looks different

astropy-4.3.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

And it seems that this one is not what is wanted on a Debian Buster system -- the source is downloaded and tried to be built locally, and when I try to use the file name as parameter, I get

$ pip3 install -v astropy-4.3.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Created temporary directory: /tmp/pip-ephem-wheel-cache-qrqvqyg5
Created temporary directory: /tmp/pip-req-tracker-h8igpuz1
Created requirements tracker '/tmp/pip-req-tracker-h8igpuz1'
Created temporary directory: /tmp/pip-install-36q1sjig
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-h8igpuz1'
astropy-4.3.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 313, in run
    self.name, wheel_cache
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 220, in populate_requirement_set
    requirement_set.add_requirement(req_to_add)
  File "/usr/lib/python3/dist-packages/pip/_internal/req/req_set.py", line 77, in add_requirement
    wheel.filename
pip._internal.exceptions.InstallationError: astropy-4.3.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl is not a supported wheel on this platform.

So, maybe (if there is no consent about a 4.3.2 bugfix version), a pragmatic workaround could be to build a proper wheel (name) for Python 3.7, x86_64, manylinux, and to upload to to pypi? This would not resolve the problem, but hide it for the majority of Linux users.

However, I have no idea why this originally happened on Windows; there the wheel name is
astropy-4.3.1-cp37-cp37m-win_amd64.whl
Is this file used when doing pip install astropy==4.3.1 on a Windows system? Can someone with more knowledge about wheels comment?

The difference is that for Mac there is a matching wheel,

No, there is no wheel for M1 and the build I showed is from source, see the log above: (It even build numpy from source!)

Collecting astropy==4.3.1
  Downloading astropy-4.3.1.tar.gz (7.6 MB)  # this is a source dist, not a wheel!!

...


Building wheels for collected packages: astropy, pyerfa
Building wheel for astropy (pyproject.toml) ... done
Created wheel for astropy: filename=astropy-4.3.1-cp37-cp37m-macosx_12_0_arm64.whl size=6488853 sha256=86f585c114c421e82f690e099d1c24d04a6153520cc53346a11dbb8e03eec8e5

@maxnoe

@maxnoe Oops, you are right; I read from the bottom and stopped with "wheel for astropy" 😄. However, also there is the question why the wheel is not used?

It seems the default pip version on debian buster (18.x) is too old to recognize the tags in that wheel. Updating pip solves it and installs the wheel on buster.

However, the workaround above (--no-build-isolation) makes the build work also with debian buster's package manager version of pip.

So you have two options that work without a new astropy release / wheel build

Update pip to make the wheel installable
Use --no-build-isolation with the package manager provided version of pip

$ docker run --rm -it debian:buster bash 
# apt update
# apt install python3 python3-pip python3-venv
# python3 -m venv venv
# source venv/bin/activate
# pip --version
pip 18.1 from /venv/lib/python3.7/site-packages/pip (python 3.7)
# pip install astropy==4.3.1  # tries to download and build source, fails due to this issue
# pip install -U pip
# pip --version
pip 22.3.1 from /venv/lib/python3.7/site-packages/pip (python 3.7)
# pip install astropy==4.3.1 # downloads wheel 
Collecting astropy==4.3.1
  Downloading astropy-4.3.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (10.7 MB)

I checked, and the workaround above for mac also works on debian with the old pip, but the preferred version would be to use the current pip which then installs the wheel (also for numpy!)

I can confirm that first updating pip solves my issue here; so for me the problem is resolved. Thank you for the help.

Does this mean we can close this issue since user has a workaround as mentioned in #13255 (comment) ?

I'd think so. It always can be re-opened if someone again stumbles upon it.

I agree reluctantly that turning off build isolation works. It makes the build process more complicated for projects which have a transitive dependency upon astropy 4 (through no fault of their own), but it seems that build complexity is not covered under LTS policy, so this is maybe the best I can hope for.

I think the main conclusion I get from this is that we should not add dependencies to astropy unless they are really essential. In the end, they just cause trouble. And that includes build dependencies.

Does this mean we can close this one as "won't fix"? 😉

Make scipy a required dependency #12448

Navigation Menu

Repository navigation

Error: cannot import name 'soft_unicode' from 'markupsafe' (v4.3) #13255

Description

Expected behavior

Actual behavior

Steps to Reproduce

System Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error: cannot import name 'soft_unicode' from 'markupsafe' (v4.3) #13255

Description

Description

Expected behavior

Actual behavior

Steps to Reproduce

System Details

Activity

github-actions commented on May 12, 2022

pllim commented on May 12, 2022

jimbo1969 commented on May 13, 2022

pllim commented on May 14, 2022

spenczar commented on Jan 11, 2023

astrofrog commented on Jan 11, 2023

spenczar commented on Jan 12, 2023

mhvk commented on Jan 19, 2023

olebole commented on Jan 26, 2023

olebole commented on Jan 26, 2023

pllim commented on Jan 26, 2023

maxnoe commented on Jan 26, 2023

maxnoe commented on Jan 26, 2023

maxnoe commented on Jan 26, 2023

olebole commented on Jan 30, 2023

maxnoe commented on Jan 30, 2023

olebole commented on Jan 30, 2023

maxnoe commented on Jan 30, 2023

olebole commented on Jan 30, 2023

pllim commented on Jan 30, 2023

olebole commented on Jan 30, 2023

spenczar commented on Jan 30, 2023

mhvk commented on Jan 30, 2023

pllim commented on Jan 30, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions