Skip to content

UnicodeDecodeError when installing scipy #13154

Closed

Description

@Max-Ole

Hi all,

I'm on Windows 11 and I have set up a fresh Python 3.11 virtual environment and used pip to install ultralytics. This in turn installs scipy, which results in the UnicodeDecodeError error:

File "<frozen codecs>", line 322, in decode
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 54: invalid continuation byte

This seems to happen together with a failure of metadata generation with a file called pyproject.toml. Note that my working directory path contains non-ascii characters, if that is of any use.

To Reproduce:

I created a new venv:
pyenv local 3.11.0b4
python3 -m venv myPyEnv
I activated the venv with a .bat script containing:
cmd /k "cd /d myPyEnv\Scripts && activate.bat && cd ..\.."
Then I ran:
pip install ultralytics

Terminal output:

C:\Users\maxol\MEINS\STUDIUM\FÄCHER\Master Arbeit\Master_Thesis_Literature\Sandbox>cmd /k "cd /d myPyEnv\Scripts && activate.bat && cd ..\.."

(myPyEnv) C:\Users\maxol\MEINS\STUDIUM\FÄCHER\Master Arbeit\Master_Thesis_Literature\Sandbox>pip list
Package    Version
---------- -------
pip        24.0
setuptools 58.1.0

(myPyEnv) C:\Users\maxol\MEINS\STUDIUM\FÄCHER\Master Arbeit\Master_Thesis_Literature\Sandbox>pip install ultralytics
Collecting ultralytics
  Downloading ultralytics-8.2.4-py3-none-any.whl.metadata (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.6/40.6 kB 2.0 MB/s eta 0:00:00
Collecting matplotlib>=3.3.0 (from ultralytics)
  Downloading matplotlib-3.8.4.tar.gz (35.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.9/35.9 MB 8.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting opencv-python>=4.6.0 (from ultralytics)
  Downloading opencv_python-4.9.0.80-cp37-abi3-win32.whl.metadata (20 kB)
Collecting pillow>=7.1.2 (from ultralytics)
  Downloading pillow-10.3.0-cp311-cp311-win32.whl.metadata (9.4 kB)
Collecting pyyaml>=5.3.1 (from ultralytics)
  Downloading PyYAML-6.0.1-cp311-cp311-win32.whl.metadata (2.1 kB)
Collecting requests>=2.23.0 (from ultralytics)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting scipy>=1.4.1 (from ultralytics)
  Downloading scipy-1.13.0.tar.gz (57.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.2/57.2 MB 6.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 2
  ╰─> [25 lines of output]
      + meson setup C:\Users\maxol\AppData\Local\Temp\pip-install-7cn8o__9\scipy_e3b2e9eac7c64e81b00f14057e4aaa67 C:\Users\maxol\AppData\Local\Temp\pip-install-7cn8o__9\scipy_e3b2e9eac7c64e81b00f14057e4aaa67\.mesonpy-jv6pmawh -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\maxol\AppData\Local\Temp\pip-install-7cn8o__9\scipy_e3b2e9eac7c64e81b00f14057e4aaa67\.mesonpy-jv6pmawh\meson-python-native-file.ini
      Traceback (most recent call last):
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\mesonmain.py", line 186, in run
          return options.run_func(options)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\msetup.py", line 363, in run
          app.generate()
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\msetup.py", line 180, in generate
          env = environment.Environment(self.source_dir, self.build_dir, self.options)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\environment.py", line 591, in __init__
          config = coredata.parse_machine_files(self.coredata.config_files, self.source_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\coredata.py", line 1164, in parse_machine_files
          parser = MachineFileParser(filenames, sourcedir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\maxol\AppData\Local\Temp\pip-build-env-hs_2kreh\overlay\Lib\site-packages\mesonbuild\coredata.py", line 1099, in __init__
          content = f.read()
                    ^^^^^^^^
        File "<frozen codecs>", line 322, in decode
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 54: invalid continuation byte

      ERROR: Unhandled python exception

          This is a Meson bug and should be reported!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

(myPyEnv) C:\Users\maxol\MEINS\STUDIUM\FÄCHER\Master Arbeit\Master_Thesis_Literature\Sandbox>

Activity

  1. Max-Ole commented on Apr 28, 2024

    @Max-Ole
    Author

    Update:

    I was able to install scipy now by running the steps below. I had success on a different computer where I didn't use special characters and hadn't tried the installation before. With these steps it then also worked on the original computer:

    1. Created a new venv in a directory without special characters in the path.
    2. Upgraded pip.
    3. Installed scipy without using the cache (--no-cache-dir)

    Note that this time there was no mention of "Preparing metadata". This might not have fixed but just skipped the meson steps.

    Terminal output:

    C:\Users\maxol\Desktop\MasterThese_venv>pyenv versions
      3.11.0b4-win32
    
    C:\Users\maxol\Desktop\MasterThese_venv>pyenv local 3.11.0b4-win32
    
    C:\Users\maxol\Desktop\MasterThese_venv>pyenv local
    3.11.0b4-win32
    
    C:\Users\maxol\Desktop\MasterThese_venv>python -m venv thesis_venv
    
    C:\Users\maxol\Desktop\MasterThese_venv>thesis_venv\Scripts\activate
    
    (thesis_venv) C:\Users\maxol\Desktop\MasterThese_venv>pip list
    Package    Version
    ---------- -------
    pip        23.2.1
    setuptools 65.5.0
    
    [notice] A new release of pip is available: 23.2.1 -> 24.0
    [notice] To update, run: python.exe -m pip install --upgrade pip
    
    (thesis_venv) C:\Users\maxol\Desktop\MasterThese_venv>python.exe -m pip install --upgrade pip
    Requirement already satisfied: pip in c:\users\maxol\desktop\masterthese_venv\thesis_venv\lib\site-packages (23.2.1)
    Collecting pip
      Obtaining dependency information for pip from https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl.metadata
      Using cached pip-24.0-py3-none-any.whl.metadata (3.6 kB)
    Using cached pip-24.0-py3-none-any.whl (2.1 MB)
    Installing collected packages: pip
      Attempting uninstall: pip
        Found existing installation: pip 23.2.1
        Uninstalling pip-23.2.1:
          Successfully uninstalled pip-23.2.1
    Successfully installed pip-24.0
    
    (thesis_venv) C:\Users\maxol\Desktop\MasterThese_venv>pip install scipy --no-cache-dir
    Collecting scipy
      Downloading scipy-1.13.0-cp311-cp311-win_amd64.whl.metadata (60 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.6/60.6 kB 645.4 kB/s eta 0:00:00
    Collecting numpy<2.3,>=1.22.4 (from scipy)
      Downloading numpy-1.26.4-cp311-cp311-win_amd64.whl.metadata (61 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB ? eta 0:00:00
    Downloading scipy-1.13.0-cp311-cp311-win_amd64.whl (46.2 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.2/46.2 MB 7.0 MB/s eta 0:00:00
    Downloading numpy-1.26.4-cp311-cp311-win_amd64.whl (15.8 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.8/15.8 MB 7.0 MB/s eta 0:00:00
    Installing collected packages: numpy, scipy
    Successfully installed numpy-1.26.4 scipy-1.13.0
    
    (thesis_venv) C:\Users\maxol\Desktop\MasterThese_venv>
    
  2. added a commit that references this issue on Apr 28, 2024
    aa42f4c
  3. added a commit that references this issue on Apr 28, 2024
    e5f32b7
  4. dnicolodi commented on Apr 28, 2024

    @dnicolodi
    Member

    I find it odd that your first installation attempt resulted in pip trying to build scipy from source, but not the second.

    Anyhow, there is a bug in meson-python where we write the native file with the wrong encoding. Fix waiting in mesonbuild/meson-python#623

  5. Max-Ole commented on Apr 28, 2024

    @Max-Ole
    Author

    I noticed that even in a fresh venv, pip cache list shows wheels that I used elsewhere on the computer. Running it now I don't see anything related to scipy in the cache, but just maybe the --no-cache-dir option made the change.

    Anyway, it's working and thanks for the fix!

  6. added a commit that references this issue on Jun 4, 2024
    d2a0850
  7. added a commit that references this issue on May 1, 2025
    c7b909c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions