Skip to content

Comments

fix(env): ensure all system site paths are used#9942

Merged
abn merged 3 commits intopython-poetry:mainfrom
abn:fix-env-site-detection
Jan 6, 2025
Merged

fix(env): ensure all system site paths are used#9942
abn merged 3 commits intopython-poetry:mainfrom
abn:fix-env-site-detection

Conversation

@abn
Copy link
Member

@abn abn commented Jan 5, 2025 •
edited by sourcery-ai bot
Loading

With this change, when using newer Python versions (>=3.12), Poetry
correctly detects and considers read-only system site packages when
an environment is loaded.

Resolves: #9878

An example of this scenario is when the OS distributions manages your Python installations, you can end up with the following.

>>> json.dumps(sysconfig.get_paths())
{"stdlib": "/usr/lib64/python3.13", "platstdlib": "/usr/local/lib64/python3.13", "purelib": "/usr/local/lib/python3.13/site-packages", "platlib": "/usr/local/lib64/python3.13/site-packages", "include": "/usr/include/python3.13", "platinclude": "/usr/include/python3.13", "scripts": "/usr/local/bin", "data": "/usr/local"}
>>> 
>>> site.getsitepackages()
['/usr/local/lib64/python3.13/site-packages', '/usr/local/lib/python3.13/site-packages', '/usr/lib64/python3.13/site-packages', '/usr/lib/python3.13/site-packages']

You can see that both platlib and purelib uses the /usr/local prefix. While the system installation site uses the /usr prefix.

Summary by Sourcery

Include system site packages when loading a Python environment.

Bug Fixes:

  • Fixed an issue where read-only system site packages were not considered when loading an environment in Python 3.12 or later.

Enhancements:

  • Improved site package detection to include system site packages that use a different prefix than purelib and platlib.

@abn abn requested a review from a team January 5, 2025 16:36
@abn abn marked this pull request as draft January 5, 2025 17:01
@abn abn marked this pull request as ready for review January 5, 2025 17:01
@abn abn force-pushed the fix-env-site-detection branch from bd1aab7 to abcc889 Compare January 5, 2025 17:54
@abn abn enabled auto-merge (rebase) January 5, 2025 18:48
@abn
Copy link
Member Author

abn commented Jan 5, 2025

@sourcery-ai review

@sourcery-ai
Copy link

sourcery-ai bot commented Jan 5, 2025

Reviewer's Guide by Sourcery

This pull request fixes a bug in Poetry where read-only system site packages were not being considered when an environment is loaded in newer Python versions (>=3.12). This is achieved by adding a fallback mechanism to include system site packages when constructing the environment's site packages.

Sequence diagram for environment path resolution

sequenceDiagram
    participant Poetry
    participant BaseEnv
    participant Python
    Poetry->>BaseEnv: Initialize environment
    BaseEnv->>Python: Get paths (GET_PATHS script)
    Python-->>BaseEnv: Return paths including system site packages
    BaseEnv->>BaseEnv: Initialize site_packages
    Note over BaseEnv: Now includes fallback paths
    BaseEnv-->>Poetry: Return initialized environment
Loading

Class diagram showing the updated BaseEnv class structure

classDiagram
    class BaseEnv {
        -_purelib: Path
        -_platlib: Path
        -_fallbacks: list[Path]
        -_site_packages: SitePackages
        +site_packages: SitePackages
        +purelib: Path
        +platlib: Path
        +fallbacks: list[Path]
        -_get_lib_dirs(): list[Path]
    }
    class SitePackages {
        +purelib: Path
        +platlib: Path
        +fallbacks: list[Path]
    }
    BaseEnv --> SitePackages: creates
    note for BaseEnv "Added fallbacks property and
modified _get_lib_dirs to include
fallback paths"
Loading

File-Level Changes

Change Details Files
Added fallback mechanism for system site packages
  • Added a fallbacks property to the BaseEnv class to store fallback paths for site packages.
  • Modified the site_packages property to include the fallback paths.
  • Updated the GET_PATHS script to include fallback paths from site.getsitepackages().
  • Modified the GenericEnv class to use the updated GET_PATHS script.
  • Updated tests to reflect the changes in site package handling.
  • Removed unused code related to the old site package handling.
 src/poetry/utils/env/base_env.py
src/poetry/utils/env/script_strings.py
src/poetry/utils/env/generic_env.py
tests/utils/env/test_env.py
src/poetry/utils/env/__init__.py
Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time. You can also use
    this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

sourcery-ai[bot]
sourcery-ai bot reviewed Jan 5, 2025
View reviewed changes
Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @abn - I've reviewed your changes and they look great!

Here's what I looked at during the review
  • 🟡 General issues: 2 issues found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Complexity: all looks good
  • 🟢 Documentation: all looks good
Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@abn abn force-pushed the fix-env-site-detection branch 2 times, most recently from 72efd57 to dc4c68c Compare January 6, 2025 12:00
Secrus
Secrus reviewed Jan 6, 2025
View reviewed changes
abn added 3 commits January 6, 2025 15:31
@abn
fix(env): unify path scripts for environments
b4dba3e
@abn
fix(env): ensure all system site paths are used
5ad79f4 
With this change, when using newer Python versions (>=3.12), Poetry
correctly detects and considers read-only system site packages when
an environment is loaded.

Resolves: python-poetry#9878
@abn
chore(env): remove unused skip_write_checks flag
3dd85d5
@abn abn force-pushed the fix-env-site-detection branch from dc4c68c to 3dd85d5 Compare January 6, 2025 14:32
@abn abn requested a review from Secrus January 6, 2025 14:32
@abn abn merged commit 5d8f880 into python-poetry:main Jan 6, 2025
72 checks passed
@abn abn deleted the fix-env-site-detection branch January 6, 2025 16:09
@radoering radoering mentioned this pull request Jan 11, 2025
Merged
@github-actions
Copy link

github-actions bot commented Feb 6, 2025

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 6, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@sourcery-ai sourcery-ai[bot] sourcery-ai[bot] left review comments

@Secrus Secrus Secrus approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test_env_system_packages_are_relative_to_lib fails on some systems

2 participants

@abn @Secrus