Build backend: Support namespace packages#13833
Conversation
9cfef8e to
a3e8113
Compare
|
I sort of like the idea of allowing people to enumerate all the expected members if |
8fc2cc4 to
8b1721f
Compare
|
From checking projects that use namespaces, no other build backend requires enumerating your modules, they allow pointing to the root and include whatever directory you've given, and I don't want to make the migration harder. I'm fine doing this more lenient approach for namespace packages particularly since you already opt-in to less coherence by using a namespace. I haven't done exhaustive testing, but with the validation we're performing for |
BurntSushi
left a comment
There was a problem hiding this comment.
LGTM. I have some concerns mostly around the UX here. Specifically, I think the docs could be beefed up. And I'm unsure of the implications of coupling "declaration of this package as a namespace package" and "coherence checks are disabled."
My initial reaction matched @zanieb's here, but I'm swayed by your follow-up in terms of being more lenient (to make migration less painful and since this is what other build backends already do).
| value_type = "bool", | ||
| example = r#"namespace = true"# | ||
| )] | ||
| pub namespace: bool, |
There was a problem hiding this comment.
So, this makes sense to me:
I'm fine doing this more lenient approach for namespace packages particularly since you already opt-in to less coherence by using a namespace.
But one thing I'm a little iffy on here is coupling the notion of "this is a namespace package" and "we will disable most coherence checks." What if someone doesn't want to disable those coherent checks and list out the modules explicitly? I'm not necessarily asking for that to be available in this PR, but rather, is there a plausible path to that?
Maybe we'd keep this setup as-is, and you'd need to opt into a coherent-checks = false option to list out the modules? Or if a list is provided, then the coherent checks are automatically re-enabled?
So I think maybe this is all fine, but I just wanted to poke about this to check if this path was considered.
There was a problem hiding this comment.
I've improved the framing. The coherence checks I'm mentioning here are really just two things: Validating that the module-name matches our expectations for either either stubs or regular package, and checking that __init__.py[i] are absent above and present inside the root module. Both of those aren't super elaborate, but also something that other build backends usually don't do. With namespace = true, we turn those two checks off, so you can have arbitrary namespace structures. Setuptools, hatchling and poetry all support these kind of packages by default by having include declarations, we make this opt-in with namespace = true, and support a smaller, better checked subset suitable for many cases without it instead.
docs/concepts/build-backend.md
Outdated
| ``` | ||
|
|
||
| Complex namespaces with more than one root module can be built by setting the `namespace` option. | ||
| Note that this option disables most coherence check, but it is still required to have a valid |
There was a problem hiding this comment.
Is it feasible to give an example of something that will be accepted here that normally wouldn't be? Reading this with my beginner eyes on (as if I didn't just read this PR), I become immediately curious what kinds of checks have been disabled on me. :-) Should I be concerned? What if I want those checks enabled?
There was a problem hiding this comment.
I've put a comparison into the reference docs. I don't want to go deep on namespace = true in the concept docs since I don't want to encourage complex namespaces.
2c24cd4 to
44b4e5e
Compare
Unlike regular packages, specifying all `__init__.py` directories for a namespace package would be very verbose There is e.g. https://github.com/python-poetry/poetry/tree/main/src/poetry, which has 18 modules, or https://github.com/googleapis/api-common-protos which is inconsistently nested. For both the Google Cloud SDK, there are both packages with a single module and those with complex structures, with many having multiple modules due to versioning through `<module>_v1` versioning. The Azure SDK seems to use one module per package (it's not explicitly documented but seems to follow from the process in https://azure.github.io/azure-sdk/python_design.html#azure-sdk-distribution-packages and https://github.com/Azure/azure-sdk-for-python/blob/ccb0e03a3de748f3aabf44be94776ba37e55791f/doc/dev/packaging.md). For simplicity with complex projects, we add a `namespace = true` switch which disabled checking for an `__init__.py`. We only check that there's no `<module_root>/<module_name>/__init__.py` and otherwise add the whole `<module_root>/<module_name>` folder. This comes at the cost of `namespace = true` effectively creating an opt-out from our usual checks that allows creating an almost entirely arbitrary package. For simple projects with only a single module, the module name can be dotted to point to the target module, so the build still gets checked: ```toml [tool.uv.build-backend] module-name = "poetry.core" ``` ## Alternatives ### Declare all packages We could make `module-name` a list and allow or require declaring all packages: ```toml [tool.uv.build-backend] module-name = ["cloud_sdk.service.storage", "cloud_sdk.service.storage_v1", "cloud_sdk.billing.storage"] ``` Or for Poetry: ```toml [tool.uv.build-backend] module-name = [ "poetry.config", "poetry.console", "poetry.inspection", "poetry.installation", "poetry.json", "poetry.layouts", "poetry.masonry", "poetry.mixology", "poetry.packages", "poetry.plugins", "poetry.publishing", "poetry.puzzle", "poetry.pyproject", "poetry.repositories", "poetry.toml", "poetry.utils", "poetry.vcs", "poetry.version" ] ``` ### Support multiple namespaces We could also allow namespace packages with multiple root level module: ```toml [tool.uv.build-backend] module-name = ["cloud_sdk.my_ext", "local_sdk.my_ext"] ``` For lack of use cases, we delegate this to creating a workspace with one package per module. ## Implementation Due to the more complex options for the module name, I'm moving verification on deserialization later, dropping the source span we'd get from serde. We also don't show similarly named directories anymore.
Co-authored-by: Andrew Gallant <[email protected]>
c1752c6 to
41eb34f
Compare
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.7.7` -> `0.7.13` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.7.13`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0713) [Compare Source](astral-sh/uv@0.7.12...0.7.13) ##### Python - Add Python 3.14.0b2 - Add Python 3.13.5 - Fix stability of `uuid.getnode` on 3.13 See the [`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250612) for more details. ##### Enhancements - Download versions in `uv python pin` if not found ([#​13946](astral-sh/uv#13946)) - Use TTY detection to determine if SIGINT forwarding is enabled ([#​13925](astral-sh/uv#13925)) - Avoid fetching an exact, cached Git commit, even if it isn't locked ([#​13748](astral-sh/uv#13748)) - Add `zstd` and `deflate` to `Accept-Encoding` ([#​13982](astral-sh/uv#13982)) - Build binaries for riscv64 ([#​12688](astral-sh/uv#12688)) ##### Bug fixes - Check if relative URL is valid directory before treating as index ([#​13917](astral-sh/uv#13917)) - Ignore Python discovery errors during `uv python pin` ([#​13944](astral-sh/uv#13944)) - Do not allow `uv add --group ... --script` ([#​13997](astral-sh/uv#13997)) ##### Preview changes - Build backend: Support namespace packages ([#​13833](astral-sh/uv#13833)) ##### Documentation - Add 3.14 to the supported platform reference ([#​13990](astral-sh/uv#13990)) - Add an `llms.txt` to uv ([#​13929](astral-sh/uv#13929)) - Add supported macOS version to the platform reference ([#​13993](astral-sh/uv#13993)) - Update platform support reference to include Python implementation list ([#​13991](astral-sh/uv#13991)) - Update pytorch.md ([#​13899](astral-sh/uv#13899)) - Update the CLI help and reference to include references to the Python bin directory ([#​13978](astral-sh/uv#13978)) ### [`v0.7.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0712) [Compare Source](astral-sh/uv@0.7.11...0.7.12) ##### Enhancements - Add `uv python pin --rm` to remove `.python-version` pins ([#​13860](astral-sh/uv#13860)) - Don't hint at versions removed by `excluded-newer` ([#​13884](astral-sh/uv#13884)) - Add hint to use `tool.uv.environments` on resolution error ([#​13455](astral-sh/uv#13455)) - Add hint to use `tool.uv.required-environments` on resolution error ([#​13575](astral-sh/uv#13575)) - Improve `python pin` error messages ([#​13862](astral-sh/uv#13862)) ##### Bug fixes - Lock environments during `uv sync`, `uv add` and `uv remove` to prevent race conditions ([#​13869](astral-sh/uv#13869)) - Add `--no-editable` to `uv export` for `pylock.toml` ([#​13852](astral-sh/uv#13852)) ##### Documentation - List `.gitignore` in project init files ([#​13855](astral-sh/uv#13855)) - Move the pip interface documentation into the concepts section ([#​13841](astral-sh/uv#13841)) - Remove the configuration section in favor of concepts / reference ([#​13842](astral-sh/uv#13842)) - Update Git and GitHub Actions docs to mention `gh auth login` ([#​13850](astral-sh/uv#13850)) ##### Preview - Fix directory glob traversal fallback preventing exclusion of all files ([#​13882](astral-sh/uv#13882)) ### [`v0.7.11`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0711) [Compare Source](astral-sh/uv@0.7.10...0.7.11) ##### Python - Add Python 3.14.0b1 - Add Python 3.13.4 - Add Python 3.12.11 - Add Python 3.11.13 - Add Python 3.10.18 - Add Python 3.9.23 ##### Enhancements - Add Pyodide support ([#​12731](astral-sh/uv#12731)) - Better error message for version specifier with missing operator ([#​13803](astral-sh/uv#13803)) ##### Bug fixes - Downgrade `reqwest` and `hyper-util` to resolve connection reset errors over IPv6 ([#​13835](astral-sh/uv#13835)) - Prefer `uv`'s binary's version when checking if it's up to date ([#​13840](astral-sh/uv#13840)) ##### Documentation - Use "terminal driver" instead of "shell" in `SIGINT` docs ([#​13787](astral-sh/uv#13787)) ### [`v0.7.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0710) [Compare Source](astral-sh/uv@0.7.9...0.7.10) ##### Enhancements - Add `--show-extras` to `uv tool list` ([#​13783](astral-sh/uv#13783)) - Add dynamically generated sysconfig replacement mappings ([#​13441](astral-sh/uv#13441)) - Add data locations to install wheel logs ([#​13797](astral-sh/uv#13797)) ##### Bug fixes - Avoid redaction of placeholder `git` username when using SSH authentication ([#​13799](astral-sh/uv#13799)) - Propagate credentials to files on devpi indexes ending in `/+simple` ([#​13743](astral-sh/uv#13743)) - Restore retention of credentials for direct URLs in `uv export` ([#​13809](astral-sh/uv#13809)) ### [`v0.7.9`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#079) [Compare Source](astral-sh/uv@0.7.8...0.7.9) ##### Python The changes reverted in [0.7.8](#​078) have been restored. See the [`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250529) for more details. ##### Enhancements - Improve obfuscation of credentials in URLs ([#​13560](astral-sh/uv#13560)) - Allow running non-default Python implementations via `uvx` ([#​13583](astral-sh/uv#13583)) - Add `uvw` as alias for `uv` without console window on Windows ([#​11786](astral-sh/uv#11786)) - Allow discovery of x86-64 managed Python builds on macOS ([#​13722](astral-sh/uv#13722)) - Differentiate between implicit vs explicit architecture requests ([#​13723](astral-sh/uv#13723)) - Implement ordering for Python architectures to prefer native installations ([#​13709](astral-sh/uv#13709)) - Only show the first match per platform (and architecture) by default in `uv python list` ([#​13721](astral-sh/uv#13721)) - Write the path of the parent environment to an `extends-environment` key in the `pyvenv.cfg` file of an ephemeral environment ([#​13598](astral-sh/uv#13598)) - Improve the error message when libc cannot be found, e.g., when using the distroless containers ([#​13549](astral-sh/uv#13549)) ##### Performance - Avoid rendering info log level ([#​13642](astral-sh/uv#13642)) - Improve performance of `uv-python` crate's manylinux submodule ([#​11131](astral-sh/uv#11131)) - Optimize `Version` display ([#​13643](astral-sh/uv#13643)) - Reduce number of reference-checks for `uv cache clean` ([#​13669](astral-sh/uv#13669)) ##### Bug fixes - Avoid reinstalling dependency group members with `--all-packages` ([#​13678](astral-sh/uv#13678)) - Don't fail direct URL hash checking with dependency metadata ([#​13736](astral-sh/uv#13736)) - Exit early on `self update` if global `--offline` is set ([#​13663](astral-sh/uv#13663)) - Fix cases where the uv lock is incorrectly marked as out of date ([#​13635](astral-sh/uv#13635)) - Include pre-release versions in `uv python install --reinstall` ([#​13645](astral-sh/uv#13645)) - Set `LC_ALL=C` for git when checking git worktree ([#​13637](astral-sh/uv#13637)) - Avoid rejecting Windows paths for remote Python download JSON targets ([#​13625](astral-sh/uv#13625)) ##### Preview - Add `uv add --bounds` to configure version constraints ([#​12946](astral-sh/uv#12946)) ##### Documentation - Add documentation about Python versions to Tools concept page ([#​7673](astral-sh/uv#7673)) - Add example of enabling Dependabot ([#​13692](astral-sh/uv#13692)) - Fix `exclude-newer` date format for persistent configuration files ([#​13706](astral-sh/uv#13706)) - Quote versions variables in GitLab documentation ([#​13679](astral-sh/uv#13679)) - Update Dependabot support status ([#​13690](astral-sh/uv#13690)) - Explicitly specify to add a new repo entry to the repos list item in the `.pre-commit-config.yaml` ([#​10243](astral-sh/uv#10243)) - Add integration with marimo guide ([#​13691](astral-sh/uv#13691)) - Add pronunciation to README ([#​5336](astral-sh/uv#5336)) ### [`v0.7.8`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#078) [Compare Source](astral-sh/uv@0.7.7...0.7.8) ##### Python We are reverting most of our Python changes from `uv 0.7.6` and `uv 0.7.7` due to a miscompilation that makes the Python interpreter behave incorrectly, resulting in spurious type-errors involving str. This issue seems to be isolated to x86\_64 Linux, and affected at least Python 3.12, 3.13, and 3.14. The following changes that were introduced in those versions of uv are temporarily being reverted while we test and deploy a proper fix for the miscompilation: - Add Python 3.14 on musl - free-threaded Python on musl - Add Python 3.14.0a7 - Statically link `libpython` into the interpreter on Linux for a significant performance boost See [the issue for details](astral-sh/uv#13610). ##### Documentation - Remove misleading line in pin documentation ([#​13611](astral-sh/uv#13611)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MC4yNi4xIiwidXBkYXRlZEluVmVyIjoiNDAuNTEuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiUmVub3ZhdGUgQm90Il19-->
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/astral-sh/uv](https://github.com/astral-sh/uv) | final | patch | `0.7.3` -> `0.7.13` | --- ### Release Notes <details> <summary>astral-sh/uv (ghcr.io/astral-sh/uv)</summary> ### [`v0.7.13`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0713) [Compare Source](astral-sh/uv@0.7.12...0.7.13) ##### Python - Add Python 3.14.0b2 - Add Python 3.13.5 - Fix stability of `uuid.getnode` on 3.13 See the [`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250612) for more details. ##### Enhancements - Download versions in `uv python pin` if not found ([#​13946](astral-sh/uv#13946)) - Use TTY detection to determine if SIGINT forwarding is enabled ([#​13925](astral-sh/uv#13925)) - Avoid fetching an exact, cached Git commit, even if it isn't locked ([#​13748](astral-sh/uv#13748)) - Add `zstd` and `deflate` to `Accept-Encoding` ([#​13982](astral-sh/uv#13982)) - Build binaries for riscv64 ([#​12688](astral-sh/uv#12688)) ##### Bug fixes - Check if relative URL is valid directory before treating as index ([#​13917](astral-sh/uv#13917)) - Ignore Python discovery errors during `uv python pin` ([#​13944](astral-sh/uv#13944)) - Do not allow `uv add --group ... --script` ([#​13997](astral-sh/uv#13997)) ##### Preview changes - Build backend: Support namespace packages ([#​13833](astral-sh/uv#13833)) ##### Documentation - Add 3.14 to the supported platform reference ([#​13990](astral-sh/uv#13990)) - Add an `llms.txt` to uv ([#​13929](astral-sh/uv#13929)) - Add supported macOS version to the platform reference ([#​13993](astral-sh/uv#13993)) - Update platform support reference to include Python implementation list ([#​13991](astral-sh/uv#13991)) - Update pytorch.md ([#​13899](astral-sh/uv#13899)) - Update the CLI help and reference to include references to the Python bin directory ([#​13978](astral-sh/uv#13978)) ### [`v0.7.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0712) [Compare Source](astral-sh/uv@0.7.11...0.7.12) ##### Enhancements - Add `uv python pin --rm` to remove `.python-version` pins ([#​13860](astral-sh/uv#13860)) - Don't hint at versions removed by `excluded-newer` ([#​13884](astral-sh/uv#13884)) - Add hint to use `tool.uv.environments` on resolution error ([#...
Unlike regular packages, specifying all
__init__.pydirectories for a namespace package would be very verbose There is e.g. https://github.com/python-poetry/poetry/tree/main/src/poetry, which has 18 modules, or https://github.com/googleapis/api-common-protos which is inconsistently nested. For both the Google Cloud SDK, there are both packages with a single module and those with complex structures, with many having multiple modules due to versioning through<module>_v1versioning. The Azure SDK seems to use one module per package (it's not explicitly documented but seems to follow from the process in https://azure.github.io/azure-sdk/python_design.html#azure-sdk-distribution-packages and https://github.com/Azure/azure-sdk-for-python/blob/ccb0e03a3de748f3aabf44be94776ba37e55791f/doc/dev/packaging.md).For simplicity with complex projects, we add a
namespace = trueswitch which disabled checking for an__init__.py. We only check that there's no<module_root>/<module_name>/__init__.pyand otherwise add the whole<module_root>/<module_name>folder. This comes at the cost ofnamespace = trueeffectively creating an opt-out from our usual checks that allows creating an almost entirely arbitrary package.For simple projects with only a single module, the module name can be dotted to point to the target module, so the build still gets checked:
Alternatives
Declare all packages
We could make
module-namea list and allow or require declaring all packages:Or for Poetry:
Support multiple namespaces
We could also allow namespace packages with multiple root level module:
For lack of use cases, we delegate this to creating a workspace with one package per module.
Implementation
Due to the more complex options for the module name, I'm moving verification on deserialization later, dropping the source span we'd get from serde. We also don't show similarly named directories anymore.