Background & Rationale

This request is inspired by RPM Package Manger’s capability to build subpackages from the same Spec File.

Here, I want to propose and discuss replication a version of this capability can be replicated within poetry to allow for simplified user experience for a python project maintainer, especially when either maintaining namespace packages and/or multi-project source trees. While strict project separation is a good thing in most cases, it might not always be the more pragmatic scenario for package maintainers.

For our purposes here, we can refer to each of theses packages as a subproject. And all subprojects are managed under a single poetry project. This means that there is only a single pyproject.toml file and a shared project root directory with either a shared source tree or independent source trees (subdirectory) for each subproject.

Description

Let us consider the scenario of multiple namespace packages being maintained in a single repository with the following structure.

    namespace-project/
    └── src
        └── namespace
            └── package
                ├── one
                │   └── __init__.py
                ├── three
                │   └── __init__.py
                └── two
                    └── __init__.py

Note that this will still apply even if different source directories exists within the root directory for each subproject.

Here the intention could be that we want to distribute 3 packages, namely, namespace-package-one , namespace-package-two and namespace-package-three.

For the purpose of this example, let us assume that namespace-package-three depends on namespace-package-one. The pyproject.toml file could look something like this.

New sections are annotated with comments detailing them and expected behaviour.

[build-system]
requires = ["poetry-core>=1.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "namespace-package"
version = "1.0.0-alpha.0"
description = ""
authors = [
    "Bender Rodriguez <[email protected]>"
]
license = "MIT"
readme = "README.md"
repository = "https://git.planetexpress.com/bender/python-namespace-package"
keywords = []
classifiers = [
    "Intended Audience :: Developers",
    "Operating System :: OS Independent",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.8",
]

# this section remains as is, but now specifies shared dependencies
[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
pre-commit = "^2.1"
flake8 = "^3.7"
black = "^19.10b0"
pytest = "^5.2"

# the following are package specific section
[tool.poetry.packages.one]
name = "namespace-package-one"  # this is optional as name would be derrived from <project.name>-<package name from section>
description = ""  # this will overide the description from the project for this package
readme = "README.one.md"  # this will overide the readme from the project for this package
packages = [  # this is mandatory for sub-packages
    # any package not included in a sub-package is added to the base package (ie. "namespace-package")
    # if the "packages" property is not explicitly configured in the base
    { include = "namespace.package.one", from = "src" }
]

[tool.poetry.packages.one.dependencies]
ujson = "^1.35"

[tool.poetry.packages.one.dev-dependencies]
pytest-mock = "^2.0"

[tool.poetry.packages.two]
packages = [ 
    { include = "namespace.package.two", from = "src" }
]

[tool.poetry.packages.two.dependencies]
psycopg2 = "^2.8.4"

[tool.poetry.packages.two.dev-dependencies]
pytest-postgresql = "^2.3.0"

[tool.poetry.packages.three]
requires = [ # this enables us to specify the relationships between sub-packages
    "one" # this could also be namespace-package-one
]
packages = [ 
    { include = "namespace.package.two", from = "src" }
]

[tool.poetry.packages.three.dependencies]
aiohttp = "^3.5"

[tool.poetry.packages.three.dev-dependencies]
beautifulsoup4 = "^4.8"
aioresponses = "^0.6"
pytest-asyncio = "^0.10"

Under this scenario, the following might be what the cli commands look like. Current behaviour will remain unaltered as these are additive changes.

$ poetry add --package one <dependency>
.. <similar to current add output>

$ poetry packages list
namespace-package-one
namespace-package-two
namespace-package-three

$ poetry build
<builds all three packages>

$ poetry build --package one
<builds only namespace-package-one>

$ poetry publish --dry-run
...
Publishing namespace-package-one (1.0.0-alpha.0) to PyPI
  - Uploading namespace-package-one-1.0.0-alpha.0.tar.gz
  - Uploading namespace-package-one-1.0.0-alpha.0-py3-none-any.whl

Publishing namespace-package-two (1.0.0-alpha.0) to PyPI
  - Uploading namespace-package-two-1.0.0-alpha.0.tar.gz
  - Uploading namespace-package-two-1.0.0-alpha.0-py3-none-any.whl

Publishing namespace-package-three (1.0.0-alpha.0) to PyPI
  - Uploading namespace-package-three-1.0.0-alpha.0.tar.gz
  - Uploading namespace-package-three-1.0.0-alpha.0-py3-none-any.whl

Variations

The above is an initial though of how it might work. That said there are variations to this that should be discussed.

1. Does a per-package dev-dependnecy section make sense?
   This only really makes sense if we want to allow for developing a single package at a time. However, this will become tricky in cases like here where "three" depends on "one". This will mean that when developing "three", dev dependencies for "one" should also be installed. If isolation is required, then multiple virtual environments will be required, which might be overkill for majority use cases for this feature.

2. Will all packages be installed under PEP-0517?
   Is it even possible to install only specific package when being installed under PEP-0517? One possible solution might be to make use of "extras" here as a way of specifying which package if any to install, but default to all.

Extensions

1. Optional Project Package
   As an extension to this, one might also want to optionally distribute a a namespace only package namespace-package (let's call this the "project package" for now) that installs the core dependencies and also allow for "extras" as we do today without requiring the distribution of the entire source tree with the binary distribution.

This means that if someone does pip install namespace-package, the maintainer might expect the the following to be installed:

1. The namespace namepace.package.
2. Packages namespace-package-one and namespace-package-three, which are required for the "default" install.

An end-user can also install the remaining package, like so - pip install namespace-package[two] which simply will install a dependency namespace-package-two.

This behaviour might not be desired in all cases, and can be considered opt-in.