Table of Contents

1. Feature request: poetry installing from private pypi (incl. CI/CD jobs)
   1. TL;DR
   2. Background
   3. Requirements
   4. Proposed solution
      1. Named repositories ready to serve as sources
      2. sources as list of source names to use
      3. package without source name installs from any index
      4. How to express value of sources
      5. Preference of sources definition
   5. Open issues
      1. Terms repository, (poetry) source, index and pypi
      2. How to prevent use of public pypi
      3. poetry search ignores currently sources
   6. Conclusions

Feature request: poetry installing from private pypi (incl. CI/CD jobs)

For serious project development, usage of one or more private pypi indexes and flexibility to use different ones during CI/CD jobs is essential.

This proposal describes possible poetry modifications to allow that.

TL;DR

Reuse existing option to define named repositories in poetry global config.

Instead of expressing sources directly in `pyproject.toml` (what lacks flexibility when indexes has to be modified), allow specification of (repository) names to be used as sources.

Introduce env. variable POETRY_SOURCES which would be space delimited list of repository names to be used as sources.

Background

• Poetry related:
  • Poetry allows definition of named repositories in global config (config.toml). Optionally username and password for basic authorization can be specified (stored in auth.toml).
  • Poetry allows specification of named sources in pyproject.toml. This can reuse basic authorization parameters from auth.toml file.
  • There is draft PR of @sdispater to allow package specification to express name of pypi to install from.
  • Poetry allows upload of the package to an index, this is what repositories are used for.
  • There is issue #625 proposing to use global repositories as fallback sources (with PR, which was rejected).
• CI/CD related:
  • the same python package might be uploaded to or read from different pypi indexes during different phases of testing.
    • during development, volatile indexes can be used (which allow replacing already existing package version with another having exactly the same version spec)
    • during staging, non-volatile indexes can be used, which would reject attempt to replace already existing package version by another instance.
  • devpi-server is good example of private pypi suitable to development. It supports multiple indexes with options to be volatile, non-volatile, use bases pointing to other indexes etc.
  • real url of pypi index to install from may differ during different phases of testing, integration and deployment.

Requirements

• For poetry sub-commands such as install, add and search allow usage of alternative pypi url(s) without need to modify pyproject.toml file.
• Changing url of any pypi index shall be easy using environment variable(s).

Regarding the last item: It is fine, if this happens using one or more poetry calls via CLI passing value of env. variable on command line.

Proposed solution

Named repositories ready to serve as sources

Use existing configuration of repositories by poetry which allows to define name, url and optionally username and password.

Anyway, named repository definition does not make it automatically available as a source unless specified explicitly (see below).

sources as list of source names to use

pyproject.toml, global config.toml, command line parameter --source or env. variable POETRY_SOURCES would allow to define list of source names to use.

The names would refer to repositories named in global config. Order of names would be significant.

package without source name installs from any index

All packages, which do not specify explicit source name, do attempt to install from any source.

Note, that at this moment package does not allow stating source name, but this might come in future.

If a package states the index name, it can install only from source with given name.

How to express value of sources

1. Global poetry config

   Global poetry config file config.toml would introduce

   [settings]
   sources = ["private", "gitlab/prod", "javl/dev", "pypi"]
   

2. Command line argument

   Commands poetry add, poetry install and poetry search would allow multiple --source options specifying source name.

3. pyproject.toml source

   As of today, following would provide complete source definition in pyproject.toml:

   [[tool.poetry.source]]
   name = 'private'
   url = 'http://example.com/simple'
   

   In case url defined in global config would be intended for real use, missing url or providing empty string would mean "use the url defined globally":

   [[tool.poetry.source]]
   name = 'private'
   

   or alternatively

   [[tool.poetry.source]]
   name = 'private'
   url = ''  # use globally defined url
   

Preference of sources definition

As multiple places defining sources can be in place, following order of preferences and processing rules shall apply:

In short:

• if pyproject.toml source does not specify url and there is no globally defined repository with given name, fail processing.
• if source name is used and it is not found in global config, fail processing.
• sources from pyproject.toml are always used as most important
• first of following methods, CLI --source, env var POETRY_SOURCES or globally configured ~sources, may contribute additional sources.

1. Use all sources defined in pyproject.toml

   Whatever source appears in pyproject.toml shall be used. Use them in given order.

   In case any given source is using empty url, require existence of globally defined repository with given name and use that url. Fail, if such global repository is missing.

2. Use all sources defined on command line

   If --source name is specified on command line call, use it. This option ca be used multiple times.

   Use those names in given order and append them to names possibly read from pyproject.toml.

   If any ~source option was used, skip further processing of env. variable or globally defined sources.

3. Use sources from env. variable POETRY_SOURCES

   Note: if --source CLI option was used, env. variable is ignored.

   Value of POETRY_SOURCES is space delimited list of repository names to use as sources.

   If any of names specified are missing in global configuration, fail.

   Use all of these names and append them to names possibly read from pyproject.toml.

4. Use of globally configured sources

   Note: if --source CLI option or env variable POETRY_SOURCES were used, globally configured sources value is ignored.

   sources value read from global config is list of repository names to be used as sources.

   If any of given names are not defined as repository, fail.

   Append all source names to names possibly read from pyproject.toml.

Open issues

Terms repository, (poetry) source, index and pypi

There seem to be some redundancy. Could we merge some of the terms?

How to prevent use of public pypi

E.g. use --default-source and point to name of repository to use as default one. By default, it would use public pypi.

poetry search ignores currently sources

Conclusions

If there is some preliminary agreement on proposed solution, we may contribute PR.

We had another, simpler proposal, which assumed only one source url. Providing such url e.g. via devpi server is feasible but it would go against concept of named sources and repositories, which are already clearly visible concepts in poetry.

We are looking forward to make poetry essential part in (at least our) CI/CD jobs and scripts.