Skip to content

Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only) #1168

@hozn

Description

@hozn
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Question

I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.

Here's the background / what I'm trying to accomplish:

  • I have a large modular python project. It is generally a main application with dependencies on other components that we maintain -- shared libraries, essentially.
  • Adding a new feature in the main application often involves changing one or more of these shared libraries.
  • During development, I'd like to be able to experiment with different code in those shared libraries and integration test it with my main application before actually writing tests for / committing / tagging / pushing / etc. the changes to those shared libraries.

Here's how I would do this with setuptools / pip:

  • Ensure that my main application (my-app e.g.) has a requirements.txt file that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, if my-app will use an updated my-lib1 version 4.1 and my-lib2 version 1.12.3, I might have the following inmy-app/requirements.txt:
    my-lib1 ^4.1
    my-lib2 ^1.12.3
    
  • Setup a virtualenv for my top-level application and activate it.
  • Change into each shared library directory and run pip install -r requirements.txt && python setup.py develop (Sometimes pip install -e . would also work, but the behavior was a little more consistent actually using python setup.py develop. Argh, the mess that is python packaging !?!)
  • Finally I'd run pip install -r requirements.txt in my main application; because I'd already done python setup.py develop I'd have installed (symlinked) in the right versions to satisfy for my-lib1 and my-lib2.
  • After finishing development, I'd package up the shared libraries and deploy them to our registry; then when my-app was pushed to build server, it'd find the versions it needed.

This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.

I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATION

If I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHS

But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.

So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/solverRelated to the dependency resolverkind/featureFeature requests/implementations

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions