-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have searched the documentation and believe that my question is not covered.
Question
I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.
Here's the background / what I'm trying to accomplish:
- I have a large modular python project. It is generally a main application with dependencies on other components that we maintain -- shared libraries, essentially.
- Adding a new feature in the main application often involves changing one or more of these shared libraries.
- During development, I'd like to be able to experiment with different code in those shared libraries and integration test it with my main application before actually writing tests for / committing / tagging / pushing / etc. the changes to those shared libraries.
Here's how I would do this with setuptools / pip:
- Ensure that my main application (
my-appe.g.) has arequirements.txtfile that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, ifmy-appwill use an updatedmy-lib1version 4.1 andmy-lib2version 1.12.3, I might have the following inmy-app/requirements.txt:my-lib1 ^4.1 my-lib2 ^1.12.3 - Setup a virtualenv for my top-level application and activate it.
- Change into each shared library directory and run
pip install -r requirements.txt && python setup.py develop(Sometimespip install -e .would also work, but the behavior was a little more consistent actually usingpython setup.py develop. Argh, the mess that is python packaging !?!) - Finally I'd run
pip install -r requirements.txtin my main application; because I'd already donepython setup.py developI'd have installed (symlinked) in the right versions to satisfy formy-lib1andmy-lib2. - After finishing development, I'd package up the shared libraries and deploy them to our registry; then when
my-appwas pushed to build server, it'd find the versions it needed.
This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.
I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):
[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATIONIf I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:
[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"
[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHSBut even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.
So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!