Development<\/h2>\n
Under development, the issues I will focus on are the following:<\/p>\n
- \n
- Dependency Management<\/li>\n
- Virtualenvs and managing them<\/li>\n<\/ul>\n
Historically, the way to do dependency management was through\n
requirements.txt<\/code>. I foundrequirements.txt<\/code> hard to manage. In that setup,\nadding a dependency and installing it was two steps:<\/p>\n- \n
- Add the package
bar<\/code> torequirements.txt<\/code><\/li>\n- Either do
pip install bar<\/code> orpip install -r requirements.txt<\/code><\/li>\n<\/ul>\nWhile focused on development, I would often forget one or both of these steps.\nAlso, the lack of a lock file was a small downside for me (could be a much\nlarger downside for others). The separation between
pip<\/code> and\nrequirements.txt<\/code> can also easily lead you to accidentally depend on packages\ninstalled on your system or in your virtualenv but not specified in your\nrequirements.txt<\/code>.<\/p>\nManaging virtualenvs was also difficult. As a virtualenv and a project are not\nrelated, you need a directory structure. Otherwise, you can’t tell which\nvirtualenv is being used for which project. You can use the same virtualenvs\nfor multiple projects, but that partially defeats the point of virtualenvs and\nmakes
requirements.txt<\/code> more error-prone (higher chances of forgetting to add\npackages to it). The approach generally used is one of the following two:<\/p>\nfoo\/\n\u251c\u2500\u2500 foo_src\/\n\u2514\u2500\u2500 foo_venv\/\n<\/code><\/pre>\nor<\/p>\n
foo_src\/\n\u2514\u2500\u2500 venv\/\n<\/code><\/pre>\nI preferred the second one as the first one nests the source code one\ndirectory deeper.<\/p>\n
A new standard -
pyproject.toml<\/code><\/h3>\nIn PEP-518<\/a>\n, python standardized\nthe
pyproject.toml<\/code> file which allows users to choose alternate build systems\nfor package generation.<\/p>\nOne such project that provides an alternate build system is\nPoetry<\/a>\n. Poetry hits the nail on the head and\nsolves my major gripes with traditional tooling.<\/p>\n
Poetry and virtualenvs<\/h3>\n
Poetry manages the virtualenvs automatically and keeps track of which project\nuses which virtualenv automatically. Working on an existing project which uses\npoetry is as simple as this:<\/p>\n
$ git clone https:\/\/gitlab.com\/ceda_ei\/verlauf\n$ poetry install\n<\/code><\/pre>\nThe
poetry install<\/code> command sets up the virtualenv, install all the required\ndependencies inside that, and sets up any commands accordingly (I will get to\nthis soon). To activate the virtualenv, simply run:<\/p>\n. "$(poetry env info --path)\/bin\/activate"\n<\/code><\/pre>\nI wrap this in a small function which lets me toggle it quickly:<\/p>\n
function poet() {\n\tPOET_MANUAL=1\n\tif [[ -v VIRTUAL_ENV ]]; then\n\t\tdeactivate\n\telse\n\t\t. "$(poetry env info --path)\/bin\/activate"\n\tfi\n}\n<\/code><\/pre>\nRunning
poet<\/code> activates the virtualenv if it is not active and deactivates it if\nit is active. To make things even easier, I automatically activate and\ndeactivate the virtualenv as I enter and leave the project directory. To do\nso, simply drop this in your.bashrc<\/code>.<\/p>\nfunction find_in_parent() {\n\tlocal path\n\tIFS="\/" read -ra path <<<"$PWD"\n\tfor ((i=${#path[@]}; i > 0; i--)); do\n\t\tlocal current_path=""\n\t\tfor ((j=1; j<i; j++)); do\n\t\t\tcurrent_path="$current_path\/${path[j]}"\n\t\tdone\n\t\tif [[ -e "${current_path}\/$1" ]]; then\n\t\t\techo "${current_path}\/"\n\t\t\treturn\n\t\tfi\n\tdone\n\treturn 1\n}\n\nfunction auto_poet() {\n\tret="$?"\n\tif [[ -v POET_MANUAL ]]; then\n\t\treturn $ret\n\tfi\n\tif find_in_parent pyproject.toml &> \/dev\/null; then\n\t\tif [[ ! -v VIRTUAL_ENV ]]; then\n\t\t if BASE="$(poetry env info --path)"; then\n\t\t\t. "$BASE\/bin\/activate"\n\t\t\tPS1=""\n\t\t else\n\t\t\tPOET_MANUAL=1\n\t\t fi\n\t\tfi\n\telif [[ -v VIRTUAL_ENV ]]; then\n\t\tdeactivate\n\tfi\n\treturn $ret\n}\n\nPROMPT_COMMAND="auto_poet;$PROMPT_COMMAND"\n<\/code><\/pre>\nThis ties in well with the
poet<\/code> function; if you usepoet<\/code> anytime in a bash\nsession, activation switches from automatic to manual and changing directories\nno longer auto-toggles the virtualenv.<\/p>\n![\"auto_poet](\"https:\/\/cedaei.com\/images\/auto_poet.webp\")
<\/p>\nPoetry and dependency management<\/h3>\n
Instead of using
requirements.txt<\/code>, poetry stores the dependencies inside\npyproject.toml<\/code>. Poetry is more strict compared topip<\/code> in resolving\nversioning issues. Dependencies and dev-dependencies are stored inside\ntool.poetry.dependencies<\/code> andtool.poetry.dev-dependencies<\/code> respectively.\nHere is an example of apyproject.toml<\/code> for a project I am working on.<\/p>\n[tool.poetry]\nname = "bells"\nversion = "0.3.0"\ndescription = "Bells is a program for keeping track of sound recordings."\nauthors = ["Ceda EI <ceda_ei@webionite.com>"]\nlicense = "GPL-3.0"\nreadme = "README.md"\nhomepage = "https:\/\/gitlab.com\/ceda_ei\/bells.git"\nrepository = "https:\/\/gitlab.com\/ceda_ei\/bells.git"\n\n[tool.poetry.dependencies]\npython = ">=3.7,<3.11"\nclick = "^8.0.1"\nquestionary = "^1.10.0"\nsounddevice = "^0.4.2"\nSoundFile = "^0.10.3"\nnumpy = "^1.21.2"\n\n[tool.poetry.dev-dependencies]\n\n[build-system]\nrequires = ["poetry-core>=1.0.0"]\nbuild-backend = "poetry.core.masonry.api"\n\n# I will talk about this section soon\n[tool.poetry.scripts]\nbells = "bells.__main__:main"\n<\/code><\/pre>\nOne of the upsides of poetry is that you don’t have to manage the dependencies\nin
pyproject.toml<\/code> file yourself. Poetry adds annpm<\/code>-like interface for\nadding and removing dependencies. To add a dependency to your project, simply\nrunpoetry add bar<\/code> and it will add it to yourpyproject.toml<\/code> file and\ninstall it in the virtualenv as well. To remove a dependency, just runpoetry remove bar<\/code>. For development dependencies, just add the--dev<\/code> flag to the\ncommands.<\/p>\nPackaging<\/h2>\n
Since poetry replaces the build system, we can now configure the build using\npoetry via
pyproject.toml<\/code>. Insidepyproject.toml<\/code>, thetool.poetry<\/code> section\nstores all the build info needed;tool.poetry<\/code> contains the metadata,\ntool.poetry.dependencies<\/code> contains the dependencies,tool.poetry.source<\/code>\ncontains private repository details (in case, you don’t want to use PyPi).<\/p>\nOne of the options is
tool.poetry.scripts<\/code>. It contains scripts that the\nproject exposes. This replacesconsole_scripts<\/code> inentry_points<\/code> of\nsetuptools<\/code>.<\/p>\nFor example,<\/p>\n
[tool.poetry.scripts]\nfoobar = "foo.bar:main"\n<\/code><\/pre>\nThis will add a script named
foobar<\/code> in yourPATH<\/code>. Running that is\nequivalent to running the following script<\/p>\nfrom foo.bar import main\n\nif __name__ == "__main__":\n main()\n<\/code><\/pre>\nFor further details, check the\nreference<\/a>\n.<\/p>\n
Poetry also removes the need for manually doing editable installs (
pip install -e .<\/code>). The package is automatically installed as editable when you run\npoetry install<\/code>. Any scripts specified intool.poetry.scripts<\/code> are\nautomatically available in yourPATH<\/code> when you activate thevenv<\/code>.1<\/a><\/sup><\/p>\nTo build the package, simply run
poetry build<\/code>. This will generate a wheel and\na tarball in the dist folder.<\/p>\nTo publish the package to PyPi (or another repo), simply run
poetry publish<\/code>.\nYou can combine the build and publish into one command withpoetry publish --build<\/code>.<\/p>\n![\"example](\"https:\/\/cedaei.com\/images\/poetry_build.webp\")
<\/p>\nUsage<\/h2>\n
This part is more user-facing rather than dev-facing. If you want to use two\npackages globally that expose some scripts to the user, (e.g.
awscli<\/code>,\nyoutube-dl<\/code>, etc.) the general approach to do so is to run something likepip install --user youtube-dl<\/code>. This install the package at the user level and\nexposes the script through~\/.local\/bin\/youtube-dl<\/code>. However, this installs\nall the packages at the same user level. Hypothetically, if you have two\npackagesfoo<\/code> andbar<\/code> which have conflicting dependencies, this causes an\nissue. If you run,<\/p>\n$ pip install foo\n$ pip install bar\n$ bar # works\n$ foo # breaks because of dependency mismatch\n<\/code><\/pre>\nWhile installing
bar<\/code>,pip<\/code> will install the dependencies forbar<\/code> which\nwill breakfoo<\/code> after warning you2<\/a><\/sup>.<\/p>\nTo solve this, there is
pipx<\/code><\/a>\n. Pipx installs\neach package in a separate virtualenv without requiring the user to activate\nsaid virtualenv before using the package.3<\/a><\/sup><\/p>\nIn the same scenario as before, doing the following works just fine.<\/p>\n
$ pipx install foo\n$ pipx install bar\n$ bar # works\n$ foo # also works\n<\/code><\/pre>\nIn this scenario, both
bar<\/code> andfoo<\/code> are installed in separate virtualenvs so\nthe dependency conflict doesn’t matter.<\/p>\nSome more things from my bashrc<\/h2>\n
\nfunction wrapper_no_poet() {\n\tlocal last_env\n\tif [[ -v VIRTUAL_ENV ]]; then\n\t\tlast_env="$VIRTUAL_ENV"\n\t\tdeactivate\n\tfi\n\t"$@"\n\tret=$?\n\tif [[ -v last_env ]]; then\n\t\t. "$last_env\/bin\/activate"\n\tfi\n\treturn $ret\n}\n\nalias wnp='wrapper_no_poet'\nalias pm='POET_MANUAL=1'\n<\/code><\/pre>\nPrefixing any command with
wnp<\/code> runs it outside the virtualenv if a virtualenv\nis active. Runningpm<\/code> turns off automatic virtualenv activation.<\/p>\n\n
\n- \n
- \n
This also allows for a nice switch between the development and production\nversions of the app. Essentially, when the virtualenv is active, you are\nusing the development script while when it is deactivated, you are using\nthe global (likely production) version. ↩︎<\/a><\/p>\n<\/li>\n
- \n
To be precise, it will warn you that it broke
foo<\/code> but will still\ncontinue with the installation ↩︎<\/a><\/p>\n<\/li>\n- \n
For development, poetry also provides
poetry run<\/code> which runs a file\nwithout having to activate the virtualenv. ↩︎<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"}}} - \n
- Either do
- Add the package