Quick Deep Learning Python Development Environment with Poetry
Trying to find a smoother developer experience compared to virtualenvs
Let’s say you want to start playing with a new python library and a framework 1 , and you don’t want it to mess up anything else you have installed on your system.
The common steps involved in trying out some new library/framework are
- Install the package (via pip)
- read the documentation
- follow the “hello world” equivalent example
- if you like what you see, include the library in your own “full sized” projects.
trips up people because they do
pip install foo
and that specific package+version combo
might conflict with a dependency
installed by yet
; this is a fairly common occurrence
in all kinds of software, not just
What you need is a self-contained, isolated environment, often called
The typical virtualenv steps to achieve the same things as above are:
$ mkdir newstuff $ cd newstuff $ python3 -m venv venv # the second venv is the directory name. can be anything of your choice, but many IDEs recognize .venv and venv and pick it up use it with your code. $ source venv/bin/activate $ pip install TTS
But, even if one becomes familiar with virtualenv, there remains a small user experience irritation.
venvdirectory hanging around along with the code
source venv/bin/activate. not exactly “intuitive”
having to remember what’s installed in this venv (you can query it
We can improve the UX with Python Poetry . You can read why poetry beats pip setup for some rationale, I’ll focus on how the typical workflow changes with it. Let’s look at how I used it today - to play with TTS .
First install OS lib dependencies 3 and then:
➜ sandbox mkdir ttslearn ➜ sandbox cd ttslearn ➜ ttslearn poetry init This command will guide you through creating your pyproject.toml config. Package name [ttslearn]: Version [0.1.0]: Description : Author [Pradeep Gowda <email@example.com>, n to skip]: License : Compatible Python versions [^3.11]: Would you like to define your main dependencies interactively? (yes/no) [yes] You can specify a package in the following forms: - A single name (requests): this will search for matches on PyPI - A name and a constraint (requests@^2.23.0) - A git url (git+https://github.com/python-poetry/poetry.git) - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop) - A file path (../my-package/my-package.whl) - A directory (../my-package/) - A url (https://example.com/packages/my-package-0.1.0.tar.gz) Package to add or search for (leave blank to skip): tts Found 20 packages matching tts Showing the first 10 matches Enter package # to add, or the complete package name if it is not listed : [ 0] TTS [ 1] tts369 [ 2] TTS2 [ 3] tts-arranger [ 4] xfyun-tts [ 5] macosx-tts [ 6] vnpy-tts [ 7] google-tts [ 8] mPython-tts [ 9] sanskrit-tts [ 10] > 0 Enter the version constraint to require (or leave blank to use the latest version): Using version ^0.17.4 for TTS Add a package (leave blank to skip): Would you like to define your development dependencies interactively? (yes/no) [yes] no Generated file [tool.poetry] name = "ttslearn" version = "0.1.0" description = "" authors = ["Pradeep Gowda <firstname.lastname@example.org>"] readme = "README.md" [tool.poetry.dependencies] python = "^3.11" TTS = "^0.17.4" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" Do you confirm generation? (yes/no) [yes] yes
The above chioces were written to
to install the dependencies I
mentioned above. I activate the “shell” which has all the dependencies
poetry shell Spawning shell within /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11 ➜ ttslearn emulate bash -c '. /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11/bin/activate'
You can see the python interpret is provided by the virtual
environment specified by the
Immediately a few things stand out:
- the menu driven approach.
The dependencies (and exact version number) are captured in the
pyproject.toml; you always know what version of the software you had installed to work “here”. (You can add more dependencies with
poetry add torch
The shell is where I do all my work, whether its the REPL with all
the deps installed or run a script in the
I can now take the new library for a spin:
(ttslearn-py3.11) ➜ ttslearn python3 Python 3.11.2 (main, Feb 26 2023, 17:21:36) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> from TTS.api import TTS >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model_name = TTS().list_models() No API token found for 🐸Coqui Studio voices - https://coqui.ai Visit 🔗https://app.coqui.ai/account to get one. Set it as an environment variable `export COQUI_STUDIO_TOKEN=<token>`
That looks good enough to know that the library is installed correctly.
If you decide to convert your
“script” into an actual application, it is very easy to do so because
Poetry handles many things for you including packaging your app for
or add custom entry
Conclusion: I set out to see if Poetry can indeed be better than virtualenv based setup. So far it shows clear upgrades over that approach.