Quick Deep Learning Python Development Environment with Poetry
Trying to find a smoother developer experience compared to virtualenvs
Let’s say you want to start playing with a new python library and a framework 1 , and you don’t want it to mess up anything else you have installed on your system.
The common steps involved in trying out some new library/framework are
- Install the package (via pip)
- read the documentation
- follow the “hello world” equivalent example
- if you like what you see, include the library in your own “full sized” projects.
Often
step 1
trips up people because they do
pip install foo
and that specific package+version combo
might conflict with a dependency
baz
installed by yet
another package
bar
; this is a fairly common occurrence
2
in all kinds of software, not just
Python.
What you need is a self-contained, isolated environment, often called
a
virtualenv
.
The typical virtualenv steps to achieve the same things as above are:
$ mkdir newstuff
$ cd newstuff
$ python3 -m venv venv
# the second venv is the directory name. can be anything of your choice, but many IDEs recognize .venv and venv and pick it up use it with your code.
$ source venv/bin/activate
$ pip install TTS
But, even if one becomes familiar with virtualenv, there remains a small user experience irritation.
-
separate
venv
directory hanging around along with the code -
having to
source venv/bin/activate
. not exactly “intuitive” -
having to remember what’s installed in this venv (you can query it
with
pip list
orpip freeze
) - etc
We can improve the UX with Python Poetry . You can read why poetry beats pip setup for some rationale, I’ll focus on how the typical workflow changes with it. Let’s look at how I used it today - to play with TTS .
First install OS lib dependencies 3 and then:
➜ sandbox mkdir ttslearn
➜ sandbox cd ttslearn
➜ ttslearn poetry init
This command will guide you through creating your pyproject.toml config.
Package name [ttslearn]:
Version [0.1.0]:
Description []:
Author [Pradeep Gowda <me@btbytes.com>, n to skip]:
License []:
Compatible Python versions [^3.11]:
Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
- A single name (requests): this will search for matches on PyPI
- A name and a constraint (requests@^2.23.0)
- A git url (git+https://github.com/python-poetry/poetry.git)
- A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
- A file path (../my-package/my-package.whl)
- A directory (../my-package/)
- A url (https://example.com/packages/my-package-0.1.0.tar.gz)
Package to add or search for (leave blank to skip): tts
Found 20 packages matching tts
Showing the first 10 matches
Enter package # to add, or the complete package name if it is not listed []:
[ 0] TTS
[ 1] tts369
[ 2] TTS2
[ 3] tts-arranger
[ 4] xfyun-tts
[ 5] macosx-tts
[ 6] vnpy-tts
[ 7] google-tts
[ 8] mPython-tts
[ 9] sanskrit-tts
[ 10]
> 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^0.17.4 for TTS
Add a package (leave blank to skip):
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file
[tool.poetry]
name = "ttslearn"
version = "0.1.0"
description = ""
authors = ["Pradeep Gowda <pradeep@btbytes.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.11"
TTS = "^0.17.4"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Do you confirm generation? (yes/no) [yes] yes
The above chioces were written to
pyproject.toml
file.
4
I ran
poetry install
to install the dependencies I
mentioned above. I activate the “shell” which has all the dependencies
installed with
poetry shell
.
poetry shell
Spawning shell within /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11
➜ ttslearn emulate bash -c '. /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11/bin/activate'
You can see the python interpret is provided by the virtual
environment specified by the
pyproject.toml
configuration.
Immediately a few things stand out:
- the menu driven approach.
-
poetry -h
andpoetry list
-
The dependencies (and exact version number) are captured in the
pyproject.toml
; you always know what version of the software you had installed to work “here”. (You can add more dependencies withpoetry add torch
-
The shell is where I do all my work, whether its the REPL with all
the deps installed or run a script in the
ttslearn
directory
I can now take the new library for a spin:
(ttslearn-py3.11) ➜ ttslearn python3
Python 3.11.2 (main, Feb 26 2023, 17:21:36) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from TTS.api import TTS
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model_name = TTS().list_models()[0]
No API token found for 🐸Coqui Studio voices - https://coqui.ai
Visit 🔗https://app.coqui.ai/account to get one.
Set it as an environment variable `export COQUI_STUDIO_TOKEN=<token>`
That looks good enough to know that the library is installed correctly.
Advanced usage:
If you decide to convert your
“script” into an actual application, it is very easy to do so because
Poetry handles many things for you including packaging your app for
distribution with
poetry build
or add custom entry
points.
Conclusion: I set out to see if Poetry can indeed be better than virtualenv based setup. So far it shows clear upgrades over that approach.
fin.