How to use new packaging standards with virtual environment tools — adapted from the official documentation of python.org and Pipenv.
When my team and I started developing and shipping new Python libraries, we ran into two roadblocks:
Since we were developing our Python library in a virtual environment (specifically, Pipenv), we had to figure out a way to use these new packaging standards with the virtual environment tool. So, we decided to build our own approach by reading the official documentation of python.org and Pipenv.
But first, let us briefly look at the last two packaging standards.
Generate requirements using pip freeze command
The earlier standard was to use pip freeze command to generate the requirements.txt file. This file would be read in setup.py file while packaging and distributing the library.
This created two key problems:
command would copy all the libraries installed in your system into the
file while you only needed the ones used in this project. More exhaustive means better packaging, right? Sorry but the answer is no — you want your library to have minimal requirements so that it does not create clashes during installation.
file, you may very well be deleting these dependencies.
Generate requirements in a virtual environment
Virtual environments allowed us to move away from the
command. For example, in Pipenv, you would use
command to generate the
pipenv lock -r > requirements.txt
How was this helpful? Didn’t we just swap out a command?
True but it solved a key pain point with the earlier standard — now our requirements file only contained the libraries that we were using in this project and not the entire system.
But this standard still had one problem — there was always a chance of misconfiguration while setting up CI/CD pipeline. Hence, errors like ending up with an outdated
file were frequent and quite plausible.
If you are interested in learning more about what has been deprecated and what are the new alternatives, you can check this blog.
New packaging standard and virtual environment tools
The new packaging standard is not directly compatible with the commonly used virtual environment tools. For example, according to the new standard, we should use the
package instead of running
python -m build
python setup.py sdist
creates a virtual environment, installs dependencies, etc. However, it fails to install dependencies from Pipfile if you are using a virtual environment tool such as Pipenv.
python -m build
Since we were developing our Python library in a virtual environment (specifically, Pipenv), we had to figure out a way to use these new packaging standards with the virtual environment tool.
Use a virtual environment like Pipenv while developing the Python library. A good guide on Pipenv can be found here.
A good place to start is to first read through Python.org’s tutorial on packaging a simple Python project. We will adopt these steps to use them with virtual environment tools.
Step 1: Creating the package files
Step 2: Creating
Step 3: Configuring metadata
We can now get requirements programmatically read Pipfile instead of generating a
Below is an example of how we can do it — an excerpt from the
file in Streamlit’s Github repository.
import sys import setuptools try: from pipenv.project import Project try: # Pipenv 2022.4.8 from pipenv.utils.dependencies import convert_deps_to_pip except: # Older Pipenv from pipenv.utils import convert_deps_to_pip except: err_msg = "Please install pipenv and try again." sys.exit(err_msg) pipfile = Project().parsed_pipfile packages = pipfile["packages"].copy() requirements = convert_deps_to_pip(packages, r=False)
We can use these requirements while calling
setuptools.setup( ... install_requires=requirements, ... )
Step 4: Generating distribution archives
Note that running PyPi’s recommended
package creates a new virtual environment. However, we want
python -m build
package to use the virtual environment created by
instead of creating its own.
What if we simply add a flag to the command python -m build -n that tells it to not create its own virtual environment?
Unfortunately, this still fails.
Note that earlier
used to have two steps — sdist and wheel.
If we add a flag to the command
, the code we wrote in the previous step to generate requirements dynamically passes in the sdist step (this step generated .tar.gz file) but fails while creating a wheel. You will get a NonExistentKey error.
python -m build -n
This might get fixed in a future version of
. Until then, you need to run sdist and wheel step separately.
python -m build -n -s
python -m build -n -w
Step 5: Uploading the distribution archives
file in your $HOME directory. The content should be as follows:
3. Install twine like
pipenv install twine --dev
4. Upload your package to PyPiTest. From inside the virtual environment run:
python -m twine upload --repository testpypi dist/*
5. Install your package from PyPiTest:
pipenv install -i https://test.pypi.org/simple/ <your-package-name>
In this post, we learned the following things about packaging in Python:
Also Published Here