Developing, Packaging and Distributing a Python Library

How to use new packaging standards with virtual environment tools — adapted from the official documentation of python.org and Pipenv. Introduction When my team and I started developing and shipping new Python libraries, we ran into two roadblocks: The packaging standards we knew had been deprecated. We could not find any good resources to use these new packaging standards in a virtual environment. Since we were developing our Python library in a virtual environment (specifically, Pipenv), we had to figure out a way to use these new packaging standards with the virtual environment tool. So, we decided to build our own approach by reading the official documentation of python.org and Pipenv. But first, let us briefly look at the last two packaging standards. How packaging standards have evolved? Generate requirements using pip freeze command The earlier standard was to use pip freeze command to generate the requirements.txt file. This file would be read in setup.py file while packaging and distributing the library. This created two key problems: command would copy all the libraries installed in your system into the file while you only needed the ones used in this project. More exhaustive means better packaging, right? Sorry but the answer is no — you want your library to have minimal requirements so that it does not create clashes during installation. pip freeze requirements.txt You may say that you’ll manually edit this file and remove the libraries that you know that you are not using. It will work, right? Sorry but the answer is no again — this is because you do not know what are the dependencies of such libraries. By deciding to manually edit the file, you may very well be deleting these dependencies. requirements.txt Generate requirements in a virtual environment Virtual environments allowed us to move away from the command. For example, in Pipenv, you would use command to generate the file. pip freeze pipenv lock -r > requirements.txt requirements.txt How was this helpful? Didn’t we just swap out a command? True but it solved a key pain point with the earlier standard — now our requirements file only contained the libraries that we were using in this project and not the entire system. But this standard still had one problem — there was always a chance of misconfiguration while setting up CI/CD pipeline. Hence, errors like ending up with an outdated file were frequent and quite plausible. requirements.txt If you are interested in learning more about what has been deprecated and what are the new alternatives, you can check this . blog New packaging standard and virtual environment tools The new packaging standard is not directly compatible with the commonly used virtual environment tools. For example, according to the new standard, we should use the package instead of running . python -m build python setup.py sdist Running creates a virtual environment, installs dependencies, etc. However, it fails to install dependencies from Pipfile if you are using a virtual environment tool such as Pipenv. python -m build Since we were developing our Python library in a virtual environment (specifically, Pipenv), we had to figure out a way to use these new packaging standards with the virtual environment tool. Development Use a virtual environment like Pipenv while developing the Python library. A good guide on Pipenv can be found . here Packaging a Python library in a virtual environment A good place to start is to first read through Python.org’s tutorial on packaging a simple Python project. We will adopt these steps to use them with virtual environment tools. Step 1: Creating the package files Step 2: Creating pyproject.toml Step 3: Configuring metadata We can now get requirements programmatically read Pipfile instead of generating a from PipFile. requirements.txt Below is an example of how we can do it — an excerpt from the file in Streamlit’s Github repository. setup.py # Older Pipenv sys.exit(err_msg) pipfile = Project().parsed_pipfile requirements = convert_deps_to_pip(packages, r=False) import sys import setuptools try : from pipenv.project import Project try : # Pipenv 2022.4 .8 from pipenv.utils.dependencies import convert_deps_to_pip except : from pipenv.utils import convert_deps_to_pip except : err_msg = "Please install pipenv and try again." packages = pipfile[ "packages" ].copy() We can use these requirements while calling function in file. setuptools.setup setup.py ... install_requires=requirements, ... ) setuptools.setup( Step 4: Generating distribution archives Note that running PyPi’s recommended package creates a new virtual environment. However, we want package to use the virtual environment created by instead of creating its own. python -m build build pipenv What if we simply add a flag to the command python -m build -n that tells it to not create its own virtual environment? Unfortunately, this still fails. Note that earlier used to have two steps — sdist and wheel. setup.py If we add a flag to the command , the code we wrote in the previous step to generate requirements dynamically passes in the sdist step (this step generated .tar.gz file) but fails while creating a wheel. You will get a NonExistentKey error. python -m build -n This might get fixed in a future version of . Until then, you need to run sdist and wheel step separately. build For sdist step, run python -m build -n -s For wheel step run python -m build -n -w Step 5: Uploading the distribution archives 1. Create accounts in and . PyPi TestPyPi 2. Create file in your directory. The content should be as follows: .pypirc $HOME 3. Install twine like . pipenv install twine --dev 4. Upload your package to PyPiTest. From inside the virtual environment run: python -m twine upload --repository testpypi dist /* 5. Install your package from PyPiTest: pipenv install -i https: //test.pypi.org/simple/ Closing thoughts In this post, we learned the following things about packaging in Python: How packaging standards have evolved? How to adopt PyPi’s recommended guidelines to package a Python library in a virtual environment? Also Published Here