Custom Python PyPI repository

Python downloads its dependencies from PyPI repositories by default. It contains latest versions (can be stable or not) and various amount of packages. We’re good right? So, whats the need of custom private package repository?

Dependency management

You can control dependencies of packages explicitly regardless of versions deprecation or latest backward incompatible versions. Of course, this can be done by defining versions in requirements.txt, but to make sure that every package we get is as we expected its better to use custom repository.

TLS v1.1, v1.0 Deprecation by PyPI servers

PyPI servers have discontinued support for TLS 1.0 / TLS 1.1 version devices to download packages.

“I am going to see about possibly organizing some scheduled ‘brown outs’ of TLSv1.0 and TLSv1.1 prior to the cut off dates to try and help folks find places that will need updates. Any scheduled brownouts will be posted to status.python.org prior to happening.”

Hence Upgrade Your Python: TLS v1.2 Will Soon Be Mandatory and it has become compulsory now. Devices that has Ubuntu 12.04 or below has TLS v1.1, which means those devices cannot download any python packages from default python package management server. Check your TLS version by running following command in your terminal.

python2 -c "import urllib2,json; print(json.loads(urllib2.urlopen('https://www.howsmyssl.com/a/check').read())['tls_version'])"

There were some workarounds like explicitly saying PIP to download from PyPI server.

pip install --index-url=https://pypi.python.org/simple/ scapy

This worked for a while (During brownout period), then it also doesn’t happen to be working.

Too many stack overflow questions!

Here we have only two options, Upgrading TLS version or using custom repository. After a while upgrading TLS also became impossible since you need to upgrade openssl and python Cryptography module. Upgrading a python module is not possible, so we stuck with a deadlock 😕. This is where custom repository comes to rescue! 😃

Building Custom Repository

You need following things to make your own custom python repository.

1. Hosting Ubuntu server
2. Python environment with TLS v1.2
3. Public Domain

Alright, lets get our hands dirty 😉

First login to your Ubuntu server and, list down all your required dependencies in a requirements.txt file like below

boto31.4.4botocore1.5.90click3.1coloredlogs8.0configobj5.0.6docutils0.13.1futures3.1.1jmespath0.9.3pexpect3.1pyOpenSSL0.15.1pyasn10.4.2pycrypto2.6.1python-dateutil2.6.1requests2.5.3s3transfer0.1.10scapy2.4.0six1.5.2speedtest-cli2.0.0urllib31.7.1virtualenv15.2.0websocket-client0.16.0wheel0.24.0ndg-httpsclientdistributepip==1.5.4

Then create Python virtual environment to install these dependencies.

pip install virtualenvvirtualenv reposource repo/bin/activatemkdir /home/batman/py-cache

Existing pip version should be 1.5.4 within virtual environment.

pip install -r requirements.txt --no-use-wheel --download="/home/batman/py-cache"

Now having all packages downloaded in your Ubuntu server, we need to organize all packages in a standard directory structure so that PIP clients can identify modules. For that I’ve written a small script, run it in py-cache directory, it will do the job for you 😃

import os

for filename in os.listdir('/home/batman/py-cache'):try:package_name = filename.split('.')[0].split('-')del package_name[-1]package_name = '-'.join(map(str, package_name))print package_nameos.system('mkdir %s' % package_name)os.system('mv %s %s/' % (filename, package_name))except Exception as e:print 'ERROR: 'print 'e'

Make sure all the packages have been moved. If anything is not moved, create a directory manually and move the tar ball inside. We’re almost there! Lets host the repository.

First install hosting client

pip install twisted

By default Host starts in Port 8080, make sure no other process owns it.

lsof -i:8080

Get your terminal into py-cache directory

twistd -n web --path .

Thats it! 👏 You’ve built your own custom repository and its now hosted in your Ubuntu server’s port 8080.

Informing PIP Clients about our Custom repository

In your client,

vim /etc/pip.conf

Add your custom repository URL

[global]index-url = http://10.1.10.69:8080

Then export an environment variable PIP_CONFIG_FILE.

export PIP_CONFIG_FILE=/etc/pip.conf

Thats it. Now usual pip installation as follow will install python package from your custom repository.

pip install scappy

Here you have full control on what dependency you need to provide and what are the clients you need to authenticate and forbid. I hope that I’ve shared an useful information with you all.

Cheers 🍻

Happy Coding!