Custom Python PyPI repository

Python downloads its dependencies from PyPI repositories by default. It contains latest versions (can be stable or not) and various amount of packages. We’re good right? So, whats the need of custom private package repository? Dependency management You can control dependencies of packages explicitly regardless of versions deprecation or latest backward incompatible versions. Of course, this can be done by defining versions in but to make sure that every package we get is as we expected its better to use custom repository. requirements.txt, TLS v1.1, v1.0 Deprecation by PyPI servers PyPI servers have discontinued support for TLS 1.0 / TLS 1.1 version devices to download packages. “I am going to see about possibly organizing some scheduled ‘brown outs’ of TLSv1.0 and TLSv1.1 prior to the cut off dates to try and help folks find places that will need updates. Any scheduled brownouts will be posted to status.python.org prior to happening.” Hence and it has become compulsory now. Devices that has Ubuntu 12.04 or below has TLS v1.1, which means those devices cannot download any python packages from default python package management server. Check your TLS version by running following command in your terminal. Upgrade Your Python: TLS v1.2 Will Soon Be Mandatory python2 -c "import urllib2,json; print(json.loads(urllib2.urlopen('https://www.howsmyssl.com/a/check').read())['tls_version'])" There were some workarounds like explicitly saying PIP to download from PyPI server. pip install --index-url= scapy https://pypi.python.org/simple/ This worked for a while (During brownout period), then it also doesn’t happen to be working. Too many stack overflow questions! Here we have only two options, Upgrading TLS version or using custom repository. After a while upgrading TLS also became impossible since you need to upgrade openssl and . Upgrading a python module is not possible, so we stuck with a deadlock 😕. This is where custom repository comes to rescue! 😃 python Cryptography module Building Custom Repository You need following things to make your own custom python repository. Hosting Ubuntu server Python environment with TLS v1.2 Public Domain Alright, lets get our hands dirty 😉 First login to your Ubuntu server and, list down all your required dependencies in a requirements.txt file like below boto3 1.5.90click 8.0configobj 0.13.1futures 0.9.3pexpect 0.15.1pyasn1 2.6.1python-dateutil 2.5.3s3transfer 2.4.0six 2.0.0urllib3 15.2.0websocket-client 0.24.0ndg-httpsclientdistributepip==1.5.4 1.4.4botocore 3.1coloredlogs 5.0.6docutils 3.1.1jmespath 3.1pyOpenSSL 0.4.2pycrypto 2.6.1requests 0.1.10scapy 1.5.2speedtest-cli 1.7.1virtualenv 0.16.0wheel Then create Python virtual environment to install these dependencies. pip install virtualenvvirtualenv reposource repo/bin/activatemkdir /home/batman/py-cache Existing pip version should be within virtual environment. 1.5.4 pip install -r requirements.txt --no-use-wheel --download="/home/batman/py-cache" Now having all packages downloaded in your Ubuntu server, we need to organize all packages in a standard directory structure so that PIP clients can identify modules. For that I’ve written a small script, run it in py-cache directory, it will do the job for you 😃 import os for filename in os.listdir('/home/batman/py-cache'):try:package_name = filename.split('.')[0].split('-')del package_name[-1]package_name = '-'.join(map(str, package_name))print package_nameos.system('mkdir %s' % package_name)os.system('mv %s %s/' % (filename, package_name))except Exception as e:print 'ERROR: 'print 'e' Make sure all the packages have been moved. If anything is not moved, create a directory manually and move the tar ball inside. We’re almost there! Lets host the repository. First install hosting client pip install twisted By default Host starts in Port 8080, make sure no other process owns it. lsof -i:8080 Get your terminal into directory py-cache twistd -n web --path . Thats it! 👏 You’ve built your own custom repository and its now hosted in your Ubuntu server’s port 8080. Informing PIP Clients about our Custom repository In your client, vim /etc/pip.conf Add your custom repository URL [global]index-url = http://10.1.10.69:8080 Then export an environment variable PIP_CONFIG_FILE. export PIP_CONFIG_FILE=/etc/pip.conf Thats it. Now usual pip installation as follow will install python package from your custom repository. pip install scappy Here you have full control on what dependency you need to provide and what are the clients you need to authenticate and forbid. I hope that I’ve shared an useful information with you all. Cheers 🍻 Happy Coding!