Pipenv & Requests Libraries Author Kenneth Reitz Talks About Python Scalability: An Interview

Kenneth Reitz is a well-known software engineer, international keynote speaker, open-source advocate, who also focuses on photography and music production.

He is well known for his many open-source projects, specifically Requests "simple, yet elegant HTTP library" and Pipenv Python Development Workflow for Humans.

We are excited to have interviewed Kenneth! In this interview he talks about the top issues Python developers face while scaling applications, shares his thoughts on async / await paradigm in Python, speaks about his project PyTheory, and the "natural affinity" for software development.

The Interview

Evrone: Alongside all your Python contributions you also have several music albums and thousands of great professional photos. What is your main interest right now?

Kenneth: My main interest at the moment is crafting for myself a simpler life. Love, relationships, friendships, stability and comfort, and the sanctuary of my home.

Evrone: Looking back at your experience being a Product Owner of Python at Heroku, what are the top issues Python developers face while scaling applications?

Kenneth: To be completely honest, the most primary issue that most engineers face is themselves — typically because of misassumptions made about concurrency models related to subprocess or threading. Horizontal vs. vertical scaling seems to be another common thread (“let’s just throw more memory at the problem”) — there seems to be a common misunderstanding of common node network limitations, background task idempotency, timeout, and retry patterns — as well as general kernel-level blocking calls on cloud service providers, such as making DNS queries. Everything should have timeouts, always. 

Over-optimization is also a common thread. Simple is better than complex, remember? Hardware is often much cheaper than engineering time, and it has the potential to cause far less stress on your team and your teammates. Don’t pre-optimize if you don’t need to. 

For example: greenlets (e.g. gevent) can often quickly solve far more problems than people realize :)

Evrone: How do you feel about switching to Windows & VS Code after so many years using Sublime and macOS? As a programmer, what do you like about the new setup and miss from the old one?

Kenneth: Not much, it’s a fluid experience. I use both on a daily basis, and they work well together. VS Code is what I use on all platforms. I am thankful for the syncing that has been introduced, as well as WSL 2.

Evrone: As the author of Pipenv, how do you see the main dependency management challenges for Python right now?

Kenneth: Confusion. Not much has changed. Political forces not unpresent towards this, I believe. There’s an undercurrent of some form of unspecified control that seems to be present. I removed myself from the situation. I don’t particularly want to speak about it any further. 

I’m very happy that Pipenv has seen the success that it has. It was such a joy to make! One of my favorite projects. That’s why the logo is a present :) 

Evrone: Do you think there is any natural affinity towards software development like some people have for playing musical instruments or drawing?

Kenneth: Yes, exactly! The ability to extrapolate tones and express them into a physical instrument is a very abstract and advanced skill, let alone express emotionality in an instrument. Software engineering is not dissimilar, if approached from the same perspective. 

There is a larger presence of “worker bee” engineers coming into the workforce, I feel, as the industry needs more and more engineers to fulfill cultural demand. So, I suspect that the “natural affinity for” / passionate programmers will be a rarer breed. I supremely hope that the value of the autodidact will not be diminished.

Evrone: Sometimes Pipenv spends hours trying to resolve the dependency graph. From a technical point of view, what exactly is happening during all that time?

Kenneth: Your internet connection is slow. Run `pipenv lock --verbose` and you can see what’s happening. or `pipenv install --no-lock` to avoid it. Basically, in order to ensure 100% reproducibility of builds across all operating systems, wheels must be downloaded for each available operating system (even those that are not your own) so hashsums can be calculated for all systems. This can take a while.

I had the idea of creating a web service that would supply all needed information quickly to the users, so this would never happen, but I felt it went against the philosophy of what Python users tend to want — and it would put me at political risk — there’s a lot of tension in the packaging space. Lots of opinions and lots of demand.

PyPI’s API does supply some of this information I believe, but this process is carried out manually for private package indexes, which is a large use case for Pipenv. 

Everything in life is a tradeoff. You can’t (always) have your cake and eat it too. Remember — the cake is a lie.

✨ 🍰 ✨

Evrone: With the "Hitchhiker's Guide to Python" under your belt, how do you see good education for a software developer? Do we need a "science" like "computer science" with all math, algorithms, and data structures or do we need something else?

Kenneth: We need better tools to enable tinkerers and explorers to just do their own thing in a simple environment. Lower the barrier to entry as much as possible. I didn’t have a credit card when I was in 7th grade, did you? Prime example: Repl.it.

Evrone: Given your experiments on async requests and async responder, what are your thoughts about the async / await paradigm in Python? How do you see the main challenges and opportunities this "async / await" thing provides to us, software developers?

Kenneth: I think async / await are great keywords and should be used sparingly and intentionally. They are not a saving grace, however, that will magically awaken superpowers for your applications. In fact, they can often introduce hidden complexity, and therefore cost (which takes many forms). 

I prefer to use these keywords in distinct, separated components of a codebase, or in microservices.

Evrone: If you could travel back in time and give only one piece of technical advice to the younger self about your "requests" masterpiece, what advice would it be?

Kenneth: You do you, man :)

Evrone: What are your thoughts on type annotations and gradual typing that were recently introduced into Python?

Kenneth: They are great for integrative-layer APIs, especially ones that will be heavily interfaced with code completion tools et cetera.

Evrone: You designed a lot of successful APIs "for humans". Any recipes for a good API design you can share with fellow developers?

Kenneth: The Design of Everyday Things by Don Norman.

Evrone: Do you have any software projects in mind that you want to implement? Something about Python and its ecosystem that needs to be fixed and improved to be more "for humans" and less "for legacy"?

Kenneth: More creative discovery. PyTheory — Music Theory for humans. Lots of things to explore here. I want to focus on the tonal / chordal / harmonic interplay of ratios mathematically between various musical systems and temperaments using numpy and scipy and see if there is any correlation between sensory perception and, say, perception of colors and emotion. 

I have put a tremendous amount of work into this codebase. It is not well presented in the readme, yet. 

Evrone: What do you think about the "Full-stack developers" hype and how good Python fits into it as a backend language of choice?

Kenneth: Bash first. Then Python. Bash is an amazing language. 

Evrone: You learned Python many years ago. How do you see the best way for a wanna-be developer to start their journey in 2020 and learn to develop software?

Kenneth: Follow your heart. Keep it simple. Listen to everyone, then disregard it. Don’t repeat yourself. Rules are meant to be broken. There’s nearly always a technical solution to any problem — but it always has a cost, be it financial, time, storage, memory, or computational. Enjoy the ride. Never forget why you are doing this in the first place. It’s so easy to get lost in the how instead of the why.

The conclusion

We were glad to speak with Kenneth and learn more about his life, thoughts & projects. At Evrone, we frequently use Python to develop custom solutions for our clients. If you love Python as much as we do — just send us a message!

Previously published at https://evrone.com/kenneth-reitz-interview. The author of the story is the Chief Editor at Evrone.