I have been both secretly, and publicly working in the HTTP client field for an entire year; here is what my newly acquired expertise got us. This will shock you, so be prepared. Seatbelt on! Let's travel back to the future. As a Python programmer, we should be aware that it is soon to be 2024, but most of the community lives like it's 2015. And we don't always know it. That's insane. Let us crawl out of the pit we inadvertently dug ourselves, . We've almost all started by using Requests, and now we are trapped and kind of forced out sometimes. The thing is... Habits die hard! together I was so lazy to migrate my projects from Requests that I decided to tackle the next Requests myself. 🏆 (I) HTTP/2 by Default, HTTP/3 Broadly Available! 👴 Internet Explorer 11 just celebrated its anniversary this year, and you'll never guess that it has ( ). tenth support for HTTP/2 integrated not activated by default* Saying that while the Python ecosystem has no HTTP client supporting HTTP/2 and HTTP/3 by default in November 2023. With great incentives! 👇 How about a client that supports them all? *: Doesn't this remind you of a particular HTTP client? HTTP/1 by default, optional HTTP/2? 🔀 (II) Multiplexed Connection We've just almost murdered Async and made you watch! Beware that reading down further will follow a in your head. " Cannot be unseen " 🚨 Some ideas are just sticky, like " faster HTTP " Async or your code will be slower. Wrong! requires Do you, too, work in a company with programs that exceed 100k lines involving synchronous HTTP requests? No way that the company would allow us to rewrite the whole thing to async? Budget and mental sanity reasons? We are here to save the day. Let's do a quick MCQ! According to you, how long does this take to run? We are fetching . The controller behind does a then return a valid output. this resource sleep(3) JSON from requests import Session

responses = []

with Session() as s:
    responses.append(s.get("https://pie.dev/delay/3"))
    responses.append(s.get("https://pie.dev/delay/3"))
    print([r.status_code for r in responses]) A) 6 seconds B) 3 seconds C) 12 seconds ✅ ¡spuoɔǝs 9 sɐʍ ɹǝʍsuɐ ǝɥ⊥ Again! from niquests import Session

responses = []

with Session(multiplexed=True) as s:
    responses.append(s.get("https://pie.dev/delay/3"))
    responses.append(s.get("https://pie.dev/delay/3"))
    print([r.status_code for r in responses]) A) 6 seconds B) 3 seconds C) 12 seconds ✅ ¡spuoɔǝs Ɛ sɐʍ ɹǝʍsuɐ ǝɥ⊥ With enabled, you have just transferred the whole async/await to the remote peer! Isn't it awesome?! No hidden magic trick under the carpet, no async, no threads! multiplexed There's a non-negligible chance you'd rather write 6 lines with a mere ! multiplexed=True The great thing here is that we made it completely transparent for our fellow developers. You are one toggle away from leveraging one or many multiplexed connections! literally 🔒 (III) Enhanced Security TLS 1.2+ was just the starting block Earlier this year, some HTTP clients decided to enforce as the minimum support. We decided to go all in; not only did we enforce TLS 1.2, but we also enforced that Mozilla regularly updates so that we remain safe against sophisticated types of attacks. TLS 1.2 the recommended cipher suite Quantum CPUs are a thing, right? OCSP! No HTTP client is, in fact, capable of providing a basic level of insurance that the peer certificate is in fact (still) valid. With the growing size of issued Let's Encrypt (short-lived) certificates, more insurance is needed. It's not bulletproof as it follows the strategy that most browsers do unless is enabled. fail-safe strict-mode The dependency chain is certified! ! SLSA Almost all the dependencies in the chain are verifiable at any moment. They all benefit from reproducible build. Each of our releases comes with SLSA provenance data (multiple.intoto.jsonl), which can be used to verify the source and provenance of the binaries with the tool. slsa-verifier slsa-verifier verify-artifact ./niquests-3.2.3-py3-none-any.whl --provenance-path multiple.intoto.jsonl --source-uri github.com/jawah/niquests --source-tag Certificates are validated against a trustable source 👇 ( ) Spoiler: Your OS 🪪 (IV) System Root CAs Almost everyone is using Certifi or has seen it pass through your usual routine. In a nutshell, Certifi is a package that ships with a static collection of certificate authorities or root CAs if you prefer. pip install ... Certifi was a mistake, a huge one. A workaround is great to move on, but only for a short period of time. Here's the cons: Security concerns This list of trusted CA is so critical that you should be scared if anything would happen. It is almost never updated by users, so your local copy is most likely too old and . may contain a revoked certificate Does not work at work! Well, Certifi does not ship with your company's certificates! So requesting internal services may come with additional painful extra steps! Also for a local development environment that uses mkcert for example! This is why you should migrate! Our implementation takes away all of those concerns by looking at your OS trust store. Which is constantly updated in the background and protected properly. ⚡ (V) Faster You may get up to 3X faster in your usual synchronous context if you ever switch to . Without multiplexing, well, it is roughly aligned to competitors, except with Requests, we are faster in all comparable scenarios. Niquests HTTP/3 is a bit less performant as of today, but still much faster than any competitor in a synchronous context. We are actively working toward making it faster than HTTP/2. Keep in mind that R̶e̶ has a lot of features and that we made every effort conceivable to keep them! Niquests 🧠 (VI) In-Memory Certificate This one should come with enthusiasm to many advanced users! Python did not allow end-users to load client mTLS certificates and associated keys without having them in a file. Requests did not allow you to pass on your root CAs without a file either. That's ; we elaborated a clever way to work around this limitation. a thing of the past now 🍰 (VII) Object-Oriented Headers We should control ourselves when eating sugar, not when pushing the " sugar syntax " as a programmer. Who did not, at the very least, waste a bit of time accessing HTTP headers from a response? Well! import niquests

r = niquests.get("https://1.1.1.1") Now, how should we access the from header or or ASAP? max_age Nel Report-To Strict-Transport-Security r.oheaders.nel.max_age
# and
r.oheaders.report_to.max_age
# also
r.oheaders.strict_transport_security.max_age You are very much awake; yes, this is now that easy! Pseudo-oriented object headers are long overdue. 😫 ( VIII) Fixed Known (Painful) Issues We've heard you, we know how painful some encounters are, especially with bugs! Here is an extract of what we've fixed for you! (in addition to the In-memory client certificate) An invalid content-type definition would cause the charset to be evaluated to True, thus making the program crash. Given proxies could be mutated when environment proxies were evaluated and injected. This package should not modify your inputs. Fixed wrongfully added to headers when the body is actually of length 0. Transfer-Encoding Function for Windows may be fooled by insufficient control on our end. proxy_bypass_registry Unattended override of manually provided Authorization if .netrc existed with an eligible entry. 🚦 (IX) Type Annotated Knows you are doing right from the start. Powerful IDEs like can easily leverage our definitions to guide you through the code. PyCharm Just spotted something weird. ...And here's why! 📝 Yes, actually typeshed already has some definitions for Requests, but they are (most of them) incomplete. ☎️ (X) Plain Better Cu̶s̶t̶o̶m̶e̶r̶ "Developer"-Care It's a cliché linked to small entities, but, it can make all the difference. You won't feel wronged or thrown away because I am deeply convinced that knowledge is best acquired by good confrontations. It's a promise, we'll never say to go away to Stackoverflow. We will assume this position forever. For us, even the most insignificant Q&A is like a real commitment to the project. We have a great turnover for responses to your concerns, proposals, or criticisms. Reports are good, always. Never be scared of filling in an issue even if you are not sure of anything. Even less to submit a PR. ➕ Bonus Main feature matrix Compatibility is compatible with Python, and PyPy 3.7+, and we remain attached to not dropping any interpreter version unless strictly required to! Niquests Versioning All releases on the same Major version will be guaranteed to be backward compatible. Unless forced to. Did you mention Async? Yes! support , while it's not true Async**, it serves the purpose of having non-blocking code in your event loop! You may combine Async with the usage of a multiplexed connection. Niquests Async Drop-in replacement for Requests You have nothing to do (most of the time) to switch to . A simple on to suffices! Or on to . Any trouble migrating? Come and let us know what happened. We will find a way; trust me. Niquests CTRL+R import requests import niquests as requests from requests import from niquests import Much more you don't know about Many topics (e.g., improvements) haven't been shared in this article; come and discover all the great things we delivered! FYI, Niquests is more resilient to most bot detectors provided you'd put the right headers (e.g., UA, Sec-CH-UA, etc..) **: We ported used in asgiref, as Niquests is mostly thread-safe. It is transparent for the end-user. sync_to_async ***: Not enabled by default. 📆 What's Next? With your support, we can project ourselves much more than 2024! Advanced proxy combination Today, we support tunneling from HTTP/1.1, meaning we can to HTTP/1.1 -> HTTP/2 but not HTTP/2 -> HTTP/2 or even less HTTP/3 -> HTTP/2 or 3. Support for DoH and DoQ DNS over QUIC is soon to be a must-have. when using QUIC. ECH True Async top to bottom Will take some time. Not urgent. Advanced scheduling for multiplexed connections Support for request priority weight. Faster! Faster! We'll never settle. This is just the starting block. — Source: https://github.com/jawah/niquests PyPI: https://pepy.tech/project/niquests Docs: https://niquests.readthedocs.io/en/latest/ — Thank you for reading! In the end, I got what I wanted, migrated all my projects, enabled multiplexing where it was pertinent, and gained a substantial boost.

This post provides insights into new product. 

10 Reasons You Should Quit Your HTTP Client

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

How I Used Python to Solve Declareless Encoding Madness

Using Angular HttpClient the right way

An Introduction to cURL: The Most Popular HTTP Client

What You Need to Know About Advanced Patterns for Symfony HttpClient

HTTP and Web Servers

HTTP For Beginners - Part 1: Definitions

How I Used Python to Solve Declareless Encoding Madness

Using Angular HttpClient the right way

An Introduction to cURL: The Most Popular HTTP Client

What You Need to Know About Advanced Patterns for Symfony HttpClient

HTTP and Web Servers

HTTP For Beginners - Part 1: Definitions

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps