Building a Simple HTTP Client Using Proxygen

This article is about proxygen - a library used to build all kinds of servers. It is open-sourced and maintained by Meta. Initially, framework was envisaged as a collection of tools used to develop proxies. Hence proxygen. However, this library is so much more.

In the first article in this series, we will go through developing the most simple HTTP client possible. There are a lot of moving pieces so I prefer to keep complexity to a minimum and only focus on what is important. If you’re reading this, you probably have a good head on your shoulders and you will be able to expand on this on your own.

First, let’s start with very general proxygen architecture. Data is either read from a socket or written to a socket. We then have HTTPSession that uses HTTPCodec to read/write data to/from the socket. Each HTTPSession will have one or more HTTPTransactions. Each HTTPTransaction interacts with a Handler as data is received/sent. You can see this illustrated in the image below.

This very simple, yet powerful set of abstractions allows handling all sorts of other protocols like HTTP2, HTTP3/QUIC/SPDY including own “home” grown protocols. But before you start working with any of these you’ll probably want to build the most basic application to make sure the project is set up correctly and that you understand all relevant concepts. I personally found this to be the most challenging part. Even though proxygen comes with it’s own build script and a set of examples, I didn’t find this the most “friendly”. Porxygen relies heavily on some other libraries (e.g. folly, libevent, glog, gflags…) so you will need to make sure that those are present on your system too. You might also want to customize which compiler and/or linker is used. I personally like to use clang compiler and toolset that comes with it.

In order to help with the above problems, I created a repository that does the following:

• introduces most dependencies as git submodules

• provides a build script to build all dependencies

• implements a simple HTTP client (KissClient, Kiss stands for keep it stupid simple)

• provides instructions how to build the client

  
  

Caveat: the project currently only works on linux. At the moment I don’t have plans to make it work on MacOS X or Windows. Having said that, I am open to accepting a pull request.

This repository should be enough for you to fork and go make your own project based on

proxygen.

If you run into any problems, please open an issue (https://gitlab.com/technologysoup/starter-kits/hello-proxygen/-/issues) or reach out via discord (https://discord.gg/vFbZFzMmDK)

Now let’s walk through the example. I strongly suggest you open code and reference it as you read the rest. The code is available in git repo:

https://gitlab.com/technologysoup/starter-kits/hello-proxygen/-/tree/main/src/kiss_client

KissClient is the class where we implement the client. It extends two other classes - proxygen::HTTPConnector::Callback and proxygen::HTTPTransactionHandler.

proxygen::HTTPConnector is used to establish new HTTP or HTTPS connections while proxygen::HTTPConnector::Callback defines callback object which will receive results. The callback object has connectSuccess and connectError methods.

HTTPTransaction represents one request/response pair in an HTTP-like protocol. It works with a Transport and a Handler to handle ingress and egress of data.

When a session is established connectSuccess will be called with a session object which in turn is used to get a new HTTPTransaction. That transaction object (txn_) is used to send headers and body.

Once the client finishes sending content (headers + body), server will send a response. At that point hooks which are implemented as part of HTTPTransactionHandler interface will be called - onHeadersComplete, onBody, onEOM,etc.

To summarize, the sequence of events is:

1. The application creates a callback and a handler object.
2. The application then creates HTTPConnector passing it the callback and connects to an address.
3. Once the connection is successfully established an HTTPSession is created using the callback hooks.
4. The session is used to create a new HTTPTransaction. The HTTPTransaction is initialized using handler object.
5. The transaction is used to send request to server.
6. Once the server response is received handler hooks are invoked.
7. When the session goes idle (meaning there is nothing more to exchange between client and server) the session is closed.

And that is it. We have a working HTTP client. You can test it by spinning up some server or using netcat.

$ ./hello-proxygen
I0606 23:00:19.731132 1463353 kiss_client.cc:44] Sending request for http://localhost:4586
I0606 23:00:19.731266 1463353 kiss_client.cc:21] Got headers for http://localhost:4586.
I0606 23:00:19.731276 1463353 kiss_client.cc:24] Got body for http://localhost:4586.
I0606 23:00:19.731279 1463353 kiss_client.cc:31] Got EOM for http://localhost:4586.

In my next article, I plan to show how to make a simple HTTP server.

If you have any questions feel free to drop a comment below.