This is the second episode of the ReST series! While the first episode centered around semantics, this installment delves into the ways in which request and response headers can enhance interactions, making them more meaningful and comprehensive. For the sake of a focused narrative, we'll concentrate on a common issue: the pagination problem.
In this episode, our primary focus will be on understanding how HTTP Range headers can be employed to address the challenges associated with pagination. By narrowing our scope, we aim to provide a clearer and more in-depth exploration of this specific aspect of RESTful API design.
What Is Pagination?
Simplistically, pagination means splitting large datasets into comprehensible smaller sets and providing means to navigate across these smaller sets. An example would be, say, in some web application, a search for users yields thousands of results.
It would be impractical to overwhelm the poor soul with all results at once. First of all, the convenience of searching for large datasets is no longer a convenience, rather it is a nuisance.
Now, this post may also come as a rant as there are multiple ways to implement it, and the way I feel it should be is not possible with current specifications.
Pagination Using Query Params
OData specification has support for pagination, and it makes use of query parameters to implement pagination. A typical OData interaction can look as below.
Request:
GET /users?$skip=0&$top=10&$inlinecount=allpages
Response:
200 Okay
X-Some-Side-Channel: count=200
Now, that’s an example of repurposing. Here, the query string is repurposed for pagination. It sounds a bit weird from the get-go. The query string, the name itself, states its purpose “for querying” and is repurposed.
There is a header prefixed X-, and such headers are called custom headers. User-agent needs to understand it, or there is a need for customization in the form of coding at the client end.
Can We Do Better?
RFC2616 had a section on content ranges as below.
14.35.2 Range Retrieval Requests
HTTP retrieval requests using conditional or unconditional GET
methods MAY request one or more sub-ranges of the entity, instead of
the entire entity, using the Range request header, which applies to
the entity returned as the result of the request
Does that sound similar to pagination?
Let’s try to model the earlier OData request using Range Retrieval Requests. Ranges are resource-specific meaning /users may be a collection of more than one, user data. A single user is a unit in the collection.
Let’s Ask: What Is a Unit for Users’ Resource?
Request:
OPTION /users
Response:
200 Okay
Accept-Ranges: users
So, the web application is telling us that users is a unit to specify the range of users. Now, since we got to know that users is the unit, let’s ask for the first 10 users.
Request:
GET /users
Range: users=0-9
Response:
206 Partial Content
Accept-Ranges: users
Content-Range: users 0-9/200
[ 0, …, 9 ]
How does this dialogue work?
Request:
In the request, the user-agent has added a Range header mentioning a specific unit discovered from earlier OPTIONS dialogue and a range in numeric form. Natural, isn’t it?
Response:
The web application responds with 206 clearly stating that the response is partial. Content-Range provides details about the data in the response. In the example, 0-9/200 indicates the first 10 users’ data is being returned out of 200, the number of users satisfying the search query.
It also reiterates the unit being users as Accept-Ranges header. Since Accept-Ranges is reiterated, the OPTIONS call can be avoided altogether as the web application can simply default to sane defaults for the range parameters if the request did not send the Range header.
At times, it might be very intensive to compute the total number of records in the returned collection. In such cases, one can simply respond with * as count. So, the response looks as below.
Response:
206 Partial Content
Accept-Ranges: users
Content-Range: users 0-9/*
[ 0, …, 9 ]
You can ask for multiple ranges too.
Request:
GET /users
Range: users=0-9,35-50
Response:
206 Partial Content
Accept-Ranges: users
Content-Type: multipart/mixed; boundary=PART
--PART
Content-Range: users 0-9
[ 0, …, 9 ]
--PART
Content-Range: 35-50
[ 35, …, 50]
What if the data requested is beyond the range?
The web application can simply state that the data is not available.
Request:
GET /users
Range: users=0-9,35-50
Response:
416 Requested range is not satisfiable
But (The Rant Part)
All this discussion about Range headers might feel like a lot of talk. The potential usefulness of Range headers is, for now, mostly theoretical. While originally mentioned in RFC 2616, the specification explicitly mentions only bytes as the unit for specifying a range. It doesn't necessarily rule out the possibility of using other units, but it also doesn't affirmatively state that it's allowed.
Even in the latest specifications on HTTP, namely RFC7231 and RFC7233, the stance remains unchanged. In reality, the sad truth is that as of now, there are no HTTP servers with native support for custom range units. It's like having a tool in the toolbox that looks promising on paper but turns out to be rarely used in practice.
Whether this will change in the future or if the theoretical potential of Range headers will forever remain just that - theoretical - only time will tell.
Further reading: