Exploring Redis II: How Redis Stream Works

This is a continuation of my previous post about Redis. In this case, I'm going to cover briefly the use of a new Redis data type called Stream, well maybe not so new but very useful.

Prerequisites

Same as before

Streams, naive description

Have you used logs in your applications? Yep, those messages that you write in console or in a file to keep track of the application's behavior. It can be very useful when trying to know what happened at a certain time of the application life cycle. Well, streams in Redis behave like this. They are like a record of all the events that you decide to append on the stream. I'm not saying that you should stop using logs and start using Redis for this. No, what I'm telling you is that it behaves very similarly. In both cases, you have a stream of data that you want to access at any time, and you could access it multiple times and from multiple clients.

In the sketch, it seems that the consumers can only get the data sequentially, but actually, they can request any range of data in the stream. It's like saying, "give me the date from 1 to 10", you can literally request data in the stream being very specific about what you want to consume. After you consume the data, other consumers could request this same data, so a request for data doesn't "pop" or "delete" it from the stream. Let's see this in action.

Setup

Almost the same setup as in the previous tutorial, the only difference you just need one instance of the redis-cli.

Adding elements to the stream

XADD twitt_feeds * date 01-07-21 username Gealber

Let's split this syntax, which at first could be a little confusing. The first part is the XADD command that allows you to append data to the stream. The twitt_feeds is the name of the stream, the other incoming character *.

Well, this is a way to say to Redis that you don't want to specify an ID of the data by yourself. Instead, you are requesting the server to assign this ID for you. And the last part,date 01-07-21 username Gealber represents the actual data that you want to store. This data comes in pairs, the first part in this case is date 01-07-21, and the other one is username Gealber.

Now go and try it by yourself. You will notice that the server will return you the generated ID. This ID generated had the following format:

<millisecondsTime>-<sequenceNumber>

Generally, you don't want to supply this by yourself, but you can, in case you want to.

Querying the data

Let's say that you want all the data in the stream; how could we request that? Take into account that maybe this is not so smart. It depends on the size of your stream, which you could also request. We'll see how to do that too.

XRANGE twitt_feeds - +

Mmmm... - and +, do these characters have a special meaning? Well, in this particular case, you are saying, "give me the data from the first data in the stream to the last one", so basically, you are requesting all the data.

Now let's say you want to fetch the last 3 data in the stream,

XRANGE twitt_feeds + - COUNT 3

In this case, you are saying, "give me from the last data to the first data in the stream, but only 3 of them", so the "last three elements". Pretty cool, right?

You could also request from a specific timestamp, given the format of the IDs, remember <millisecondsTime>-<sequenceNumber>. You could request elements in a range of time, for example

XRANGE twitt_feeds 1625161017441 1625162015154

These two long numbers represent the milliseconds on the ID here. You are requesting data between 1625161017441 and 1625162015154. In a real application, this is pretty useful. You could request info that happened in a range of times.

Requesting size of stream

There are cases when you may want to know the size of your stream, which could be pretty convenient in case you ask for all the data in your stream and the stream is pretty large. Knowing its size beforehand could make you give up on this idea of "I want it all", instead request in a smart way, but that's just me. If you want it all, you could get it all. Anyway, here it is:

XLEN twitt_feeds

This should return you the number of elements in your stream.

Redis Stream in action, a good resource to look at

I recently discovered this series, Redis Stream in action, and it is fascinating how this guy applies this simple idea into a practical app, definitely something to look at.

Conclusion

I won't elaborate more on this because I think there's better documentation on this part that this article could supply. Really there's a lot of information about this feature of Redis; my purpose was to make you aware of it. Now go and implement some pretty cool applications using this, so you won't forget about it. I will also provide you some documentation that you could consume and learn from it. In the next part, I will use Redis as a reliable queue, awesome, right? 😎

Bibliography

1. Stream Intro
2. How to use Redis for real-time stream processing
3. How to use Redis Stream

Also published here.