This post is about one of the topics we get asked about the most: guaranteeing webhook ordering. At first glance it seems like a simple, and easy-to-implement idea — just send the webhooks in order. Why do people want guaranteed ordering? Before we get over to explaining the challenges, let's first talk about why people want it in the first place. Consider for example a basic billing system where you have two types of entities, a and a . A holds information about the person making the payment, for example, their home address and telephone number. A holds information about the payment method (e.g. a credit card) being used, and a reference to its associated . customer card customer card customer Now imagine you have a form on the website that lets a new customer add their personal details and card information all at once. This form, on the backed will, created them both and trigger two webhooks and . customer.created card.created If you don't ensure ordering, the webhook endpoint may receive either one before the other. So it could potentially receive the endpoint webhook, referencing the , before the endpoint ever got a note of the being created in the first place. card.created customer customer What's the obvious solution to this? Just ensure the order of delivery! Guaranteeing order when webhooks fail The first challenge with ensuring ordering comes when considering failed deliveries. What should happen when delivery fails? Should we block the whole queue of messages? Or should we only ensure ordering when things work and there are no errors? If we block the whole queue, then a single failure when sending a minor webhook would completely distrust webhook delivery, and thus the whole service. Think about it, they won't be getting any messages because of a bug in the delivery of one type of message. This is obviously not good. The alternative then is to ensure ordering only when there are no errors. This is simple enough, but the problem is that we don't really ensure ordering. If your customers need to ensure out-of-order delivery in case of failures, they may as well just process out-of-order delivery in general. It's the same code. So in this case ensuring ordering doesn't add any value. Guaranteeing order doesn't really work The other, more significant, challenge with ensuring webhook ordering is that even if you do send webhooks in order, they may not be processed in order. For example, let's assume we have two events: and , and you send first and second. Now, let's assume the service consuming the webhooks looks roughly like this pseudo-Python code: first second first second def handler_for_event_first(payload): # This will not be a sleep, but some other slow call (db, external API) sleep(1) do_stuff(payload) return HTTP_OK_200 def handler_for_event_second(payload): do_stuff(payload) return HTTP_OK_200 So there are two handlers, one for that's very slow, and one for that's fairly fast. Note: we've used the directive to emulate slowness, in reality it will be some other slow call. first second sleep Looking at this code, it means that even if you send first, it will in practice be processed after and not before. This is because even though the handler will be called first, the function is slower so a good chunk of it will be processed second. In a more real scenario it will probably manifest as a race condition rather when in addition to being out of order, the order will also be very indeterministic. first second We can easily work around this by only sending once finishes processing (we get a 200 for the webhook handler), though this brings forward two additional problems: second first if the handler takes one second to complete (not that rare), our webhook delivery will essentially be limited to one webhook per second, which is terrible, and it'll often be even worse. Best practice when ingesting incoming webhooks is to do some basic validation, put the webhook in a queue for later processing, and immediately return a successful HTTP response. This means that waiting on to finish doesn't actually fix the problem, because unless the queue is processed in order (again, waiting for to finish before attempting to process ), it will suffer from the same issues. first first second So what can we do? As you saw above, even if you put the onus on your customers, educate them about webhooks best practices, and have them try to process everything in order; it's still quite fragile. So what can you do? The best solution is to design your webhooks in a way that doesn't require ordering. One common solution is using what's called "thin payloads", which are essentially webhooks with identifiers and some additional metadata (like which properties have changed), which give your customers enough information about what's going on, but still have them fetch the most recent information using the API. Another solution (which you should be doing regardless), is including the entity's modification date (or modification counter) in the payload, so the customer can check whether this event is newer or older than what they currently have stored. While not recommended, you can also ensure the ordering of events by attaching a monotonically increasing sequence number to events and have your customers ensure the processing order on their end by tracking this sequence number. This is still susceptible to all of the issues described above, but it just shows that ordering can be done even without ordered sending. In conclusion We see a lot of webhooks implementations at Svix, and this question comes up quite often. Though I can't recall even one example where forcing the delivery order of webhooks was the right solution for our customer or their customers. However, as discussed above, there is a solution to the underlying problem. You can design your payloads so that your customers have the information they need to process them regardless of ordering or to follow the ordering constraints their systems were designed to follow. Also published here.
