Too Long; Didn't Read
At Highlight, we face a unique challenge when it comes to adapting to the load of our customers' web apps. Because we record web apps, we need to be able to handle bursts of traffic as they come from their users. We had to run some database migrations that ran an ALTER command with an aggressive table lock. The initial approach worked fine for a while. But as our customer base grew, we ran into key issues with this implementation. We needed to find a way to buffer data, we settled on a producer-consumer message strategy using Apache Kafka.