Dashboard fatigue is a real thing. I’ve experienced it first hand! Before Agoda, I would’ve told myself that there’s no such thing. But after Agoda — well now I know there is too much of a good thing.
At Agoda, we liked to experiment. We were driven by data. But how much is too much? In this post, I’ll talk about my experiences at Agoda to do with Dashboards and data. Say, for example, a dashboard monitoring sign ups — your KPI (Key performance indicator) — do you look at it every day? When does it stop becoming relevant and start becoming something you don’t care about? What about if I added four more dashboards about other monitoring KPIs? Would they all still be relevant? Would you be looking at them every day?
Part I — New and Shiny
When I first joined the Agoda Homes team, we were a new product and were building it from scratch. We didn’t have dashboards because our KPI was to build the onboarding process. Just a side note: if you’ve ever signed up with Agoda Homes to list your Home (like Airbnb), that was built by my team. And then we finished. But now we needed to find a way to display how many sign-ups we were getting a day. We should build a dashboard. Most other teams have them! So, we used a platform called Grafana and made our dashboard to show sign-ups over time.
In the first few weeks — we were all very excited and interested. How many sign-ups did we have yesterday? How many today? Wow, it’s increasing! Our product is working. Awesome! This dashboard was displayed on TV hanging on a wall. Agoda has several TVs around each team so you can always glance at your dashboard to see how your product is performing. We were, of course, very proud.
Part II — Add Add Add
It wasn’t just enough to know how many signs ups we were doing. As we’re a developer team, we also needed to understand how our product is performing regarding speed. What APIs are working fast and what is working slow? What is the First Meaningful Paint? When do you put the critical mark — the point at which your application is first usable? Where do users drop off in the funnel? How many users add photos? How many users edit their properties?
So you can already see, we were adding a lot of dashboards. These weren’t all added at once, but every time we added a new feature, we usually thought about how we can add or improve our dashboards. There was a lot we were monitoring. And it eventually got too much. Having your TVs flick through five different monitoring pages makes you switch off and stop watching.
What Happened Next
This is something that shouldn’t happen — but did. We didn’t put alerts on our dashboards because we were so sure someone would be monitoring it. The Product Owner thought the development team was monitoring it. The development team thought that person X and QA were monitoring it. Person X believed that the rest of the team was monitoring. Etc. Etc. There is no one to blame but ourselves as a collective.
What happened was that one day our sign ups just dropped. No one noticed. No one reacted. It wasn’t for another couple of days until finally, someone saw the odd trend in our dashboard. We had dashboard fatigue. We were no longer paying attention to all the data we were collecting. It was overwhelming. We fixed the bug — but probably lost a couple of customers in return.
Advice for Future Me
Concentrate on having one focused dashboard that’s as simple as possible. Colours representing states makes your life easier and means that the dashboard can be understood at a glance. Avoid bar graphs and having dashboards too detailed (unless they are for following up on a problem). Add alerts for when something goes seriously wrong.
Originally published at www.alexaitken.nz on August 6, 2018.