What work does a data engineer actually do? Let me tell you one thing: it’s not what you think they should be doing, especially not the part where they are running around collecting data for you or building yet another one of those dashboards that will only be used for a few weeks.
“We can’t control spending on campaigns because we don’t have any data.”
“I wasted hours combining this report in Excel because I don’t have the data I need.”
“I've asked for this report five times now and I still haven't received it.”
"I don't have any user visability, I don't know what they do"
“The data was wrong and now I've made some bad decisions.”
“WHERE IS MY DATA?!”
I've heard these so many other complaints over the years, blaming the data team for not functioning properly, and they prompted me to ask the question: what is an analyst's job?
Before you hire another data engineer/analyst or you decide you need to hire a hands-on analyst, data engineer or other data functions, here are a few points that you will never hear from your data hires themselves, and this is why we never get the right data when we need it. When the data fails, please don’t blame your engineers/analysts again – they're limited in what they can do and we both know it.
1. API connectors
Don’t ask them to build you more API connectors. One of the jobs analysts and data engineers hate the most is building API connectors to Google ads, Facebook ads, Google analytics and whatnot – they just hate it.
Most don’t even think it’s their job to actually collect the data. Now, I must be honest here; they're wrong. It is their job, but luckily, there are so many services out there that can help you collect and store data. There are even services that you can use on your own AWS without needing to share your data with a third party.
2. Dashboard Revision
You can’t expect them to keep adapting your dashboard after your second revision. Yep, that's right – you had your chance and you failed to build your data strategy.
It's not their job to keep adapting it to keep you and your boss happy with your gazillion KPIs that you'll use perhaps once a year. Dashboards need to be action-driven. Design them cleverly and use them for at least three months before you change them, and for God's sake, learn how to build a simple dashboard yourself already!
3. They are not mind readers
No, analysts are not mindreaders; you can’t expect them to read your mind. They won't know what you did or didn't mean when you asked or forgot to ask, “Ahh I can’t filter it by country?” If you didn’t write down that you needed a country filter, well, you won’t get one. It doesn’t mean that you have a bad dashboard, it's just not what you wanted because you didn't request specifics. Next time you ask for a dashboard, make sure your request includes everything you need – don't assume someone will just read your mind and add the measures or dimensions you didn't write down.
4. Repeat dashboards are not analysis
Dashboards are for making decisions quickly, while analysis is for investigating. Now, dashboards should only be changed every quarter, or whenever you feel the data has become irrelevant or your data universe has expanded so much that now you think you could be using it to make better decisions. Analysis, on the other hand, needs to be a one-time deep dive to get a direction, so it needs to be defined correctly. If you don’t know how, take a look at the article I created about this a year ago. We tend to think that a one-time analysis should feature on our dashboards, but in reality, we don't use them on a daily or weekly basis – sometimes not even monthly – so there is no reason for it to be on our dashboards permanently.
5. Data costs money!
Yes, running queries and obtaining data costs money. The more data you request, the higher the costs are on your infrastructure. With the size of data increasing, data engineers have found ways to reduce costs, often by storing data in “graveyards”, but if you're asking for MAUs, DAUS or any other KPI that requires the processing of a lot of data, it will cost your company serious money.
6. Learn to build dashboards
You should build your own dashboards. It's an analyst's job to run the analysis for you and help you find the hidden gems. If they're busy building dashboards all day, they'll never be able to help you find your growth levers. The same goes for a super busy data engineer. While his job may be to make sure the data arrives, QA the data and inform you if anything goes wrong, his main priority should be to think how he can improve user privacy and ensure data is being used for its intended purpose.
Free your engineers to do what they care about
To sum it up, when working with engineers, analysts, and data scientists, you should be working together, not apart. You should involve them in everything you do, whether it's thinking about a new feature or changing your approach to marketing. This will help to ensure they are always aware of things. All those times in the past when we killed a feature or stopped a campaign, the data team may have flagged the data as being wrong or “looking strange”, investing hours into understanding what happened in the process. The problem was that the data team wasn't made aware of what was going on.
Free your engineers to do what they care about – and no, that's not building dashboards or setting new APIs. Allow your analyst to find the gems in your data, gems that can help you turn your business around. They'll thank you for that.
Originally published at https://www.taleaboutdata.com/