"Data is the new oil. It’s valuable, but if unrefined it cannot really be used." -Clive Humby
I recently became very interested in Data Science and Data Engineering; how they compare and complement. I initially assumed Data Engineering was a subset of Data Science but after extensive research, I found out just how much the two fields differ.
In this article, I hope to discuss the differences and similarities between data science and data Engineering.
Data
To fully understand the relationship between Data Science and Data Engineering, you have to understand the one thing that links them both; Data.
Data is a word that has become commonplace in today's society, with so many reports of data leaks, the inappropriate collection of data by big tech companies, and so on.
Data is information that is collected and stored in a format that can be processed by a computer. It can be in various forms such as numbers, text, images, and videos, and it can be collected, stored, and analyzed to extract insights and inform decisions.
Now, why do so many companies want data, and what's so special about it?
Data is important to companies because it allows them to make informed decisions about their operations and strategies. By analyzing data, companies can gain insights into the behavior of their users, and insights gotten from their users can then be used to make their products way more efficient and useful for users.
Data scientists and engineers are the people responsible for collecting the data, making it useful, analyzing it, gaining insights & trends from it, and passing on the information mined to the management in order to permit informed decision-making. Now let's see how they differ.
Data Science
Data Science was termed The Sexiest Job of the 21st Century by the Harvard Business Review and its claim to the title is arguably legitimate. Data Science is the process of using scientific methods, algorithms, and systems to analyze and extract value from data.
In other words, the data scientist is the individual responsible for gaining insights from data and making abstract mathematical models from the data in order to enable prediction.
Now let us look at the data engineer.
Data Engineering
Data Engineering is the process of designing, constructing, and maintaining the pipelines and infrastructure that collect, store, process, and analyze data.
The Data Engineer is the individual responsible for ensuring that data required by Data Scientists to analyze and gain insights from is available in the right and accurate format. Data is infuriatingly complex and disordered when it is collected and in order for Data Scientists to efficiently gain insights from it, the data needs to be pre-processed once insights have been made, Data Scientists then formulate an abstract mathematical model from it which is commonly known as a Machine Learning Model and this said abstraction needs to be post-processed in order to be deployed and integrated into the product. All the tasks described are performed by data engineers.
An analogy to describe the relationship between the Data Scientist and the Data Engineer
Imagine you placed a bet with a friend on the outcome of a football game but you wanted to cut out the luck factor, which is ever so present in uninformed guesses, and be extremely sure that the team of your choice wins the game and you win the bet.
A data engineer would collect the data on the two teams involved in the bet, data points such as; the number of games won, possession rate per game, and results of previous clashes between the two teams, and create an ETL pipeline where the data would be collected, cleaned and stored for the data scientist.
The Data Scientist would then perform something called Predictive Analysis using Machine Learning; this means the data scientist would simply feed the data prepared by the data engineer into an algorithm that then generates a mathematical Abstraction called a Machine Learning model, the Machine learning model then predicts the team expected to win the bet, and just like that your guess becomes less of guess and more of a data-informed decision.
Summary
As you can, hopefully, extrapolate from the description between Data Scientists and Engineers above, a Data Scientist is similar to a star football player and a Data Engineer is like his very talented coach who keeps him fit and provides him with tactics to win a game.
Also published here.
