Data Observability: The First Step Towards Being Data-Driven

I started my career as a first-generation analyst focusing on writing SQL scripts, learning R, and publishing dashboards. As things progressed, I graduated into Data Science and Data Engineering where my focus shifted to managing the life-cycle of ML models and data pipelines. 2022 is my 16th year in the data industry and I am still learning new ways to be productive and impactful. Today, I am now the head of a data science & data engineering function in one of the unicorns and I would like to share my findings and where I am heading next.

When I look at the big picture, I realized that the problems most companies face are quite similar. Their vision towards being data-driven has turned into a BHAG — pronounced “bee hag” (Big Hairy Audacious Goal).

We data folks like patterns, so here are my findings:

1. During 5 out of 10 review meets, I have witnessed people question the reliability of the data/report/dashboard. Additionally, HODs will also try to convince others that their data is the most accurate or reliable :)

   
   

2. A lot of times, HOD comes and says that the data is not updated. The data team is already working to fix the report/data table.

   
   

3. A new product got launched the week before, however, we are yet to figure out the performance. The data team is working on a query change and will soon update the CXO team.

   
   

4. Everyone has built expertise around writing complicated ML (machine-learned) models, however very few talk about or deploy inference monitoring. There is a high probability of model drift or performance drift in the coming weeks/months if not monitored or observed efficiently.

   
   

5. Very few companies deploy solutions or models to detect performance anomalies.

   
   

The list is long, I am sure you can relate or add more to this.

In a nutshell, I found that data reliability is a BIG challenge and there is a need for a solution that is easy to use, understand, and deploy, and also not heavy on investment.

I am Jatin Solanki and I am on a mission to build and develop a solution to make your data reliable.

What is needed to make your data more reliable?

Complexities around data infrastructure are surging as companies gear to get a competitive edge and out-of-the-box offerings.

Every company goes through a data maturity matrix. In order to reach a level where you deploy AI models or self-service models, you need to invest in a robust foundation.

In my opinion, the foundation begins with a reliable data source or defining source of truth. Your data models won’t be impactful if it’s ingested with bad data. You know it’s garbage in garbage out.

On a high level, here are a few checks you can implement to ensure data reliability:

• Volume: It ensures all the row/events are captured or ingested.

  
  

• Freshness: Recency of the data. If your data gets updated every xx mins, this test will ensure its updated and raises an incident if not.

  
  

• Schema Change: If there is a schema change or a new feature that was launched, your data team needs to be aware to update the scripts.

  
  

• Distribution: All the events are in an acceptable range. e.g if a critical shouldn’t contain null values, then this test ensures to raise an alert for any null or missing values.

  
  

• Lineage: This is a must-have module, however, we always underplay these ones. Lineage provides a handy info to the data team of the upstream and downstream.

  
  

• Reconciliation: I would say recon or finding deltas between two given datasets. This could be used to understand the difference between stagingand production OR between source and destination . This could be effective in running some financial recon too, like payment gateway to the sales table.

• 
  

What next? How do we implement this?

The most common question people face with:

Build versus Buy

I am a big fan of open-source tech, however, in some critical modules, I prefer buying an out-of-the-box solution because it’s scalable and already tested in the market. Developing in-house might cost you around US2k per month and it includes a few hours of engineer’s time along with cloud cost.

If you are inclined toward buying an out-of-the-box solution, here are a few factors that should be part of your checklist.

• Should be able to connect to popular sources which require minimal config.
• Extract information automatically without the need for additional code.
• No-code or CLI (I leave it to you)
• Lineage and Catalog module.
• Data Reconciliation along with scheduling feature.
• Anomaly detection
• Of course, Of course, all the tests we discussed earlier along with alerts should be in a position to tell where to debug.

A robust platform provides easy access to all the incidents and also evaluates the data health.

It should be in a position to automatically detect my critical data assets and apply hygiene checks.

The only platform to group alerts instead of pushing 100+ alerts.

At last, the solution should help you reduce data quality incidents and make your data more reliable.

So, do I need a data observability platform?

If your answer to any of the below questions or scenarios is “Yes”, then you should procure or deploy a data observability solution right away.

1. Dashboard not getting updated on a regular basis?
2. Don’t know which report is accurate?
3. Business stakeholders are the first to learn about data incidents.
4. Questions during a meeting on the performance stats.
5. Have at least 2 members in the data team.
6. Deployed a business intelligence tool.

As software developers have leveraged on DataDog, Dynatrace, etc kind of solutions to ensure web/app uptime, data leaders should invest in data observability solutions to ensure data reliability.

Also published here.