Real-Time Data Quality Monitor Earns a 54 Proof of Usefulness Score by Building an Open-Source Data Observability Dashboard

In this interview, we catch up with Pradeep Kalluri to discuss the Real-Time Data Quality Monitor, an open-source solution designed to provide visibility into streaming data pipelines. We look at how this project leverages machine learning to offer a cost-effective alternative to enterprise data observability tools.

What does Real-Time Data Quality Monitor do? And why is now the time for it to exist?

An open-source real-time data quality monitoring dashboard that tracks 6 quality dimensions (completeness, timeliness, accuracy, consistency, uniqueness, validity) across streaming data pipelines. Built with Apache Kafka, dbt, and ML-powered anomaly detection using Isolation Forest, it processes 332K+ orders with sub-10ms latency and 93%+ quality scores. A cost-effective alternative to enterprise data observability tools, potentially saving companies over £100k annually.

Now’s a good time for Real-Time Data Quality Monitor to exist because data complexity is increasing, and engineering teams need accessible, low-latency observability solutions that can catch quality issues before they impact business decisions without the barrier of enterprise-level pricing.

Who does your Real-Time Data Quality Monitor serve? What’s exciting about your users and customers?

Data engineers and analytics teams who need real-time visibility into data quality without expensive enterprise tools like Monte Carlo or Bigeye

What technologies were used in the making of the Real-Time Data Quality Monitor? And why did you choose the ones most essential to your tech stack?

The system is architected around Apache Kafka for high-throughput data streaming and dbt for robust data transformation. To ensure high-fidelity monitoring, the stack incorporates machine learning via Isolation Forest for anomaly detection, allowing the system to identify irregularities with sub-10ms latency.

What is the traction to date for Real-Time Data Quality Monitor? Around the web, who’s been noticing?

The project showcases its capabilities through a live dashboard that processes over 15,000 orders across six quality dimensions with high accuracy. Additionally, the open-source GitHub repository features ensemble ML testing, and the methodology behind processing 332K+ orders has been detailed in a technical article submitted for publication.

What excites you about this Real-Time Data Quality Monitor's potential usefulness?

Most companies discover data quality issues only after they've impacted business decisions. This tool catches problems in real-time at sub-10ms latency, making data observability accessible to teams that can't afford £50k+ enterprise solutions. The ML anomaly detection automatically learns what "normal" looks like for your pipeline, reducing alert fatigue while catching genuine issues.