Too Long; Didn't Read
In this article, we first explain the requirements for monitoring your big data analytics pipeline. We go into the key aspects that you need to consider to build a system that provides holistic observability. This article covers some key ideas to gain observability into big data applications. The user personas for the data application observability can be broadly bucketed into the categories described below. Each kind of user has different requirement for data freshness, query latency, type and granularity of information. These insights lead to:lower infrastructure cost by improving resource utilization.increased user productivity by ease of debugging and troubleshooting.