In the world of Data Analytics and Business Intelligence, the data teams, also called as the “ ”, who build solutions that are needed by business users (red) and work with engineering teams (blue), essentially build the infrastructure for Data. purple teams BI teams predominantly work on building flows or pipelines that deliver reports and essential dashboards for business user consumption. There are many new generation tools that help the data teams build these end-user solutions, like , , and , or industry leaders who were in the “data analytics” space for a while, like Tableau or PowerBI. Mode Superset Lightdash Analysts building these solutions must prepare their data from diverse sources, ensuring the data is sanitized for querying. A set of tools or transformations aimed at performing a cleansing step in the workflow called “Data Prep.” With the advent of large language models, discussing AI has been a common trend across the software engineering stack. But what if I say: Using practices, we could automate the cleansing of the data step? Enabling you to export a cleaner version of the dataset with minimal effort! Data-centric AI In this blog, we will discuss how using Data-centric AI, you can easily prepare your data for BI tools to ensure reliable conclusions from your subsequent data analysis. Data Analyst Workflow Several years ago, data analysts had to manually collect, clean, and analyze data, which was a time-consuming process that limited their ability to gain valuable insights. Today, the data analysis landscape has undergone a significant transformation with the introduction of data preparation tools such as , , etc. Alteryx Tableau These efficient tools have simplified the workflow, enabling analysts to seamlessly integrate data from multiple sources, automate data cleaning tasks, and generate visually appealing and insightful representations of data. Data Analysis After Manual Data Preparation Data prepped using the tools are analyzed using BI tools for identifying specific business queries. For instance, consider of customer requests within a bank where customers log issues they are encountering in a customer service portal, which a human or automated task manager then labels. this dataset Imagine if a business analyst is to determine the number of customer requests appearing for a particular issue category. Below is the result he/she would be seeing - with category showing customer issues. beneficiary_not_allowed 111 Similarly, if an analyst wants to find out how many instances of issues are related to the word , a quick analysis would return below visual representation. Notice the number of issues for the category. ATM change_pin It looks simple and straightforward, but if you dig deeper into , you can find the categorization of customer requests is wrong in a few cases. the dataset For example: Text Label (as per dataset) Label (ideally) My card is almost expired. How fast will I get a new one, and what is the cost? apple_pay_or_google_pay card_about_to_expire Real-world data, for the most part, is messy and unstructured, which makes it hard to deduct values through statistics. As we want humans and machines to make decisions driven by data, it would be critical for the data to be well-labelled, clear of any erroneous data, and de-duplicated. Data-Centric AI It's crucial to ensure that the data used in analyses is accurate, up-to-date, and free from duplicates. Failure to do so can result in incorrect decisions and conclusions. For instance, an empty location field in user profile data or inconsistent formatting of the location field can lead to errors. Therefore, maintaining data quality is critical for effective Data Analytics. Data-centric AI is the discipline of systematically engineering the data used to build an AI system. Most data in the real world is unstructured or labeled wrongly. A quality dataset with the right set of labeled training data leads to an efficient model, which could predict better outcomes. Better outcomes yield a better customer experience. To learn more, you can refer to the course from MIT. Data-centric AI Introducing Cleanlab is an open-source project that helps you clean data and labels by automatically detecting issues in the dataset. Cleanlab uses - based on a paper that talks about estimating uncertainty in dataset labels by Curtis Northcutt (also co-founder of ) and others. Cleanlab confident learning Cleanlab.ai Cleanlab basically enhances a data analysis workflow by inducing AI. Auto-Clean Your Data Using Cleanlab Studio Cleanlab Studio is a no-code tool that is built on top of the open-source package Cleanlab —it helps with prepping the data for an analysis workflow. You can also import data from your data warehouses like , , or Cloud Object Stores like . Databricks Snowflake AWS S3 Step 1: Sign up for access to . Cleanlab Studio You will be logged on to a dashboard with some sample datasets and projects. Step 2: Click on “Upload Dataset” to initiate the upload wizard. You can upload from your computer, URL, API, or a Data warehouse like and Snowflake. the dataset Databricks Cleanlab Studio automatically infers your data schema and modality, i.e., text, image, voice, or tabular. Once you confirm the details, you will be shown a screen with the uploaded dataset and associated errors (if any!) encountered while uploading the data. Note: Some datasets might take a few minutes to upload. Cleanlab will inform you once the dataset is fully uploaded to Cleanlab Studio by Email. Step 3: Based on the type of dataset, you can use a specific machine-learning task to identify problems with the data. Currently, Cleanlab Studio supports several ML classification tasks related to text, tabular, and image data. Specific to classification, it can be one of K classes or one to N of K classes. In this dataset, each customer request falls under a specific category. It would be a “Multi-Class” classification. Cleanlab studio will auto-detect choose the text and label column. You can correct it if needed. Using fast models might not produce the best results; in the interest of the time, choosing Fast is an option. Hit “ ” Clean my data! Step 4: Cleanlab Studio runs an ensemble of models on the dataset and presents an issue overview! As pointed out earlier, the dataset had miscategorized data and outliers, which might not add value to the overall decision-making process when analyzed. You can also take a look at meta-analytics of the issues identified by Cleanlab Studio on the dataset by switching to the analytics view on the top. Step 5: The interesting part of Cleanlab Studio is not just exporting a cleaned dataset but offering an issue-oriented view of your data. The missing data prep workbench that a data analyst and business intelligence user has wanted for years. You can sort each issue by keyboard-assisted actions provided in Cleanlab Studio OR export an “Export Cleanset” by clicking the button below. Data Analysis After AI-Assisted Data Preparation Let us examine the same data analysis with the cleaned dataset. It appears that there are discrepancies in the numbers between the and categories. While this is a smaller dataset, it's important to note that these data corrections could lead to significantly different estimates and potential business decisions at a larger scale. cancel_transfer visa_or_mastercard Similarly, you can find customer requests for some categories disappear as issues are marked appropriately. If you're a data analyst or part of the business intelligence community, Cleanlab Studio can revolutionize your data preparation workflow. Try today, and experience the power of AI-assisted data cleaning for more reliable and accurate data analysis. Cleanlab Studio Conclusion Cleanlab Studio is a no-code Data Preparation workbench used by thousands of engineers, analysts, and data scientists at Fortune 500 companies. This innovative platform was pioneered at MIT to train more reliable and accurate Machine Learning models using real-world, erroneous data. You can join our for more information. Slack Community

This story contains new, firsthand information uncovered by the writer.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

This writer has a vested interest be it monetary, business, or otherwise, with 1 or more of the products or companies mentioned within.

Enhancing Data Preparation With AI for Business Intelligence

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps