Data & AI ROI: 2 Key Metrics for Executives

In the rush to embrace AI (artificial intelligence) or LLM, many companies are putting the cart before the horse. As AI tools like GPT and others dominate conversations about improving workflow, there's a real risk of history repeating itself. Remember the machine learning hype a few years ago? It led to a wave of data scientist layoffs when companies realized that developing effective models was more challenging and time-consuming than anticipated.

To avoid falling into the same trap with AI, managing directors and executives need to focus on two key performance indicators (KPIs) that will guide their data and AI initiatives in the right direction. These KPIs will help ensure that your investments in data and AI translate into tangible business value.

Data Utilization

Data Utilization is perhaps the most crucial KPI for managing boards. It indicates how effectively data is being used across the organization and how much of it is gathering digital dust and burning money.

Definition: Data Utilization is the percentage of collected and stored data that is actively used for meaningful purposes.

Calculation: (Amount of data actively used / Total amount of stored data) x 100

• Note: Exclude data that must be stored for legal compliance (e.g., financial or health-related data) from this calculation.

When examining your Data Utilization KPI, ask yourself:

1. Is the data we're collecting and storing adding value, or is it just taking up space?

   I know of companies that collect everyday GBs of data but use only 40% of it, and the rest is to have just in case; I was part of this group in the past, but I don’t believe in it any longer.

2. Can we aggregate or delete old, unused data?

   Many companies have been keeping raw data for over a year. Does this make sense? Old dashboards and reports that were created, used once, and since then forgotten.

3. Is data going unused because it's inaccurate or irrelevant?

   Collecting wrong or missing data is a waste of effort and money and requires the attention of the management to address the wasted efforts, we are talking here about efficnese or as Winnie-the-Pooh says, “A fish in the sea?”

4. Are there untapped opportunities to leverage our existing data?

   We collect so much, we don’t use it, are we missing something, can we use it to improve our models? Can we use it to enhance the user experience?

Example: I suggested this metric to a friend who worked in a gaming company; his outcome was that only 45% of its consented customer data was being actively used (The actively used definition they set was the data was accessed in the past six months) for personalization and marketing. Since they applied the KPI as part of the company health metrics dashboard, they have managed to save 30% in their data costs by removing outdated, bad, and irrelevant data, they increased the usage of data to 85%, and relief many teams from collecting data without a purpose. The best part was that they helped the ML team to focus more on their models by leveraging hidden data because one of the side effects was a full data catalog that was available for the entire company.

Data ROI

Data ROI is a vital indicator of the return on your data investments. It helps you understand whether your data initiatives are generating value or burning resources.

Definition: Data ROI measures the financial return generated by data-driven initiatives relative to the investment in data collection, storage, and analysis.

Calculation: (Value generated from data-driven initiatives - Cost of data initiatives) / Cost of data initiatives

• Note: We quantify here fixed costs of the employees, the storage of the data, the processing costs, on the other side we calculate the uplift of having the data.

When assessing your Data ROI, consider:

1. Are our data products (like AI models or dashboards) helping users find what they need?

   Are we generating money with the data, or maybe we save costs, the data utilization for example is a cost saver that can generate money saved each month.

2. Are we maximizing our financial investment in data to generate profit?

   We pay for expensive services to store, process, and visualize our data. Does that help us use the data better to gain profit? Maybe actually we lose money.

3. Why are we processing or producing data without clear value?

   If we have a dashboard, do we know what is the value of having it, how much money does it save us or increase for us? If not, does it make sense to look at numbers just for the numbers?

4. Can we optimize our data operations to improve ROI?

   Where can we cut or reduce the load on our computing systems to save costs or increase the speed of other, more relevant data?

Example: I mentored the COO of an e-commerce company on implementing the ROI KPI. Here's how it played out:

Initial Implementation:

• They applied the KPI to their main data stream, not the entire company data, due to time and people limitations.
• The recommendation model generated a 12% uplift in sales, around $79K.
• With model costs at only $14.4K per year, the initial ROI was an impressive 464%.

Evolution and Adaptation:

• As the model grew and consumed more data, the ROI decreased.
• The uplift wasn't as significant, while costs increased.
• The ROI calculation dropped to 75%.

Result: The ability to track this KPI allowed the team to quickly identify the declining ROI. They were able to react swiftly, developing and testing a different model to improve performance."

To start tracking these KPIs:

1. Conduct a data audit to understand what data you have and how it's being used.
2. Implement data governance policies to ensure data quality and relevance.
3. Invest in data cataloging and metadata management tools to make data more discoverable and usable.
4. Develop a system for tracking costs and benefits associated with data initiatives.
5. Regularly review these KPIs with your data and business teams to drive improvements.

The Path to AI Success: Avoiding Pitfalls and Reaping Long-Term Benefits

Recent studies underscore the growing importance of AI in business strategy while also revealing a significant gap in effective implementation; I tend to think the AI strategy should be part of the data strategy but needs to be a significant part of it going forward by setting the vision to how it will be used by the organization to increase profits. Gartner's 2023 CEO survey shows that 82% of CEOs are increasing investments in digital initiatives, including AI and data analytics. However, only 31% feel adequately prepared to use these technologies effectively. Similarly, Deloitte's State of AI in the Enterprise report reveals that while 94% of business leaders see AI as critical for future success, only 27% describe their organizations as truly AI-fueled.

These statistics highlight the need for better metrics to guide AI and data initiatives. By focusing on Data Utilization and Data ROI, companies can avoid common pitfalls and position themselves for long-term success.

Common Pitfalls to Avoid:

1. The 'collect everything' syndrome: Many organizations fall into the trap of amassing vast amounts of data without a clear purpose. This leads to bloated storage costs and reduced data quality, ultimately hampering AI effectiveness.
2. Vanity metrics obsession: It's easy to get caught up in impressive-sounding AI capabilities that don't translate to business value. Remember, the goal is not to have the most advanced AI but to have AI that drives tangible results.
3. Ignoring hidden costs: Failing to account for the full cost of data initiatives, including storage, processing, and maintenance, can lead to misleading ROI calculations. This can result in continued investment in projects that aren't truly delivering value.

Long-Term Benefits of Focusing on Data Utilization and ROI:

Consistently monitoring these KPIs doesn't just provide short-term insights; it cultivates a data-driven culture that can yield substantial long-term benefits:

1. Improved decision-making agility: By focusing on utilizing the right data effectively, organizations can respond faster to market changes and emerging opportunities.
2. Enhanced customer experiences: More targeted and effective use of data leads to better personalization and service, driving customer satisfaction and loyalty.
3. Optimized resource allocation: Ensuring investments in data and AI initiatives deliver maximum value allows for more efficient use of both financial and human resources.
4. Competitive edge in talent acquisition and retention: Data professionals are drawn to organizations that demonstrate a commitment to effective data practices, helping you attract and keep top talent.

By avoiding these pitfalls and focusing on the long-term benefits, managing directors can guide their organizations towards true AI success. Remember, in the age of AI, it's not about having the most data or the fanciest algorithms – it's about leveraging these tools effectively to drive business success. Data Utilization and Data ROI are your compass in this journey.

Looking ahead, the importance of these KPIs will only grow. As AI becomes more sophisticated and pervasive, the ability to effectively utilize data and generate tangible returns will be the key differentiator between market leaders and laggards. Gartner predicts that by 2025, 70% of organizations will shift their focus from big to small and wide data, making AI more flexible and improving the decision-making process. This shift will make Data Utilization an even more critical metric to track.

I see in the near future a rise of a new function. I hope not, but someone will have to be the Data Operation Manager, who has a full look at the data and how it’s utilized; maybe it’s the new future for the CDOs (Chief Data Officers) that in recent years have lost the shine and less and fewer companies hiring them. Maybe this is where everything went wrong, thinking of the CDO as an engineer, the analyst, and the owner of the data, while he really needed to be an operation function with knowledge of data.

Conclusion

As Tom Davenport, distinguished professor at Babson College and MIT, notes, 'The key to getting value from AI is to focus on the business outcomes, not the technology itself. Managing directors must look beyond the hype and focus on extracting real value from their data. By monitoring Data Utilization and Data ROI, you'll ensure that your organization's data and AI initiatives are built on a solid foundation, driving genuine business impact rather than just following trends.

Remember, the goal isn't to have the most data or the fanciest AI models – it's to leverage data and technology to drive business success. Start measuring these KPIs today, and you'll be well-positioned to lead your organization into the true era of data-driven decision-making.

Call to Action

I'm passionate about helping organizations unlock the true potential of their data initiatives. If you're interested in learning more about implementing these KPIs or discussing how they can benefit your specific organization, I'd love to hear from you. Reach out to me directly on Linkedin for a personalized consultation. Let's work together to optimize your data strategy and drive tangible results for your business.

For more insights on data strategies and ROI models, check out my previous articles on Hackernoon. Your journey towards data-driven success starts with a conversation – I'm here to guide you every step of the way.