What is Ad hoc Analysis and Reporting, and Why Should you be Careful with it?

This article originally appeared on the 3AG blog.

Ad hoc analysis (definition):

In a business context, ad hoc analysis is some form of analysis, often performed by non-technical users to answer a particular business question. With the proliferation of various self-serve BI tools, ad hoc analysis has become much more accessible to broader business users.

Ad hoc reports (definition):

Ad hoc reports are reports developed in response to a specific business question or problem. They are generally not meant to be used beyond their intended analysis, hence the use of the latin term ad hoc "for this".

So what is ad hoc analysis and ad hoc reporting, really?

In a world where businesses live or die on data, self service BI is the information super-food we’ve been waiting for. Or so the rhetoric around this popular tool would have us believe.

So what is ad hoc analysis? Exactly what the name implies — reporting prepared for a specific or immediate need.

Also known as data exploration, exploratory data analysis and data discovery, this form of self service business intelligence has many names, but one big promise: to radically reduce the time organizations spend on data collection, analysis, and reporting.

Central to this promise is that any employee who wants to can collect and analyze company data on their own. Regardless of training or experience, anyone is supposed to be able to quickly become proficient in data collection, organization and analysis.

Does the promise reflect reality? And do organizations benefit from employees’ widespread use of self service business intelligence tools?

What self service BI does well

Maybe; it depends on what you want it to do. Ad hoc analysis is:

• agile and easy to use; no one will waste much time figuring out how to use it
• readily accessible because most such tools live in the cloud
• flexible: users can interact with ad hoc reporting in many ways (e.g., drilling down or through data on customizable tables, charts, and crosstabs)
• built from reusable components, so it’s easy to create more and more ad hoc reporting, or have multiple reports interact

These are all good things. However, they can’t guarantee organizations will gain either full understanding of their business data or consistent, superior value from it.

Where the ad hoc analysis promise fails

The fact is, the primary attraction of self service BI tools—making ad hoc reporting both easier and more accessible—is also what makes it potentially dangerous. Why?

They don’t go far enough. Say a company selling umbrellas sees sales increase dramatically one week; using exploratory data analysis, a team learns this spike resulted from unusually rainy weather. Too often, teams will stop at explaining a past event such as this and think they’ve gained valuable information. Most of the time, this approach merely confirms something already known or easily guessed, without running a report at all.

They’re treated as official reports and widely shared. Too often, ad hoc reports are confused with official ones; they may become corporate gospel, especially if they’re nicely formatted; they may then be used to support major business initiatives. The only reports organizations should consider official are those planned by qualified experts for specific or regular reporting needs (e.g. monthly reports).

An ad-hoc approach to data analysis can be extremely subjective. If everyone in an organization can imagine and create ad hoc reporting, odds are some will be severely skewed by confirmation bias: that is, when a person stops researching when the information they’ve already gathered supports the result they want.

Considered in light of the DELTA Plus Model, the above describes an organization displaying level-two data maturity: Able to analyze data to explain certain observations, such companies also own much useful data. But this data is usually siloed, its disparate pieces accessed by team members with strikingly different skill sets who don’t communicate with one another.

Self service business intelligence may ultimately, despite the hype, leave a bad taste in your mouth. It absolutely won’t help you build a smart, sophisticated, data-led organization. And it certainly won't help your team find ways to optimize the business and better understand customers.

Empty calories

Articles addressing self service BI tools generally focus on how it helps users with little or no data analysis knowledge very quickly become data professionals; promises of fast results abound.

Fast results aren’t necessarily good results, however, just as fast food isn’t necessarily good food. Fast food may not harm one person, when eaten in moderation; but when a community consumes it often or exclusively, health issues will inevitable result. Likewise, ad hoc analysis might not harm an organization if one person does it occasionally.

But when a company’s predominant approach to mining and analyzing data is to have staff, regardless of role or expertise, create ad hoc reporting, that company will suffer. It will suffer, for the time and the financial resources wasted, as well as for the high potential for making poor decisions based on such reports.

For exploratory data analysis to be useful and accurate, a lot of work has to be done in advance, like:

• preparing and cleaning data, to make sure it’s accurate
• building a data warehouse, so all subsets and components are accessible and can interact
• establishing standard business rules, so metrics are measured consistently throughout an organization
• de-duplicating data, to save space and to prevent errors occurring if values stored in duplicated sections aren’t all consistently updated
• regular refreshes to keep data current
• creating infrastructure to maintain these safeguards’ effectiveness

Having data analysis novices perform this kind of work may be likened to thinking you can win a marathon although you eat only junk food and rarely work out.

One step towards future—and present—success

Ad hoc analysis works best when handled those who understand its strengths and limitations. Key to this understanding is knowing self service business intelligence may be used to set the stage for predictive analytics, but it’s only one step in a larger journey towards becoming a data-led organization.

Companies that commit to a sophisticated and centralized BI plan that includes exploratory data analysis when appropriate will spend less time and money, get exponentially more accurate and useful data, and maintain sharper focus on their strategic business goals. These companies will be the ones that thrive.