The Problems with Big Data and How AI Can Help, an Interview with Andrew Gryaznov, CTO at HyperC

What is wrong with Big Data, how can classical AI solve these problems, and why is it possible now? I had the opportunity to sit down and talk about these issues in greater detail with Andrew Gryaznov, CTO at HyperC and the expert in AI for business efficiency.

To begin, let's put ourselves in the shoes of business owners and the management team. Why all this Big Data vs. AI should even bother them?

In every business, many hidden decisions are made daily, poorly controlled and with a financial impact no one planned for. Imagine a company with a sales department where reps are selling products and a set of additional services which they decide to offer for every case.

Very often, this decision-making process is not formalized and analyzed. The company just has sales projections in the financial model, but the execution is based on hidden uncontrolled decisions and actual profitability is not measured.

Machines can make way more decisions than humans, do it faster, and keep the efficiency at higher levels. For now, most likely, you'll still need a human to control the machine and make sure it made the correct thing. This model fosters a more efficient and rapid change in the company, which translates to increased ROI and overall efficiency. This is where technology can step in and help. 

Okay, and what is wrong with Big Data compared to AI?

“Big Data” and tools were supposed to solve the business efficiency improvement problem. But it turned out that expectations were far from reality. Businesses ended up collecting tons of historical data they can't really use as the data had low quality while the tools were too complex and costly.

I believe that the only data that can prove useful long-term is what I call the “telemetry data” - which is never input by the user manually but is generated in real-time by monitoring systems. You could think of automatic systems registering a call or an email but requiring no clicks or form-typing from the salesperson. 

So the reality is that all of our “clean data” and “clean process” is only in our head and we have a ton of disparate telemetry and dirty unusable data. Here is where we need to turn back to classical AI for help.

How can this work in real life?

Let's assume that we have dentistry, which is a network of several clinics. How is it usually organized? Most likely, such a company will have some long-term lease agreements, doctors on a payroll, marketing, and sales processes to build the funnel and move customers through it. 

In turn, there are also startups in the field, and we work with one of them. This company uses an innovative business model that heavily depends on machine-based decision-making. First, they do not have employees on payroll; each doctor is a contractor. Second, they do not have clinics and long-term lease agreements. Instead, they lease offices weekly.

They match their contractor dentists with lease offices using a model that takes the demand, contractors' schedules, and historical data to get the highest yield. The problem becomes more complex as you add more doctors, office spaces, and decision points like doctor’s ranking or ability to negotiate contract terms. Mathematically speaking, the problem is NP-hard (and the accompanying planning problem is PSPACE-complete) which in simple terms is impossible to solve by humans.

As a result, the business model that looks weird and less efficient with costly contractors and short-term leases turns out to be much more effective. However, it can't work without the AI automatically making endless business decisions and attacking NP-hard problems with universal heuristics and machine learning. 

Another example is the hardware manufacturing business. Before the modern days, the vendors planned their production based on consumer demand only. It was enough until the chip shortage period that we are facing in recent years. Here, "shortage" does not mean there are no chips at all, but there are chips at different prices that are higher than before. So, today to create a successful hardware production financial model, you have to map consumer demand to chip prices and include all the details of the manufacturing process. This is another NP-hard problem that humans just can't solve, but the machine can.

Can you please describe how AI solves these problems with some technical details?

Problems like those I've mentioned above are best modeled using graph databases. AI can analyze the business data and process represented as a graph, use machine learning to find dependencies and shortcuts, and the best routes from the starting point of the business process to the goal, i.e., optimizing processes for the highest margins.

Shortly speaking, finding the shortest path in the graph of such a database will be the best possible solution that can then be translated into Excel model so that humans can double-check it and make amendments if needed. 

Why is this possible now, and was not possible before?

This is actually an interesting topic that leads us to the notion of AI winter. In the early 1970s, there was a lot of buzz around AI. People were excited by the exact same topics we are covering today. They wanted AI to help them in decision-making and improving business efficiency. However, it turned out that it was impossible, given the level of tech progress at the time. This led to disappointment and reduced funding and interest in AI.

I like to compare the attempt to deliver a business-usable AI model in the 1970s as if people tried to build and land a rocket in the Stone Age. That is beyond impossible. To put that in perspective, today’s smartphone has more memory than all models of Symbolics AI machines produced, combined. Today we are ready to really implement that AI, and we are doing it right now.

What is the adoption curve of this technology right now?

I would say we’re still in the beginning of a new era. 

There are three types of companies from the technical standpoint: new ventures that do not have established processes and technology stack, companies with some sort of processes and automatization in place, and more prominent companies that have established processes for new tech implementation.

New companies fight for efficiency, and their founders and management teams are often open-minded. The idea of having successful automated decision-making seems natural for them. Large companies can also understand the benefits, but their protocols slow down innovations, though they are still possible. 

The biggest group consists of companies of the second type. They are not yet too big for innovations, but still unsure how to do them right. The main challenge for them is implementing the change management process as one of the important steps towards maturity.

So, we have some startups implementing AI right now, more prominent companies that are slowly moving towards it, and the majority will join in the upcoming years.

In conclusion, what the action plan for the AI implementation within a specific company should be?

The first step to successful AI implementation is to write down all decisions people make within the business. This will uncover an enormous amount of uncontrolled, hidden decisions that directly affect the company's margins.

You need to understand that you control only a tiny fraction of your business and that new young open-minded companies are starting to build their processes using innovative AI. Once you do it, you'll be ready to join the AI bandwagon to stay competitive in the upcoming years.