Creating a smart algorithm is not yet a given for many entrepreneurs and small businesses, who might lack the resources to launch a successful Artificial Intelligence program.
While (very) large corporations benefit from the vast amount of data available to them, as well as a combination of business, technological and regulatory expertise, most SMEs have no such luck.
This should however not discourage the brave and bold who seek to embark on a data-science journey. And though there are no shortcuts, there are clear steps to undertake in order to start an A.I project, which I’ve laid out below in the hope that it’ll clear up some of the mysticism surrounding the creation of an “intelligent” algorithm.
Please note that this is not meant to be a technical guide, and that definitions will be kept to a minimum for brevity's sake.
Before even thinking about A.I, some key questions must be answered by a variety of key company executives : do they want to disrupt their market by creating a different type of value proposition? Do they seek to be “best in class”? Maybe their aim is to stay level on a competitive market? Or even catch up to the current leader?
These questions have to be answered BEFORE any sort of A.I project is set in motion. Operational teams will otherwise be left to aimlessly dig through data, looking for a story to tell. Given the nature of most industries, though, they would be chasing a moving target, rewriting history as the data comes in. Yes, data is fun. Yes, it’s interesting. But it serves no purpose on its own. Starting with it instead of clear goals creates only solutions in search of a problem.
Hint : A good strategy starts with disgruntled customers in mind, not technologies.
The decisions, of course, do not stop at the formulation of a strategy; strategy only answers the question “Who am I?”. “What am I doing?” is an altogether different question, and needs to be answered more operationally through the identification of use cases.
Let’s say a company’s long-term strategy is to be the most “trusted” player in its industry. It could enact this strategy by ensuring that all customer queries are answered without errors, that they are answered quickly, that all callers are greeted by a human instead of a robot… hundreds of such ideas could be (should be) systematically identified, evaluated, clustered, prioritised, and discussed in workshops or through small consulting engagements (woo!).
Some will go straight to Sales, Marketing, Accounting, HR… to potentially be dealt with using a traditional operational/statistical approach; and some will be deemed fit for a solution involving A.I, in one of its many forms. I advise looking at three specific issues to start with :
Hint : It is more productive for companies to look at A.I through the lens of business capabilities rather than technologies.
The data question emerges immediately after an Machine Learning project (the term “Machine Learning” is much more accurate than “A.I” at this point of the conversation) is deemed operationally sound and technologically feasible. Or, should I say, the data questionS…
What type of data is needed ?
There technically isn’t a lot of different types of data : it can be numerical (historically the easiest to use due to its tabular nature), text, images, videos, sound... Just about anything that can be recorded could be deemed to be data. Ultimately, computers are agnostic about the type of input you give them; it’s all just numbers to them.
How much data is needed ?
There is no specific amount of data-points that can be prescribed as it varies wildly from project to project; but a start-up which has just launched and has no more than 300 clients probably does not naturally have the resources to launch a ML project. Furthermore, note that data can be formed of either a lot of data-point (“Big N”), a lot of details for each data point (“Big D”), or both. Both is best.
Do we have that data?
Data is either “available” or “to collect” (yes, I’m over-simplifying, sue me). Collection can either be done internally, which can be incredibly time-consuming (we’re talking months and major restructurations), or gathered through external sources (predicting umbrellas demand, for example, would use weather data freely available to all).
Hint : Unique data, rather than cutting-edge modeling, is what creates a valuable A.I solution.
Any self-respecting project requires continuous risk-assessment in order to address issues before they arise (PMO 101). This should come early in the project, and be done thoroughly and continuously throughout its entire life-time. Below are 10 questions to ask to get started:
Hint : Risk assessment are best performed by outside players.
Once all the matters above have been settled (if not, go back to step 1), it’s time to pick the most adapted solution, technologically speaking, to address the previously identified issue.
Doing it this late in the process might seem odd, yet makes perfect sense when one realises that the technologies below are highly adaptive, and that starting with just one in mind would only shrink the horizon of possibilities. In any case, anyone with some experience in the matter will have thought of the right tool to use during the previous steps of the project.
Below is a shallow look at the 7 main categories of machine learning. Concrete examples will be given in a later article, but in the meantime can be found all around the web :
Supervised Machine Learning
Linear algorithms : LAs model predictions based on independent variables. It is mostly used for finding out the relationship between variables and forecasting, and works well for predictions in a stable environment. → More details : Linear regression, Logistic Regression, SVM, Ridge/Lasso…
Ensemble methods : Ensemble learning is a system that makes predictions based on a number of different models. By combining individual models, the ensemble model tends to be more flexible (less bias) and less data-sensitive (less variance). → More details : Random Forest, Gradient boosting, AdaBoost…
Probabilistic Classification : a probabilistic classifier is able to predict, given an observation of an input, a probability distribution over a set of classes, rather than only outputting the most likely class that the observation should belong to. → More details : Naive Bayes, Bayesian Network, MLE…
Deep Learning : For supervised learning tasks, deep learning methods make time-consuming feature engineering tasks irrelevant by translating the data into compact intermediate representations of the data to increase precision (think categories). It is great for classifying, processing and making predictions based on non-numerical data. → More details : CNN, RNN, MLP, LSTM…
Unsupervised Machine Learning
Clustering : “Clustering” is the process of grouping similar data-points together. The goal of this unsupervised machine learning technique is to find similarities in the data point and automatically group similar data points together. → More details : K-means, DB Scan, Hierarchical Clustering …
Dimensionality reduction : dimensionality reduction, aka dimension reduction is the process of reducing the number of random variables under consideration by obtaining a set of principal variables. It also makes very pretty graphs. → More details : PCA, t-SNE…
Deep learning : For unsupervised learning tasks, deep learning methods allow the categorisation of unlabeled data. Any more details would substantially lengthen this article (Soz). → More details : Reinforcement Learning, Autoencoders, GANs...
Hint : all models are wrong, but some are useful.
As previously mentioned, few companies can afford to fully develop their own algorithms and deploy them at a large scale unaided. The unlucky 99% have to make complicated choices regarding their ability to invest a large amount with low chances of success, or invest a smaller amount, with a high chance of success but a very sticky relationship with a provider/partner.
There are no easy answers. Just hard choices. Below are a few options :
Build
Buy
Partner
Hint : BBP choices depend highly on both the executive strategy and the industry to which a company belongs.
A recently built algorithm cannot be released into the wild untested. Indeed, most machine learning systems operate as “black boxes” (again, a useful over-simplification), and something that may have worked on one set of training data may not work as well (if at all) on another set for a variety of reasons. Thankfully, a handful of tools have been devised to ensure that models works as intended :
All the above matter insofar as they allow a team to inform a pre-prepared scorecard. External teams are also useful here as they will not be biased by the costs already sunk on the project.
Hint : beware of the “rage-to-conclude”.
The deployment phase of an A.I project is probably the most crucial of all, given the resources invested. Indeed, the transition from prototypes to production systems can be expected to be expensive and time-consuming, but should at least not run into major issues if the risk analysis was done properly (that is sadly rarely the case).
There aren’t many ways to ensure that a deployment goes smoothly beyond an efficient project management team. The two main techniques are as follows :
Furthermore, and though I would encourage all first-time projects to move slowly and avoid breaking things, keeping an eye out for scaling potential can often be key to future successes.
Hint : Do not be afraid of redesigning workflows around the new solution.
Time to “Render to Caesar the things that are Caesar’s”, and share the success of the Machine Learning algorithm built with the world. If the endeavor wasn’t a success, it’s worth discussing too, if only as a learning opportunity for the entire organisation.
There are a few key groups which ought to be made aware of either successes and failures, and so for a wide variety of reasons :
Hint : Under-hyping tends to be more productive than over-hyping.
I have bad news. And good news.
The bad news is that the work never ends. The good news is that the work never ends.
Indeed, letting an algorithm run unchecked for a long period of time once its put into production would be foolish. The world change, as do people and their habits, leading to their data to change. Within months, it is likely that the initial data used to train an algorithm is no longer relevant.
The challenge here is that it might be very hard to notice at first that the algorithm is no longer representative of the world and needs to be reworked using the expertise gained during the construction of version 1.0 of the algorithm. Talk about a Sisyphean task.
Hint : The world changes a lot faster than you think. And an algorithm cannot see that.
The very nature of a data-driven project is that it cannot be generic. This means spending both time and money to create something unique to cross a moat created by an early adopter. But because A.I capabilities tend to grow at an exponential rate within large companies, the moat just keeps getting wider. This is the challenge that many companies now face, and why they need a solid process to get started : their margin for error is quickly shrinking.
As such, I have a few final recommendations :
Good luck out there.
This article was originally made for The Pourquoi Pas, a blog providing in-depth analyses of today’s technological challenges. Click here to check it out.