Understanding Anomaly Detection and How It Can Benefit Your Company

Anomaly detection can help you identify upcoming trends before your competitors. It can flag fraudulent transactions by monitoring online store traffic, and spot violence in public places, giving your security team the chance to interfere before people get hurt.

Interested? There are dedicated anomaly detection companies that can help you build and integrate custom software tailored to spot behavioral deviation in your sector of operations.

So, what is anomaly detection? How does it work? And how can you incorporate it into your company’s processes and workflows?

Content Overview

• What is anomaly detection?

• How does anomaly detection work?

• Key anomaly detection use cases

• Getting started with anomaly detection

• How ITRex can help with anomaly detection

  
  

What is anomaly detection?

Anomaly detection is a type of data mining that analyzes a company’s data to detect data points that deviate from the established baseline (e.g., the standard behavior of the dataset). These outliers typically indicate incidents, such as technical glitches in equipment, changes in customer preference, and other types of abnormalities, enabling companies to act before the damage is done.

What is an anomaly?

An anomaly is an inconsistent data point that deviates from a familiar pattern. Even though it doesn’t always represent a significant concern, it’s worth investigating to prevent possible escalations. For example, a spike in product sales can be a result of a successful marketing campaign, or it can point to a change in trends and customer behavior, which companies will have to adapt to.

Business data anomalies fall into three outlier categories:

• A global outlier is a data point that is located abnormally far from the rest of the data. Let’s assume that you receive $7,000 in your bank account each month. If you suddenly get a transfer of $50,000, that would be a global outlier.

  
  

• A contextual outlier deviates from the rest of the data within the same context. For instance, if you live in a country where it typically snows in winter and the weather is warm in the summer, then heavy snowfalls in winter are normal. But experiencing a snowfall during the summer would be a contextual outlier.

  
  

• A collective outlier is when a subset of data points deviates from the entire dataset. For example, if you observe unusual drops in sales of several seemingly unrelated products, but then you realize this is somehow connected, then your observations are combined into one collective outlier.

Why do we need artificial intelligence in anomaly detection?

Most companies deal with large volumes of structured and unstructured data, with the latter comprising up to 90% of information generated within a company’s walls. It’s impossible to process all this information manually and generate meaningful insights — especially if we talk about unstructured data, which is made up of images, transactions, free-form text, etc.

Research shows that machine learning techniques are the best choice for processing large unstructured datasets. This field has an extensive number of algorithms, and you can select what suits you best. You can also combine several ML techniques together for optimal results.

How does anomaly detection work?

There are three main types of AI and machine learning-based anomaly detection techniques.

• Supervised anomaly detection. Here, ML models are trained on and tested with a fully labeled dataset containing normal and anomalous behavior. The approach works well when detecting deviations that were a part of a training dataset, but the technology stumbles when facing a new anomaly that it hasn’t seen in training. Supervised techniques require manual effort and domain expertise as someone needs to label the data.

  
  

• Unsupervised anomaly detection. This method doesn’t need manual data labeling. The models assume that only a small percentage of data points that significantly differ from the rest of the data constitute anomalies. Unsupervised techniques can still excel at identifying new anomalies that they didn’t witness during training because they detect outliers based on their characteristics rather than on what they learned during training. However, these algorithms are rather complex and their architecture is a black box, meaning that users will not receive an explanation of how the tool made its decisions.

  
  

• Semi-supervised anomaly detection. These techniques involve both labeled and unlabeled data, which reduces the expenses of manual annotation. Also, a semi-supervised anomaly detection model can still learn after deployment and detect anomalies that it hasn’t seen in training. And as with unsupervised techniques, these models can also work with unstructured data.

AI-based anomaly detection methods

Anomaly detection relies on artificial intelligence and its subtypes, including machine learning. Here are five ML techniques that are frequently deployed within this context.

Autoencoders

Autoencoders are unsupervised artificial neural networks that compress data and then reconstruct it to resemble the original form as closely as possible. These algorithms can effectively ignore noise and reconstruct text, images, and other types of data. An autoencoder has two parts:

• Encoder, which compresses the input data

• Decoder, which decompresses the data in proximity to its original form

  
  

  

When using an autoencoder, pay attention to the code’s size, as it will determine the compression rate. Another important parameter is the number of layers. With fewer layers, the algorithm will be faster, but it could work on fewer features.

Bayesian networks

This technique is a type of probabilistic graph-based model that calculates probability based on Bayesian inference. Nodes in a graph correspond to random variables, while edges represent conditional dependencies that allow the model to make inferences.

Bayesian networks are used in diagnostics, causal modeling, reasoning, and more. In anomaly detection, this method is particularly useful for detecting subtle deviations that are hard to spot using other techniques. This method can also tolerate missing data during training and will still have solid performance if trained on small datasets.

Density-based models

This is an unsupervised ML clustering technique that detects patterns relying purely on spatial location and distances between neighbors. It compares the density value of a data point to the density of its neighboring data points. An outlier (an anomaly) will have a lower density value than other data populations.

Support vector machine (SVM)

This is a supervised ML algorithm that is commonly used for classification. However, SVM extensions can also operate in an unsupervised environment. This technique uses hyperplanes to divide data points into classes.

Even though SVM typically works with two or more classes, in anomaly detection, it can analyze single-class problems. It learns “the norm” for this one class and determines whether a data point can belong to this class or whether it is an outlier.

Gaussian Mixture Models (GMM)

GMM is a probabilistic clustering technique. This technique classifies data into different clusters based on probability distribution. It assumes that data points belong to a mix of Gaussian distributions with unknown parameters, and it detects anomalies by spotting data in low-density regions.

Key anomaly detection use cases

Now that you know how anomaly detection works behind the scenes and the AI techniques it relies on, it’s time to study some anomaly detection examples in different industries.

Anomaly detection in healthcare

Anomaly detection can benefit the medical sector by helping physicians identify any problems with patient health, detect escalations in inpatients and notify the medical staff before it is too late, and aid in diagnosis and treatment selection. All this reduces the manual work and the cognitive load that doctors experience.

However, anomaly detection has its unique challenges in healthcare.

One issue is that it can be difficult to establish the baseline (i.e., the normal behavior) when it comes to different medical diagrams. For instance, an electroencephalogram of a healthy person varies based on individual characteristics. Researchers identified considerable variations in children, and there are differences in adults depending on the age group and gender.

Another aspect is that ML models must be highly accurate, as people’s lives will depend on their performance.

Medical anomaly detection algorithms can analyze the following information:

• Vital signs and other parameters measured by medical IoT devices
• Medical images, such as X-rays and CT scans, which can bear signs of benign and malignant tumors, infections, and other health conditions
• Health insurance claims, helping identify and block any fraudulent activity. This could be a game-changer in medical insurance since up to 10% of annual Medicare and Medicaid expenses are currently going toward fraudulent claims

One example of anomaly detection comes from a research team in South Africa. They successfully combined autoencoder and extreme gradient boosting techniques to monitor the physiological variables of COVID-19 patients and detect any anomalies that would indicate health degradation.

Another team focused not only on detecting anomalies but also on explaining why the tool flagged them as such. So, first, they used anomaly detection techniques to spot deviations and then deployed aspect mining algorithms to outline a set of features in which a certain data point is considered an outlier.

Anomaly detection in entertainment

Sports and entertainment environments rely on extensive video-based security monitoring with hundreds of cameras. Therefore, it wouldn’t be possible for security teams to spot and timely react to accidents if the footage was reviewed manually. Thanks to machine learning, algorithms can analyze videos streaming from each camera at the facility and detect security violations.

As ML models continue to learn on the job, they could spot threats and violations that your human operators couldn’t have noticed. These algorithms can detect vandalism, unrest among spectators, smoke, suspicious objects, and more, and alert security staff so that they have time to act and prevent liabilities and reputational damage.

One such project comes straight out of our portfolio. A US-based entertainment company with gaming rooms located all around the country turned to ITRex to build an ML-driven anomaly detection solution that would integrate into their cloud-based video surveillance system. This application would catch any dangerous and violent behavior, such as breaking slot machines. It would also streamline the administrative process by spotting forgotten items and machines that are out of order.

Our team built a customized ML model using a variational autoencoder. We aggregated a training dataset of 150 videos depicting physical violence and property damage and preprocessed these videos with the OpenCV framework. Then, we relied on the torchvision library to normalize and augment the data and used it to train the ML algorithm.

The resulting solution relied on cross-validation to spot anomalies. For instance, it could identify malfunctioning slot machines by “reading” the error message on the screen and validating it against the available screen templates. The final solution integrated seamlessly into the client’s cloud-based security system, monitored slot machines 24/7 and notified the security personnel whenever it detected an anomaly.

Anomaly detection in manufacturing

As manufacturing processes become increasingly automated, machinery becomes more complex and the facilities grow larger. Consequently, the traditional monitoring approaches aren’t sufficient anymore.

Anomaly detection techniques can depict different deviations from the norm at your facility and notify you before they escalate and even learn to distinguish between minor issues and pressing concerns.

There are numerous anomaly detection benefits for manufacturing. These tools can spot the following issues:

• Equipment malfunctioning. In collaboration with the manufacturing Internet of Things (IoT) sensors, AI algorithms can monitor various device parameters, like vibration, temperature, etc., and catch on any deviations from the norm. Such changes can indicate that the equipment is overloaded, but it can also mean the beginning of a breakdown. The algorithm will flag the equipment for further inspection. This is also called predictive maintenance.

  
  

• Equipment underutilization. ML-based anomaly detection solutions can see which devices stand idle for a prolonged period and urge the operator to balance load distribution.

  
  

• Safety hazards. By monitoring security camera feeds, anomaly detection software can spot employees who are not abiding by the factory’s safety protocols, endangering their own well-being. If your employees use wearables for safety monitoring, machine learning can analyze sensor data to spot exhausted and ill workers and encourage them to take a break or sign out for that day.

  
  

• Infrastructural issues. ML algorithms can detect water or gas leakage and any other infrastructural damage and notify the corresponding site manager.

An example of a manufacturing anomaly detection solution comes from Hemlock Semiconductor, a US-based producer of hyper-pure polysilicon. The company deployed anomaly detection to get visibility into their processes and record any deviations from optimal production patterns. The company reported saving around $300,000 per month in resource consumption.

Anomaly detection in retail

Anomaly detection can help retailers identify unusual patterns of behavior and use these insights to improve operations and protect their business and customers. AI algorithms can catch on to changing client demands and alert retailers to stop acquiring products that will not sell while restocking items that are in demand. Also, anomalies can represent business opportunities at early stages, allowing retailers to capitalize on them before the competition. In the case of eCommerce, website owners can deploy anomaly detection models to monitor traffic to spot unusual behavior that might signal fraudulent activity.

Additionally, retailers can use anomaly detection techniques to secure their premises. At ITRex, we conducted a series of PoCs to build a solution that can detect expressions of violence, such as fights, in videos streamed by security cameras placed in shopping malls and other public places. The solution relies on the 3D convolutional neural networks anomaly detection method, which was trained on an extensive fight dataset. This type of ML algorithm is known to perform well on action detection tasks.If you are interested in such a solution, we can show you the full demo to begin with. Then, our team will fine-tune the algorithm and adjust its settings to match the specifics of your location and business, and we will integrate it seamlessly into your existing security system.

Getting started with anomaly detection

As you can see, training custom AI models for spot-on anomaly detection can be a technical challenge. That’s why our team prepared a five-step guide for companies eyeing the novel technology. Scroll down for some expert tips — and consider downloading our business guide to artificial intelligence if you’re new to AI or seek more information about AI applications and project costs.

Step 1: Determine how you will approach anomaly detection

There are two options here. You are either looking for specific anomalies in your data, or you want to flag everything that deviates from the standard behavior. What you choose here will impact your training data and restrict the selection of AI techniques.

If you want to catch every event that deviates from the baseline, then you will train the model on a large dataset representing normal behavior. For instance, if you are working on driving and traffic safety, then your dataset will be composed of videos showing safe driving.

Suppose you are looking to detect specific anomalies — for instance, car crashes, but not minor violations, such as driving through a red light. In this case, your training dataset will include videos or images of car crashes.

Step 2: Aggregate and pre-process the training dataset

The outcome of the previous step will help you decide which type of data you need.

Collect the data from your internal company sources or use publicly available datasets. Then, clean this data to eliminate duplicates and any incorrect or unbalanced entries. When the dataset is cleaned, you can use scaling, normalization, and other data transformation techniques to make the set suitable for AI algorithms. Split your dataset into three parts:

• Training data to teach the models
• Validation data to evaluate the model’s performance during training
• Testing data to score the performance after completing the training process

For more information, check out our detailed guide on how to prepare data for machine learning.

Step 3: Choose your anomaly detection technique

This step is only relevant if you want to build a custom solution. You or your tech vendor will select the best-suited artificial intelligence technique to tackle the business problem. There are three key factors to consider here:

• The task at hand (refer to Step 1 above). If you want to detect specifically defined anomalies, Variational Autoencoder (VAE) is a great option.

  
  

• The technical requirements. This might include the accuracy and detail levels you’re aiming to achieve. For instance, if you want to train an ML model that spots anomalies in videos, deciding on the optimum frame rate is key, since different algorithms analyze frames at different speeds. As long as the anomaly you want to detect can occur within one second, it’s recommended that you study every frame in a video clip, and using slower algorithms, such as VAE, becomes impractical. Singular Value Decomposition (SVD), on the other hand, can do the job considerably faster.

  
  

• The size of your training dataset. Some models, such as autoencoders, can’t be properly trained on small datasets.

Step 4: Build/buy and train the model

You can either purchase ready-made anomaly detection software or implement a custom system that will correspond to your unique needs and be tailored to the type of anomalies that you are interested in.

You can opt for an off-the-shelf anomaly detection system when you have limited financial resources, no custom training dataset, or no time for model training, and you can find a vendor who already offers a solution that can detect the type of anomalies you are concerned with. But keep in mind that these solutions have built-in assumptions regarding data characteristics and they will perform well as long as these assumptions hold. However, if your company data deviates from that baseline, the algorithm might not detect anomalies with the same accuracy.

If you have sufficient data to train AI algorithms, you can hire a machine learning development company to build and train a custom anomaly detection solution. This option will be designed to meet your business needs and fit into your processes. Another great benefit is that you can still optimize this solution even after deployment. You can tweak its settings to make it work faster or focus on different parameters, depending on your changing business requirements.

Step 5: Deploy and monitor the solution

You will deploy the anomaly detection solution either locally or in the cloud. If you work with ITRex, we will have two options for you to choose from:

• Cloud-based anomaly detection, where we aggregate data from your software systems, devices, and third-party services and transmit it to the cloud for storage and processing to take the load off your local resources.

  
  

• Edge anomaly detection, where ML algorithms analyze your data locally and only upload a part of the data to the cloud. This approach is best suited for mission-critical systems that don’t tolerate delays, such as autonomous vehicles and medical IoT solutions.

ML algorithms continue to learn on the job, which allows them to adapt to new types of data. But this also means that they can acquire bias and other undesirable tendencies. To avoid this scenario, you can schedule an audit to re-evaluate the algorithms’ performance and implement the necessary adjustments.

How ITRex can help with anomaly detection

At ITRex Group, we have vast experience with machine learning models, such as Beta-Variational Autoencoders (Beta-VAE) and Gaussian Mixture Models (GMM), IoT, data analytics, and data visualization. We’ve implemented these technologies in different industries, so we are aware of the specifics that heavily-regulated sectors, such as healthcare, bring. We use a combination of open-source and proprietary technology, such as data mining tools and ML frameworks, to develop custom solutions and integrate them into your business processes.

Our custom AI-based solutions for anomaly detection can work with both tasks — detecting predefined anomalies and spotting any deviation from the established standard behavior. You can opt for the cloud to save on infrastructure, or we can make the system run locally to support critical applications that don’t tolerate latency.

Interested in implementing an anomaly detection system? Drop us a line! We can help you build and train a custom ML model. Even if you opt for a ready-made solution, if it’s open-source and has an API, we can still re-train it on your own data so that it fits your system better!