An Introduction to AI-Based Visual Inspection for Defect Detection

“Why should I care about a cool new technology until it’s solving any of my problems?” – this was the exact conversation I had with the executive of a water purification plant over a warm cup of coffee. 

What started as idle chit chat ended up evolving into an insightful and meaningful discussion about  AI technologies for defect detection and quality control. This conversation significantly changed my vision and approach.

The goal of this article is to describe how to implement AI-based visual inspection for defect detection. It’s not about the coolness of artificial intelligence—no, it’s about practical solutions to specific problems.

In this article, we will dicuss:

1. Concept of AI-Powered Visual Inspection
2. How to Integrate AI-Based Visual Inspection System
3. Visual Inspection Use Cases
4. Key Takeaways

Concept of AI-Powered Visual Inspection

Artificial intelligence in manufacturing is a trendy term. When it comes to describing AI-based defect detection solutions, it’s often about some kind of visual inspection technology based on deep learning and computer vision.

WHAT IS DEEP LEARNING IN A VISUAL INSPECTION?

Deep learning is an aspect of machine learning technology that’s powered by artificial neural networks. The operating principle of deep learning technology is teaching machines to learn by example. By providing a neural network with labeled examples of specific types of data, it’s possible to extract common patterns between those examples, and then transform it into a math equation. This helps to classify future pieces of information.

With visual inspection technology, integration of deep learning algorithms allows differentiating parts, anomalies, and characters, which imitate a human visual inspection while running a computerized system. 

So, what does this mean exactly? Let’s use an example.

If you were to create a visual inspection software for automotive manufacturing, you should develop a deep learning-based algorithm and train it with examples of defects it must detect. With enough data, the neural network will eventually detect defects without any additional instructions.

Deep learning-based visual inspection systems are good at detecting defects that are complex in nature. They not only address complex surfaces and cosmetic flaws—but also generalize and conceptualize the parts’ surfaces.

How to Integrate AI-Based Visual Inspection System:

1. STATE THE PROBLEM

Visual inspection development often starts with a business and technical analysis. The goal here is to determine what kind of defects the system should detect.

Other important questions to ask include:

• What is the visual inspection system environment?
• Should the inspection be real-time or deferred? 
• How thoroughly should the visual inspection system detect defects, and should it distinguish them by type?
• Is there any existing software that integrates the visual inspection feature, or does it require a development from scratch?
• How should the system notify the user(s) about detected defects?
• Should the visual inspection system record defects detection statistics?
• And the key question: Does data for deep learning model development exist, including images of “good” and “bad” products and the different types of defects?

Data science engineers choose the optimal technical solution and flow to proceed based on the answers they receive.

2. GATHER & PREPARE DATA

Data science engineers must gather and prepare data required to train a future model before deep learning model development starts. For manufacturing processes, it’s important to implement IoT data analytics. When talking about visual inspection models, the data is often video records, where images processed by a visual inspection model include video frames. There are several options for data gathering, but the most common are:

1. Taking an existing video record provided by a client
2. Taking open-source video records applicable for defined purposes
3. Gathering data from scratch according to deep learning model requirements

The most important parameters here are the video record’s quality. Higher quality data will lead to more accurate results. 

Once we gather the data, we prepare it for modeling, clean it, check it for anomalies, and ensure its relevance.

3. DEVELOP DEEP LEARNING MODEL 

The selection of a deep learning model development approach depends on the complexity of a task, required delivery time, and budget limitations. There are several approaches:

3.1 Using a deep learning model development service (e.g: Google Cloud ML Engine, Amazon ML, etc.)

This type of approach makes sense when requirements for defect detection features are in line with templates provided by a given service. These services can save both time and budget as there is no need to develop models from scratch. You just have to upload data and set model options according to the relevant tasks. 

What’s the catch? These types of models are not customizable. Models’ capabilities are limited to options provided by a given service.

3.2 Using pre-trained models

A pre-trained model is an already created deep learning model that accomplishes tasks similar to the one we want to perform. We do not have to build a model from scratch as it uses a trained model based on our data.

A pre-trained model may not 100% comply with all of our tasks, but it offers significant time and cost savings. Using previously trained on large datasets models lets us customize these solutions according to our problem. 

3.3 Deep Learning model development from scratch

This method is ideal for complex and secure visual inspection systems. The approach may be time and effort-intensive, but the results are worth it. 

When developing custom visual inspection models, data scientists use one or several computer vision algorithms. These include image classification, object detection, and instance segmentation.

Many factors influence the choice of a deep learning algorithm(s). These include:

• Business goals
• Size of objects/defects 
• Lighting conditions
• Number of products to inspect
• Types of defects
• Resolution of images

An example of defect categories:

source: NECAM

Let’s say that we’re developing a visual inspection model for quality assessment in buildings. The main focus is to detect defects on the walls. An extensive dataset is necessary to obtain accurate visual inspection results, as the defect categories might be incredibly diverse, from peeling paint and mold to wall cracks. The optimal approach here would be to develop an instance segmentation-based model from scratch. A pre-trained model approach is also viable in some cases.

Another example is a visual inspection for pharmaceutical manufacturing, where you want to differentiate air bubbles from particles in products like highly viscous parental solutions. The presence of bubbles is the only defect category here, so the required dataset will not be as extensive as in the example above. The optimal deep learning model development approach might be to use a model development service over developing one from scratch.

4. TRAIN & EVALUATE 

The next step after developing the visual inspection model is to train it. In this stage, data scientists validate and evaluate the performance and result accuracy of the model. A test dataset is useful here. For a visual inspection system, it may be a set of video records that are either outdated or similar to ones we want to process after deployment.

5. DEPLOY & IMPROVE

When deploying a visual inspection model, it’s important to consider how software and hardware system architectures correspond to a model capacity.

Software 

The structure of visual inspection-powered software bases itself on the combination of web solutions for data transmission and a Python framework for neural network processing. 

The key parameter here is data storage. There are three common ways to store data: local server, a cloud streaming service, or serverless architecture. 

A visual inspection system involves the storage of video records. The choice of a data storage solution often depends on a deep learning model functionality. For example, if a visual inspection system uses a large dataset, the optimal selection may be a cloud streaming service.

Hardware

Depending on the industry and automation processes, devices required to integrate visual inspection system may include:

• Camera. The key camera option is real-time video streaming. Some examples include IP and CCTV.
• Gateway. Both dedicated hardware appliances and software programs work well for a visual inspection system.
• CPU / GPU. If real-time results are necessary, a GPU would be the better choice than a CPU, as the former boasts a faster processing speed when it comes to image-based deep learning models. It is possible to optimize a CPU for operating the visual inspection model, but not for training. An example of an optimal GPU might be the Jetson Nano. 
• Photometer (optional). Depending on the lighting conditions of the visual inspection system environment, photometers may be required.
• Colorimeter (optional). When detecting color and luminance in light sources, imaging colorimeters have consistently high spatial resolutions, allowing for detailed visual inspections. 

Deep learning models are open to improvement after deployment. A deep learning approach can increase the accuracy of the neural network through the iterative gathering of new data and model re-training. The result is a “smarter” visual inspection model that learns through increasing the amount of data during operation.

Visual Inspection Use Cases

HEALTHCARE

In the fight against COVID-19, most airports and border crossings can now check passengers for signs of the disease.

Baidu, the large Chinese tech company, developed a large-scale visual inspection system based on AI. The system consists of computer vision-based cameras and infrared sensors that predict the temperatures of passengers. Now the technology, operational in Beijing’s Qinghe Railway Station, can screen up to 200 people per minute. The AI algorithm detects anyone who has a temperature above 37.3 degrees.

Another real-life case is the deep learning-based system developed by the Alibaba company. The system can detect the coronavirus in chest CT scans with 96% accuracy. With access to data from 5,000 COVID-19 cases, the system performs the test in 20 seconds. Moreover, it can differentiate between ordinary viral pneumonia and the coronavirus.

AIRLINE

According to Boeing, 70% of the $2.6 trillion aerospace services market is now dedicated to quality and maintenance. In 2018, Airbus introduced a new automated, drone-based aircraft inspection system that accelerates and facilitates visual inspections. This development reduces aircraft downtime while simultaneously increasing the quality of inspection reports.

AUTOMOTIVE

Toyota has recently agreed to a $1.3 billion settlement due to a defect that caused cars to accelerate even when drivers attempted to slow down, resulting in 6 deaths in the U.S. Using the cognitive capabilities of visual inspection systems like Cognex ViDi, automotive manufacturers can analyze and identify quality issues much more accurately and resolve them before they occur.

COMPUTER EQUIPMENT MANUFACTURING

The demand for smaller circuit board designs is growing. Fujitsu Laboratories has been spearheading the development of AI-enabled recognition systems for the electronics industry. They report significant progress in quality, cost, and delivery.

TEXTILE

The implementation of automated visual inspection, along with a deep learning approach, can now detect issues of texture, weaving, stitching, and color matching.

For example, Datacolor’s AI system can consider historical data of past visual inspections to create custom tolerances that match more closely to the samples.

We will conclude with a quotation from the general manager we mentioned earlier: “It makes no difference to me whether the suggested technology is the best, but I do care how well it’s going to solve my problems.”

Key Takeaways:

Written by Serhii Maksymenko, Data Science Solution Architect MobiDev. This article is based on our technology research and experience providing software development services.

Previously published at https://mobidev.biz/blog/ai-visual-inspection-deep-learning-computer-vision-defect-detection