How to Create an Image Processing App in 72 hours

In November 2018, Volvo ran a hackathon that gathered talents from all over the globe to generate innovative ideas focused on undercarriage inspection automation. Our team created the best solution from scratch after 72 hours of live coding! Here’s how we did it.

This was the first hackathon by Volvo Group Connected Solutions and Volvo Сonstruction Equipment that brought together multidisciplinary teams to hunt for the next big innovation in image intelligence. Over 80 developers, designers, and data scientists joined in one place at Lindholmen Science Park to create a solution that would help with image processing of the machinery parts of Volvo excavators.

The challenge

After some time, moving machinery parts need to be periodically inspected and sooner or later replaced if they become worn out. In order to make things work, a lot of manual operations and measurements by engineers are required. Combine this with finalizing the information in technical documentation and sharing it with everyone and you’ll get a quite
complex and time consuming procedure.

Thus, the aim of the hackathon was to implement the best solution to automate and streamline this process with new technologies using a smartphone camera. 

This is where image intelligence and machine learning comes into play. By detecting and measuring the real time status of all the integral parts, it can predict whether parts should be replaced, fixed or left alone. It also indicates the amount of working hours that are remaining for the part. Let’s dive deeper.

What we came up with

This is where image intelligence and machine learning comes into play. By detecting and measuring the real time status of all the integral parts, it can predict whether parts should be replaced, fixed or left alone. It also indicates the amount of working hours that are remaining for the part. Let’s dive deeper.

We’ve developed a sophisticated solution for the diagnosis and maintenance of construction equipment.

The mobile app we’ve developed can be used by any person, even without any real technical background in order to take these measurements. The measurements have to be highly precise and taken at a construction site.

For example, thanks to the cameras used over the track shoe, you can detect the height of certain parts of the excavator and then algorithms can compare them with the normal height to define the level of wear or if they are worn out. Thus engineers can predict how many working hours are
left for this part or other parts to be replaced based on formula calculations.

We manually segmented over 500 pictures of the parts in order to develop a deep learning model and help with predictions based on height and object recognition.

Solution features

The final features in the end were highly valuable. First because it has the ability to create an end-to-end process based on the information from the measurement stage collected with a smartphone camera, and second
because it has the ability to share this information automatically with service dealers. 

During the live demo, we were also able to show that the accuracy and precision of the object recognition forecast was highly impressive. After 72 hours we gained up to 5 mm accuracy of object recognition while the standard measurement tools that Google or Apple provide show approximately only 10 mm accuracy. A measuring error of 10 mm constitutes the difference between a new and a worn-out part.

Conclusion

Our team created a deep learning model using an object with a known size as a reference to ensure the desired accuracy. In the short time available for development, we managed to create a working app prototype with an appealing look and feel, intuitive and handy navigation, and embedded logics of integration with existing Volvo construction services.

The solution that we presented is a whole new way of working with technical information documentation. It avoids manual interaction
and collection of information that might be lost or neglected for some reason due to human error. It automates and significantly streamlines the whole process. It also enables businesses to increase their uptime of different machinery such as excavators and trucks or different kinds of heavy machinery, in a way that decreases the downtime of operations. All you need is one smartphone camera. No deep insight or sophisticated measurement tools are needed to make this happen.

Vision of the future

We also considered how the first development of this application could further be improved and extended. The usage of this technology is amazing in terms of predictive maintenance. We only needed to inspect a few parts of the excavator undercarriage. However, the app could be extended to other uses based on different rights and parts within one brand.

Funding could be expanded in order to build the platform for predictive maintenance because it decreases downtime. Decreasing costs right now is quite a mission for everyone, especially in these turbulent times. It's a challenge to be able to find a way to automate processes, make operations
smooth enough, and avoid any kind of downtime.

The future is coming for these technologies and also for predictive maintenance in terms of the global automotive industry.

Thanks to Tatiana Osetrova, Project Manager at Sigma Software for the story!