Along with my team, I’ve been involved in a project called , a novel approach that uses Deep Learning to identify marine-plastic and I wanted to share the findings: DeepPlastic 85% mean-average precision! Using our model, we can now detect on average 85% (mAP) of all epipelagic plastic in the ocean. We achieve this level of precision based on a Neural Network architecture called . Below we’ve attached a video containing examples of marine plastic that we’ve run our model on: YOLOv5-S Where’s the Paper and The Code? This article would mainly serve as a no-code summary of our research paper. If you’d like to skip straight to the nitty-gritty details of the neural network or the code, you can access them here: arxiv pre-print: https://arxiv.org/abs/2105.01882 Codebase and Dataset: https://zenodo.org/record/5562940#.YY1uML3MI-R If you’d like me to write another blog post explaining the code, data augmentation, etc., let me know in the comments below! How Did I Get Involved In This? I’ve always been passionate about the oceans. After graduating from school at California State University, Monterey Bay, I knew that I wanted to build tools/software that can help protect the ocean. After I collected, annotated, and curated the data along with the model, I realized that the precision metrics were significant, and there could be a manuscript that could be published here. So, I asked for help from , an expert in Marine plastics, and and for help with the writing and publishing of the manuscript. Dr. Sara-Jeanne Royer Jay Lowe Olivier Poirion Abstract The quantification of positively buoyant marine plastic debris is critical to understanding how concentrations of trash gather across the world’s oceans and identifying high concentration garbage hotspots in dire need of trash removal. Currently, the most common monitoring method to quantify floating plastic requires the use of a manta trawl. Before analysis, the need for physical removal incurs high costs and requires intensive labor–preventing scalable deployment of a real-time marine plastic monitoring service across the oceans. Without better monitoring and sampling methods, the total impact of plastic pollution on the environment as a whole, and details of impact within specific oceanic regions, will remain unknown. This study presents an automated workflow that utilizes videos and images captured within the epipelagic layer of the ocean as input and produces real-time quantification of marine plastic for accurate quantification and removal. YOLOv5-S was the best performing model, which operates at a Mean Average Precision (mAP) of 0.851 and an F1-Score of 0.89 while maintaining near real-time speed. In addition, our method can utilize” off the shelves” camera equipment and standard low-cost GPUs to monitor/quantify epipelagic plastic in near real-time. Goal For The Project We wanted to build a generalized object detector capable of identifying and quantifying sub-surface plastic around the world. Now that we understand our goal and how to achieve it, let us jump into the workflow. Curating The Dataset Finding a dataset that contained annotated pictures of marine debris was incredibly hard. There was no dataset in existence that had images of marine plastic in the epipelagic layer of the ocean. So, I decided to create one. I bought a GoPro Hero 9, A wetsuit, and snorkeling equipment and headed out to various locations in California along with 2 Plastic bags and 2 Plastic Bottles. The locations I visited were: Lake Tahoe, Bodega Bay, San Francisco Bay. Here, I shot videos of the plastic in 4K and later broke them into images frame-by-frame. [All plastics used were sanitized and were removed from the environment after I finished capturing the videost] The initial dataset was more than 100,000 images which I then painstakingly went through one by one and chose the best images and annotated them using supervise.ly. The final dataset, along with images scraped from the internet, was a whopping 4000 images. I tried my best to replicate real-world scenarios such as occlusion and brightness by burying objects in sand or placing them across from the sun. ( ) Photo by on ; ( ) Photo by on ; ( ) Image by Author Bottom Right Naja Bertolt Jensen Unsplash Top Right Nariman Mesharrafa Unsplash Top Left, Bottom Right Data Formatting Images were resized into 416x416 and converted into the format the Darknet and YOLOv5 PyTorch require. Data Augmentation Since the final dataset only contained 4000 images, I thought the best way to increase the size of the dataset was to augment it. So, I used Flip, Rotate, Brightness to replicate oceanic environments. I also used B&W to make the model not generalize towards color and cutouts to simulate occlusion. Building the Neural Network Building a Neural Network was a straightforward task. I had two goals for model selection: The model had to be somewhat accurate, and the model had to be fast. Fast enough to be used on Buoys and UAVs. I tried many models, such as Faster R-CNN, EfficientDet, SSD, etc., but stuck with two models: YOLOv4-Tiny and YOLOv5-S. Interested in a code run down for YOLOv5? Let me know in the comments below or reach out to me. Things to know/Tuning Hyperparameters: I used an called ADAM to set a decaying learning rate.Used a package called (Weights and Biases) and continuously monitored the loss.I used a as the final layer and only used called trash_plastic.I used paperspace.com and Google Colab pro with to train the model.Used transfer learning from weights trained on and Adaptive Learning Rate W&B softmax a single class NVIDIA v100 GPUs Underwater Scenes Deep Sea Debris — JAMSTEC JEDI Dataset. All the code used for the models including architectures can be found at , YOLOv4 YOLOv5 Interested in our code? Find it here Results After a lot of experimenting with training methods, data augmentations, and fine-tuning hyperparameters, we finally reached a point where the results were good enough to be used in real-world deployments. Best model : Precision: 96%, Mean-Average-Precision: 85%, F1-Score: 0.89, Inference Speed: 2.1 milliseconds/img. YOLOv5-S Photos by , ( ) Photo by on ; All Other Images by Author (Bottom Row) JAMSTEC JEDI (Second Row First Image) Bottom Right Naja Bertolt Jensen Unsplash What’s Next? At the moment, we are in the process of getting our paper published. We are trying to get the model in the hands of other researchers for testing and developing innovative ways to synthesize more data. If you’re interested, want to contribute, or want to chat, You can reach me here: . gautamtata.blog@gmail.com This article was first published here

Google

NVIDIA

2022 - HackerNoon Contributor of the Year - Climate Change

2022 - HackerNoon Contributor of the Year - Computer Vision

2022 - HackerNoon Contributor of the Year - Deep Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Climate Change

Nominated for 2022 - HackerNoon Contributor of the Year - Deep Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Computer Vision

Too Long; Didn't Read

Get hands-on learning from ML experts on Coursera 

How Deep Learning Can Help Quantify, Monitor, and Remove Marine Plastic: The DeepPlastic Way

How Deep Learning Can Help Quantify, Monitor, and Remove Marine Plastic: The DeepPlastic Way

Too Long; Didn't Read

Companies Mentioned

@gautamtata

RELATED STORIES