We are proud to release “Supervisely Country Roads” dataset, share our research as tutorial on how to build semantic segmentation model for drivable area and provide , so you will understand how to build, download and use custom models with . So you also can reproduce all steps below easily. source codes Supervisely Motivation Self-driving industry is huge today, companies invest in hardware, software, and labeled data. And it’s not only about self-driving cars, but it’s also about robots-couriers to deliver stuff, road quality inspection and other apps. The main purpose of this post is to give the quickest possible guide to the most common task in self-driving industry — drivable area recognition. What we want is to … Take training data, spend as little time as possible on data conversion / processing routines Pick a model, something Unet-like, train & run it Visualize the results in jupyter notebook Homework: try to improve our initial solutions by leveraging public datasets and more complex neural network architectures What we don’t want is to … Spend hours to download gigabytes of annotated images and to convert them from one format to another one Know anything about data formats Try various github repos until we find the right one — implementation of semantic segmentation model that actually works Now that our preferences are defined, let’s think about overall approach to address the task. There are two important challenges to deal with: It’s highly desirable to perform the entire research within single environment. In this case, there will be no need to do unnecessary data conversion and to switch between different tools and github repos Though there are public datasets (cityscapes or mapillary) that we can use to extract drivable area segmentation masks, our dataset is better to be lightweight, trivial and easy to use. So, doing all the work inside platform solves the first challenge. To address the second — we have released “Supervisely Country Roads” dataset. Supervise.ly All steps of this tutorial will be done inside Supervisely without any coding. The high level plan is the following: you will add “Supervisely Country Roads” dataset to your account, use DTL (Data Transformation Language) to create augmented training dataset, train NN and apply it to test images, then you can download NN weights and run inference right in . jupyter notebook Let’s get it started! Step 1. Add “Supervisely Country Roads” dataset to your account. Cityscapes and Mapillary datasets are mostly about city environment. But there are a lot of cases when autonomous vehicles have to drive on country roads: autonomous harvesters, trucks and agriculture robots. Other reason why you can not use mentioned datasets for this task directly is that many classes (e.g. “road shoulder”) are marked as neutral. It leads to inaccurate prediction of drivable area on country roads. We at had a project on this topic and we are happy to share our internal dataset and few experiments with community. We believe this contribution will be useful for many researchers. We release 1000 labeled images and 500 test images. We tried to keep the variety of labeled images. DeepSystems To get this dataset just go to “Import” -> “Datasets Library” page, choose “Supervisely country roads” dataset and define the name of the resulting project as “country_roads_labeled”. That’s all. Here are the few examples of labeled images: Step 2. Prepare training dataset Inside Supervisely we have special module called DTL (Data Transformation Language). It allows to merge datasets, make class mappings, filter objects and images, and of course apply various augmentations to your data. You just have to stack the data operations together. So you can build custom computational graph (analogy is the graph of neural networks layers), each your image and its annotations will go through this graph and the results will be saved to new project. In most cases automatic augmentations significantly increase the accuracy. We prepared DTL query for you. Just go to “DTL” page, copy&paste json file ( ), and press “Start” button. As a result of DTL query new project with 12k images will be created. It will contain original images, flipped version, and number of another augmentations: random crops, brightness, contrast and color transformations, and blurring. This example illustrates how it is easy and quick to prepare training dataset with our platform. link to github You will find the detailed description of applied transformations below. Step 3. Train neural network for semantic segmentation Before train NN you have to connect computer with GPU to your account. Go to “Cluster” page and click “Add node” button. Then execute following command on you computer. Docker image will be pulled, and Supervisely Agent (our task manager) will be started on your computer. It will automatically connect your PC to your account and you will be able to use your GPU for all NN related tasks (train/inference/deploy). Source codes also available in our . public git repo Let’s train segmentation neural network to recognize drivable area. Just go to the “Neural Networks” -> “Model Zoo” page, choose “UNet V2 (VGG weights)” and click “Add model” button. This model will be added to your account. Then click “Train” button near the model, choose training project “training_01”, define the name of resulting NN (latest checkpoint) as “nn_roads”, choose training configuration from template (it will be automatically added after you add model from Model Zoo) and press “Start training” button. Find example in our docs with tons of screenshots . here After training model “nn_roads” will appear in your models list. During training you can monitor metrics in real time: Step 4. Visualize predictions Let’s add test images to your account from “Import” -> “Datasets Library” -> “Supervisely country roads [test]” as project “country_roads_test”. Now you can press “Test” button near your model (that you have just trained), choose “country_roads_test” project, define the name of resulting project that will contain test images and neural network predictions and press “Start inference” button. Here are few examples of NN predictions on test images: Step 5. Download final model and use it the way you want (e.g. jupyter notebook) The idea is that you can use Supervisely to perform a lot of research experiments without any coding. But how to use the model in production? Option1. Deploy as API (we described the entire process in our ). In this tutorial we would like to show you another way — option 2: you can download NN weights and then use our source codes to run it right in . Notebook contains the example of inference. previous post jupyter notebook By default model is stored on your machine. Just click “Upload to Supervisely” button and then “Download” button ( ). As a result the tar archive will be downloaded. link to docs Here is the with example for this tutorial. Just follow instructions in (just unpack the model to certain directory and run docker container). link to our github readme.md Conclusion In this post we have shown how to build drivable area segmentation model from scratch. Although, the task gives you a taste of what self-driving companies are working on, production ready solutions require more challenges to overcome. In a more complicated scenarios we have to build a model capable of segmenting multiple classes of objects like cars, pedestrians, bikes, road landmarks and so on. In practice, constructing “multi class datasets” requires time & efforts due to the need to label / merge / filter / clean the data from various sources. Multiple classes of objects from various datasets. In a multi class settings, some of the objects are underrepresented. For example, on a road scenes, bikes are far less frequent object than cars. This makes it challenging to train a neural network that achieves required level of accuracy for the rare classes of objects. Class imbalance problem. Production level solutions should be stable in way that “edge case” images are recognized in a proper way. So we need to identify these “edge case” images first, then label them and finally, retrain the model. The process described is continuous and, sometimes, called Active Learning. It’s highly desirable to automate it as much as possible. Continuous model improvement. If you have any questions, feel free to ask in our public . We are happy to help! Slack We will address the challenges above in the next posts. Meanwhile, If you found this article interesting, give it some 👏, so that more people could see it!
