Hello, today I'd like to explain briefly how we use artificial intelligence to count sunflower seeds in a photo taken with a mobile device. Agenda: 1. Business needs 2. Data preparation 3. Model structure 4. Used libs and tools 5. Results 6. Error analysis 7. Fails/Hypotheses 8. Conclusion 9. References 1. Business needs Fortunately for me, I am working at . Where I am developing Computer Vision (CV) and other models to solve business problems and challenges. One of them is to count seeds on sunflower. Kernel Kernel – the world’s leading and the largest in Ukraine producer and exporter of sunflower oil, and major supplier of agricultural products from the Black Sea region to world markets. Kernel exports its products into more than 80 countries of the world. It is a common task for agronomist to count seeds on sunflower and corn, as these calculations will be used to predict "biologycal harvest". Separately, agronomists calculate weight of 1000 kernels, which will further be used to estimate the overall yield per area of field. Moreover, yield prediction is the primary function of agro analytics. The faster and more precise they make a forecast, the more money the company will earn. These inspections should be carried out for every filed. And this is how it works now: 1. Agronomists go to the field 2. Take about 10 sunflowers from different parts of the field 3. Split every sunflower into 4 quarters 4. Count the seeds and multiply them by 4 to find the total number of seeds on every sunflower 5. Take random 1000 seeds and weight them 5. We get the density of plants on the field per hectare from other inspections (or use planned density), btw we also will use AI to solve this task, and this could be topic for next post 6. And calculate (density per ha * avg seed quantity on sunflower * weight of 1000) / 1000 = yield weight per ha Here is an example of a common sunflower photo White (unformed) and black (formed) seeds can be seen, however only black seeds will be used to produce sunflower oil. One disadvantage of this calculation is that agronomists must count black, formed seeds on sunflower, which is very inaccurate and inconvenient. Furthermore, each agronomist may count a different number of seeds on the same sunflower because some seeds may be unformed (white, non-pollinated by bees) and it is difficult to determine which kernel is formed and which is not. 2. Data preparation To solve any Machine Learning problem, we need to get data and train the model. And for every specific task you cant find any free available labeled dataset. In our case, the dataset should contain hundreds of sunflower photos with label for every seed related to right class (black or white, for easy understanding), so we gathered our own dataset. Our agronomists have made about 1000 photos of different sunflower hybrids from different regions of Ukraine. And subsequently, another team labeled photos in free tool for data labeling. There should be for each image, as well as , and CVAT bounding_box (purple box) black seeds (green points in the picture below) white seeds (red points in the example below). Following this, we can check distribution of kernels in sunflowers looks normal with mean value - 1271. And correlation between number of black and white seeds. Weak dependence exists, but it can not be used further. 3. Model structure For business needs, all calculations should be done on a mobile device. That's why I used with encoder pretrained on What does this all means? U-net [1] mobilenet_v2 [2] imagenet . is Neural Net structure where NN takes an input image and returns a segmentation map (in our case heatmap), with downsampling and upsampling data. Our heatmap represents probability that pixel is related to some kernel. U-net Where up/down sampling blocks used weights from mobilenet_v2 Imagenet is a dataset with millions of images and tens thousands of classes. Frankly speaking, for this task I trained 2 NN and 1 algorithm. The first NN cut the sunflower from the image, like this: The second NN build 2 heatmaps, one per class (black and white). For training, input for this net is cropped image (from previous NN), and target is heatmap with builded gaussians on labeled points. For black kernels it looks like this: According to this heatmap, the lighter the pixel, the more model is sure that pixel is related to kernel. And the last algorithm (Laplacian of Gaussian approach) searches for centers of probability blobs (groups of bright pixels that are adjacent to each other), and on next image I used green points as the blob center above the cropped image. With white kernels prediction looks like this: 4. Used libs and tools For training, I used Python libs such as: and for training models - for image augmentation - to use pretrained models - to work with blobs on heatmap - to tune hyperparams for blob detection - to track training experiments PyTorch PyTorch Lightning albumentations segmentation_models_pytorch scikit_image hyperopt wandb 5. Results How do we validate the whole solution? To calculate mean absolute percentage error (MAPE) I made prediction for every image and compared the number of found seeds by model and number of seeds found by assessors. As a result, we got normal distribution with mean 0, which is extremely good. Blue bars - are for black kernels and MAPE is 3.6%, Orange bars - are for white kernels and MAPE is 8.8%. 6. Error analysis The biggest errors will have unusual sunflowers, that have with abnormally large number of kernels, or too many grey kernels, or with rounded edges, as it can be seen in the example below: Here, the model found 16% more than the assessor, and as we can see, the majority of extra seeds are on the edge of the sunflower. Or biggest error -257% for white seeds in this case: The total number of white seeds is 14, and the model found 50. As a  percentage, it is a really big number, but in absolute terms, it is not critical. 7. Fails/Hypotheses Once, I have found mathematically explained seeds positions in sunflowers and some other plants with fractal structure. Positions could be explained by which has an equation in polar coordinates Fermat's spiral where is index number of kernel and - constant, I guess for every sunflower it is different. As result, we should get the following positions n c But in practice it is doesn't work. I have modified that equations and added new constants a, b and the best result that I got was this: where blue - dot is point. So it is close in some way but not applicable for to real case. (0, 0) 8. Conclusion Currently, this task has been speeded up and standardized in our company in order to exclude human factor from counting. The MAPE is acceptable for business needs and this solution is working in mobile application now. For me, it is still unbelievable that we can count more than 1000 kernels, separate points into 2 different classes and all of this can be done via mobile device. Many thanks to for mentor ship in learning and support. Markiyan Kostiv 9. Referances 1. (2015) 2. (2019) U-Net: Convolutional Networks for BiomedicalImage Segmentation MobileNetV2: Inverted Residuals and Linear Bottlenecks

ORANGE

Polar

Target

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

With AI, You Can Count 1000+ Sunflower Seeds In Seconds

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

22/01/2018: Biggest Stories in the Cryptosphere

41 Stories To Learn About Agriculture

Agrotech: Making Agriculture Easier for Workers

All You Need is AI in Agriculture

An App that Identifies Pests and Plant Diseases on the Go

22/01/2018: Biggest Stories in the Cryptosphere

41 Stories To Learn About Agriculture

Agrotech: Making Agriculture Easier for Workers

All You Need is AI in Agriculture

An App that Identifies Pests and Plant Diseases on the Go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps