Founded by two ex-Coinbase employees and backed by top investors including Andreessen Horowitz, Coinbase, IDEO, and Kindred Ventures, Warbler Labs is the core development team supporting the growth and success of the Goldfinch network

Warbler Labs

Computer vision has captured the interest of entrepreneurs all over the world, and for good reason: the capabilities of modern AI tech makes previously impossible ideas into real products. Detecting and classifying objects in photos and videos has found applications in many different areas and hundreds of systems — from security cameras with built-in facial recognition and detection of diseases based on x-ray scans to simple mobile apps. But enough with the intro, in this article I wanted to share the ‘behind the scenes’ of how a computer vision product is developed, especially a complex one. I have had an opportunity to work on Birdsy — a bird recognition app for people who want to see who visits their backyards while they are not looking. Birdsy is a complex, AI-powered app with real-time object detection and classification that needs to be able to run on weak hardware, detect bird types and sex with high accuracy. Given all that, the road from the initial idea to publishing the app in app stores was complicated — and captivating at the same time. We have faced many hurdles — both from the business and development side of view — that I’ve decided to share in one place to hopefully help entrepreneurs and AI developers who might be facing the same project. The task at hand Birds have had millions of years of evolution to blend in perfectly with their environment to avoid predators and, in this case, bird watchers, making it harder for them to admire wildlife. While observing certain bird species face to… beak can be problematic, monitoring them through a video camera from the comfort of your home is a lovely way to enjoy our winged friends, especially if AI removes the need for looking through hours of video footage and sends you alerts when a bird has entered the camera’s view, automatically detecting what bird species it is. There are two parts to Birdsy: Object detection — a neural network in charge of detecting the presence of a bird or a mammal Object classification — a neural network in charge of determining which bird or mammal species was captured Low-quality cameras for object detection To make the service more approachable and easy to use, any camera can be used for bird watching. This is where we ran into the first problem: low-quality cameras as they are the most affordable and the most widespread. While the ‘no camera limit’ is great for users, it presented a challenge for us, as the object detection model runs using the camera chipset. Where someone gets a good deal, others get the short end of the stick, in this case ‘others’ being our CV developers. Working with a cheap camera means working with a cheap chipset which makes it impossible to use the default neural network architecture. Compared to a top-of-the-line, gold standard of computer vision video cameras (NVIDIA Jetson Nano) which allows using around 120 layers of default YOLO v4, the cameras we had to work with allowed only for 22 layers. Where a full YOLO v4 neural network provides great recognition results, a stripped-down version performs poorly. We have tested both and were unpleasantly surprised with how low the model depth was when running it using a cheap chipset. Testing default YOLO v4 vs Reduced We started with training the default YOLO v4 model and testing it on the customer’s dataset The results we achieved were satisfactory - 95% mAp, which in the world of computer vision is more than enough to launch a model into production. After retraining the model to fit the camera’s parameters, the detection quality dropped significantly. But where machines fail humans advance. We have tested the neural network on test data and visually evaluated false positives and false negatives. This highlighted where the network lacked knowledge and where it made the most mistakes. We ain't nothin but mammals, or false positives The network was eager to detect people, especially people’s hands, as animals (and we don’t blame it, humans ARE animals after all). While, from a biological point of view, this is correct, the end-user is more interested in looking at birds than their neighbors, so we had to work on teaching the network to ignore people and focus on birds and mammals instead. To do this, we have added negative examples, including pictures of people at various angles as well as human hands. What lurks in the dark The cameras have two modes: regular daytime mode which produces full-color images, and nighttime infrared mode, which produces black and white images. When the camera switched to infrared, the model produced a lot of false positives: tree leaves moving in the wind insects, like moths, that are attracted by a warm camera fountains Users would be less than happy to be woken up by a notification, get excited to look at an owl or a fox, but end up looking at a recording of a moth banging its body against a camera lens. In order to reduce sleep interruptions only to a minimum, we collected instances of false positives in nighttime settings and marked them by hand. Instagram vs Reality Ever heard of social media being called a ‘highlight reel’ where people present the best version of themselves? Who knew the same could be true for wild animals. Photos of birds that can be obtained from open sources, like Google Images and Youtube videos, are usually high quality, very sharp, and depict specimens at their best, looking at the camera or at least positioned front on, with nothing in between the bird and the camera obstructing the view. Reality is not always as pretty. Cameras produce low-quality images that can make it hard to understand what’s going on even for a human eye, bad weather conditions like rain, snow, or dust can obstruct the view, and we are sure birds sense when someone wants to capture them and position themselves in the most ridiculous way possible. The dataset that the client provided (consisting of shart images found on the Internet) was not of very much use for this project. We needed to collect a set of images of birds in real conditions using the client’s cameras to show the model what birds look like, and not how they are presented on social media. really What we have now So, after doing all of the above: manually going through every instance where the network was wrong teaching the network to ignore people and not detect them as birds or mammals working with black and white images and false positives collecting a dataset of birds in real-life conditions, under various angles, and in different weather conditions We have managed to achieve a 97,5% mAP for object detection. This is a very high result for a computer vision model as the unwritten rule for any CV model going into production is to have over 94% mAP. Achieving perfection (almost) While the results we have achieved now are more than enough to be used in the final product, there’s still room for improvement. the network mistakes flowers for birds birds’ view from the back is still sometimes a mystery birds partially covered by random objects birds close to the camera After enough images for each group are collected, we expect the mAP to increase and reach 98,5%. Image classification model The next step in getting to know your backyard visitors is to pass the image with a bird to an object classification model. Its goal is to recognize the bird species and its sex. As some bird species live exclusively on certain continents, we have decided to create two models - one for those who live in North America, and one for those in Europe. One step at a time: a lesson in multistage neural networks Initially, the problem of object classification was solved using a ‘head-on’ approach: the network was shown photos of all the different species, both males and females, from which it tried to learn what they look like and how they differ from each other. This resulted in very poor accuracy scores, in other words, the network made a whole lot of mistakes when identifying bird and mammal species. The network was trying to learn too many aspects at the same time. Many bird species look very similar to each other and differ from one another by a single patch of differently colored feathers or a differently shaped beak. Retaining all of this information, along with what different sexes of the same species look like, is too difficult under given circumstances. The network would often mix up bird species while determining the more broad bird type correctly. For example, the difference between a hooded warbler and a kentucky warbler in a patch of black feathers: The network would label a hooded warbler as a kentucky warbler, producing a wrong result, but being generally correct: both ARE warblers. For the sake of time, the client decided it was more important to detect the overall bird type rather than its particular species, so that’s where we started. After model evaluation, we have decided to implement a multistage approach: all bird species were grouped into more general categories: eagles, sparrows, crows, pigeons, etc. the network would first determine the overall bird type the second step is to detect which species it is the last would be to determine if the bird is male or female By grouping the bird species, we have managed to decrease the number of classes from 98 to 49, which greatly improved the accuracy score as the network simply didn’t have as many classes to choose from. Practice makes perfect, even for machines When you come across a new concept, you read books or watch educational videos to grasp it. If you fail, you ask your friend to explain it to you or you visit a seminar on the topic. In other words, you try to accumulate more information about it to understand it better. The same goes for neural networks. The better you need one to understand what a warbler looks like, the more images of warblers you need to show it. The more data it has looked at, the better accuracy scores will be. The multistage approach we’ve chosen not only improved the accuracy of the object classification model but also made it possible to analyze the dataset and determine where the network lacked learning data. The ugly duckling problem After the object classification model was launched, we were surprised to receive results that were much worse than what the tests showed us: the model could not determine the bird species or type correctly. The problem ran deeper than this: our computer vision developer in charge of the whole project, who has learned all of the bird species himself while working on it, also failed to determine what the birds were when he received the images incorrectly labeled by the network. It turned out that July is not the best time to launch a bird classification model as it is the time where teenage birds learn to fly and leave their nests. Remember the ugly duckling story? Well, it’s true for most birds, fledglings look nothing like adult birds, and it is hard to know what bird you are looking at if it’s still young. We have collected images of young birds over the summer and plan on training the classification network to determine the different bird species at different ages. Teamwork makes the network work Birdwatchers are a passionate bunch, they know how to identify a bird by the shape of its single feather. They possess knowledge that our classification network dreams of having, so why not bring the two together and form a bird-loving alliance the world has never seen before? Currently, the classification network doesn’t just tell the user the bird species, it shows the degree of confidence along with other guesses. The user can confirm the network’s guess or correct it, thus helping us train it - one bird at a time. After running the user feedback system for 3 months, we have collected over 20 thousand images. This data is invaluable to us since the photos are made in real-life conditions (bad weather conditions, at night, etc.) and are marked up by experts. When you stare into the abyss, the abyss chirps back at you It is worth noting that during this project, we have become the bird experts ourselves. Looking at birds all day long, while essentially educating a virtual child on all the little differences between different types of sparrows, makes one an instant platinum member of the birdwatching community. If all else fails, our CV team members can easily find themselves in ornithology. On a serious note, looking through thousands of images of birds, be it for dataset markup or analyzing where the network makes the most mistakes, we have delved deep into this project and came out the other end not only with a bunch of bird knowledge but a better understanding of how complex image recognition and classification systems work, how to best implement them, how to analyze a large dataset and find its weak points. This project was invaluable for us as an opportunity to research and work with the latest computer vision technologies, work with real-time customer feedback, and polish our problem-solving skills when working with outdated code.

Creating a Bird Detection AI: From Ideation to Product Launch

একটি পাখি সনাক্তকরণ এআই তৈরি করা: আইডিয়া থেকে পণ্য লঞ্চ পর্যন্ত

Erstellen einer Vogelerkennungs-KI: Von der Idee bis zur Produkteinführung

Creación de una IA para la detección de aves: desde la idea hasta el lanzamiento del producto

Créer une IA de détection d'oiseaux : de l'idéation au lancement de produit

बर्ड डिटेक्शन एआई बनाना: विचार से उत्पाद लॉन्च तक

鳥検出 AI の作成: アイデアから製品発売まで

조류 감지 AI 만들기: 아이디어 구상부터 제품 출시까지

Criando uma IA de detecção de pássaros: da idealização ao lançamento do produto

Создание ИИ для обнаружения птиц: от идеи до запуска продукта

Kuş Algılama Yapay Zekası Oluşturma: Fikir Geliştirmeden Ürün Lansmanına

Tạo AI phát hiện chim: Từ ý tưởng đến ra mắt sản phẩm

创建鸟类检测人工智能：从构思到产品发布

Our Humble Helpers: Familiar Talks on the Domestic Animals by Jean-Henri Fabre, is part of the HackerNoon Books Series. You can jump to any chapter in this book here . INCUBATION CHAPTER VI.  INCUBATION Incubator “Incubation means lying upon. The brooding bird does in fact crouch or lie upon her eggs, warming them with the heat of her body for a number of days with indefatigable patience. When a hen wishes to set, she makes it known by her repeated cluckings, little cries of maternal anxiety, by her ruffled feathers, her restless movements, and particularly by the perseverance with which she stays on the nest, even when it has no eggs, where she has been in the habit of laying. 1 “Some hens with wandering dispositions go back to the instincts of their wild race. They leave the hen-house and seek a hedge or thicket, where they select a hiding-place to suit them, and there make a little hollow in the earth which they line as well as they can with a mattress of dry grass, leaves, and feathers. That is a nest in the rough, without art, a shapeless construction in comparison with the clever masterpiece of the chaffinch and goldfinch. It is, furthermore, worthy of remark that all the domestic birds, as if man’s intervention had destroyed their skill by freeing them from want, fail to display in the construction of their nests the admirable resourcefulness shown by most wild birds. Here might be repeated the saying, as true for man as for beast, necessity is the mother of invention. Sure of finding, when the time comes for laying, the basket stuffed with hay by the hand of the housewife, the domestic fowl does not trouble herself to build a nest, an undertaking in which the tiniest bird of the fields shows itself a consummate architect. At the most, when her adventurous disposition makes her prefer the perilous shelter of the hedge to the safe retreat of the poultry-yard, the hen, gleaning with her beak a few straws and leaves, and plucking, if need be, some of her own feathers, succeeds in making, for her period of brooding, a disordered heap rather than a nest. There, every day, unknown to all, she goes and lays her egg. Then for three whole weeks she is not to be seen, or only at intervals. That is the time of incubation. At last, some fine day, she reappears, very proud, at the head of a family of young chickens, peeping and pecking around her.” “I should like,” said Emile, “to have some hens that set like that in the fields and then come home again some day with their family of little chickens.” “I must admit it is a sight worthy of interest, that of a hen that has stolen her nest returning to the farmhouse at the head of her newly hatched young chickens. Her eyes shine with satisfaction; her clucking has something joyful about it. ‘Look,’ she seems to say to those who welcome her, ‘see how [ ]fine, alert, and vigorous these young chickens are; they are all mine; I raised them there all alone in a corner of the hedge, and now I bring them to you. Am I not a fine hen?’ Yes, my dear biddy, you are a fine hen, but also an imprudent one. In the fields prowl the weasel and the marten which, if you are absent a moment, will suck the blood of your little ones; in the fields the fox is watching to wring your neck; in the fields there are cold, rain, bad weather, grave peril for your shivering family. You would do better to remain at home. 46 “The greater number follow this prudent advice and do not leave the poultry-yard. In the semi-obscurity of a sheltered quiet corner is placed the egg-basket, lined with a bed of hay or of crumpled straw. In it are put from twelve to fifteen eggs, the largest and freshest being chosen, and preferably those not more than a week old. If they were two or three weeks old they would not be sure to hatch, as in many of them the germ would have become too old and would have lost the power to develop. These arrangements made, the eggs are left to the setting hen without being touched again. “Whoever has not seen a setting hen has missed one of the most touching sights in this world: the devotion of the mother-bird to her eggs, her self-forgetfulness even to the point of sacrificing her own life. Her eyes shine with fever, her skin burns. Eating and drinking are forgotten, and in order not to leave her eggs a moment a hen might even let herself die of hunger on the nest if some one did not [ ]come every day and gently take her off and make her eat. Others, less persevering, leave the basket of their own accord, snatch up a little food, and immediately go back to the nest.” 47 “Do hens keep up that tiresome setting very long?” asked Emile. “It takes twenty or twenty-one days for the young chickens to come out of the shell. During the whole of that time, night and day, the mother remains squatting on the eggs, except for the rare moments that she spares, as if grudgingly, for the necessities of nourishment. Her only distraction in this complete retirement is to turn the eggs over every twenty-four hours and change their place, moving those outside into the center, and vice versa, so that all may have an equal share of heat. That is a delicate operation, and it must be left to the hen’s care to move the eggs with her beak. Let us be careful not to interfere with our clumsy hands, for the bird knows better than we how to manage it.” “If the hen is so careful to move the eggs every day and give them all the same amount of heat,” said Jules, “it must be heat alone that makes them hatch?” “Yes, my friend, simply the heat of the mother makes the eggs hatch. That is why the hen can be dispensed with and the eggs hatched by artificial heat, provided it be well regulated, gentle, and continued for a long time without interruption. The Egyptians, an ancient people of great skill, practised this method thousands of years ago. They put the [ ]eggs by hundreds of dozens into a sort of oven gently heated for three weeks, the period of natural incubation. At the end of that time the peepings of the countless brood did not fail to announce the success of the operation.” 48 “What a big family that oven-hatched brood must have been!” exclaimed Emile. “It would have taken a hundred hens to set on all the eggs, but in this way they were all hatched at once.” “A setting hen ceases to lay, and it was doubtless in order not to interrupt the beneficent daily production of eggs that the Egyptians invented artificial incubation in an oven. For the same reason sometimes with us recourse is had to this means, especially where the raising of poultry is made a business; only the incubation is no longer carried out in an oven but in ingeniously contrived incubators. In a drawer, on a bed of hay, the eggs are placed in a single layer. Above, and separated from the brooder by a sheet-iron partition, is a bed of water, which a lamp, kept always alight, warms and maintains at the temperature that the hen’s body would give; that is to say, forty degrees centigrade. In twenty-one days under this warm ceiling the eggs hatch just as they would under the hen.” “Oh, Uncle,” cried Emile, “I should really like to have an incubator like that in a corner of my room and watch the progress of the hatching every day by opening the drawer.” “What you would like to do, others, more skilful, have already done, not only opening the drawer but [ ]breaking an egg each day so as to see how things are going. I told you that the germ of the bird is a round spot of dull white, the cicatricle, which by its mobility is always on top at the surface of the yolk, no matter what the position of the egg. After five or six hours of incubation you can already distinguish in the center of the cicatricle a minute glairy swelling which will be the head, and a line which will be the backbone. Pretty soon there begins to beat, at regular intervals, the organ most necessary to life, the heart, which chases through a network of fine veins the blood formed, little by little, out of the substance of the yolk, and distributes it everywhere to furnish materials to the other organs just coming into being. It is toward the second day that these first heart-beats, destined to continue henceforth until death, become apparent. Thus irrigated with running flesh—for blood is nothing else—this organism thenceforward makes rapid progress. The eyes show themselves and form a large black spot on each side of the head; the quills of the large feathers form in their sheaths; the scales of the feet are outlined in a bluish tint; the bones, at first gelatinous, acquire firmness by becoming incrusted with a small quantity of stony matter. [ ]From the tenth day all the parts of the young chicken are well formed. The little being, softly suspended in its hammock by means of the two suspending cords that untwist little by little to give more room as it grows, is bent over on itself, the head folded against the breast and hidden under its wing. Note, my friends, that it is precisely this attitude of deep sleep inside the egg that the hen assumes when she wants to sleep. Crouched on her perch, she again folds her head on her breast and tucks it under her wing, just as she did when she was a little chicken in its shell. 49 50 “In the meantime the little bird keeps growing on the yellow and white matter; matter which soaks and penetrates it and, vivified by the air, becomes its blood and its flesh. One day it breaks the thin membrane under the shell, and there it is more at ease with the increase of space given it by the air-chamber. Now an attentive ear can distinguish feeble peepings inside the shell; it is the seventeenth or eighteenth day. A couple of days more, and the young chicken, summoning all its strength, will apply itself to the arduous work of deliverance. A pointed callosity, made expressly for the purpose, has formed on the upper part of the tip-end of the beak. Here is the tool, the pick, for opening its prison; a tool for that particular purpose and of very short duration, which will disappear as soon as the shell is pierced. With this provisional pick, the little chicken begins to hammer the shell; perseveringly it pushes, strikes, scratches, until the stone [ ]wall yields. For the most vigorous it takes several hours. Oh, joy! the shell is broken; there is the young chicken’s little head, and all yellow velvety down, and still wet with the moisture of the egg. The mother comes to its aid and completes its deliverance; others, weaker or less skilful, take twenty-four hours of painful effort to free themselves. Some even exhaust themselves in the undertaking and perish miserably in the egg without succeeding in breaking the shell.” 51 “Those are the very ones the mother ought to help,” said Jules. “She would be careful not to, for fear of a worse accident than a difficult birth. How could she direct her blows accurately enough not to wound the tender little chicken just inside the shell? The slightest false move would cause a wound, and at so tender an age any wound is death. We ourselves, with all the dexterity and care possible, could not, without danger, help the bird in distress; it can be tried as a last resort, but the chance of success is very small. The young chicken is the only one capable of carrying through this delicate deliverance if strength does not fail it. The hen knows this wonderfully well, and so does not interfere except to finish freeing the prisoner when half out of its shell. Let us hope that things will turn out as we wish, and that on the twenty-first day the whole family may be warmed under the mother’s wings without mortal accident at the moment of hatching. “From the instant of leaving the shell the young [ ]chickens already know how to peck food and how to run around the mother who, clucking, leads the way. They have besides a little fur of downy hair that clothes them warmly. This development is not found in all birds; far from it. Pigeons, for example, come naked from the egg and do not know how to eat; the father and mother have to feed them by disgorging a mouthful of food into their beaks. The young of the warbler, chaffinch, goldfinch, tomtit, lark, in fact of nearly all the field birds, are naked, very weak, at first blind, and completely incapable of feeding themselves, even with the food just under their beaks. The parents, with infinite tenderness, have for a number of days to bring it to them and put it into their beaks.” 52 “That is a difference that has always struck me,” commented Jules. “Little sparrows open their mouths wide to receive the food offered them, but for a long time they do not know how to take it even if it is put at the very end of their beak. On the contrary, little chickens easily pick up from the ground for themselves the seeds and worms that the mother digs up for them.” “I will tell you, if you do not already know,” continued Uncle Paul, “that the young of the duck, turkey, goose, and, among wild birds, the partridge and quail, have the same precocity as those of the hen. They are clothed with down on coming out of the egg, and know how to eat. One of the causes of this difference in the way young birds act immediately after [ ]hatching comes from the size of the egg. The chick is formed wholly from the substances contained in the egg; the larger the egg in proportion to the size of the animal, the stronger and more developed the young. Therefore the kind with the largest eggs are clothed at the time of hatching; they can run and know how to eat, unaided. Where the eggs are relatively small the young are hatched weak, naked, blind, and for a long time, motionless in their nest, demand the mother’s beakful of food. 53 “The largest egg known is that of an enormous bird that formerly lived in the island of Madagascar, and of which the species appears to-day to have been completely destroyed. This bird is called the epyornis. It was three or four meters tall and thus rivaled in stature a very long-legged horse or, better still, the animal called a giraffe. Such birds ought to lay monstrous eggs; such in fact they are; their length is three decimeters and a half and their capacity nearly nine liters.” “Nine liters!” exclaimed Emile. “Oh, what an egg! Our large vinegar jug only holds ten liters. Certainly the young that come from that ought to know how to run and to eat.” “To equal in bulk the egg of the epyornis it would take one hundred and forty-eight hen’s eggs.” “I think they could make a famous omelet with only one of those eggs.” “A fine large one could be made, too, with an ostrich-egg, which in size represents nearly two dozen [ ]hen’s eggs. It need not be added that young ostriches know how to run and to eat as soon as they come out of the shell. 54 “Those are the largest eggs; now let us consider the smallest ones. They are those of the humming-bird, a charming creature whose splendid plumage would outshine the most brilliant costly metals, precious stones, and jewels. There are some as small as our large wasps and that certain spiders catch in their webs just as the spiders of our country catch gnats. Their nest is a cup of cotton no bigger than half an apricot. Judge then the size of the eggs. It would take three hundred and forty to make one hen’s egg, and fifty thousand to make one laid by the epyornis.” “I imagine the little humming-birds in their nest must be all naked at first and blind, taking their food from their mother’s beak.” “From the smallness of the egg it could not be otherwise.” About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Jean-Henri Fabre (2022). Our Humble Helpers: Familiar Talks on the Domestic Animals. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/67073/pg67073-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org , located at https://www.gutenberg.org/policy/license.html .

WARBLER LABS

warblerlabs.com

-- employees

Since n.d.

COMPANY RANKING

WARBLER LABS

EVERGREEN INDEX #11667

Representing this company? Click here to claim and customize this page!

Warbler Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

WARBLER LABS

warblerlabs.com

-- employees

Since n.d.

COMPANY RANKING

WARBLER LABS

EVERGREEN INDEX #11667

RELATED COMPANIESRANK

STORIES ABOUT WARBLER LABS

Representing this company? Click here to claim and customize this page!

Warbler Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!