What do you get when you mix sugar water, 3D printing and deep learning?

A rewarding close-up concatenated video of cute hummingbirds visiting your own backyard!

The demo :

Here is the reward. Look at the cute little hummingbirds!

The hardware setup in the garden

Hummingbird feeder

Hummingbirds have long been a favorite for gardeners. Over the years, people have developed many varieties of bird-feeders to attract them to their backyards. Those design are pretty cool and can prevent bees from stealing the sugar water from hummingbirds.

I use an off-the-shelf hummingbird feeder, Perky-Pet Pinch-Waist Glass Hummingbird Feeder 203CP (8oz). You can order it from Amazon using the link above but I get mine from a local Fred Meyer.

Camera

To take close-up videos of hummingbirds, we need a small and high quality camera. I use an off-the-shelf camera, REMALI 4K Ultra HD Sports Action Camera.

3D printed camera mount on the hummingbird feeder

At this point, the camera and the hummingbird feeder are still separate. So, I need to make custom hardware to secure the camera to the hummingbird feeder. Here is the model that I make, which is designed to have the camera facing down to take close-up videos. The model is uploaded to Google Drive here . You can have various 3d view of it in your browser if you log in your Gmail first and then click the link above.

Bolts, nuts, string and rubber bands

Finally, I use bolts and nuts(6mm diameter) to secure the camera mount on the hummingbird feeder and hang the mount on a stand in the backyard using a string. I also add some rubber bands to further secure the camera.

Software :

This is what we want from our software: to be able to extract frames that have hummingbirds in them.

Determining whether there is any hummingbird in an image

Deep learning and computer vision

Very likely, hummingbirds only come to your bird-feeders for a few minutes a day. So, most of the video frames only captures the background and are not interesting. It would be nice if there were a method to automatically detect, extract and concatenate frames in which hummingbirds are present. To do this, we need some computer vision techniques.

At first, I apply a pre-trained RESNET to the whole image and examine the likelihood of it being a hummingbird. However, it does not work well. It is probably because there are so many other things in the image and they confuse the classifier.

Then, I note that the regions where hummingbirds show up in the images are normally the regions close to the flowers of the bird-feeder. So, I can just crop out the regions close to each of the two flowers and apply the same classifier on them. And it works great. Here is the Python code and the results.

Code to detect whether an image contains any hummingbirds. It also includes an evaluation snippet.

Classification output. x-axis is the index of the image, y-axis is the likelihood that we think the image contains a hummingbird. A green dot means the image contains a hummingbird and a red dot means otherwise. From the plot, we see that a threshold of 0.05 gives us only 2 mis-classified examples.

We only have two misclassification among 47 examples (i.e. accuracy > 95%). For the two misclassified examples, one is a blurred image due to high-speed movement of the hummingbird and the other one contains only the beak of the hummingbird.

Use FFMPEG to clip videos and combine videos in python

To clip video, I use FFMPEG . Here is a script in Python to achieve that.

from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip

t1 = 22*60 + 40 # Start timestamp in secondst2 = 23*60 + 40 # End timestamp in secondsffmpeg_extract_subclip("video1.MOV", t1, t2, targetname="video1.mp4")

To concatenate the videos, FFMPEG works great too. Here is a Python script that does the concatenation.

import osos.environ["FFMPEG_BINARY"] = "/usr/local/bin/ffmpeg"from moviepy.editor import *

clips = []

for filename in [ 'video1.mp4','video2.mp4', 'video3.mp4' ]:clips.append(VideoFileClip(filename))

video = concatenate_videoclips(clips, method='compose')video.write_videofile('combined2.mp4')