Source code here: https://github.com/cleuton/drone-facerec
This project is based on two other Github projects:
I ran this using Ubuntu (Linux). To run using MS Windows or MacOS some fixes must be done. Be my guest to convert it and submit a pull request!
This software controls a Tello drone, capturing the video streaming generated by its onboard camera. He intercepts each frame and tries to recognize faces that appear on the image, using a Convolutional Neural Network Model that I trained, based on images from the dataset Labeled Faces in the Wild, and some pictures of mine.
It is a proof of concept of the use of artificial intelligence with drones and, why not, Internet of Things (IoT). Note that performance can be low, after all, I am using a laptop and a cheap and simple drone. With more resources, better performance of the solution is possible.
To use this solution, you will need:
I will teach you how to do these 3 things.
There are a few things you need to do before using this application. First it was made exclusively for Drone Tello, but it applies to any other drone that has a python programming API.
The second thing to do is to be able to control your drone using only the original easyTello project and then try to use this project here. Why? Well, it's not as simple as it looks ...
If you just bought your Tello drone, you need to check which version of Firmware it has. This can be done through the Tello app (if you haven't downloaded it, it's better to do it now).
Then, turn on the drone, connect to his WiFi network with your Smartphone (TELLO ...), open the app and click on the settings button, as in the image:
Then, click on More:
Click on the three dots button:
Now, check the firmware version number. If it's 1.3... then you'll need to update it!
The process is done in two parts: Download the firmware to the Smartphone and update on the drone. To download the new firmware version, disconnect from the drone's WiFi and connect to the Internet. Using the Tello app, click the Update button to download the firmware. After downloading, connect to the drone's WiFi and click the Update button again.
After the process, check the new firmware version.
Warning: An indication that the API is out of date is when the drone does not recognize the command streamon!
If you have updated the firmware correctly, I recommend that you try to run the original easyTello project, following the instructions in that repository. This confirms that the tracking API is working correctly. Then, you can download and run this repository here.
conda env create -f conda-env.yml
Activate that environment before running this project:
conda activate drone-facerec
To do facial recognition, it is necessary to train a neural network. I didn't insert the original FaceREC_CNN project here, not even the model file that I trained. I recommend that you clone the original project and train your own model, and then copy the HDF5 file created to the easyTello folder, within this project.
Download and extract the shape_predictor_68_face_landmarks file, and place it in the FaceRec_CNN project and also inside the easytello folder for this project!
Generating a model is simple:
1. Get multiple photos of the same person's face
2. Convert the photos
./ easytello / faceprocessor.py
3. Get people's names and model file
Before trying with the real drone, I suggest that you run a local test using your webcam:
ffmpeg -f video4linux2 -s 640x480 -r 15 -i /dev/video0 -vcodec libx264 -f h264 -an udp://localhost:11111
Use the program that I included facetest_ffmpeg.py(
) to capture the images and do the recognition. If everything works, if he recognizes you, then everything is ready to run on the Drone.
If you want, you can run this project with FFMPEG:
Warning: To do facial recognition, do not use the original Easytello project! I modified the project to communicate with my facial recognition script faceprocessor.py(
./ easytello / faceprocessor.py
) script controls the drone. It basically does 2 things: puts it in command mode and starts capturing video. But you can do so much more! You can make the drone take off, go forward, or turn around and then land. There are some commented commands that you can use. If you want to know more about the commands that Tello accepts, see the documentation.
Warning: Do not try to connect your smartphone to the drone, as it only serves a single connection!
Make sure you have placed the file h5 in the easyTello folder! Make sure the name is the same as it is in the faceprocessor.py file (
Warning: If you make the drone take off, make sure there is room for it! If it hits the ceiling or walls, it could be damaged! And be careful when trying to use Tello outdoors (not recommended). If he moves more than 10 meters, he may be out of range of WiFi and collide with something. Theoretically, if he loses contact, he lands automatically.
Previously published at https://github.com/cleuton/drone-facerec/blob/master/english_version.md