Offline ,Real-Time Face Recognition in Node.js using Python atop 99.38% accuracy model

Well if you are this kind of a guy, you may head directly to code:

https://github.com/malikshubham827/get-me-through

Note: For accuracy please see this.

This is about my experience in building above stated project:

A Free, Offline, Real-Time, Open-source(MIT License) web-app to assist organisers of any event in allowing only authorised/invited people using Face-Recognition Technology or QR Code.

So yes Face Recognition is just a part of an app, and also the crux of the project. Have a cup of coffee and enjoy it(I tried my best). I have included a lot of gifs to make it interesting(inspired from Siraj Raval ‘s videos).

Well it’s common to see and hear the projects related to deep learning in Python and not in Node.js . Well it’s because there are a lot of libraries in Python like Numpy, Pandas, tensorflow etc for efficient calculations, and the gap is big enough to abridge. I know Node.js, okay a little, and wanted to use it to build a project to learn it better while fiddling with Machine Learning.

It all started during an online competition in which there was a question to leverage the AzureML API to do a task(face expressions ). I saw that they also had the Face Recognition API but with limits. Also first I have to upload the image to their service, then they would compute the result then send me back. It sounds little slow. I wanted to try it out but it was not available in my country at that time. I want to thank the team that it gave me a new way, an idea to build something. Till then I was just learning the things which are already built. So there’s some sense of security, a backup option in case something goes wrong. I also read an article

Unconventional way of learning a new programming language_Learning a new language by contributing to open source projects_hackernoon.com

by my senior Sahil Dua about learning a new language. I thought it was a great opportunity to apply what I learnt in a brand new open-ended project.

So what then? I plugged in the charging cord in my Macbook because I know it’s not gonna be a short trip. I took my pal Google with me to go on a search spree to find resources, tit-bits to start the project. I found some questions online where people asked the same question if it is possible, but the answers aren’t up to the mark. That time I realised that I am pretty much on my own. I stumbled across a nice series by Adam Geitgey:

Machine Learning is Fun! Part 4: Modern Face Recognition with Deep Learning_Update: This article is part of a series. Check out the full series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6 and…_medium.com

I was following him since his first post, but somehow lost contact as continued to dive into different things. I read this awesome article and came to know that he has also created a python package face_recognition for that. I downloaded it tried it. Well things didn’t go as smoothly as you read the previous line.

This was my condition when installing dlib. I don’t know it was somehow having a trouble in installing at the very last step. I spent many hours understanding the reason behind, I even didn’t go to the gym that day.

I was at the verge of abandoning the project. But then I came to know that it was all because some conflicts because of anaconda package python path or some sort of that. I was still learning to become familiar with the eco-system, I have to make a choice to keep anaconda and abandon project or remove anaconda. So I removed anaconda completely, spent a day complete to completely remove all the different versions of Python downloaded by different packages and only left with the system version. I then downloaded the Python3 correctly using Homebrew. I then followed to install dlib and at the end of the day I was able to run it.

But then the next problem, How to integrate it into Node.js? Then I again have two options: Learn a python framework like Django, Flask etc. to continue the project or start a seemingly open-ended project which maybe not possible though. I somewhere read the following line and it stuck my mind.

If you believe me I just googled these words to find the link as I remember it, I think I read it on Quora. So like my senior’s article this is also inspiring so I chose to go on with Node.js which I knew a little by doing some projects on web.

So I was again on a voyage to find how to integrate Python and Node.js. I came to know about child processes in Node.js. But again my situation was something like this:

Reading documentation without much examples. Reading blogs and finding the same thing everywhere. But this time I was determined enough to get it done at any cost(read internship). As the internship season is about to start, so I have to finish it asap to start preparation, else I would have enough time to create another such project in my next summer but on my own again without any guidance except Google. If you are a little familiar with Machine Learning then you might relate to this:

If the learning rate is too low then it might take more than required time to reach the optimum point or get stuck at local minimum or if it’s too high it might even miss it , but if it is correct or altered according to conditions then the algorithm finds the optimum point quickly.

For me as an algorithm internship is the correct learning rate. So I had to complete the project as soon as possible with this attitude:

So I slogged, some days worked till 5:45 a.m. in the morning(that’s whole night in front of a computer screen getting things to work, smashing bugs with your keyboard).

That was an amazing time, I did a lot of stupid mistakes too. Not refreshing the tab as I was altering the server-side code. After a few times, I was changing client-side code but still not refreshing the window. I don’t why, maybe I was feeling a little bit sleepy in a sweet temperature of 22 degrees Celcius on my cozy bed. Those were amazing moments, searching StackOverflow for unnecessary logic error which I later fixed by just refreshing the tab.

Finally I was able to coordinate python and Node.js and finally they both be like:

(Confession: I didn’t even download Pokemon Go, but I love Pikachu)

So who’s Pikachu here? Python or Node.js. Left to the readers.

I was kind of in awe to build fully offline web-app capable of everything without internet connectivity after that AzureML incident. So I have to find alternatives of APIs or make them myself. As you might know it is a common thing in Computer Science, time is inversely proportional to space. So I was trying to minimise time to get things work. Sending the photo/video to cloud service is time and bandwidth consuming,and in turn I was increasing space. Although their cloud service is much better, but I also love to code more if I am coding what interests me. You may accept this after reading the installing instructions of my repository.

I searched a lot of packages, tried them one by one, getting them to work but sometimes giving up and again searching. One thing was how to integrate QR Code as a fallback option in case Face Recognition fails. There are a lot of packages to generate QR Code but not for QR Code scanning. I tried a few finally found Instascan by Chris Schmich. Well I do fell in problems with the package. The thing is I was using OpenCV3 in python for face recognition and the same camera( there is only one 🙃 on laptop) for QR Code scanning. I need small video frame for scanning, but it also resized face recognition video too, tackled it btw later. There is also some problem with closing camera stream, if scanning closed without result. Well this is not such a problem. I can stop the face recognition process, do QR Code scanning and again start face recognition process. Simple enough, huh? But if you are a careful reader then you might have noticed 2 things:

1. If it seems easy, you’re doing it the wrong way.
2. I am trying to minimise time(the main reason for not waiting the API to be available).

So I read the code and process behind it to solve my problem.

Parsing the libraries

Yeah, I was in this segment 3 times(reading the libraries to understand the intricacies involved, atleast for me).

1. Reading face_recognition package to learn how it is working to adapt it to my use case, helped others too in the mean time.
2. Read a little Instascan to find how it is working with camera and in general how to use camera in a web app. I used Modal by MaterializeCSS to show QR Scanner. There are many cases to handle: what if user stops camera by some other way like by clicking outside of modal or close it. I changed the code, ran it many times each time squashing a few bugs. At once I ran out of RAM(just close to full) in my mac and it just freezed for a few moments. After a lot of tries finally successful, but meanwhile found another bug.
3. This time the bug is in the Modal of Materialize framework. The callbacks aren’t working. Googled it, github issues, stackoverflow no help. I found the code involved, tried to understand it, tried to run a few times with console.log() statements, getting sense what is happening and finally getting around the bug by moulding the code( got the feeling of being a hacker, bypassing the code 😅). I have also heard that its forms are also not that good, would fiddle with them in another web-app on the roadmap.

Event++

This is the amazing bug. It is due to jQuery. The number of events are increasing on each click. 1,2,3,4,5,6,7,8… My real-time notifications flooded the entire right column where they are shown, from top to bottom. I learnt that I have to use jQuery on() instead of click() and use socket.on() events outside of event handlers.

So finally after much struggle back-end is performing pretty well with front-end:

So at this time I thought that, this is it. But an idea struck me what if I add a database support to it so that the user of machine can do CRUD operations. I thought to leave it to the user( if any) to use what they want SQL, NoSQL. I thought to add one myself and what if I could make money 🤑 with that( you can start flying too early with big visions on getting small successes 😄). Giving organisation complete setup to automate arriving/leaving of visitors(Face Recognition,with QR Code, NO internet required, NO API limits, 3-step authentication) just a few clicks away. But:

1. I learnt everything from the community of open-source people.
2. I don’t think I can sell it going door-to-door. I am too lazy for that. I would prefer developing a few more of this kind.
3. I recognise a scene from The Social Network where Mark developed an app and release it for free, even though he had pretty good offers( I think from Microsoft) and here I am with a small web-app which I have not even created from scratch.

So I tried to integrate it with MongoDB as I am a little familiar with it. Another reason being I have not been taught SQL in college. It would be in my upcoming 5th semester. So I would leave it for another project maybe a fork or port of this. So I developed it, integrated it, spent a few days, learning, applying, debugging etc.

Finally I saw this:

Front-end, Back-end and Database working together in unison. The middle one is Back-end of course(power of both Python and Node.js). You may also do your other ML tasks like training a model using Python, in Node.js as I was able to integrate OpenCV3(it requires binaries to be installed), face_recognition, numpy, pandas with dataset and stored the result in .csv format in my python process. So you may also leverage it to get something done as long as you have required hardware, to build something maybe a game-changing idea.

I would leave it to the reader to choose which one is Front-end and Database 😇.

Signing.Off();

Link to the project: https://github.com/malikshubham827/get-me-through

Although, it may seem that why I have written this post even though my codebase is not so long? What’s so exciting? Well to many of you it may seem just a few hundreds of lines of code, but for me to integrate all the pieces, side by side learning, updating previous knowledge owing to deprecation warning, squashing each bug one-by-one on my on, in this open-ended project( according to me), never done before(ML with Python in Node.js, well maybe but I haven’t searched a lot for that). For me it’s a big thing. I hope it might help someone somewhere. I also wrote it to relive those moments of frustration and temporary happiness on getting one thing work and other broken. Well that’s way of life.

If you liked my post please recommend and share to help others find it! Feel free to leave a comment below to criticise me for some mistakes to avoid them in future. Do visit my repo!😃

Edit: Thank you all for your support and Sahil Dua for spotting the typos.

Special Thanks to awesome guys at icons8 for providing me a picture to be used as the featured image for this post. Do checkout their other icons.