Tech Startups Beating FBI at Facial Recognition

So Don’t Panic. Or do? Idk

One night, HiringSolved founder and CEO Shon Burton had a crazy dream that he was Iron Man. He was wearing the mechanized super-suit complete with all the rockets, laser beams, and targeting helmet.

Anti-climatically, he wasn’t anywhere near a battlefield. He was at an HR conference.

Like many of us, Shon had fallen victim to dreaming about work.

Disappointed at first, Shon then noticed his helmet was doing something funny. Anywhere he looked, a targeting reticle would lock onto someone’s face and populate a floating field containing all of that person’s professional details.

Name, contact information, title, achievements… you name it.

When he woke up he thought “this is totally doable”.

“We were willing and excited to see what could be done — we have great data and interesting capabilities so we threw it together with the engineering team to see what happens. It became one of our summer labs projects.” Burton said.

Meanwhile…

A recent Georgetown Law study about the FBI’s use of facial recognition suggests 1984 has arrived and we are becoming members of The Party.

Articles with titles like Cops Have a Database of 117M Faces. You’re Probably in It from Wired help paint the picture that Big Brother is here.

Does advanced image processing create a “perpetual lineup” for local law enforcement, FBI, CIA and other federal agencies as Georgetown suggests?

Or did the Tony Starks of the world get there first?

“Face recognition error rates have decreased over the last twenty years by three orders of magnitude” — Facebook Research

With Big Data Comes Great Responsibility

Background on HiringSolved

HiringSolved plays primarily in the talent acquisition world. Their engineering team builds products like RAI — a conversational AI that helps recruiters discover candidates on the web. PROPHET — a Chrome extension that predicts and uncovers information about anyone whose page you visit, and TalentFeed — a Google-esque search engine that crawls the web as well as contracted company candidate databases to assemble and organize everyone into consistent professional profiles.

Tony Stark vs. The Law

Tenets of Facial Recognition

“The conventional pipeline consists of four stages: detect → align→ represent→ classify.” — Research at Facebook

Detecting: Finding a face

The FBI

In law enforcement, this stage relies on things like eye position or skin texture. This creates issues differentiating black people among other things.

Tech Startup

Unlike FBI methods to find the face in an image, HiringSolved’s face detection step doesn’t use anything except the direction of light.

This method is known as Histogram of Oriented Gradients or HOG. For the first step in this method, the system desaturates an image making it black-and-white rendering color irrelevant.

Then the system replaces all the pixels with arrows based on the procession of light-to-dark where the pixels once were, enabling the system to detect distinct features of a face based on the patterns created.

Aligning: Centering with face forward

The FBI

The FBI has trouble with faces that are obscured or not aligned properly.

For an accurate read,FBI systems have to capture a face straight-on. They cannot handle images more than 15 degrees off the center axis.

FBI Tech Designed for Mugshots

Tech Startup

The tech used by HiringSolved can align any face in any position using a two-part aligning method.

The first part is an algorithm based on a method of “face-landmark estimation.” This breaks a face down into 68 points which each reference a facial landmark.

The second part is an algorithm that conducts “affine transformation.” This rotates, scales, and shears the image so everything is front and center.

Any processed image should now have the eyes and mouth in roughly the same spots so it is prepped for encoding.

Representing: Converting image into numeric set

The FBI

The FBI may not do this crucial time-saving step.

A document from their face recognition program suggests they may skip representation altogether:

“All search submissions to the IPSFRP system MUST include a single frontal photo of the search subject. Further, face images must be compressed within the search transaction using JPEG encoding.”

Also, their system compares pre-determined measurements of features like distance between the eyes.

Tech Startup

Matching millions of image files to each other takes forever so tech startups use a set of numbers to represent and processes matches much much faster.

HiringSolved’s system uses a machine learning technique that decides which 128 measurements on a face to log, unlike FBI systems that use measurements determined by humans.

HiringSolved then takes the aligned face and captures facial measurements for representation. This capture is known as “embedding” in machine learning circles.

For the best results in the following classification step, the rule-of-thumb is to embed 20+ images per person for higher accuracy.

Classifying: finding the closest match in database

The FBI

The FBI doesn’t find just one match. Instead the system finds the top 50. Out of those top 50 matches, it claims that one of them is a match only 85 percent of the time, according to an Electronic Frontier Foundation (EFF) study.

The EFF report comments that a system like this “will return a lot of images of the wrong people.”

The FBI relies on databases consisting solely of driver’s license photos and mugshots. So, their system can only really reference one or two photos of a person which lowers their measurement accuracy.

Tech Startup

HiringSolved has a similar issue to the FBI since their image database also has a low per-person count.

Instead of relying on a sub-par image volume, engineers elected to build a live trainer for the classifier.

The trainer boots on a MacBook, asks for the user’s name and company, and then captures a short video. This takes 50 images of the user for the embedding.

Once the classifier is trained using the 50 images, it can be accessed by a computer’s camera via a server.

Since HiringSolved is working towards a mobile facial recognition product, they built the classifier app for a Droid Turbo 2.

||

v

After training the classifier with multiple people’s faces, the device is able to find a match in two seconds!

Fear the FBI?

Should we fear our government overlords?

No. Not yet.

While the FBI has access to face data, it’s too thinly-spread and they aren’t able to use it efficiently.

If a small tech startup can can develop highly accurate facial recognition software for use on a camera phone relatively cheaply, rivaling the 98% accuracy rate that Facebook currently touts, we need to trust the Tony Starks (and Facebooks, Snapchats, or Googles) of the world are using their powers for good.

Some entities, like Facebook, have a major inherent advantage with a high volume of indexed images while others struggle to create gimmicks that collect face data (see Microsoft’s “How Old Do I Look?” app).

Getting people to give you a facial embedding is hard.

If only there was a way to coerce people. We could make it fun. Free. And social. Let people invite their friends. Spread it. Let them put silly cartoon items on their face. Then promise we’re deleting it all.

To be fair, Snapchat claims they can’t “Recognize a Specific Face”.

So why worry? It’s not like they have a secret engineering wing or something like that.

(Disclaimer: I wrote this story in the fall of 2016 so there’s a good chance that federal agencies have begun co-opting the latest face rec. techniques by now.)