How to Spot a DeepFake in 2021

Wondering about the best ways to spot a deepfake? In this video, learn about a breakthrough US Army technology that uses artificial intelligence to find deepfakes.

Watch the video

References

►Read the full article: https://www.louisbouchard.ai/spot-deepfakes

►Test your deepfake detection capacity: https://detectfakes.media.mit.edu/

►DeepFakeHop: Chen, Hong-Shuo et al., (2021), “DefakeHop: A Light-Weight High-Performance Deepfake Detector.” ArXiv abs/2103.06929

►Saab Transforms: Kuo, C.-C. Jay et al., (2019), “Interpretable Convolutional Neural Networks via Feedforward Design.” J. Vis. Commun. Image Represent.►OpenFace 2.0: T. Baltrusaitis, A. Zadeh, Y. C. Lim and L. Morency, "OpenFace 2.0: Facial Behavior Analysis Toolkit," 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 2018, pp. 59-66, doi: 10.1109/FG.2018.00019.

Video Transcript

00:00

While they seem like they've always been there, the very first realistic deepfake didn't appear

00:05

until 2017.

00:07

It went from these first-ever resembling fake images automatically generated to today's

00:13

identical copy of someone on videos, with sound.

00:16

The reality is that we cannot see the difference between a real video or picture and a deepfake

00:22

anymore.

00:23

How can we tell what's real from what isn't?

00:25

How can audio files or video files be used in court as proof if an AI can entirely generate

00:32

them?

00:33

Well, this new paper may provide answers to these questions.And the answer here may again

00:37

be the use of artificial intelligence.

00:40

The saying "I'll believe it when I'll see it" may soon change for "I'll believe it when

00:45

the AI tells me to believe it..."

00:47

I will assume that you've all seen deepfakes and know a little about them.

00:51

Which will be enough for this article.

00:53

For more information about how they are generated, I invite you to watch the video I made explaining

00:58

deepfakes just below, as this video will focus on how to spot them.

01:04

More precisely, I will cover a new paper by the USA DEVCOM Army Research Laboratory entitled

01:09

"DEFAKEHOP: A LIGHT-WEIGHT HIGH-PERFORMANCE DEEPFAKE DETECTOR."

01:14

Indeed, they can detect deepfakes with over 90% accuracy in all datasets and even reach

01:20

100% accuracy in some benchmark datasets.

01:24

What is even more incredible is the size of their detection model.

01:27

As you can see, this DeFakeHop model has merely 40 thousand parameters, whereas the other

01:33

techniques yielding much worse accuracy had around 20 million!

01:37

This means that their model is 500 HUNDRED times smaller while outperforming the previous

01:43

state-of-the-art techniques.

01:44

This allows the model to quickly run on your mobile phone and allows you to detect deep

01:49

fakes anywhere.

01:50

You may think that you can tell the difference between a real picture or a fake one, but

01:54

if you remember the study I shared a couple of weeks ago, it clearly showed that around

01:59

50 percent of participants failed.

02:01

It was basically a random guess on whether a picture was fake or not.

02:05

There is a website from MIT where you can test your ability to spot deefakes if you'd

02:10

like to.

02:11

Having tried it myself, I can say it's pretty fun to do.

02:14

There are audio files, videos, pictures, etc.

02:17

The link is in the description below.

02:19

If you try it, please let me know how well you do!

02:21

And if you know any other fun apps to test yourself or help research by trying our best

02:26

to spot deepfakes, please link them in the comments.

02:29

I'd love to try them out!

02:31

Now, if we come back to the paper able to detect them much better than we can, the question

02:36

is: how is this tiny machine learning model able to achieve that while humans can't?

02:42

DeepFakeHop works in four steps.

02:44

Step 1:

02:45

At first, they use another model to extract 68 different facial landmarks from each video

02:50

frame.

02:51

These 68 points are extracted to understand where the face is, recenter, orient and resize

02:57

it to make them more consistent, and then extract specific parts of the face from the

03:02

image.

03:03

These are the "patches" of the image we will send our network, containing specific individual

03:09

face features like the eyes, mouth, nose.

03:12

It is done using another model called OpenFace 2.0.

03:16

It can accurately perform facial landmark detection, head pose estimation, facial action

03:22

unit recognition, and eye-gaze estimation in real-time.

03:26

These are all tiny patches of 32 by 32 that will all be sent into the actual network one

03:32

by one.

03:33

This makes the model super efficient because it deals with only a handful of tiny images

03:38

instead of the full image.

03:40

More details about OpenFace2.0 can be found in the references below if you are curious

03:45

about it.

03:46

Step 2 to 4 (left to right, blue, green, orange):

03:47

More precisely, the patches are sent to the first PixelhHop++ unit named Hop-1, as you

03:53

can see.

03:54

Representing the step one in blue.

03:55

This is an algorithm called Saab transform to reduce the dimension.

03:58

It will take the 32 by 32 image and reduce it to a downscaled version of the image but

04:04

with multiple channels representing its response from different filters learned from the Saab

04:10

transform.

04:11

You can see the Saab transform as a convolution process, where the kernels are found using

04:16

the PCA dimension reduction algorithm replacing the need of backpropagation to learn these

04:21

weights.

04:22

I will come back to the PCA dimension reduction algorithm in a minute as it is repeated in

04:26

the next stage.

04:27

These filters are optimized to represent the different frequencies in the image, basically

04:32

getting activated by varying degrees of details.

04:35

The Saab transform was shown to work well against adversarial attacks compared to basic

04:40

convolutions trained with backpropagation.

04:43

You can also find more information about the Saab transformation in the references below.

04:47

If you are not used to how convolutions work, I strongly invite you to watch the video I

04:52

made introducing them:

04:56

I said Saab transforms worked well on adversarial attacks.

05:00

These adversarial attacks happen when we "attack" an image by changing a few pixels or adding

05:06

noise that humans cannot see to change the results of a machine learning model processing

05:11

the image.

05:12

So to simplify, we can basically see this PixelHop++ Unit as a typical 3 by 3 convolution

05:19

here since we do not look at the training process.

05:21

Of course, it works a bit differently, but it will make the explanation much more straightforward

05:26

as the process is comparable.

05:28

Then, the "Hop" step is repeated three times to get smaller and smaller versions of the

05:33

image with concentrated general information and more channels.

05:37

These channels are simply the outputs, or responses, of the input image by filters that

05:42

react differently depending on the level of detail in the image, as I said earlier.

05:48

One new channel per filter used.

05:50

Thus, we obtain various results giving us precise information about what the image contains,

05:55

but these results are smaller and smaller containing less spatial details unique to

06:00

that precise image sent in the network, and therefore have more general and useful information

06:06

with regard to what the image actually contains.

06:09

The first few images are still relatively big, starting at 32 by 32, being the initial

06:15

size of the patch and thus contains all the details.

06:18

Then, it drops to 15 by 15, and finally to 7 by 7 images, meaning that we have close

06:24

to zero spatial information in the end.The 15 by 15 image will just look like a blurry

06:29

version of the initial image but still contains some spatial information, while the 7 by 7

06:35

image will basically be a very general and broad version of the image with close to no

06:40

spatial information at all.

06:43

So just like a convolutional neural network, the deeper we get, the more channels we have

06:48

meaning that we have more filter responses reacting to different stimuli, but the smaller

06:53

they each are, ending with images of size 5x5.

06:56

Allowing us to have a broader view in many ways, keeping a lot of unique valuable information

07:02

even with smaller versions of the image.

07:05

The images get even smaller because each of the PixelHop units is followed by a max-pooling

07:12

step.

07:13

They are simply taking the maximum value of each square of two by two pixels, reducing

07:17

the image size by a factor of four at each step.

07:20

Then, as you can see in the full model shown above, the outputs from each max-pooling layer

07:24

are sent for further dimension reduction using the PCA algorithm.

07:26

Which is the third step, in green.

07:28

The PCA algorithm mainly takes the current dimensions, for example, 15 by 15 here in

07:34

the first step, and minimizes that while maintaining at least 90% of the intensity of the input

07:40

image.

07:41

Here is a very simple example of how PCA can reduce the dimension, where two-dimensional

07:45

points of cats and dogs are reduced to one dimension on a line, allowing us to add a

07:51

threshold and easily build a classifier.

07:54

Each hop gives us respectively 45, 30, and 5 parameters per channel instead of having

08:00

images of size 15 by 15, 7 by 7, and 3 by 3, which would give us in the same order 225,

08:07

49, and 9 parameters.

08:11

This is a much more compact representation while maximizing the quality of information

08:16

it contains.

08:17

All these steps were used to compress the information and make the network super fast.

08:22

You can see this as squeezing all the helpful juice at different levels of details of the

08:27

cropped image to finally decide whether it is fake or not, using both detailed and general

08:32

information in the decision process (step 4 in orange).

08:35

I'm glad to see that the research in countering these deepfakes is also advancing, and I'm

08:39

excited to see what will happen in the future with all that.

08:43

Let me know in the comments what you think will be the main consequences and concerns

08:47

regarding deepfakes.

08:48

Is it going to affect law, politics, companies, celebrities, ordinary people?

08:53

Well, pretty much everyone...

08:55

Let's have a discussion to share awareness and spread the word to be careful and that

09:00

we cannot believe what we see anymore, unfortunately.

09:02

This is both an incredible and dangerous new technology.

09:06

Please, do not abuse this technology and stay ethically correct.

09:10

The goal here is to help improve this technology and not to use it for the wrong reasons.