What Did AI Bring to Computer Vision?

In this video, I will openly share everything about deep nets for computer vision applications, their successes, and the limitations we have yet to address.

Watch the video

References

Read the article: https://www.louisbouchard.ai/ai-in-computer-vision/

Yuille, A.L., and Liu, C., 2021. Deep nets: What have they ever done for vision?. International Journal of Computer Vision, 129(3), pp.781–802, https://arxiv.org/abs/1805.04025.

Video Transcript

00:00

if you clicked on this video you are

00:01

certainly interested in computer vision

00:04

applications

00:04

like image classification image

00:06

segmentation object detection

00:08

and more complex tasks like face

00:10

recognition image generation or even

00:12

star transfer application

00:14

as you may already know with the growing

00:16

power of our computers

00:17

most of these applications are now being

00:19

realized using similar deep neural

00:21

networks

00:22

what we often refer to as artificial

00:25

intelligence models

00:26

there are of course some differences

00:28

between the deep nets used in these

00:30

different vision applications

00:31

but as of now they all use the same

00:34

basis of convolutions

00:35

introduced in 1989 by yan loken the

00:38

major difference here

00:40

is our computation power coming from the

00:42

recent advancements of gpus

00:44

to quickly go over the architecture as

00:46

the name says convolution

00:48

is a process where an original image or

00:50

video frame which is

00:51

our input in a computer vision

00:53

applications is convolved

00:55

using filters that detect important

00:57

small features of an image

00:59

such as edges the network will

01:01

autonomously learn

01:02

filter values that detect important

01:04

features to match the output we want to

01:06

have

01:07

such as the object's name in a specific

01:09

image sent as input

01:11

for a classification task these filters

01:13

are usually of size

01:15

3x3 or 5x5 pixel squares allowing them

01:18

to detect the direction of the edges

01:20

left right up or down just like you can

01:23

see in this image

01:24

the process of convolution makes a dot

01:26

product between the filter and the

01:28

pixels it faces

01:29

it's basically just a sum of all the

01:31

filter pixels multiplied with the values

01:34

of the images pixels

01:35

at the corresponding positions then it

01:38

goes to the right and does it again

01:40

convolving the whole image once it's

01:42

done these convolved features

01:44

give us the output of the first

01:46

convolution layer which we call this

01:48

output a feature map

01:49

we repeat this process with many other

01:51

filters giving us multiple feature maps

01:54

one for each filter used in the

01:56

convolution process

01:57

having more than one feature map gives

01:59

us more information about the image

02:01

and especially more information that we

02:03

can learn during training

02:04

since these filters are what we aim to

02:06

learn for our task

02:08

these feature maps are all sent into the

02:10

next layer

02:11

as input to produce many other smaller

02:14

sized

02:14

feature maps again the deeper we get

02:17

into the network the smaller these

02:18

feature maps gets

02:20

because of the nature of convolutions

02:21

and the more general the information of

02:23

these feature maps become

02:25

until it reaches the end of the network

02:27

with extremely general information

02:29

about what the image contains disposed

02:31

of our many feature maps

02:33

which is used for classification or to

02:35

build a latent code

02:37

to represent information present in the

02:39

image in the case of a gan architecture

02:41

to generate a new image

02:43

based on this code which we refer to as

02:45

encoded information

02:47

in the example of image classification

02:49

simply put

02:50

we can see that at the end of the

02:52

network these small feature maps contain

02:54

the information about the presence of

02:56

each possible class telling you whether

02:58

it's a dog a cat

03:00

a person etc of course this is super

03:03

simplified

03:04

and there are other steps but i feel

03:06

like this is an accurate summary of

03:07

what's going on

03:08

inside a deep convolutional neural

03:10

network

03:11

if you've been following my channel and

03:13

posts you know that deep neural networks

03:15

proved

03:16

to be extremely powerful again and again

03:18

but they also have weaknesses and

03:20

weaknesses

03:21

that we should not try to hide as with

03:24

all things in life

03:25

deep nets have strength and weaknesses

03:27

while strengths are widely shared

03:30

the latter is often omitted or even

03:32

discarded by companies

03:34

and ultimately by some researchers this

03:36

paper

03:37

by alan yule and chenxileo aims to

03:40

openly share

03:41

everything about deep nets for vision

03:43

applications their success and the

03:45

limitations we have to address

03:47

moreover just like for our brain we

03:50

still do not fully understand their

03:52

inner workings

03:53

which makes the use of deep nets even

03:55

more limited since we cannot maximize

03:57

their strength

03:58

and limit weaknesses as stated by o

04:00

hobart

04:01

it's like a road map that tells you

04:02

where cars can drive but doesn't tell

04:04

you when or where

04:06

cars are actually driving this is

04:08

another point they discuss

04:09

in their paper namely what is the future

04:12

of computer vision algorithms

04:14

as you may be thinking one way to

04:16

improve computer vision applications is

04:18

to understand our own visual system

04:20

better starting with our brain which is

04:23

why

04:24

neuroscience is such an important field

04:26

for ai

04:27

indeed current deep nets are

04:29

surprisingly different than our own

04:31

vision system

04:32

firstly humans can learn from very small

04:35

numbers of examples

04:37

by exploiting our memory and the

04:39

knowledge we already acquired

04:40

we can also exploit our understanding of

04:43

the world and its physical properties to

04:45

make

04:45

deductions something that a deep net

04:47

cannot do in 1999

04:50

gupp nick ital explained that babies are

04:52

more like tiny scientists

04:54

who understand the world by performing

04:56

experiments

04:57

and seeking causal explanations for

05:00

phenomena rather than

05:01

simply receiving stimulus from images

05:03

like current

05:04

deep nets do also we humans

05:07

are much more robust as we can easily

05:10

identify an object from any viewpoint

05:12

texture it has occlusions it may

05:14

encounter and novel context

05:16

as a concrete example you can just

05:18

visualize the annoying captcha you

05:20

always have to fill in

05:22

when logging into a website this captcha

05:24

is used to detect butts since they are

05:26

awful

05:27

when there are occlusions like this as

05:30

you can see here

05:31

the deep net got fooled by all the

05:33

examples because of the jungle context

05:35

and the fact that a monkey

05:37

is not typically holding a guitar this

05:39

happens because it's certainly not in

05:41

the training data set

05:43

of course this exact situation might not

05:45

happen very often

05:46

in real life but i will show some more

05:48

concrete examples

05:49

that are more relatable and that already

05:52

happened later on

05:53

in the video deep nets also have

05:55

strength that we must highlight

05:57

they can outperform us for face

05:59

recognition tasks since

06:00

humans are not used to until recently

06:03

seeing more than a few thousands of

06:05

people

06:05

in their whole lifetime but this

06:07

strength of deep nets also comes with a

06:09

limitation

06:10

where these faces need to be straight

06:12

centered clear

06:14

without any occlusions etc indeed

06:17

the algorithm could not recognize your

06:19

best friend at the alwyn party

06:21

disguised in harry potter having only

06:23

glasses and a lightning bolt on the

06:25

forehead

06:26

where you would instantly recognize him

06:28

and see

06:29

whoa that's not very original it looks

06:31

like you just put glasses on

06:33

similarly such algorithms are extremely

06:36

precise radiologists

06:37

if all the settings are similar to what

06:39

they have been seeing

06:40

during their training they will

06:42

outperform any human

06:43

this is mainly because even the most

06:45

expert radiologists have only seen

06:48

a fairly small number of ct scans in

06:50

their lives as they suggest

06:51

the superiority of algorithms may also

06:54

be because they are doing a low priority

06:56

task

06:57

for humans for example a computer vision

06:59

app on your phone can identify the

07:02

hundreds of plants in your garden much

07:04

better than

07:05

most of us watching the video can but a

07:08

plant expert

07:08

will surely outperform it and all of us

07:11

together as well

07:12

but again this strength comes with a

07:14

huge problem

07:15

related to the data the algorithm needs

07:17

in order to be this powerful

07:19

as they mentioned and as we often see on

07:21

twitter and article titles

07:23

there are biases due to the data set

07:25

these deep nets are trained on

07:28

since an algorithm is only as good as

07:31

the data set it is evaluated on

07:33

and the performance measures used this

07:35

dataset limitation

07:37

comes with the price that these deep

07:39

neural networks are much

07:40

less general purpose flexible and

07:43

adaptative

07:44

than our own visual system they are less

07:46

general purpose

07:47

and flexible in the way that contrary to

07:50

our visual system

07:51

where we automatically perform edge

07:53

detection binocular stereo

07:55

semantic segmentation object

07:57

classification scene classification and

07:59

3d

08:00

depth estimation deep nets can only be

08:02

trained to achieve

08:03

one of these tasks indeed simply by

08:05

looking around

08:06

your vision system automatically

08:08

achieves all these tasks with extreme

08:10

precision

08:10

where deep nets have difficulty

08:12

achieving similar precision on one of

08:14

them

08:14

but even if this seems effortless to us

08:17

half of our neurons are at work

08:19

processing the information

08:20

and analyzing what's going on we are

08:23

still

08:23

far from mimicking our vision system

08:26

even with the current depth of our

08:27

networks

08:28

but is that really the goal of our

08:30

algorithms will it be better to just

08:33

use them as a tool to improve our

08:35

weaknesses i couldn't say

08:36

but i am sure that we want to address

08:39

the deep nets limitations

08:40

that can cause serious consequences

08:43

rather than omitting them

08:44

i will show some concrete examples of

08:47

such consequences

08:48

just after introducing these limitations

08:50

but if you are too intrigued you can

08:52

skip

08:52

right to it following the timestamps

08:54

under the video and come back to the

08:56

explanation

08:57

afterwards indeed the lack of precision

08:59

we previously mentioned

09:01

by deepnets arises mainly because of the

09:03

disparity between the data we use to

09:05

train our algorithm

09:06

and what it sees in real life as you

09:09

know an algorithm needs to see a lot of

09:11

data

09:11

to iteratively improve at the task it is

09:14

trained for

09:14

this data is often referred to as a

09:17

training data set

09:18

this data disparity between the training

09:20

data set and the real

09:22

world is a problem because the real

09:23

world is too complicated to accurately

09:25

be represented

09:27

in a single data set which is why deep

09:29

nets are less additive than our vision

09:31

system

09:32

in the paper they call this the

09:34

combinatorial complexity explosion of

09:36

natural images the combinatorial

09:38

complexity

09:39

comes from the multitude of possible

09:41

variations within a natural image

09:43

like the camera pose lighting texture

09:46

material

09:47

background the position of the objects

09:49

etc biases can appear at any of these

09:52

levels of complexity

09:53

the data set is missing you can see how

09:56

these large data sets now seem

09:57

very small due to all these factors

10:00

considering that having only

10:02

let's say 13 of these different

10:04

parameters and we allow only

10:06

1000 different values for each of them

10:08

we quickly jump to this number of

10:10

different images

10:11

to represent only a single object the

10:14

current data sets only cover a handful

10:16

of these multitudes of possible

10:18

variations for each object

10:20

thus missing most real-world situations

10:22

that it will encounter in production

10:24

it's also worth mentioning that since

10:26

the variety of images is very limited

10:28

the network may find shortcuts to

10:31

detecting some objects as we saw

10:33

previously with the monkey where it was

10:34

detecting a human instead of a monkey

10:37

because of the guitar in front of it

10:39

similarly you can see that it's

10:40

detecting a bird here

10:42

instead of a guitar probably because the

10:44

model has never

10:45

seen a guitar with a jungle background

10:48

this is called

10:49

overfitting to the background context

10:51

where the algorithm does not focus on

10:53

the right thing

10:54

and instead finds a pattern in the

10:56

images themselves rather than on the

10:58

object of interest

11:00

also these data sets are all built from

11:03

images taken by photographs

11:04

meaning that they only cover specific

11:06

angles and poses that do not transfer to

11:09

all orientation possibilities in the

11:11

real world

11:12

currently we use benchmarks with the

11:14

most complex data sets possible to

11:16

compare the current algorithms and rate

11:18

them

11:19

which if you recall are very incomplete

11:21

compared to the real world

11:23

nonetheless we are often happy with 99

11:26

accuracy for a task on such benchmarks

11:29

firstly the problem is that this

11:31

one-person error is determined on a

11:32

benchmark data set

11:34

meaning that it's similar to our

11:35

training data set in the way that it

11:37

doesn't

11:37

represent the richness of natural images

11:40

it's normal because it's impossible to

11:42

represent the real world in just a bunch

11:44

of images

11:45

it's just way too complicated and there

11:47

are too many situations possible

11:49

these benchmarks we use to test our data

11:52

set to determine whether or not

11:53

they are ready to be deployed in the

11:55

real world application are not really

11:57

accurate to determine how well it will

11:59

actually perform

12:00

which leads to the second problem that

12:02

is how it will actually perform

12:04

in the real world let's see that the

12:06

benchmark data set is huge

12:08

and most cases are covered and we really

12:11

have 99

12:12

accuracy what are the consequences of

12:14

the one percent of cases where the

12:16

algorithm fails in the real world

12:19

this number will be represented in

12:21

misdiagnosis

12:22

accidents financial mistakes or even

12:25

worse

12:25

death such cases could be a self-driving

12:28

car

12:28

during a heavy rainy day heavily

12:30

affecting the death sensors

12:32

used by the vehicle causing it to fail

12:34

many depth estimations

12:36

would you trust your life to this

12:37

partially blind robot taxi

12:40

i don't think i would similarly would

12:42

you trust a self-driving car at night to

12:44

avoid

12:44

driving over pedestrians or cyclists

12:47

where even yourself had difficulty

12:49

seeing them

12:50

these kinds of life-threatening

12:51

situations are so broad

12:53

that it's almost impossible that they

12:55

are all represented in the training data

12:57

set

12:57

and of course here i use extreme

12:59

examples of the most relatable

13:01

application

13:02

but you can just imagine how harmful

13:04

this could be

13:05

when the perfectly trained and tested

13:07

algorithm misclassifies your ct scan

13:09

leading to misdiagnosis just because

13:12

your hospital has different settings in

13:13

their scanner or because you didn't

13:15

drink enough water

13:16

or die anything that would be different

13:19

from your training data

13:20

could lead to a major problem in real

13:22

life even if the benchmark

13:24

used to test it says it's perfect also

13:27

as it already happened

13:28

this can lead to people in

13:30

underrepresented demographics being

13:32

unfairly treated by these algorithms

13:34

and even worse this is why i argue that

13:37

we must focus on the task

13:38

where the algorithms help us and not

13:41

where they replace

13:42

us as long as they are that dependent on

13:44

data

13:45

this brings us to the two questions they

13:47

highlight how can we efficiently test

13:49

these algorithms to ensure that they

13:51

work on these enormous data sets

13:53

if we can only test them on a finite

13:55

subset and two

13:57

how can we train algorithms infinite

13:59

size data sets so that they can perform

14:01

well

14:02

on the truly enormous datasets required

14:04

to capture the combinatorial complexity

14:07

of the real world

14:08

in the paper they suggest to rethink our

14:11

methods for benchmarking performance

14:13

and evaluating vision algorithms and i

14:15

agree entirely

14:17

especially now where most applications

14:19

are made for real life users instead of

14:21

only academic competitions

14:23

it's crucial to get out of these

14:24

academia evaluation metrics

14:26

and create more appropriate evaluation

14:28

tools we also have to accept that data

14:31

bias exists

14:32

and that it can cause real world

14:34

problems of course we need to learn to

14:36

reduce these biases

14:38

but also to accept them biases are

14:40

inevitable due to the combinatorial

14:42

complexity of the real world

14:44

that cannot be realistically represented

14:46

in a single data set of images

14:48

yet thus focusing our attention without

14:51

any play of words with transformers

14:53

on better algorithms that can learn to

14:55

be fair

14:56

even when trained on such incomplete

14:58

data sets

14:59

rather than having bigger and bigger

15:01

models trying to represent the most data

15:04

possible

15:05

even if it may look like it this paper

15:07

was not a criticism of current

15:08

approaches

15:09

instead it's an opinion piece motivated

15:11

by discussions with other researchers in

15:14

several disciplines

15:15

as they state we stress that views

15:17

expressed in the paper

15:18

are our own and do not necessarily

15:20

reflect

15:21

those of the computer vision community

15:23

but i must say

15:24

this was a very interesting read and my

15:27

views are quite similar

15:28

they also discuss many important

15:30

innovations that happen over the last 40

15:32

years in computer vision

15:34

that is definitely worth reading as

15:36

always the link to the paper

15:38

is in the description below to end on a

15:40

more positive note we are nearly

15:42

a decade into the revolution of deep

15:44

neural networks that started in 2012

15:47

with alexnet and the imagenet

15:49

competition since then

15:51

there has been immense progress on our

15:53

computation power

15:54

and the deep net architectures like the

15:56

use of batch normalization

15:58

residual connections and more recently

16:00

self-attention

16:01

researchers will undoubtedly improve the

16:03

architecture of deep nets but we shall

16:05

not forget that there are other ways to

16:07

achieve intelligent models than going

16:09

deeper and using more data of course

16:12

these ways are yet to be discovered

16:14

if this story of deep neural networks

16:16

sounds interesting to you

16:18

i made a video of one of the most

16:19

interesting architecture

16:21

along with a short historical review of

16:23

deep nets i'm sure you'll love it

16:25

thank you for watching

Also published on: https://www.louisbouchard.me/ai-in-computer-vision/