From Amputee to Cyborg with this AI-Powered Hand 🦾

Researchers used AI models based on the current neural networks (RNN) to read and accurately decode the amputee’s intent of moving individual fingers from peripheral nerve activities.

The AI models are deployed on an NVIDIA Jetson Nano as a portable, self-contained unit. With this AI-powered nerve interface, the amputee can control a neuroprosthetic hand with life-like dexterity and intuitiveness.

Watch the video

►Subscribe to my newsletter: http://eepurl.com/huGLT5

References

[1] Nguyen & Drealan et al. (2021) A Portable, Self-Contained Neuroprosthetic Hand with Deep Learning-Based Finger Control: https://arxiv.org/abs/2103.13452​

[2]. Luu & Nguyen et al. (2021) Deep Learning-Based Approaches for Decoding Motor Intent from Peripheral Nerve Signals: https://www.researchgate.net/publication/349448928_Deep_Learning-Based_Approaches_for_Decoding_Motor_Intent_from_Peripheral_Nerve_Signals

[3]. Nguyen et al. (2021) Redundant Crossfire: A Technique to Achieve Super-Resolution in Neurostimulator Design by Exploiting Transistor Mismatch: https://experts.umn.edu/en/publications/redundant-crossfire-a-technique-to-achieve-super-resolution-in-ne​ 

[4]. Nguyen & Xu et al. (2020) A Bioelectric Neural Interface Towards Intuitive Prosthetic Control for Amputees: https://www.biorxiv.org/content/10.1101/2020.09.17.301663v1.full

Video Transcript

00:00

in this video i will talk about a

00:02

randomly picked application of

00:03

transformers from the 600 new papers

00:06

published this week

00:07

adding nothing much to the field but

00:09

improving the accuracy by 0.1 percent on

00:11

one benchmark by tweaking some

00:13

parameters

00:14

i hope you are not too excited about

00:16

this introduction because that was just

00:18

to mess with the transformers recent

00:19

popularity

00:20

of course they are awesome and super

00:22

useful in many cases

00:23

and most researchers are focusing on

00:25

them but other things exist in ai

00:28

that are as exciting if not more you can

00:31

be sure

00:31

will cover exciting advancements of the

00:33

transformers architecture applied to nlp

00:36

computer vision or other fields

00:38

as i think it is very promising but

00:40

covering these new papers making slight

00:42

modifications to them is not as

00:44

interesting to me

00:45

just as an example here are a couple of

00:47

papers shared in march

00:49

applying transformers to image

00:51

classification and since they are all

00:53

quite similar and i already covered one

00:55

of them

00:56

i think it is enough to have an overview

00:58

of the current state of transformers

00:59

in computer vision now let's enter the

01:02

real subject

01:03

of this video which is nothing related

01:05

to transformers or even

01:06

gans in that case no hot words at all

01:09

except maybe

01:10

cyberpunk and yet it's one of the

01:12

coolest applications of ai

01:14

i've seen in a while it attacks a real

01:16

world problem and can change the lives

01:18

of many people

01:19

of course it's less glamour than

01:21

changing your face into an anime

01:23

character

01:23

or a cartoon but it's much more useful i

01:26

present you the portable

01:28

self-contained neuroprosthetic hand with

01:30

deep learning based

01:31

finger control by anguian drilan ital

01:34

before diving into it i just wanted to

01:36

remind you of the free nvidia gtc event

01:39

happening

01:40

next week with many exciting news

01:42

related to ai

01:43

and the deep learning institute giveaway

01:45

i am running if you subscribe to my

01:46

newsletter if you are interested i

01:48

talked about this giveaway with much

01:50

more details in my previous video

01:52

also i just wanted to announce that from

01:54

now on all new youtube members will have

01:57

a specific role on my discord channel as

01:59

a thank you for your support

02:00

now let's jump right into this unique

02:03

and amazing paper

02:04

this new paper applies deep learning to

02:06

a neuroprosthetic hand to allow

02:08

real-time control of individual finger

02:11

movements

02:11

all done directly within the arm itself

02:14

with

02:15

as little as 50 to 120 milliseconds of

02:18

latency

02:19

and up to 99 accuracy an arm amputee who

02:22

has lost his hand for 14 years can move

02:24

its cyborg fingers

02:26

just like a normal hand this work shows

02:28

that the deployment of deep neural

02:30

network applications embedded directly

02:32

on wearable biomedical devices is first

02:35

possible

02:36

but also extremely powerful here deep

02:38

learning is used to process and decode

02:40

nerve data acquired from the amputee to

02:43

obtain dexterous finger movements

02:45

the problem here is that in order to be

02:47

low latency this deep learning model

02:50

has to be on a portable device with much

02:52

lower computational power

02:53

than our gpus fortunately there has been

02:56

recent development of compact hardware

02:58

for deep learning users to fix this

03:00

issue

03:01

in this case they use the nvidia jetson

03:03

nano module

03:04

specifically designed to deploy ai in

03:07

autonomous applications

03:08

it allowed the use of gpus and powerful

03:10

libraries like tensorflow and pytorch

03:13

inside the arm itself

03:14

as they state this offers the most

03:16

appropriate trade-off

03:18

among size power and performance for our

03:20

neural decoder implementation

03:22

which was the goal of this paper address

03:24

the challenge of

03:26

efficiently deploying deep learning

03:28

neural decoders

03:29

on a portable device using real-life

03:31

applications

03:32

towards long-term clinical uses of

03:35

course there are a lot of technical

03:36

details that i will not enter into

03:38

as i am not an expert like how the nerve

03:40

fibers

03:41

and bioelectronics connect together the

03:44

microchip's designs that allows the

03:46

simultaneous neural recording

03:48

and stimulation or the implementation of

03:50

software and hardware

03:51

to support this real-time motor decoding

03:54

system you can read a great explanation

03:56

of these in their papers

03:58

if you'd like to learn more about it

03:59

they are all linked in the description

04:01

of the video

04:02

but let's dive a little more into the

04:04

deep learning side of this insane

04:06

creation

04:07

here their innovation leaned to

04:09

optimizing the deep learning motor

04:10

decoding to reduce as much as possible

04:13

the computational complexity

04:15

into this jetson nano platform this

04:17

image shows

04:18

an overview of the data processing flow

04:20

on the gesture nano

04:21

at first the data in the form of

04:23

peripheral nerve signals

04:25

from the amputee's arm is sent into the

04:28

platform

04:29

then it is pre-processed this step is

04:31

crucial to cut

04:32

raw input neural data into trials and

04:35

extract their main features

04:36

in the temporal domain before feeding to

04:39

the models

04:40

this preprocessed data correspond to the

04:42

main features

04:43

of one second of past neural data from

04:46

the amputee

04:47

cleaned from all noise sources then

04:50

this process data is sent into the deep

04:52

learning model

04:53

to have a final output controlling each

04:55

finger's movement

04:56

note that there are five outputs one for

04:58

each finger

04:59

to quickly go over the model they used

05:01

as you can see it starts with a

05:03

convolutional layer

05:05

this is used to identify different

05:06

representations of data input

05:09

in this case you can see the 64 meaning

05:11

that there are

05:12

64 convolutions made using different

05:15

filters

05:15

so 64 different representations these

05:18

filters are the network parameters

05:20

learned during training to correctly

05:22

control the hand when finally deployed

05:25

then we know that time is very important

05:27

in this case since we want fluid

05:28

movements of the fingers

05:30

so they opt for gated recurrent units or

05:33

gru to represent this time dependency

05:35

aspect when decoding the data

05:37

grews will allow the model to understand

05:40

what the hand was doing in the past

05:41

second

05:42

what is first encoded and what it needs

05:44

to do next

05:45

what is decoded to stay simple gru's are

05:49

just an improved

05:50

version of recurrent neural networks or

05:52

rnns solving computational problems

05:54

rnns had with long inputs by adding

05:57

gates to keep only the relevant

05:59

information

06:00

of past inputs in the recurrent process

06:02

instead of washing out

06:04

the new input every single time it's

06:06

basically allowing the network to decide

06:08

what information should be passed to the

06:10

output

06:11

as in recurrent neural networks the one

06:13

second data here

06:14

in the form of a 512 features is

06:17

processed iteratively

06:19

using the repeated gru blocks each dru

06:22

block

06:22

receives the input at the current step

06:25

and the previous output to produce the

06:27

following output

06:28

we can see gru's as an optimization of

06:30

the basic recurrent neural network

06:32

architecture finally this decoded

06:35

information is sent to linear layers

06:37

basically just propagating the

06:38

information and condensing it

06:40

into probabilities for each individual

06:42

finger

06:43

they studied many different

06:44

architectures as you can read in their

06:46

paper

06:47

but this is the most computationally

06:49

effective model they could make

06:50

yielding incredible accuracy of over 95

06:53

percent for the movement of the fingers

06:56

now that we have a good idea of how the

06:57

model works and know that it's accurate

07:00

some questions are still left such as

07:02

what does the person using it

07:04

feels about it does it feel real does it

07:06

work

07:07

etc in short is this similar to a real

07:10

arm

07:10

as the patient himself said i feel like

07:13

once this thing is fine tuned as

07:15

finished products that are out there

07:17

it will be more lifelike functions to be

07:19

able to do everyday tasks

07:21

without thinking of what positions the

07:23

hand is in

07:24

or what mode i have the hand programmed

07:26

in it's just like if i want to reach and

07:28

pick up something

07:29

i just reach and pick up something

07:31

knowing that it's just like my able hand

07:34

for every functions i think we will get

07:36

there i really do

07:38

please just take one more minute of your

07:40

time to watch this short

07:41

touching video where the amputee uses

07:44

the hand

07:44

and shares his honest feedback

07:50

is it pleasurable playing with it oh

07:52

yeah

07:53

it's just really cool like this is this

07:58

is crazy cool

08:00

to me these are the most incredible

08:02

applications that we can work on

08:04

with ai it directly helps real people

08:07

improve their lives quality

08:09

and there's nothing better than that i

08:11

hope you enjoyed watching this video

08:13

and don't forget to subscribe to the

08:14

channel to stay up to date with

08:16

artificial intelligence news

08:18

thank you for watching and as he just

08:20

said in the video

08:21

i will say the same about ai in general

08:24

this is

08:24

crazy cool