> [You can find me on Twitter @bhutanisanyam1](http://twitter.com/bhutanisanyam1), connect with me on [Linkedin here](https://www.linkedin.com/in/sanyambhutani/)  
> [Here](https://becominghuman.ai/a-self-driving-new-year-33284e592f35) and [Here](https://hackernoon.com/a-self-driving-new-year-2-d1bbc5a83570) are two articles on my Learning Path to [Self Driving Cars](https://hackernoon.com/tagged/self-driving-cars)

> [You can find the Markdown File Here](https://github.com/init27/MIT-6.S094-Deep-Learning-for-Self-Driving-Cars)

> [You can find the Lecture 1 Notes here](https://medium.com/init27-labs/mit-6-s094-deep-learning-for-self-driving-cars-2018-lecture-1-notes-807be1a50893)  
> [Lecture 2 Notes can be found here](https://hackernoon.com/mit-6-s094-deep-learning-for-self-driving-cars-2018-lecture-2-notes-e283b9ec10a0)  
> [Lecture 3 Notes can be found here](https://hackernoon.com/mit-6-s094-deep-learning-for-self-driving-cars-2018-lecture-3-notes-deep-reinforcement-learning-fe9a8592e14a)  
> [Lecture 5 Notes can be found here](https://medium.com/@init_27/mit-6-s094-deep-learning-for-self-driving-cars-2018-lecture-5-notes-deep-learning-for-human-5cb0f53e4f15)

> These are the [Lecture 4](https://www.youtube.com/watch?v=CLOAswsxudo) notes for the [MIT 6.S094: Deep Learning for Self-Driving Cars](https://selfdrivingcars.mit.edu) Course (2018), Taught by [Lex Fridman](https://twitter.com/lexfridman).

All Images are from the Lecture Slides.

### Computer Vision: Teaching Computers to See.

![](https://hackernoon.com/hn-images/1*YxkzAc1QwaSJHxm6Swi8Fw.png)

Computer Vision, as of Today is Deep [Learning](https://hackernoon.com/tagged/learning). Majority of the successes of our understanding of images, utilise Neural Networks.

*   Supervised [Learning](https://hackernoon.com/tagged/learning): Annotated data is provided by Humans. NN parses through the data maps from the raw sensory data to image categories and for a sanity check, it should perform well on a test set.

![](https://hackernoon.com/hn-images/1*KuSIFsaKvEVAqXjMtidvJQ.png)

Raw Sensory data: For the machine, images are in the form of numbers.

The images in the form of 1 channel or 3 channel numerical arrays, are taken as input by the NN, the output is produced by regressing or by classifying the image into various categories.

![](https://hackernoon.com/hn-images/1*pcpUdNJe0rWIrGg8L_OMuA.png)

We must be careful about our assumptions for what is easy and hard with Perception.

**Human Vision Vs Computer Vision.**

![](https://hackernoon.com/hn-images/1*U719aymTJg7PVNERWN_P7A.png)

1.  Structure of the Visual Cortex is in layers. As information is passed from our eyes to the brain, higher and higher order representation are formed. This is the inspiration behind Deep NN for images. Higher and higher representations are formed through the layers.   
    The early layers, taking in the raw pixels, finding edges. Further finding more abstract features by connecting these edges. Finally, finding higher order semantic meaning.
2.  Deep Learning is hard for Computer Vision:

![](https://hackernoon.com/hn-images/1*hVXlFofI5DJIMNSTgbIAjA.png)

*   Illumination variability is one of the biggest challenges in driving.

![](https://hackernoon.com/hn-images/1*PefR1HUW40ckJR-FIV4DYw.png)

*   Pose Variablity: NN are not good with representing pose. Objects in 2D plane of colour and texture look very different when the objects are rotated.

![](https://hackernoon.com/hn-images/1*i3SGzTySxqFd4tj8G-3n-w.png)

*   Inter-class variablity: For Classification problems, there is a lot of variability inside the classes and very little between the classes.

![](https://hackernoon.com/hn-images/1*1YEhXzcwANTdTfE0uf3PZw.png)

*   Occlusion: When part of the object is occluded by another object and we are tasked with identifying the object that is occluded.

![](https://hackernoon.com/hn-images/1*W30FXvziH9kcZDkKSMZvuA.png)

*   Philosophical Ambiguity: Image classfication !=Understanding.
*   Unsupervised Learning
*   Reinforcement Learning.

### Image classification Pipeline:

![](https://hackernoon.com/hn-images/1*jUu1Dvp-ezhDLgSM4FUAgA.png)

There is a bin with different categories inside each class. Those bins have a lot of examples of each. Task: Bin a new image into one of these classes.

Famous Datasets:

![](https://hackernoon.com/hn-images/1*NmY46s2lvQraj5IXLJfrAw.png)

*   MNIST
*   ImageNet

![](https://hackernoon.com/hn-images/1*2nXjWxFUZt__whbQRK14Nw.png)

*   CIFAR-10  
    One of the Simplest ones, containing 10 categories is commonly used to explore CNNs.

Trivial Example:

![](https://hackernoon.com/hn-images/1*-higFyNdjJvp15wTYShpuw.png)

*   Comparing images by subtracting the pixel intensity matrices and then sum up the differences of every element. If the sum is high, the images are different.   
    If we use this methodology, we can get 35% accuracy using L2 diff, 38% with L1 diff these are better than the random accuracy of10%.

![](https://hackernoon.com/hn-images/1*czz9e3Rd7L0SZTOLoGEL3g.png)

*   K-Nearest Neighbours:   
    Instead of finding the 1 image that is the closest to our dataset, we try to find k-closest images and bin them into k-classes. We vary the k from 1–5 and see how that changes the problems.

![](https://hackernoon.com/hn-images/1*8NVfvEr976S7t4yQ7zIy7A.png)

*   With k=7, we achieve 30% accuracy.   
    Human level accuracy is 95%.  
    With CNNs, we get 97.75% accuracy.

#### Working of Neural Networks:

![](https://hackernoon.com/hn-images/1*gSpVjRXsA0fLt3Df9zyx1g.png)

*   Each input signal is weighed, biased and added.
*   These are then inputted to a non-linear activation function.
*   Put together in more and more layers, these form a Deep NN.

![](https://hackernoon.com/hn-images/1*w-50YTIcul9hgW8OX5BDgA.png)

*   The NN Trains by making a forward pass, evaluating how close are examples are to the ground truth and then punishing the weights that resulted in incorrect decisions and rewarding the weights that had resulted in the correct decisions.

![](https://hackernoon.com/hn-images/1*IZgsCq8Ny18b_EvH2FPBig.png)

*   Ex: MNIST-For the Case of 10 cases, the outputs is 10 different values.
*   Each Neuron gets excited based on the class it represents.
*   The output is assigned to the class that shows the highest value of activation.

[**Convolutional Neural Networks**](https://hackernoon.com/convolutional-neural-network-in-5-minutes-8f867eb9ca39)

![](https://hackernoon.com/hn-images/1*zl0VrQAfpMz8VHXSFkxmGw.png)

When the NN is tasked to learning a complex task with large data and large number of objects, CNNs work efficiently.

‘Trick-Spatial Invarince’:  
An Object in the top left corner is the same as the object in the bottom right corner of an image. So we learn the same features across the image.

![](https://hackernoon.com/hn-images/1*WT92r_Cl2mV2NkOkJWeoVQ.png)

Convolution operation: Instead of the Fully connected layers; Here a 3rd dimension of depth is present. So the block take 3 input volumes and produce 3D output volumes

They take a slice of the image, ‘a window’ and slide it through the image. They apply the same weights to slice/window of an image to generate outputs. We can make many such filters.

![](https://hackernoon.com/hn-images/1*oxMa-pUYxeBIxN7y8sJZXw.png)

Parameters on each of these filters are shared. (If a feature is useful in one place, it’s useful everywhere) This allows parameter reduction significantly. The re-use of spatial features.

*   The Depth: The number of filters.
*   Stride: Pixels skipped when applying the filter
*   Padding: Adding 0 values on the borders to the input for the Convolutional layer.

Example:

![](https://hackernoon.com/hn-images/1*KinbvXl7QpjSMiDbU1Qtug.png)

*   Left Column: 3 Input Channel.
*   Filter W0: 2 channels of filter, each of size 3x3.
*   The 3x3 filters are to be ‘learned’
*   These are slided across the image to produce the output.
*   The operations are applied to produce the output.

**Convolution**

![](https://hackernoon.com/hn-images/1*y1YSSYRMteyZWvVWav0eiw.png)

*   The image is inputted.
*   We perform the identity filter to generate the Convolved image.
*   We perform different other filters to generate the Edges.
*   We can detect any kind of pattern and produce the outputs.
*   Depending on the filters, you’d have equal number of outputs, each showing where the patterns were found.

![](https://hackernoon.com/hn-images/1*gjM2QAWmgzbQy1l_6TWusA.png)

*   Task: Learn the useful patterns required for the classification task.
*   The filters have higher and higher order for representation.
*   Starting with edges, finally to the high semantic meaning spanning the images.

![](https://hackernoon.com/hn-images/1*U300iCYAYK6V5Biqa8Q-og.png)

*   Spanning Images: Pooling.   
    Taking the output of a convolutional operation and reducing the resolution of that by condensing the information, for example considering the max values in Max-Pooling.
*   Note: The ‘reduction in spatial resolution’ has detrimental effects when it comes to scene segmentation, but is better at finding higher order representation in images for classification.
*   Many such Convolutional Layers form a CNN.

![](https://hackernoon.com/hn-images/1*zW4fhjLmKABtNeD9IdugWA.png)

*   The Fully Connected layers allow us to apply this to particular domains.

**ImageNet Case Study**

![](https://hackernoon.com/hn-images/1*xYDmhdcJ9EGstz9R2KDgNg.png)

*   Task: Classification of one of the largest dataset of images.   
    14M+ Images  
    21k+ Categories  
    With Many Sub-Classes
*   These provide a good challenge to the check the Pose, Intra Class Variability, Illumination.
*   **Networks:**

![](https://hackernoon.com/hn-images/1*akfhZS4bMtBevd3jWshRiQ.png)

*   AlexNet 2012: First Significant Improvement.
*   ZFNet 2013
*   VGGNet 2014

![](https://hackernoon.com/hn-images/1*_x50STFcoXXW2SB-NspMNQ.png)

*   GoogLeNet 2014  
    Inception Modules were introduced.   
    Idea: It used the idea that different sized convolutions provide different value for the [network](https://hackernoon.com/tagged/network), doing different convolutions and concatenating.   
    Smaller convolutions: Features that are very Local/High res in texture.  
    Larger Convolutions: Higher/More Abstract Features.  
    Result: Fewer Parameters and Better Performance.

![](https://hackernoon.com/hn-images/1*YpRAepXS2R7Jusb9DV-vlA.png)

*   ResNet 2015  
    Inspiration: (Doesn’t always hold) Network depth increases representation power.   
    ‘Residual blocks’ allow creating much ‘Deeper’ Networks.  
    Residual Block:  
    \- Repeat a Simple Network Block, similar to RNNs.  
    \- Pass input along without transformation, along with the ability to learn the weights.  
    \- Every layer takes in the input of previous layer and the raw, untransformed data to learn something new.
*   CUImage 2016

![](https://hackernoon.com/hn-images/1*ynL9gNuqG2wO4HgXzj6lWA.png)

*   SENet 2017  
    Squeeze and Excitation Network:  
    \- Added a parameter to each channel of a convolutional block so that the network can adaptively adjust the weighting of each channel based on each feature map/input to the network.  
    \- Trick: Allow the Network to learn the weighting on each individual channels.   
    \- Note: This is applicable to any architecture. Since, it simply parametrises which filter to pick based on the content.

![](https://hackernoon.com/hn-images/1*o-IGEnXPgjAaYA_vD5kcmw.png)

*   ILSVRC Challenge Evaluation for Classification  
    Top 5 guesses.  
    Human Error is 5.1%  
    Surpassed in 2015.

![](https://hackernoon.com/hn-images/1*mdhBCpd6CZop7oqYoLj8JA.png)

*   Capsule Networks:  
    \- Inspiration: Consider what assumptions are made by the network and what information is thrown away.   
    \- CNNs, due to their spatial invariance-throw away the hierarchy between simple and complex objects.   
    \- Future challenge: Design NN that work with Rotational.

![](https://hackernoon.com/hn-images/1*rX34vEwIJj9Y6PcTEN3iig.png)

*   We take a CNN, change the last layers to apply them.

### Object Detection

![](https://hackernoon.com/hn-images/1*DfeDq_nsnk328DzAQDM_Qg.png)

Note: CNN produce a pixel level heat map of activations based on convolutions

**Scene understanding**

![](https://hackernoon.com/hn-images/1*mQSoSREvMOnOm--SMW-etw.png)

*   Goal: Classify every single pixel in the 2D projection, for the real 3D world.
*   Challenge: Marking boundaries at pixel level.
*   Use-Cases:  
    \- Precise boundaries of Object Matters in Medical, Driving.   
    \- In Driving, to mark exact boundaries of environment. ‘Fuse’ this with data from sensors. So fusing the semantic knowledge with 3D location in real world.

![](https://hackernoon.com/hn-images/1*r5jZl30yzM9b-FRPEBerMg.png)

*   FCN 2014:  
    \- Repurpose the ImageNet pretrained Nets  
    \- Replaced the Fully connected layers with decoders that upsampled the image to produce a heat map.  
    \- Connections are skipped to improve the coarseness of upsampling.

![](https://hackernoon.com/hn-images/1*yXdBVXGusK46sjcnFiQGVQ.png)

*   SegNet 2015:  
    \- Applied this to Driving Context.

![](https://hackernoon.com/hn-images/1*qGcv0l0kil6Sujll7-9fFA.png)

*   Dilated Convoltions 2015:  
    \- Convolution operation as pooling operation reduces the resolution significantly.   
    \- ‘Gridding’ maintains the local high-res textures while still capturing the spatial windows necessary.

![](https://hackernoon.com/hn-images/1*k3o-RXCbLa9wD2vFtDFPkg.png)

*   DeepLab v1,v2 2016:  
    \- Added Conditional Random Fields (CRFs): Post processing to smooth the segmentation by looking at the underlying image intensities.

**Key Aspects of Segmentation**

*   Fully Convolutional Network.
*   Conditional Random Forests.
*   Dilated Convolutions.

![](https://hackernoon.com/hn-images/1*cnuqDYlIXZPgc8D0H0FCgA.png)

ResNet-DUC 2017:

*   Dense upsampling Convolution, instead of bilinear upsampling to learn the upscaling features.
*   Hybrid Dilated Convoltion:  
    Convolution is spread apart from Input to output.
*   Trick: Parametrisation of The upscaling features.

**FlowNet**

![](https://hackernoon.com/hn-images/1*5CJii3phPuVKAmSrN9kIqg.png)

The methods discussed here, disregard the temporal dynamics, which is relevant in the case of Robotics.

*   Flow is helpful in approximating how each pixel in the input image moved wrt output image.
*   The Optical flow produces a direction of where the pixel moved and the magnitude of movement.   
    This allows us to take information detected from the First Frame and propagate it forward.
*   This is extremely slow for ‘Colouring book annotation’ (90 Minutes for 1 image).
*   FlowNetS-stacks two images as input.
*   FlowNetC-convolute separately, combine with correlation layer.

Challenge: Segmentation of images through time.

FlowNet 2 2016:

![](https://hackernoon.com/hn-images/1*5X8_jL2MNWrbyzYp9v_OYQ.png)

*   Combines FlowNetC and FlowNetS
*   Produces smoother flow field.
*   Preserves fine motion detail.
*   Runs at 8–140fps.
*   Process:  
    \- Stacking Networks as an approach.   
    \- Ordering of Dataset matters

### SegFuse

**Dataset:**

![](https://hackernoon.com/hn-images/1*Y5gJ16VWCp7JjkxVicey7A.png)

*   Original video of driving in 1080p in a 8k 360 degree video.

![](https://hackernoon.com/hn-images/1*pGjyjfC06gEOR87FqFSMhA.png)

*   Ground Truth for the Training Set, for every single frame.

![](https://hackernoon.com/hn-images/1*ZYJLm7XKhrFttvUDlzj6UA.png)

*   Output of the SOTA Segmentation
*   Optical Flow

**Task:**

![](https://hackernoon.com/hn-images/1*HhTeMzitc2k-WR57BEHYlg.png)

*   Use the original video, ground truth, segmentation from a SOTA Network and improve the segmentation.
*   Use output of the Network to help propagate the information better.   
    Can we figure out ways to use Temporal Information?

> [You can find me on Twitter @bhutanisanyam1](http://twitter.com/bhutanisanyam1), connect with me on [Linkedin here](https://www.linkedin.com/in/sanyambhutani/)  
> [Here](https://becominghuman.ai/a-self-driving-new-year-33284e592f35) and [Here](https://hackernoon.com/a-self-driving-new-year-2-d1bbc5a83570) are two articles on my Learning Path to Self Driving Cars

> [Subscribe to my Newsletter for a Weekly curated list of Deep learning, Computer Vision Articles](https://tinyletter.com/sanyambhutani)

Flow

Stacks

Twitter

YouTube

2022 - HackerNoon Contributor of the Year - Computer Science

2022 - HackerNoon Contributor of the Year - Deep Learning

2022 - HackerNoon Contributor of the Year - Machine Learning

Connect with me on Twitter

Nominated for 2022 - HackerNoon Contributor of the Year - Deep Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Computer Science

Too Long; Didn't Read

Build an AI-based text classifier with Tensorflow & NoSQL

MIT 6.S094: Deep Learning for Self-Driving Cars 2018 Lecture 4 Notes: Computer Vision

MIT 6.S094: Deep Learning for Self-Driving Cars 2018 Lecture 4 Notes: Computer Vision

Too Long; Didn't Read

Companies Mentioned

Coins Mentioned

@init_27

RELATED STORIES