The software used to manage networks hasn’t kept up with the dramatic expansion of networks themselves

Forward Networks

forwardnetworks

A holy grail of computer vision is the complete understanding of visual scenes. Progress in perceptual tasks began with the task of identifying what’s in the image, for example, whether there’s a man or not. Next, if there’s a man, we want to know where it is in the image. can draw a bounding box around it or label every pixel, e.g., whether the pixel belongs to a man or not. These tasks are called image classification, object detection, and image segmentation. AI What else can we say about the image? Was the man talking to a friend or running with a ball? To answer this question, AI needs to understand the interactions and relationships between objects in an image. This task is called Scene Graph Generation (SGG). Let’s discover how relationships between objects can be described. Relationships connect two objects. These relationships can be actions (jumping over), spatial (is behind), verbs (wear), prepositions (with), comparative (taller than), or prepositional phrases (drive on). Combining the objects and relationships, we create a directed graph representation (figure below). Objects can be represented by bounding boxes, from object detection, or by masks from panoptic segmentation. But panoptic segmentation has advantages, such as it covers the full scene with background and has better accuracy compared to the coarse bounding boxes. So, gives us a deeper scene understanding. Panoptic Scene Graph Generation (PSG) Understanding scenes would enable important applications such as image search, question answering, and robotic interactions. Although It seems easy for us, there’s a big perception gap between robotic systems and humans. This technology will allow them to do more intellectual work. Now, let’s dive into how this AI looks under the hood. We expect a set of triplets as an output of our neural net. Every triplet consists of two objects and their relationship. Such an output raises a couple of challenges for our network architecture. For example, should we predict all triplets simultaneously or just one by one? Can we solve the task in two stages: first performing object detection, then pairwise relationship prediction between these predicted objects? Or is it better to use a one-stage end-to-end neural net? There are multiple ways to approach this task. Let’s look at a promising one using a Transformer. The transformer has established itself as a surprisingly versatile network. It fits us because by using an attention mechanism, it can understand the relationship between sequential elements that are far from each other. As shown in the figure above, the PSGTR model first extracts image features from a CNN backbone and then feeds the features along with trainable queries and position encoding into a transformer encoder-decoder. Here, we expect the queries to learn the representation of scene graph triplets so that for each triplet query, the object/relationship/object predictions can be extracted by three individual Feed Forward Networks (FFNs), and the segmentation task can be completed by two panoptic heads for both objects, respectively. To train, the model authors created a dataset consisting of object/relationship/object triplets. The model predicts all triplets at once and is trained end-to-end with a set loss function that performs bipartite matching between prediction and ground truth. This is one way an . You can check the paper and code linked below for better understanding. AI understands a scene Reference Yang, Jingkang and Ang, Yi Zhe and Guo, Zujin and Zhou, Kaiyang and Zhang, Wayne and Liu, Ziwei. Panoptic Scene Graph Generation. Code: https://github.com/Jingkang50/OpenPSG Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, Li Fei-Fei. Visual Genome.

How An AI Understands Scenes: Panoptic Scene Graph Generation.

Your grandfather wears those comfy slipper-y shoes all day, every day, and they’re starting to get holes in the toe and a detached sole. For his birthday, you’d love to surprise him with a new pair of those. You jump on your favorite retail website and land in the search box, only to realize that your aging human brain doesn’t remember the name of those. Not slippers, not loafers…some Indian name. If after scratching your head, your best guess is still “soft shoes my grandpa wears”, all is not lost. Because if the search engine is on the ball, it will simply dig a little deeper, harnessing the power of data science to decipher what you’re actually looking: your underlying . search intent Slippers, shoes, or something else? In the digital world nowadays, search intent is a very hot topic. To succeed in providing an excellent user experience, website managers must know more than just which keywords people are entering. Now, the best practice is for search engines to have the ability to what users want, even if these humans can’t come close to articulating it. fully comprehend That’s where “brainy” neural network logistics come in. Neural network–based search technology facilitates “understanding” of search intent and related themes to find exactly what people need. A neural network takes into account the to improve search . So when you search for something like “soft shoes my grandpa wears”, a savvy search engine will instantly read between the lines and show you all the moccasins your granddad could ever need. context of words and connected ideas relevance What is a neural network? Neural networks were conceived as mathematical models by of the University of Chicago (1944). This concept exists in two realms: Warren McCullough and Walter Pitts Animal brains, including humans’, have biological neural networks and feedback loops that can process input data from the senses — sights, sounds, and scents — plus learn from the surroundings. Nature: Computer scientists have figured out how to replicate the model of neurons in natural neural networks. Artificial neural networks (ANNs) are the basis for artificial intelligence (AI) and machine learning, sitting at the center of many contemporary AI tools. These versions, complete with artificial neurons, allow computers to more effectively process an input dataset. As for the “intelligent” machines involved, the quality of their output is limited to the material they receive, but they do “think” like an animal does. Technology: Three Processing Layers With naturally occurring networks, information is processed through nodes (computational units) that sit in various layers. An artificial neural net is also a series of nodes, organized in layers and connected through inputs and outputs. Neural networks work their magic through three layers: this first layer is where data is received before being passed along to the next-layer nodes. Input layer: where weighted connections and non-linear activation functions generate the output  (this level could include multiple layers). Hidden layer: where the finished data is expressed. Output layer: The number of layers in a neural network is a clue to its classification. A basic neural network has two or three layers. One that has at least two layers — which adds some complexity — is technically a neural network. A very large neural network is a deep-learning tool; IBM’s definition is that more than three layers (including the input and output) constitutes a deep-learning algorithm. deep How ANNs Work Artificial neural networks contain a number of weighted responses that trigger different reactions within the hidden layer. These responses dictate what gets produced as the outcome. To do their jobs, a little training data is in order. Neural networks (or neural nets, for short) are “trained” by adjusting weighting and testing on different types of outcomes. A neural network must have the right rules and weighted responses to do the particular job for which it’s intended. For example, the ChatGPT neural network contains nodes in the hidden layer that allow it to effectively process language. Different Types of Neural Networks There are multiple kinds of artificial neural networks. Some are generative and can “learn” as they work. Some simply focus on automating and improving tasks such as facial recognition or data analysis. Some are fantastic for aiding image processing and automation; others are excellent for enhancing . Different applications of neural networks can also be tailored for the type of use case. search and discovery Here are six types of neural network with their key data points: Feed Forward Feed-forward neural networks are linear: they process information in one direction until an output is ready. This is the simplest form of neural network architecture. When used alone, as opposed to as part of a modular network, they don’t contain the feedback loops required to build artificial intelligence. This architecture is often used in classification software like spam filter, where each step in the analysis brings us closer to the final result without intermediate results needing to be reused later downstream. Recurrent In terms of complexity, recurrent neural networks (RNNs) pick up where feed-forward networks leave off, reprocessing their own outputs to generate more-accurate future outputs. This creates a feedback loop, — a recurring process — in the hidden layer. Because an RNN can create more-valuable insights by reusing its output — pushing data back through the network to contextualize and “understand” the information — it’s ideal for utilizing with AI-powered tools like ChatGPT. Convolutional (CNNs) are often used for pattern recognition, image recognition, and image classification applications. This subset of machine learning categorizes data as it’s processed through multiple convolutional layers, breaking it down and recategorizing based on importance. The resulting information can then be used to differentiate similar images from one another, spotting critical pixel differences, for instance, that the human eye might not see. One consideration: if a CNN isn’t set up optimally, it can consider some information “unimportant”. Convolutional neural networks That’s where a convolutional neural network comes in handy. de Deconvolutional Deconvolutional neural networks (DNNs) work in the opposite way to CNNs. They start with “raw” data and work backward toward usable image output. DNNs can be used to identify information discarded by or missed when using CNNs, and they can be used in conjunction with CNNs to self-test outcomes. Modular Modular neural networks are made up of multiple neural networks that work separately toward a common goal by using ensemble learning, a method that connects different models to pool results and test outcomes before presenting findings. An organization can use this variation of network technology to ensure accurate outcomes. For example, a finance company might utilize it to generate more-accurate stock-market predictions as the different networks produce different outcomes; the findings can then be used to accurately peg averages and outliers. Transformers Did you know that the in ChatGPT and GPT-4 stands for transformer? The transformer, the element that makes generative AI so powerful, is a relatively new form of that utilizes modular neural networks. Transformers use something called a self-attention layer, along with feed-forward neural networks and RNNs, to focus on complex tasks such as language processing. T machine learning Take your search to neural levels As in many industries, AI is steadily changing the fabric of real-world search. Neural networks are being pressed into service to improve software’s comprehension of what consumers want. These tools can help among people, content, and data, as well as connections between user interests and search queries (both current and past). ecommerce identify relationships At Algolia, we harness machine learning and (NLP) to get at the heart of search intent; then the best query results can be and provided to your users. natural language processing ranked Want to smarten your search engine with the ? Maybe improve your (image recognition or computer vision skills), or perhaps optimize your ? advantages of neural search visual search voice search To assist your decision-making, let’s look at your needs and weigh the options. with us today. Get in touch Also published . here

What are the Types of Neural Networks and How Do They Work?

RNN is one of the popular neural networks that is commonly used to solve natural language processing tasks, other neural networks are Feed-Forward Neural Network which is used for regression and classification problems. The last one is a Convolutional Neural Network or CNN which can be used for image classification and object detection. Table of Contents: What is an RNN? How does an RNN work? 5 Common Use Cases of RNNs What is an RNN (Recurrent Neural Network)? RNN stands for Recurrent Neural Network, this is a type of artificial neural network that can process sequential data, recognize patterns and predict the final output. This Neural Network is called Recurrent because it can repeatedly perform the same task or operation on a sequence of inputs. An RNN has an internal memory that allows it to remember or memorize the information of the input it received and this helps the system to gain context. Therefore if you have sequential data like a time series, then an RNN will be a good fit to process that data. This can not be done by a CNN or Feed-Forward Neural Networks since they can not sort the correlation between previous input to the next input. Popular products like Google's voice search and Apple's Siri use RNN to process the input from their users and predict the output. How do RNNs Work? The logic behind an RNN is to save the output of the particular layer and feed it back to the input in order to predict the output of the layer. Below is a simple example of how you can convert a Feed-Forward Neural Network into a Recurrent Neural Network (RNN). On the left side of the image, represent the following: X - is the input layer. h - is the hidden layer and it holds the information for the previous output and feeds it back to itself. y - is the output layer A, B, C are the parameters to improve the output of the model. Here is an example of a fully connected Recurrent Neural Network. Recurrent Neural Networks share the same weight parameter within each layer of the network while Feed-Forward Networks tend to have different weights across each node. Note: Now that you understand what a recurrent neural network is, let’s look at the common use case of RNNs. 5 Common Use Cases of Recurrent Neural Networks 1. Machine Translation RNN can be used to create a deep learning model that can translate a text from the source language into the target language without human intervention. For example, you can translate a text from your local language to the English language. Resources for Machine Translation https://hackernoon.com/neural-machine-translation-using-open-nmt-for-training-a-translation-model-1129a3a2a2d3 https://www.youtube.com/watch?v=vI2Y3I-JI2Q 2. Text Generation RNNs can also be used to create a deep learning model to generate text. A trained model learns the likelihood of occurrence of a word/character based on the previous sequence of words/characters used in the text. You can train a model at the character level, n-gram level, sentence level, or paragraph level. Resources for Text Generation. https://hackernoon.com/generate-stories-using-rnns-pure-mathematics-with-code-cb6f1e967b22 https://www.tensorflow.org/text/tutorials/text_generation 3. Image Captioning Image captioning is the process of generating text that describes the content of a given image. The image's content can show the object and the action of the particular object on the image. For example, in the image below the trained deep learning model by using RNN can describe the image as . "A lady with the green coat is reading a book under a tree" Resources for Image Captioning. https://hackernoon.com/summarizing-most-popular-text-to-image-synthesis-methods-with-python-34d3xkn https://towardsdatascience.com/a-guide-to-image-captioning-e9fd5517f350 4. Speech Recognition This is also known as Automatic Speech Recognition (ASR) that can process human speech into a written or text format. Don't confuse speech recognition with voice recognition; speech recognition mainly focuses on transforming voice data into text, while voice recognition identifies the voice of the user. Products like Alexa, Cortana, Google Assistant and Siri are examples of Speech recognition technologies that are used daily by different users. Resources for Speech Recognition. https://hackernoon.com/how-to-do-speech-recognition-in-python-bk1234w9 https://heartbeat.fritz.ai/a-2019-guide-for-automatic-speech-recognition-f1e1129a141c 5. Time-Series Forecasting You can use an RNN to create a time series prediction model that predicts the future outcome after being trained from the historical time-stamped data. A good example is in the stock market. You can use the stock market data to create a machine learning model that can predict the future stock price in the market based on what the model learns from the historical data. This can help investors to make a data-driven decision for their investments. Resources for Time Series Forecasting. https://hackernoon.com/time-series-forecasting-with-tensorflowjs-co3rz32ms https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ There is more to cover in Recurrent Neural Networks, I recommend you take one of the following courses to learn more about RNNs. https://machinelearningmastery.com/crash-course-recurrent-neural-networks-deep-learning/ https://www.coursera.org/projects/simple-recurrent-neural-network-keras If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in Machine Learning & Data Science? in the footer below. Subscribe to our newsletter

What is an RNN (Recurrent Neural Network) in Deep Learning?

The Hitchhikers's Guide to PyTorch for Data Scientists

Build an Abstractive Text Summarizer in 94 Lines of Tensorflow !! (Tutorial 6) This tutorial is the sixth one from a series of tutorials that would help you build an abstractive text summarizer using tensorflow , today we would build an abstractive text summarizer in tensorflow in an optimized way . Today we would go through one of the most optimized models that has been built for this task , this model has been written by , this is to his model , I have used his model model on different datasets (in different languages) and it resulted in truly amazing results , so I would truly like to thank him for his effort dongjun-Lee the link I have made multiple modifications to the model to enable it to enable it to run seamlessly on google colab ( ) , and i have hosted the data onto google drive ( ) , so no need to download neither the code , nor the data , you only need a google colab session to run the code , and copy the data from my google drive to yours ( ) , and connect google drive to your notebook of google colab link to my model more on how to link google drive to google colab more on this 0- Intro 0-A About Series This is a series of tutorials that would help you build an abstractive text summarizer using tensorflow using multiple approaches , we call it abstractive as we teach the neural network to generate words not to merely copy words . We have covered so far (code for this series can be found ) here 0. (how to use google colab with google drive) Overview on the free ecosystem for deep learning Overview on the text summarization task and the different techniques for the task Data used and how it could be represented for our task (prerequisites for this tutorial) What is seq2seq for text summarization and why Mulitlayer Bidirectional LSTM/GRU Beam Search & Attention for text summarization 0-B About the Data Used The data that would be used would be news and their headers , it can be found on my google drive , so you just copy it to your google drive without the need to download it ( ) more on this We would represent the data using word embeddings , which is simply converting each word to a specific vector , we would create a dictionary for our words ( ) more on this (prerequisites for this tutorial) 0-C About the Model Used There are for this task , they are built over a corner stone concept , and they keep on developing and building up . different approaches Today we would start building this corner stone implementation which is a type of network called RNN , which is arranged in an Encoder/Decoder architecture called seq2seq ( ), then we would build the seq2seq in a mulitlayer bidirectional structure , where the rnn cell would be a LSTM cell ( ) , then we would add an attention mechanism to better interface the encoder with the decoder ( ) , then to generate better output we use the ingenious concept of beam search ( ) more on this more on this more on this more on this the code for all these different approaches can be found here so lets get started !! Model Structure our model can be seen to structured into different blocks these blocks are Initialization Block : Here we would initialize the needed tensorflow & , and here would define our that would be used throughout the model placeholders variables RNN cell Embedding Block : Here we would define the embedding matrix used in both the & the encoder decoder Encoder Block : Here we would define the ( ) that forms the encoder part of our model , and we state as an input to the decoder part multilayer bidirectional RNN more on this output the encoder Decoder Block : Here the decoder is actually portioned into 2 distinct parts ( ) which is used to better interface the encoder with the decoder , this would be used in phase Attention Mechanism more on this training ( ) which is used to generate better output from our model , this would be used in phase BeamSearch more on this testing Loss Block : This block would only be used in phase , here we would apply clipping to our gradients , and we would actually run our optimizer (Adam Optimizer is used here) , and here is the place where we would apply our gradients to the optimizer. training 1- Initialization Block First we would need to import the libs that we would use import tensorflow as tf
from tensorflow.contrib import rnn  #cell that we would use Before Building our Model Class we need to get define some tensorflow concepts first So we tend to define placeholders like this X = tf.placeholder(tf.int32, [None, article_max_len])
# here we define the input x as int32 , with promise to provide its # data in runtime
#
# we also provide its shape , where None is used for a dimension of # any size and for the variables we tend to define them as global_step = tf.Variable(0, trainable=False)
# a variable must be intialized , 
# and we can set it to either be trainable or not Then lets build our Model Class class Model(object):
    def __init__(self, reversed_dict, article_max_len, summary_max_len, args, forward_only=False):
        self.vocabulary_size = len(reversed_dict)
        self.embedding_size = args.embedding_size
        self.num_hidden = args.num_hidden
        self.num_layers = args.num_layers
        self.learning_rate = args.learning_rate
        self.beam_width = args.beam_width we would pass an obj called args that would actually contain multiple parameters from embedding size (size of word2vector) num_hidden (size of RNN) num_layers (layers of RNN) ( ) more on this Learning Rate BeamWidth ( ) more on this Keep Prob we would also need to initialize the model with other paaremetrs like reversed dict (dict of keys , each key wich is a num points to a specific ord) ( ) more on how to build your reversed dict article_max_len & article_summary_len (max length of article sentence as input and max length of summary sentenceas output) Forward Only (bool value to indicate training or testing phase) ( ) Forward Only = False → training phase then to continue the initalization if not forward_only: #training phase
            #keep_prob as variable in training phase
            self.keep_prob = args.keep_prob
else: #testing phase
            #keep_prob constant in testing phase
            self.keep_prob = 1.0
   
        #here we would use LSTM as our cell
        self.cell = tf.nn.rnn_cell.BasicLSTMCell 
  
        #projection layer that would be used in decoder in both 
        #training and testing phase
        with tf.variable_scope("decoder/projection"):
              self.projection_layer =  tf.layers.Dense(self.vocabulary_size, use_bias=False)
   
        #define batch size(our data would be provided in batches)
        self.batch_size = tf.placeholder(tf.int32, (), name="batch_size")
  
        #X as input , define as length of articles 
        self.X = tf.placeholder(tf.int32, [None, article_max_len])
        self.X_len = tf.placeholder(tf.int32, [None])
  
        #define decoder (input , target , length) 
        #using the summary length 
        self.decoder_input = tf.placeholder(tf.int32, [None, summary_max_len])
        self.decoder_len = tf.placeholder(tf.int32, [None])
        self.decoder_target = tf.placeholder(tf.int32, [None, summary_max_len])
  
        #define global step beginning from zero 
        self.global_step = tf.Variable(0, trainable=False) 2- Embedding Block : Here we would represent both our and the using word2vector ( ) inputs articles that would be the embedded inputs decoder inputs more on this we would define our variables for embedding in a variable scope , we would name it embedding with tf.name_scope("embedding"):
            #if training , 
            #and you enable args.glove variable to true
            if not forward_only and args.glove:
                #here we use tf.constant as we won't change it
                #get_init_embedding is a function 
                #that returns the vector for each word in our dict
                init_embeddings = tf.constant(get_init_embedding(reversed_dict, self.embedding_size), dtype=tf.float32)
            else: #else random define the word2vector for testing
                init_embeddings = tf.random_uniform([self.vocabulary_size, self.embedding_size], -1.0, 1.0)
            self.embeddings = tf.get_variable("embeddings", initializer=init_embeddings)
            #then define for both encoder input
            self.encoder_emb_inp = tf.transpose(tf.nn.embedding_lookup(self.embeddings, self.X), perm=[1, 0, 2])
            #and define for decoder input
            self.decoder_emb_inp = tf.transpose(tf.nn.embedding_lookup(self.embeddings, self.decoder_input), perm=[1, 0, 2]) 3- Encoder Block : Here we would actually define the multilayer bidirectional lstm for the encoder part of our seq2seq ( ) , we would define our variables here in a name scope that we would call “encoder”. more on this Here we would use the concept of , we would use it after each cell in our architecture , it is used to randomly activate a subset of our net, and is used during training for regularization. Dropout with tf.name_scope("encoder"):

            fw_cells = [self.cell(self.num_hidden) for _ in range(self.num_layers)]

            bw_cells = [self.cell(self.num_hidden) for _ in range(self.num_layers)]

            fw_cells = [rnn.DropoutWrapper(cell) for cell in fw_cells]

            bw_cells = [rnn.DropoutWrapper(cell) for cell in bw_cells] Now after defining the forward and backward cells , we would need to actually connect them together to form the bidirectional structure , so we would use , which takes all of the following parameters as its inputs stack_bidirectional_dynamic_rnn forward cells Backward Cells Encoder emb input (input articles in word2vector format) X_len (length of articles) Using time_major = True is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. encoder_outputs, encoder_state_fw, encoder_state_bw = tf.contrib.rnn.stack_bidirectional_dynamic_rnn(
fw_cells, bw_cells, self.encoder_emb_inp,
sequence_length=self.X_len, time_major=True, dtype=tf.float32) Now we would need to actually use the output from this function , we mainly need 2 main outputs stack_bidirectional_dynamic_rnn encoder_output (would be used in attention calculation) ( ) more on attention encoder_state (would be used for the initial state of the decoder) so to get encoder_output we simply self.encoder_output = tf.concat(encoder_outputs, 2) then to get encoder_state , we would combine both (encoder_state_c) & (encoder_state_h) of both the forward & backward using LSTMStateTuple encoder_state_c = tf.concat((encoder_state_fw[0].c, encoder_state_bw[0].c), 1)

encoder_state_h = tf.concat((encoder_state_fw[0].h, encoder_state_bw[0].h), 1)

self.encoder_state = rnn.LSTMStateTuple(c=encoder_state_c, h=encoder_state_h) 4- Decoder Block : Here the decoder is divided into 2 parts Training part (to train attention model) ( ) more on attention model testing/running part (for attention & beam search) ( ) more on beam search so lets first define out (name scope) & (variable scope) for both parts , we would also define a multilayer cell structure that would be also used for both parts with tf.name_scope("decoder"), tf.variable_scope("decoder") as decoder_scope:
            decoder_cell = self.cell(self.num_hidden * 2) 4.a Training Part (Attention Model) First we need to prepare our attention structure , here we would use BahdanauAttention would be used inside the attention calculation ( ) encoder_output more on attention model attention_states = tf.transpose(self.encoder_output, [1, 0, 2])
attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(
                    self.num_hidden * 2, attention_states, memory_sequence_length=self.X_len, normalize=True) then we would further define the decoder cell (as from the first step in decoder , we just defined the decoder cell as a simple multilayer lstm , now we would add attention) , to do this we would use , which combines attention_mechanism with decoder cell AttentionWrapper decoder_cell = tf.contrib.seq2seq.AttentionWrapper(decoder_cell, attention_mechanism,                                                                 attention_layer_size=self.num_hidden * 2) Now we would need to define the inputs to the decoder cell , this input actually comes from 2 sources ( ) more on seq2seq encoder output (used within initial step) decoder input (summary sentence in the training phase) so lets first define the initial state that would come from the encoder initial_state = decoder_cell.zero_state(dtype=tf.float32, batch_size=self.batch_size)

initial_state = initial_state.clone(cell_state=self.encoder_state) now we would combine both the initial state with the decoder input (summary sentence) , here to use the BasicDecoder , we need to provide the decoder input through a helper , this helper would combine all of (decoder_emb_inp , decoder_len) together helper = tf.contrib.seq2seq.TrainingHelper(self.decoder_emb_inp, self.decoder_len, time_major=True)

decoder = tf.contrib.seq2seq.BasicDecoder(decoder_cell, helper, initial_state)

outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(decoder, output_time_major=True, scope=decoder_scope) now for the last step of the training phase , we would need to define the outputs (logits)from the decoder , to be used within the loss block for training #just use the rnn_outputs from aall the outputs
self.decoder_output = outputs.rnn_output

#then get logits , by performing a transpose on decoder output
self.logits = tf.transpose(self.projection_layer(self.decoder_output), perm=[1, 0, 2])

#then reshape the logits 
self.logits_reshape = tf.concat(
                    [self.logits, tf.zeros([self.batch_size, summary_max_len - tf.shape(self.logits)[1], self.vocabulary_size])], axis=1) 4.b Testing/Running part (Attention & Beam search) Here in this phase , there are 2 main goals divide the encoder output & encoder states & x_len (article length) to parts to actually perform the beam search methodology ( ) more on beam search build a decoder independent on decoder input , as in the testing phase we don’t have the summary sentence as our input , so we would need to build the decoder in a different way than above first lets divide encoder output & encoder states & x_len (article length) to parts to actually perform the beam search methodology , here we would use variable that was already defined above beam_width tiled_encoder_output = tf.contrib.seq2seq.tile_batch(
                    tf.transpose(self.encoder_output, perm=[1, 0, 2]), multiplier=self.beam_width)

tiled_encoder_final_state = tf.contrib.seq2seq.tile_batch(self.encoder_state, multiplier=self.beam_width)
                
tiled_seq_len = tf.contrib.seq2seq.tile_batch(self.X_len, multiplier=self.beam_width) then lets define the attention mechanism (just like before , but taking the tiled variables into consideration) attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(
                    self.num_hidden * 2, tiled_encoder_output, memory_sequence_length=tiled_seq_len, normalize=True)

decoder_cell = tf.contrib.seq2seq.AttentionWrapper(decoder_cell, attention_mechanism,                                                                 attention_layer_size=self.num_hidden * 2)

initial_state = decoder_cell.zero_state(dtype=tf.float32, batch_size=self.batch_size * self.beam_width)
                
initial_state = initial_state.clone(cell_state=tiled_encoder_final_state) then lets define our decoder , but here we would use the , this takes into consideration all of BeamSearchDecoder Decoder cell (previously defined) Embedding word2vector (defined in embedding part) projection layer (defined in the beginning of class) decoder initial state (previously defined) beam_width (user defined) start token & end token decoder = tf.contrib.seq2seq.BeamSearchDecoder(
cell=decoder_cell,
embedding=self.embeddings,
start_tokens=tf.fill([self.batch_size], tf.constant(2)),
end_token=tf.constant(3),
initial_state=initial_state,
beam_width=self.beam_width,
output_layer=self.projection_layer ) then all what is left to do , is to define the outputs , that would actually directly reflect to the real output from the whole seq2seq architecture , as this phase is where prediction is actually computed outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(
                    decoder, output_time_major=True, maximum_iterations=summary_max_len, scope=decoder_scope)

self.prediction = tf.transpose(outputs.predicted_ids, perm=[1, 2, 0]) 5- Loss Block : This block is where training actually occurs , here training actually occurs through multiple steps calculating loss ( ) more on loss calculation calculating gradients and applying clipping on gradients ( ) more on exploding gradients applying optimizer (here we would use Adam optimizer) First we define our name scope , and we would specify that this block would only work through the training phase with tf.name_scope("loss"):
            if not forward_only: Second we would calculate the loss ( ) more on loss calculation crossent = tf.nn.sparse_softmax_cross_entropy_with_logits(
                    logits=self.logits_reshape, labels=self.decoder_target)

weights = tf.sequence_mask(self.decoder_len, summary_max_len, dtype=tf.float32)

self.loss = tf.reduce_sum(crossent * weights / tf.to_float(self.batch_size)) Third we would calculate our gradients , and apply clipping on gradients to solve the problem of exploding gradients ( ) more on exploding gradients ( from tutorial 4 ) Occurs with deep networks (i.e: networks like in our case) , when we apply back propagation, the gradients would get too large . Actually this error can be solved rather easy , using the concept of , which is simply setting a specific threshold , that when the gradients exceed it , we would clip it to a certain value . Exploding Gradients : with many layers gradient clipping params = tf.trainable_variables()
gradients = tf.gradients(self.loss, params)
clipped_gradients, _ = tf.clip_by_global_norm(gradients, 5.0) Forth we would apply our optimizer , here we would use Adam optimizer , here we would use the previously defined learning_rate optimizer = tf.train.AdamOptimizer(self.learning_rate)

self.update = optimizer.apply_gradients(zip(clipped_gradients, params), global_step=self.global_step) Next Time if GOD wills it , we would go through the code needed to divide our data into batches needed code to use this model for training Then after we are done with this core model implementation , if GOD wills it , we would go other modern implementations for text summarization like pointer generator Using reinforcement learning with seq2seq ( ) more on different implementations for seq2seq for text summarization All the code for this tutorial is found as open source . here I truly hope you have enjoyed reading this tutorial , and i hope i have made these concepts clear , all the code for this series of tutorials are found here , you can simply use google colab to run it , please review the tutorial and tell me what do you think about it , hope to see you again <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Build an Abstractive Text Summarizer in 94 Lines of Tensorflow !! (Tutorial 6)

What is a Recurrent Neural Network or RNN, how it works, where it can be used? This article tries to answer the above questions. It also shows a demo implementation of a RNN used for a specific purpose, but you would be able to generalise it for your needs. Knowhow. Python, CNN knowledge is required. CNN is required to compare why and where RNN performs better than CNN? No need to understand the math. If you want to check then go back to my earlier article to check what is a CNN. We will begin with the word use of the word “Recurrent”. Why is it called Recurrent? In english the word recurrent means: occurring often or repeatedly In the case of this type of Neural Network it’s called Recurrent since it does the same operation over and over on sets of sequential input. We will discuss about the meaning of the later in the article. operation Why do we need RNN? You might be wondering by now, we have vanilla networks like Convolutional ones which perform very well. Why do we need another type of a network? There is a very specific use case where RNNs are required. In order to explain RNNs you need to first understand something called a sequence. Let's talk about first. sequences Sequence is a stream of data (finite or infinite) which are interdependent. Examples would be time series data, informative pieces of strings, conversations etc. In a conversation a sentence means something but the entire flow of the conversation mostly means something completely different. Also in a time series data like stock market data, a single tick data means the current price, but a full days data will show movement and allow us to take decision whether to buy or sell. CNNs generally don’t perform well when the input data is interdependent in a sequential pattern. CNNs don’t have any sort of correlation between previous input to the next input. So all the outputs are self dependent. CNN takes in an input and outputs based on the trained model. If you run 100 different inputs none of them would be biased by the previous output. But imagine a scenario like sentence generation or text translation. All the words generated are dependent on the words generated before (in certain cases, it’s dependent on words coming after as well, but we will discuss that later). So you need to have some bias based on your previous output. This is where RNNs shine. RNNs have in them a sense some memory about what happened earlier in the sequence of data. This helps the system to gain context. Theoretically RNNs have infinite memory, meaning they have the capability to look back indefinitely. By look back I mean all previous inputs. But practically they can only look back a last few steps. (we will discuss this later) Just to draw a correlation with humans in general, we also don’t take in place decisions. We also base our decisions on previous knowledge on the subject. (**over simplified, hard to say I understand even 0.1% of human brains**) Where to use a RNN? RNNs can be used in a lot of different places. Following are a few examples where a lot of RNNs are used. 1. Language Modelling and Generating Text Given a sequence of word, here we try to predict the likelihood of the next word. This is useful for translation since the most likely sentence would be the one that is correct. 2. Machine Translation Translating text from one language to other uses one or the other form of RNN. All practical day systems use some advanced version of a RNN. 3. Speech Recognition Predicting phonetic segments based on input sound waves, thus formulating a word. 4. Generating Image Descriptions A very big use case is to understand what is happening inside an image, thus we have a good description. This works in a combination of CNN and RNN. CNN does the segmentation and RNN then used the segmented data to recreate the description. It’s rudimentary but the possibilities are limitless. 5. Video Tagging This can be used for video search where we do image description of a video frame by frame. Lets Dig Deep! We will be following the below mentioned sequence of topics to finish the document. Each section builds on top of another so don’t read this as a reference. Feed-forward Networks Recurrent Networks Recurrent Neuron Backpropagation Through Time (BPTT) RNN Implementation Feed-forward Networks Primer Feed-forward networks channel information through a series of operations which take place in each node of the network. Feed-forward networks pass the information directly through each layer exactly once. This is different from other recurrent networks. We will talk about them in a later section. Generally feed-forward nets take an input and produce an output from it. This is also mostly a supervised learning step and the outcome most likely will be a classification. It behaves similarly to how a CNN behaves. Outputs can be expected to be classes like cats or dogs as labels. A feed-forward network is trained on a set of pre labelled data. The objective of the training phase is to reduce the error while the feed-forward network tries to guess the class. Once training is done, the weights are used to classify new batches of data. A typical feed-forward network architecture One important thing to note here. In a feed-forward network whatever image is shown to the classifier during test phase, it doesn’t alter the weights so the second decision is not affected. This is one very important difference between feed-forward networks and recurrent nets. Feed-forward nets don’t remember historic input data at test time unlike recurrent networks. It’s always point in time decision. They only remember things that were shown to them during the training phase. Recurrent Networks Recurrent networks, on the other hand, take as their input not just the current input example they see, but also what they have perceived previously in time. Let’s try to build a to start with the explanation. In simple terms there is a input layer, a hidden layer with certain activations and finally we receive an output. multi layer perceptron A sample multi layer perceptron architecture If we increase the number of layers in the above example, input layer takes the input. Then the first hidden layer does the activation passing onto the next hidden layers and so on. Finally it reaches the output layer which gives the output. Each hidden layer has its own weights and biases. Now the question is can we input to the hidden layers. Each layer has its own weight (W), biases (B), Activation Functions (F). These layers behave differently and technically would be challenging to merge together. To be able to merge them, lets replace all the layers with the same weights and biases. It will look something like this. Now we can merge all the layers together. All the hidden layers can be combined into a single recurrent layer. So they start looking somewhat like this: We will provide input to the hidden layer at each step. A recurrent neuron now stores all the previous step input and merges that information with the current step input. Thus it also captures some information regarding the correlation between current data step and the previous steps. The decision at a time step affects the decision taken at time . This is very much like how we as humans take decisions in our life. We combine the present data with recent past to take a call on a particular problem at hand. This example is excessively rudimentary but in principle it aligns with our decision making capability. t-1 t This really intrigues me as to whether we as humans are intelligent or we have a very advanced neural network model. Our decisions are just the training data that we have been collecting throughout our life. Thus can we digitise our brains once we have a fairly advanced model and systems capable of storing and computing them in reasonable time periods. So what happens when we have models better and faster than our brains training on data from millions of people? Funny anecdote from another : article a person is haunted by their deeds Let’s come back to the problem at hand and rephrase the above explanation with an example to predict what the next letter is after a sequence of letters. Imagine in the word . The word is of 8 letters. namaskar : a traditional Indian greeting or gesture of respect, made by bringing the palms together before the face or chest and bowing. namaskar If we were trying to figure out the 8th letter after 7 letters were fed to the network, what would have happened. The hidden layer would have gone through 8 iterations. If we were to unfold the network, it would be a 8 layer network, one layer for each letter. So you can imagine that a normal neural network is repeated multiple times. The number of times you unroll has a direct correlation with how far in the past it can remember. But more on this later. how recurrent neural networks work #deeplearning4j #dl4j Recurrent Neuron Here we will look in more depth regarding the actual neuron that is responsible for the decision making. We will be using the example described above. We will try to figure out the 8th letter given all the previous 7 letters. Total vocabulary of the input data is {n,a,m,s,k,r}. In real world you will have more complex words or sentences. For simplicity we will use this simple vocabulary. namaskar In the above diagram, the hidden layer or the RNN block applies a formula to the current input as well as the previous state. In this case the letter from namaste has nothing preceding it, so we will move on to the next letter which is . During the time of letter and the previous state which was letter the formula is applied by the hidden layer. We will go through the formula in a bit. Each state when an input passes the network is a time step or a step. So if at time t, the input is , then at time t-1, the input is . After applying the formula to both and , we get a new state. n a a n a n n a The formula for the current state can be written like this: is the new state and is the previous state. is the input at time . We now have a sense of the previous inputs after it has gone through the same formula from the previous time steps. We will go through 7 such inputs to the network which passes by the same weights and same function at each step. ht ht-1 xt t Now let’s try to define in a simple fashion. We will take as the activation function. The weights are defined by the matrix and the input is defined by the matrix . So the formula looks like: f() tanh Whh Wxh The above example takes only the last step as memory and thus merging with the data of last step. To increase the memory capacity of the network, and hold longer sequences in memory, we have to add more states to the equation, like , etc. Finally the output can be calculated as during test time: ht-2 ht-3 where is the output. The output is compared to the actual output and then an error value is computed. The network learns by back propagating the error via the network to update the weights. We will talk about backpropagation in the next section. yt Backpropagation Through Time (BPTT) This section considers that you are aware of Backpropagation as a concept. If you need to understand Backpropagation then please visit this to read more. link So now we understand how a RNN actually works, but how does the training actually work? How do we decide the weights for each connection? And how do we initialise these weights for these hidden units. The purpose of recurrent nets is to accurately classify sequential input. We rely on the backpropagation of error and gradient descent to do so. But a standard backpropagation like how used in feed forward networks can’t be used here. The problem with RNNs is that they are cyclic graphs unlike feed-forward networks which are acyclic directional graphs. In feed-forward networks we could calculate the error derivatives from the layer above. In a RNN we don’t have such layering. The answer lies in what we had discussed above. We need to unroll the network. We will unroll it and make it look like a feed-forward network. Unrolling a RNN We take a RNN’s hidden units and replicate it for every time step. Each replication through time step is like a layer in a feed-forward network. Each time step layer connects to all possible layers in the time step . Thus we randomly initialise the weights, unroll the network and then use backpropagation to optimise the weights in the hidden layer. Initialisation is done by passing parameters to the lowest layer. These parameters are also optimised as a part of backpropagation. t t+1 An outcome of the unrolling is that each layer now starts maintaining different weights and thus end up getting optimised differently. The errors calculated w.r.t the weights are not guaranteed to be equal. So each layer can have different weights at the end of a single run. We definitely don’t want that to happen. The easy solution out is to aggregate the errors across all the layers in some fashion. We can average out the errors or even sum them up. This way we can have a single layer in all time steps to maintain the same weights. RNN Implementation Here is a sample code where we have tried to implement a RNN using Keras models. Here is the direct link to the . We are trying to predict the next sequence given a set of text. gist This model was built by . I have updated the code slightly to fit the requirements of this article. The code is commented as you go along, it’s pretty self explanatory. Yash Katariya Conclusion Well so we have come to the end of this article. What we have discussed so far is just a basic implementation of a RNN. There are so many things that we need to cover to get a full understanding on this topic. I’ll be writing a second article within a week. I will try to cover the following topics. Vanishing and Exploding Gradients The Problem of Long-Term Dependencies Long Short Term Memory networks(LSTM) LSTM Gate Bidirectional RNNs Deep (Bidirectional) RNNs GRU (Gated Recurrent Unit) Cells If you want me to cover things apart from this, please drop a message in the comments section. RNNs are really powerful stuff, and it is very close to how a human brain seems to work. I will be looking out for more development in this area and also am personally working on this. Any improvement I’ll surely share here. So please follow me either here on or on to stay updated. Medium Twitter If you liked this article, please hit the 👏 button to support it. This will help other Medium users find it. Share this on Twitter to help out reach as many readers as possible.

RNN or Recurrent Neural Network for Noobs

(but sometimes the results look like one) [epistemic status: I work in a lab dedicated to biologically plausible neural circuits, so I’m informed on the problem, but probably still biased. There’s probably going be a follow-up post to this once I get a bunch of rebuttals from people smarter than me.] Update: As expected, I got a bunch of rebuttals and have adjusted my position accordingly I keep seeing and comparing Deep (DL) and the brain. This offends my sensibilities a bit, because although there are from DL that resembles certain areas of the brain, DL is not a good overall description of the brain. DL explicitly passes the buck on biological plausibility (like almost every other cognitive modelling approach) and implies that it’s “neurons” can be implemented biologically, it’s just that no one has bothered yet. I think the problem goes much deeper and that DL is missing a lot of the key features of the brain, which makes it a poor analogical target. academic non-academic articles Learning many results The brain is low power DL is power hungry. Alpha GO consumed the power of , not to train, but . The is an attempt to satiate this hunger, but it’s still not even close to the . IBM’s TrueNorth chip is another example of trying to bring low-power computation, but . Specifically, True North only implements feed-forward networks and has no on-chip learning. 1202 CPUs and 176 GPUs just to run TenserFlow Processing Unit brain’s power consumption of 20W it’s capabilities are quite limited when compared to other Neuromorphic hardware The brain can’t do back-prop Back-propagation is the foundation of all DL. Although there is evidence that errors being propagated through multiple layers is happening in the brain, no one has come up with a method for back-propagation (back-prop)that doesn’t rely on information propagating backwards through unidirectional synapses. I personally think it’s only a matter of time before a biologically plausible method is discovered, but until then it is unwise to ignore the implementation details and the restrictions it might place on what can be learned. The brain uses spikes to communicate Although it is possible to , these spikes are not leveraged for specific computational advantages. As far as I know (and I still have some reading to do), spiking computation has yet to be used anywhere for the learning in DL. convert DL networks into spiking neurons for use on neuromorphic hardware Neurotransmitters aren’t just spike transporters and neurons aren’t just spike-machines DL completely ignores the role of neurotransmitters. However, neurotransmitters have been shown to be computationally significant in adapting the receptive features on the fly, something which DL has really hard time doing. networks The brain is noisy Given the choice between neuron redundancy and neuron performance, evolution chose to make the brain redundant. , which isn’t surprising when you consider the warm, biologically variable environment they’re in. Although certain DL networks can cope with loss of their nodes, DL isn’t known for it’s robustness to noisy input or noisy training data. Neurons are noisy In conclusion, there are a lot of features of the brain that DL is omitting, thus using DL as an analogy for neural circuitry isn’t ideal. The alternative to comparing with DL is using a modelling paradigm that takes these challenges into account. At the time of writing, the only approach I know of is the Neural Engineering Framework (NEF) from the laboratory I belong to, but I’m sure as research marches forward other frameworks will emerge. “But Sean,” you cry with a gleam of mischief in your eye, “don’t most NEF models from the lab you belong to suffer from the same problems as DL? The models usually stop at LIF neurons, which aren’t realistic neurons at all! Why don’t you use more complex neuron models, i.e. things like , multi-compartmental neurons, and neurogenesis?” dendritic computation glial cells That’s a work in progress. There are 2 people ( and ) of the 16-strong (CNRG) that I belong to who are working on the problem of more complex neuron models. is working on biologically plausible back-prop and the moment he has a break-through you can be sure I’ll be shoving it in everyone’s face. Aaron Voelker Peter Duggins Computation Neuroscience Research Group Eric Hunsberger As for neurogenesis, There’s no good computational model of what neurogenesis does and the CNRG lacks the resources to do that sort of basic research. Dendritic and glial cell computation has no one working on it, because we’re only 16 people. If that bothers you, maybe you want to join us? is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Stories, News & Mentions about FORWARDNETWORKS

Forwardnetworks’s Stories on HackerNoon