In this Post we are going to use and (behind the scenes) Deep learning for Natural Language Processing / Understanding! real Machine Learning In this post we are going to use the conversational AI solution both for the NLP/U engine and for the RASA dialogue part RASA — Is an Open Sourced Python implementation for NLP Engine / Intent Extraction / Dialogue → in which all of the above run on / On premise → your machine NO CLOUD! As a side note, we are using machine learning and data science extensively in our product at and we are hiring :) (PM me) Lemonade https://bit.ly/2NG88T0 The inspiration/reference to this post was the great tutorial video made by: , thanks! I tried to break it down and used my own version which is slightly different on Mac, so I hope you’d enjoy it and find it useful. Justina Petraityte skip to TL;DR — here First here is a short introduction (if you are well familiar with chatbot/dialogue architecture or want to get down to business , skip to ) here When the input is text, for example: “I want to order pizza” chatbot first the bot needs to run a NLP ( ) engine in order to parse the sentence (create a structure out of it or “try to understand it” if you may) Natural Language Processing …but what’s next? let’s assume that the bot classify it successfully to the correct “Intent”, in our case lets say the “Intent” is: order_pizza, the common sense says that we must have (from the user), the full name, address, pizza size, topping, what else? for this very purpose we need a “Dialogue management” component, in this component we would configure the various “Intents” that our bot is supporting and for each “Intent” (the next part — part 2) we would specify which “Slots”/”entities” are mandatory and which are just optional — this configuration is often called: “Domain definition”. If you already noticed in our example, the full name, address, pizza size and topping (slots) are mandatory, by mandatory we mean that our bot would continue to interact with our user until this information is fulfilled and only then it would be able to complete the task — pizza ordering! Top 3 Most Popular Ai Articles: 1. Finding the genre of a song with Deep Learning 2. Tensorflow demystified 3. Bots, MODs & Multiplayer Co-op: why Slack is game-like — NOT gamified (I’m going to create a more detailed video on it — so stay tuned :)) And now let’s get down to business. Build the NLU side To get the best results run with (and no later), better to work in a (if you getting errors) try creating the env with: Python 3.6.4 virtualenv pip3 install virtualenvpython3 -m virtualenv <env_name>source <env_name>/bin/activate Make sure you run the above command with python3 pointing out to the python 3.6.4 mentioned You can choose to install it: pip3 install rasa_nlu==0.12.3 If you want to be able to debug your code and dive in the implementation you should clone it: git clone :RasaHQ/rasa_nlu.git git@github.com cd rasa_nlu #go to the relevant release (0.12.3):git reset --hard d08b5765e92ea27741926b1246e5e22713158987 pip install -r requirements.txtpip install -e . Either way create a working directory in which we will place all our files for this tutorial: mkdir rasa_democd rasa_demovim requirements.txt If you don’t want to use vim, just create the file requirements.txt . and edit it in a text editor (sublime or notepad) If you cloned the code skip to . configuration In the file “requirements.txt” add the content below: Now run (for the location of the file): pip install -r requirements.txt Configuration First lets create the configuration files and the training file. Lets start with the configuration file. Create a dir named config: mkdir configcd configvim config.json If you don’t want to use vim, just create the file requirements.txt . and edit it in a text editor (sublime or notepad) In this file “config.json” add the content below: If you are using a newer version of Rasa i.e configuration is done via yml, create config/config.yml: Go back to the parent workspace directory — in our case “rasa_demo” Training data Now let’s create the training data, for that matter, examples for sentences that we think our user is going to say and to which Intent and entities our chatbot should break it. First let’s create a directory “data” and in it create a file “training_data.json” run: cd ../mkdir datacd !$vim training_data.json If you don’t want to use vim, just create the file requirements.txt . and edit it in a text editor (sublime or notepad) with this content: There is a great tool (rasa_nlu_trainer) you can use to add new examples/Intents/entities. To install it, run in terminal: npm i -g rasa-nlu-trainer If you don’t have npm and nodejs go to and follow the links to npm and nodejs in the installation part. here Now launch the trainer: rasa-nlu-trainer -v <path to the training data file> In our example we the file under the data directory: rasa-nlu-trainer -v data/training_data.json Here is a screenshot of the trainer: In the screenshot we can see the Intent part: “order_pizza” and the user input as the text: “I want to order large pizza”. In addition I marked the word “large” and now I get an option to add it as an entity: “size” which its value in this example is: “large” For this example we have only 2 examples per Intent (btw this is the minimum you must have for each Intent otherwise you’d get an error in training time. Machine Learning Training Now we are ready to actually train our NLU model. Machine Learning Create a file: “nlu_model.py” in the parent workspace directory with this code: If you are using the newer version of Rasa (no NLU conversters/config via yml), use this nlu_model.py: Now run (make sure that you are in the parent workspace directory): python nlu_modle.py Notice that in the above code the call to run() function is commented out. Now try to uncomment it and comment out the call to the function train(). Run again: python nlu_modle.py You should see the output: {u’entities’: [], u’intent’: {u’confidence’: 0.78863250761506243, u’name’: u’order_pizza’}, ‘text’: u’I want to order pizza’, u’intent_ranking’: [{u’confidence’: 0.78863250761506243, u’name’: u’order_pizza’}, {u’confidence’: 0.21136749238493763, u’name’: u’greet’}]} As you can see our model has successfully classified the text to the Intent: “order_pizza”, with confidence: “0.788…” — to get better probability, simply add more examples in the training data. ! we’ve just built an Interpreter for our chatbot — half way :) Congratulations You can take a break and get to the second part later — don’t forget to :) clap Now you are ready to build the dialogue side. (link will be added soon) Go to part 2 of the guide (BTW We are building amazing AI & chatbots at ) Lemonade Troubleshooting If you’re experiencing issues! git clone the latest version, install all the requirements like : this Run: virtualenv -p python3 <env name> python3 -m virtualenv <env_name> pip3 install And now you ready Just make sure you have a training file (similar to the above) config file like this — config.yml: pipeline:- name: "tokenizer_whitespace"- name: "ner_crf"- name: "intent_featurizer_count_vectors"- name: "intent_classifier_tensorflow_embedding"batch_size: [64, 256]epochs: 1500embed_dim: 20 2. Now train: python3 rasa_nlu/train.py \--config `pwd`/config/config.yml \--data `pwd`/data/training_data.json \--path `pwd`/projects/ 3. Now run: python3 rasa_nlu/server.py --path `pwd`/projects --config config/config.yml Make sure the config.yml file is in the same place. If you have issues with Python pkgs follow : this virtualenv -p python3 <env name> python3 -m virtualenv <env_name> pip3 install More issues? If you get the below error when trying to run the NLU (> python nlu_model.py): “OSError: Can’t find model ‘en’” It means that you haven’t got the spacy “en” language model downloaded. run: python3 -m spacy download en Or: python -m spacy download en
