CoLaborator

colaborator

Vor kurzem erst entdeckt und für gut befunden: Googles Plattform um schnell mal Scripts zu erstellen und in Form eines Jupyter Notebooks verwalten. Alle Möglichkeiten sind unter: ersichtlich. Ausgaben über die , Nutzung von , installieren weiterer Libs über und und und. Python https://colab.research.google.com matplotlib NumPy pip Zur Nutzung benötigt man eine Einladung. Beim Aufruf des o.g. Links wird man darauf hingewiesen und kann eine Einladung anfordern. Bei mir hat das keine 24 Stunden gedauert. Einfach mal ausprobieren. Hier mal ein Beispielnotebook von mir: Webscraping mit Python und BeautifulSoup Momentan ist (leider) nur Python 2.7.x verfügbar. Für eine _Jupyter_artige Arbeit aber auf jeden Fall mehr als empfehlenswert. Besonders die nahtlose Integration in finde ich super. Man kann aus Drive heraus einfach ein neues Notebook erstellen und natürlich auch den Link dazu wie bei Google Drive für andere Dateien gewohnt, freigeben und teilen. Tolle Sache. Google Drive Google Hoffen wir mal, dass dieses Produkt weiterentwickelt und gepflegt wird. Man weiß ja, dass Google schnell mal bestehende Produkte beerdigt. Hier will Google aber die Möglichkeiten des oder auf eine einfache Plattform heben. Deshalb bin ich optimistisch, dass dieses Projekt nicht eingestampft wird. Machine Learning Deep Learning Daumen hoch dafür! Anregungen, Korrekturen und weitere Meinungen wie immer Willkommen! Stay tuned. :-) Postet on mosenturm.de

Colaboratory — Google — Research at Google — am Beispiel Webscraping mit Python

70-Page Report on the COCO Dataset and Object Detection [Part 2]

An overview of ten Google AI tools that developers, businesses and analysts can use. It would be unfair to call Google just a search giant - from a system for finding relevant sites, it has quickly become a driving force for innovation in key IT sectors. In recent years, Google has become frighteningly entrenched in almost everything digital, whether it's electronics (smartphones, tablets, laptops), basic software (Android, Chrome OS), or intelligent software powered by Google AI. In this article, we will discuss what tools are provided by Google AI for developers, researchers, and organizations. I. For developers 1. TensorFlow Tensor tools are used to create highly accurate and well-defined machine learning models by data analysts. It consists of deep learning software libraries which are open source. 2. ML Kit ML Kit is a mobile SDK currently available on Android and iOS. It leverages Google's machine learning capabilities in mobile apps to solve real-life problems. ML Kit will help you succeed in many tasks. More than a hundred languages ​​are supported, including Hindi, Arabic, Chinese, not to mention European. Human faces can be recognized by ML KIT as well as text, faces, QR codes and other objects in images. In addition, there is an API to add your own machine learning models to TensorFlow Lite and integrate them into applications. 3. Google Open Source Google continues to open source code. And with the active participation of the connecting community, Google's work is getting even better. Some of the large projects by google open source are mentioned below- ClusterFuzz: In February 2019, the company opened the source code for the ClusterFuzz platform to test code using a server cluster. ClusterFuzz automates tasks such as notifying developers, creating an issue, tracking a bug, and closing reports after a fix. AutoFlip: In February of this year, Google introduced AutoFlip , an intelligent video reshaping system that detects the most important objects in the frame and cuts the video accordingly. Due to this, from landscape-oriented videos, you can get vertical, portrait ones, which are convenient to watch on mobile phones without rotating the screen. 4. Colab In short, Colaboratory or Colab is an online Python code editor and compiler. Colab is similar to Jupyter Notebook, with an extra feature of online sharing. The idea of ​​working at Colab is summarized in the following video. At the same time, it is allowed to use one NVidia Tesla K80 GPU for free for 12 hours (after 12 hours, the calculation is thrown off, but you can have time to calculate a small model). II. For researchers When diving into a new field, research on the topic under study is especially necessary. Google has made it easier to find a dataset by offering the following tools. 5. Moderated Google Cloud Datasets The fundamental problem with any ML model is to train it on "correct" data. Google Cloud Public Datasets are Google curated datasets that are periodically updated through analysis from multiple studies. The datasets are available in different formats such as image, audio, video or text. It’s intended to help a wide range of researchers with different requirements and use cases. Depending on the need, anyone can search the dataset using the tool. 6. Google search on datasets In September 2018, Google launched search for datasets (Dataset Search). The issue contains information about the resource where the set is published, the license, the date of the update, the description and the formats available for download. Despite the fact that the name of the datasets does not contain the name of the city, the datasets matches the request and contains information on weather changes in St. Petersburg and the Leningrad region. But note that it was updated quite a long time ago. 7. Crowdsource Another goal of Google is to improve the accuracy of datasets by offering users interesting tasks, such as recognizing different categories of images such as pictures, letters, newspapers, illustrations, and more. In this case, Google exploits users, offering a game from where Yandex gives - albeit small - but money. Improve your products with Google Crowdsource. On the other hand, you can learn how Google Crowdsource works with images, handwriting recognition, facial expressions, translation validation and image tags. Crowdsource analysis results can be transferred to your own ideas for data markup as part of a game approach. III. For organizations By closely monitoring the market, Google is identifying how its services can transform a potentially successful idea into an accomplished goal and is creating custom tools for organizations. 8. Cloud TPU Machine learning, despite all the advances, requires high-performance hardware. To this end, the company has created a tensor processor (Tensor Processing Unit). Cloud TPU enables enterprises to offer their customers the best possible service by reducing hardware costs. The tensor processor belongs to the class of neural processors with an application specific integrated circuit. The architecture is tailored for linear algebra tasks, and, for example, allows Google Photos to process up to 100 million photos per day with one processor. 9. Cloud AI Cloud AI facilitates the use of existing business models by Google, it enables to customize them thus helps in integrating machine learning capabilities into the existing business. Cloud AI consists of three components: Hub, Building Blocks, and Platform. Hub is a kind of library with AI ready-to-use components that can be used or shared for experimenting on the models. Building block component helps developers with speech processing, computer vision, language and structured data, to be added to their application. The AI ​​Platform enables scientists, engineers, and data developers to quickly deploy their ideas through multiple services such as AI Platform notebooks, deep learning containers, labeling services, and others. 10. Cloud AutoML Many popular brands like Disney, Imagia, and Meredith use Cloud AutoML. The niche of the tool is affordable machine learning for businesses with little experience in this area, but initially well-labeled data. Conclusion The advances in AI technology solutions are staggering. Google's involvement in AI innovation is really high, several tools like ML Kit, TensorFlow, Fire Indicators have been developed by Google. And you can search for datasets not only on Kaggle, but also through a specialized search.

10 Google Artificial Intelligence Tools Available to Everyone

What Exactly is a Chatbot? are computer programs designed to simulate or emulate human interactions through artificial intelligence. You can converse with chatbots the same way you would have a conversation with another person. They are used for various purposes, including customer service, information services, and entertainment, just to name a few. Chatbots What is a Rule-based chatbot? A rule-based chatbot is a chatbot that is guided in a sequence; they are straightforward; compared to Artificial Intelligence-based chatbots, this rule-based chatbot has specific rules. We will use a straightforward and short method to build a rule-based chatbot. Advantages of Rule-based chatbot Very easy to develop. Time-saving. Disadvantages of Rule-based chatbot Not power and advance as AI-based Chatbot. Requires a large number of codes to make a full-scale chatbot. Project Prerequisites: In this Python project, you just need to know basic Python. That includes. Python print() function Python Input / Output function Python Flask framework Python functions Basic HTML/CSS What will you learn? End to End development of Python chatbot. Basic Input/Output implementation. Flask Framework Basic Getting Started First, let make a very basic chatbot using basic Python skills like input/output and basic condition statements, which will take basic information from the user and print it accordingly. First, we will variables store user information. Step 1: User_Name  = None
User_Age  = None
User_Job = None Then, we will take input from the user. Step 2: print("Hello, I'm a Chatbot \n")
User_Name =  input("What is your name? ")
print("How are you {0}. \n".format(User_Name))

User_Age = input("What is your age? ")
print("Oh, so your age is {0}. \n".format(User_Age))

User_Job = input("What is your job profile? ")
print("So you're a  {0}. \n".format(User_Job)) When we run the above program, we will get the following output: Hello, I'm a Chatbot 
What is your name? pythonscholar
How are you pythonscholar. 
What is your age? 26
Oh, so your age is 26. 
What is your job profile? python developer
So you're a python developer. So now, let’s start creating a real chatbot and deploy it on Flask. We will use the Python library, which is mainly developed for building chatbots. ChatterBot What is ChatterBot, and how does it work? ChatterBot is a machine learning library that helps to generate an automatic response based on the user’s input. It uses a Natural Language Processing-based algorithm to generate repossessed based on the user’s contexts. You guys can refer to official documents for more information, or you can see the for it. Also, you can see the below flow chart to understand better how ChatterBot works. ChatterBot's GitHub code How To Install ChatterBot In Python? Installing ChatterBot in Python is very easy; it can be done using the pip command in the following steps. Type below command in terminal: pip install chatterbot chatterbot_corpus If you guys are using , you need to use the below command to install it on Google Colab. Google Colaboratory notebook !pip install chatterbot chatterbot_corpus So let’s start building a chatbot by following the below steps. First, we will import the ChatterBot library into our Python file. Step 1: from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTraine Then, we will name our chatbot. It can be anytime as per our need. Step 2: chatbot=ChatBot('Pythonscholar') We will start training our chatbot using its pre-defined dataset. Step 3: # Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Now, let us train our bot with multiple corpus
trainer.train("chatterbot.corpus.english.greetings",
              "chatterbot.corpus.english.conversations" ) chatterbot.corpus.english.greetings and chatterbot.corpus.english.conversations are the pre-defined dataset used to train small talks and everyday conversational to our chatbot. Then, we will check how our chatbot is responding to our question using the below code. Step 4: response = chatbot.get_response("How are you?")
print(response) Now let’s run the whole code and see what our chatbot responds to. Whole Source Code: from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

chatbot=ChatBot('Pythonscholar')

# Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Now let us train our bot with multiple corpus
trainer.train("chatterbot.corpus.english.greetings",
              "chatterbot.corpus.english.conversations" )
response = chatbot.get_response("How are you?")
print(response) The Output will be as follow: I am doing well. How to deploy a chatbot on Flask First, we will install the Flask library in our system using the below command: pip install flask And for Google Colab use the below command, mostly Flask comes pre-install on Google Colab. !pip install flask But first, let’s understand what the Flask framework in Python is. What is Python Flask? The Flask is a Python micro-framework used to create small web applications and websites using Python. Flask works on a popular templating engine called Jinja2, a web templating system combined with data sources to the dynamic web pages. Now start developing the Flask framework based on the above ChatterBot in the above steps. We have already installed the Flask in the system, so we will import the Python methods we require to run the Flask microserver. Import necessary methods of Flask and ChatterBot. Step 1: from flask import Flask, render_template, request
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer Then, we will initialize the Flask app by adding the below code. Step 2: #Flask initialisation
app = Flask(__name__) Flask( ) is used to create the Flask class object so that Python code can initialize the Flask server. name Now, we will give the name to our chatbot. Step 3: chatbot=ChatBot('Pythonscholar') Add training code for a chatbot. Step 4: # Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Now, let us train our bot with multiple corpus
trainer.train("chatterbot.corpus.english.greetings",
              "chatterbot.corpus.english.conversations" ) We will create the Flask decorator and a route in this step. Step 5: @app.route("/")
def index():
	render_template("index.html") The route() is a function of a Flask class used to define the URL mapping associated with the function. Then we make an index function to render the HTML code associated with the index.html file using the render_template function. In the next step, we will make a response function that will take the input from the user, and also, it will return the result or response from our trained chatbot. Create a function to take input from the user and respond accordingly. Step 6: @app.route("/get", methods=["POST"])
def chatbot_response():
	msg = request.form["msg"]
	response = chatbot.get_response(msg)
	return response Then, we will add the final code that will start the Flask server after interpreting the whole code. Step 7: if __name__ == "__main__":
    app.run() The complete code will look like this: from flask import Flask, render_template, request
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

#Flask initialization
app = Flask(__name__)

chatbot=ChatBot('Pythonscholar')

# Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Now let us train our bot with multiple corpus
trainer.train("chatterbot.corpus.english.greetings",
              "chatterbot.corpus.english.conversations" )

@app.route("/")
def index():
	
	return render_template("index.html")



@app.route("/get", methods=["GET","POST"])
def chatbot_response():
    msg = request.form["msg"]
    response = chatbot.get_response(msg)
    return str(response)

if __name__ == "__main__":
    app.run() As per Jinja2 implementation, we have to create two folders for storing HTML and CSS files; while working with the Jinja2 engine, it is necessary to make a folder with a name template to add HTML files, and for other files like CSS, javascript, or image we have to make a folder with the name static but it optional but creating template folder is compulsory. First, we will make an HTML file called index.html inside the template folder. <!DOCTYPE html>
<html>

<head>
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css')}}" />
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
</head>

<body>
    <div class="row">
        <div class="col-md-10 mr-auto ml-auto">
    <h1>Pythonscholar ChatBot</h1>
    <form>
        <div id="chatbox">
            <div class="col-md-8 ml-auto mr-auto">
                <p class="botText"><span>Hi! I'm Pythonscholar.</span></p>
            </div>
        </div>
        <div id="userInput" class="row">
            <div class="col-md-10">
                <input id="text" type="text" name="msg" placeholder="Message" class="form-control">
                <button type="submit" id="send" class="btn btn-warning">Send</button>
            </div>
        </div>
    </form>
</div>
</div>

<script>
    $(document).ready(function() {
        $("form").on("submit", function(event) {
            var rawText = $("#text").val();
            var userHtml = '<p class="userText"><span>' + rawText + "</span></p>";
            $("#text").val("");
            $("#chatbox").append(userHtml);
            document.getElementById("userInput").scrollIntoView({
                block: "start",
                behavior: "smooth",
            });
            $.ajax({
                data: {
                    msg: rawText,
                },
                type: "POST",
                url: "/get",
            }).done(function(data) {
                var botHtml = '<p class="botText"><span>' + data + "</span></p>";
                $("#chatbox").append($.parseHTML(botHtml));
                document.getElementById("userInput").scrollIntoView({
                    block: "start",
                    behavior: "smooth",
                });
            });
            event.preventDefault();
        });
    });
</script>
</body>

</html> We will not understand HTML and jquery code as jquery is a vast topic. We will create a style.css file, save it in the static folder, and use the below code. body {
    font-family: Garamond;
}

h1 {
    color: black;
    margin-bottom: 0;
    margin-top: 0;
    text-align: center;
    font-size: 40px;
}

h3 {
    color: black;
    font-size: 20px;
    margin-top: 3px;
    text-align: center;
}
.row {
    display: flex;
    flex-wrap: wrap;
    margin-right: -15px;
    margin-left: -15px;
}

.ml-auto{
    margin-left:auto !important;
}
.mr-auto{
    margin-right:auto !important;
}

.col-md-10,.col-md-8,.col-md-4{
    position: relative;
    width: 100%;
    min-height: 1px;
    padding-right: 15px;
    padding-left: 15px;
}
.col-md-8{flex:0 0 66.666667%;max-width:66.666667%}
.col-md-4{flex:0 0 33.333333%;max-width:33.333333%}
.col-md-10{flex:0 0 83.333333%;max-width:83.333333%}

.form-control {
    background: no-repeat bottom,50% calc(100% - 1px);
    background-image: none, none;
    background-size: auto, auto;
    background-size: 0 100%,100% 100%;
    border: 0;
    height: 36px;
    transition: background 0s ease-out;
    padding-left: 0;
    padding-right: 0;
    border-radius: 0;
    font-size: 14px;
}
.form-control {
    display: block;
    width: 100%;
    padding: .4375rem 0;
    padding-right: 0px;
    padding-left: 0px;
    font-size: 1rem;
    line-height: 1.5;
    color: #495057;
    border:none;
    background-color: transparent;
    background-clip: padding-box;
    border-bottom: 1px solid #d2d2d2;
    box-shadow: none;
    transition: border-color .15s ease-in-out,box-shadow .15s ease-in-out;
}
.btn {
    float: left;
    text-align: center;
    white-space: nowrap;
    vertical-align: middle;
    user-select: none;
    border: 1px solid transparent;
    padding: .46875rem 1rem;
    font-size: 1rem;
    line-height: 1.5;
    border-radius: .25rem;
    transition: color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;
}
.btn-warning {
    color: #fff;
    background-color: #f08f00;
    border-color: #c27400;
}
.btn.btn-warning:active, .btn.btn-warning:focus, .btn.btn-warning:hover {
    box-shadow: 0 14px 26px -12px rgba(255,152,0,.42),0 4px 23px 0 rgba(0,0,0,.12),0 8px 10px -5px rgba(255,152,0,.2);
}

button, input, optgroup, select, textarea {
    margin: 0;
    font-family: inherit;
    font-size: inherit;
    line-height: inherit;
    overflow:visible;
}
#chatbox {
    background-color: cyan;
    margin-left: auto;
    margin-right: auto;
    width: 80%;
    min-height: 70px;
    margin-top: 60px;
}

#userInput {
    margin-left: auto;
    margin-right: auto;
    width: 40%;
    margin-top: 60px;
}

#textInput {
    width: 87%;
    border: none;
    border-bottom: 3px solid #009688;
    font-family: monospace;
    font-size: 17px;
}

#buttonInput {
    padding: 3px;
    font-family: monospace;
    font-size: 17px;
}

.userText {
    color: white;
    font-family: monospace;
    font-size: 17px;
    text-align: right !important;
    line-height: 30px;
    margin: 5px;
}

.userText span {
    background-color: #009688;
    padding: 10px;
    border-radius: 2px;
}

.botText {
    color: white;
    font-family: monospace;
    font-size: 17px;
    text-align: left;
    line-height: 30px;
    margin: 5px;
}

.botText span {
    background-color: #ef5350;
    padding: 10px;
    border-radius: 2px;
}

#tidbit {
    position: absolute;
    bottom: 0;
    right: 0;
    width: 300px;
} We are all set to run our Flask application. So let’s start our chatbot by running a Python file. Once you run the Python file it will create a local server with the address , and when you open this URL on the browser, you can see the user interface of our chatbot. http://127.0.0.1:5000/ This is our Chatbot output. Congratulations, we have successfully built a chatbot using Python and Flask. Also, you guys can check our other articles below. How to Create Mad Libs game in Python What Is Python? 11 Things To Know Before Python Programming Also published on: https://pythonscholar.com/python-projects/rule-based-chatbot-in-python-using-flask/

Using Flask to Build a Rule-based Chatbot in Python

Uso de Flask para construir un chatbot basado en reglas en Python

Object Detection Using Single Shot MultiBox Detector (A Case Study Approach)

“Notebooks” are web applications that run on the browser which basically give you, the user, an interface to a virtual machine that runs an out-of-the-box environment. They are based on the project — an open-source project that brings machine learning development to your browser! No more virtual environment set-up hassle from now on. Which is great news since a lot of developers want to deal only with data science projects and not get derailed from that by needing to configure and set up their computers. Jupyter Notebook By using “notebooks”, you don’t have to set up an environment to begin with. Building machine learning applications now comes out of the box with notebooks. Why They’re Useful :They are useful because they reduce the time that a developer takes to set up his/her environment: install packages, update them to the required versions, and any other network or file-structure configurations that take time to set-up. And all that needs to be done BEFORE jumping into developing the apps themselves. Now, with notebooks, everything is set-up and ready to go in a matter of seconds. Ease of use : Notebooks can be shared easily and built upon incrementally. This allows projects to be worked on in a flexible manner. You don’t need to waste time anymore with getting the latest version of your project and sending in your updated version or dealing with conflicting versions and editing mistakes. It works like a Google Drive document which is being constantly updated (or a Git repository). Collaboration Notebooks are easy ways to bundle and upload to a shareable web page. The tools are evolving, the theory of data science is developing and Jupyter Notebooks now reduces deployment time to a matter of seconds. This makes it an invaluable tool for education platforms which showcase projects and tutorials. Other collaborative online projects can take advantage of this trait and leverage it to its needs. Google has built some tools which make use of Jupiter Notebooks. Portability: Jupyter notebooks were built initially for Python, R and Julia. Those are the most popular languages used in Machine Learning. Python being used in Google’s machine learning framework Tensoorflow and in Facebook’s PyTorch. R and Julia are reputable as well and have been used for a long time. Nowadays, the Jupyter Notebook has extended its reach to many other programming langurages including JavaScript, C++ and Ruby. Multiple Programming Languages: the interface has a modern look and provides many useful controls. Code can also be run one block at a time, which makes debugging easier. Certain subsystems of the notebook server are designed to be extended or overridden by users. This means that the notebooks are customisable. Nice Interface: How Do I Use Them? There is an increasing number of third parties who make use of notebooks on their platforms. , or “Colab” for short, is a product from Google Research. Colab enables users to write and execute python code through the browser. It is especially well suited to machine learning, data analysis and education. Colaboratory is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources. Colaboratory Collaboratory is great for testing some machine learning projects that you found on github. : an joint educational platform and experimentation platform. Users create they’re own solutions and implement machine learning algorithms. The site has numerous “kernels” from which people can investigate how their peers tackled a problem and what they used to optimise their results. This platform is a great hub for community and learning within the machine learning space. Kaggle Sites like Kaggle brings community together making it easy for machine learning developers to learn from each other. : for any private uses (in which you prefer to avoid sharing your work via github or any other public platform) you can spin up your own server that serves Jupyter Notebooks. You can also customise them to fit your project needs or even your own branding! Personal server Being an open-source project, there is always the option of tayloring your own custom Jupyter Notebook-based solution. Conclusion The Notebooks have enabled users to access computing resources and to prototype machine learning agents quickly and without hassle in out-of-the-box environments, reducing the set-up time to 0. It is a huge leap that made it possible for communities to access and tweak their own implementations, improving on one-another and establishing machine-learning development further into the open-source space.

Intro to Jupiter Notebooks

Building an End-to-End Speech Recognition Model in PyTorch with AssemblyAI

These days, machine learning and computer vision are all the craze. We’ve all seen the news about self-driving cars and facial recognition and probably imagined how cool it’d be to build our own computer vision models. However, it’s not always easy to break into the field, especially without a strong math background. Libraries like PyTorch and TensorFlow can be tedious to learn if all you want to do is experiment with something small. In this tutorial, I present a simple way for anyone to build fully-functional object detection models with just a few lines of code. More specifically, we’ll be using , a Python package built on top of PyTorch that makes the process easy and open to programmers at all levels. Detecto Quick and easy example To demonstrate how simple it is to use Detecto, let’s load in a pre-trained model and run inference on the following image: First, download the Detecto package using pip: pip3 install detecto Then, save the image above as “fruit.jpg” and create a Python file in the same folder as the image. Inside the Python file, write these 5 lines of code: detecto core, utils, visualize

image = utils.read_image( )
model = core.Model()

labels, boxes, scores = model.predict_top(image)
visualize.show_labeled_image(image, boxes, labels) from import 'fruit.jpg' After running this file (it may take a few seconds if you don’t have a CUDA-enabled GPU on your computer; more on that later), you should see something similar to the plot below: Awesome! We did all that with just 5 lines of code. Here’s what we did in each: Imported Detecto’s modules Read in an image Initialized a pre-trained model Generated the top predictions on our image Plotted our predictions Detecto uses a from PyTorch’s model zoo, which is able to detect about 80 different objects such as animals, vehicles, kitchen appliances, etc. However, what if you wanted to detect custom objects, like Coke vs. Pepsi cans, or zebras vs. giraffes? Faster R-CNN ResNet-50 FPN You’ll be glad to know that training a Detecto model on a custom dataset is just as easy; again, all you need is 5 lines of code, as well as either an existing dataset or some time spent labeling images. Building a custom dataset In this tutorial, we’ll start from scratch by building our own dataset. I recommend that you do the same, but if you want to skip this step, you can download a sample dataset (modified from Stanford’s ). here Dog Dataset For our dataset, we’ll be training our model to detect an underwater alien, bat, and witch from the competition, as shown below: RoboSub Ideally, you’ll want at least 100 images of each class. The good thing is that you can have multiple objects in each image, so you could theoretically get away with 100 total images if each image contains every class of object you want to detect. Also, if you have video footage, Detecto makes it easy to split that footage into images that you can then use for your dataset: detecto.utils split_video

split_video( , , step_size= ) from import 'video.mp4' 'frames/' 4 The code above takes every 4th frame in “video.mp4” and saves it as a JPEG file in the “frames” folder. Once you’ve produced your training dataset, you should have a folder that looks something like the following: images/
|   image0.jpg
|   image1.jpg
|   image2.jpg
|   ... If you want, you can also have a second folder containing a set of validation images. Now comes the time-consuming part: labeling. Detecto supports the PASCAL VOC format, in which you have XML files containing label and position data for each object in your images. To create these XML files, you can use the open-source tool as follows: LabelImg pip3 install labelImg labelImg # Download LabelImg using pip # Launch the application You should now see a window pop up. On the left, click the “Open Dir” button and select the folder of images that you want to label. If things worked correctly, you should see something like this: To draw a bounding box, click the icon in the left menu bar (or use the keyboard shortcut “w”). You can then drag a box around your objects and write/select a label: When you’ve finished labeling an image, use CTRL+S or CMD+S to save your XML file (for simplicity and speed, you can just use the default file location and name that they auto-fill). To label the next image, click “Next Image” (or use the keyboard shortcut “d”). Once you’re done with the entire dataset, your folder should look something like this: images/
|   image0.jpg
|   image0.xml
|   image1.jpg
|   image1.xml
|   ... We’re almost ready to start training our object detection model! Getting access to a GPU First, check whether your computer has a . Since deep learning uses a lot of processing power, training on a typical CPU can be very slow. Thankfully, most modern deep learning frameworks like PyTorch and Tensorflow can run on GPUs, making things much faster. Make sure you have PyTorch downloaded (you should already have it if you installed Detecto), and then run the following 2 lines of code: CUDA-enabled GPU torch

print(torch.cuda.is_available()) import If it prints True, great! You can skip to the next section. If it prints False, don’t fret. Follow the below steps to create a notebook, an online coding environment that comes with a free, usable GPU. For this tutorial, you’ll just be working from within a Google Drive folder rather than on your computer. Google Colaboratory 1. Log in to Google Drive 2. Create a folder called “Detecto Tutorial” and navigate into this folder 3. Upload your training images (and/or validation images) to this folder 4. Right-click, go to “More”, and click “Google Colaboratory”: You should now see an interface like this: 5. Give your notebook a name if you want, and then go to Edit ->Notebook settings -> Hardware accelerator and select GPU 6. Type the following code to “mount” your Drive, change directory to the current folder, and install Detecto: os google.colab drive

drive.mount( )

os.chdir( )

!pip install detecto import from import '/content/drive' '/content/drive/My Drive/Detecto Tutorial' To make sure everything worked, you can create a new code cell and type to check that you’re in the right directory. !ls Train a custom model Finally, we can now train a model on our custom dataset! As promised, this is the easy part. All it takes is 4 lines of code: detecto core, utils, visualize

dataset = core.Dataset( )
model = core.Model([ , , ])

model.fit(dataset) from import 'images/' 'alien' 'bat' 'witch' Let’s again break down what we’ve done with each line of code: Imported Detecto’s modules Created a Dataset from the “images” folder (containing our JPEG and XML files) Initialized a model to detect our custom objects (alien, bat, and witch) Trained our model on the dataset This can take anywhere from 10 minutes to 1+ hours to run depending on the size of your dataset, so make sure your program doesn’t exit immediately after finishing the above statements (i.e. you’re using a Jupyter/Colab notebook that preserves state while active). Using the trained model Now that you have a trained model, let’s test it on some images. To read images from a file path, you can use the function from the module (you could also use an image from the you created above): read_image detecto.utils Dataset image = utils.read_image( )
predictions = model.predict(image) labels, boxes, scores = predictions print(labels) print(boxes) print(scores) # Specify the path to your image 'images/image0.jpg' # predictions format: (labels, boxes, scores) # ['alien', 'bat', 'bat'] #           xmin       ymin       xmax       ymax # tensor([[ 569.2125,  203.6702, 1003.4383,  658.1044], #         [ 276.2478,  144.0074,  579.6044,  508.7444], #         [ 277.2929,  162.6719,  627.9399,  511.9841]]) # tensor([0.9952, 0.9837, 0.5153]) As you can see, the model’s predict method returns a tuple of 3 elements: labels, boxes, and scores. In the above example, the model predicted an alien ( ) at the coordinates [569, 204, 1003, 658] ( ) with a confidence level of 0.995 ( ). labels[0] boxes[0] scores[0] From these predictions, we can plot the results using the module. For example: detecto.visualize visualize.show_labeled_image(image, boxes, labels) Running the above code with the image and predictions you received should produce something that looks like this: If you have a video, you can run object detection on it: visualize.detect_video(model, , ) 'input.mp4' 'output.avi' This takes in a video file called “input.mp4” and produces an “output.avi” file with the given model’s predictions. If you open this file with or some other video player, you should see some promising results! VLC Lastly, you can save and load models from files, allowing you to save your progress and come back to it later: model.save( ) model = core.Model.load( , [ , , ]) 'model_weights.pth' # ... Later ... 'model_weights.pth' 'alien' 'bat' 'witch' Advanced usage You’ll be happy to know that Detecto isn’t just limited to 5 lines of code. Let’s say for example that the model didn’t do as well as you hoped. We can try to increase its performance by augmenting our dataset with and defining a custom : torchvision transforms DataLoader torchvision transforms

augmentations = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip( ),
    transforms.ColorJitter(saturation= ),
    transforms.ToTensor(),
    utils.normalize_transform(),
])

dataset = core.Dataset( , transform=augmentations)

loader = core.DataLoader(dataset, batch_size= , shuffle= ) from import 0.5 0.5 'images/' 2 True This code applies random horizontal flips and saturation effects on images in our dataset, increasing the diversity of our data. We then define a DataLoader object with ; we’ll pass this to instead of the Dataset to tell our model to train on batches of 2 images rather than the default of 1. batch_size=2 model.fit If you created a separate validation dataset earlier, now is the time to load it in during training. By providing a validation dataset, the method returns a list of the losses at each epoch, and if , then it will also print these out during the training process itself. The following code block demonstrates this as well as customizes several other training parameters: fit verbose=True matplotlib.pyplot plt

val_dataset = core.Dataset( )

losses = model.fit(loader, val_dataset, epochs= , learning_rate= , 
                   lr_step_size= , verbose= )
                   
plt.plot(losses)
plt.show() import as 'validation_images/' 10 0.001 5 True The resulting plot of the losses should be more or less decreasing: For even more flexibility and control over your model, you can bypass Detecto altogether; the method returns the underlying torchvision model used, which you can mess around with as much as you see fit. model.get_internal_model Conclusion In this tutorial, we showed that computer vision and object detection don’t need to be challenging. All you need is a bit of time and patience to come up with a labeled dataset. If you’re interested in further exploration, check out or visit the for more tutorials and use cases! Detecto on GitHub documentation Previously published at https://medium.com/@alankbi/build-a-custom-trained-object-detection-model-with-5-lines-of-code-713ba7f6c0fb

Build a Custom-Trained Object Detection Model With 5 Lines of Code

Pytorch is a deep learning framework; a set of functions and libraries which allow you to do higher-order programming designed for Python language, based on . Torch is an open-source machine learning package based on the programming language . It is primarily developed by Facebook’s artificial-intelligence research group and Uber’s probabilistic programming language software is built on it. Torch Lua Pyro PyTorch is more “pythonic” and has a more consistent API. It also has native model exports which can be used to speed up inference. PyTorch shares many commands with , which helps in learning the framework with ease. ONNX numpy At its core, PyTorch provides two main features: An n-dimensional Tensor, similar to Numpy but can run on GPUs Automatic differentiation for building and training neural networks If you’re using anaconda distribution, you can install the Pytorch by running the below command in the anaconda prompt. conda install pytorch-cpu torchvision-cpu -c pytorch The rest of the article is structured as follows: What is Colab, Anyway? Setting up GPU in Colab Pytorch Tensors Simple Tensor Operations Pytorch to Numpy Bridge CUDA Support Automatic DifferentiationConclusion If you want to skip the theory part and get into the code right away, Niranjankumar-c/DeepLearning-PadhAI Colab — Colaboratory is a research tool for machine learning education and research. It’s a Jupyter notebook environment that requires no setup to use. Colab offers a free GPU cloud service hosted by Google to encourage collaboration in the field of Machine Learning, without worrying about the hardware requirements. Colab was released to the public by Google in October 2017. Google Colab Getting Started with Colab Sign in with your Google Account Create a new notebook via File -> New Python 3 notebook or New Python 2 notebook You can also create a notebook in Colab via Google Drive Go to Google Drive Create a folder of any name in the drive to save the project Create a new notebook via Right click > More > Colaboratory To rename the notebook, just click on the file name present at the top of the notebook. Setting up GPU in Colab In Colab, you will get 12 hours of execution time but the session will be disconnected if you are idle for more than 60 minutes. It means that for every 12 hours Disk, RAM, CPU Cache and the Data that is on our allocated virtual machine will get erased. To enable GPU hardware accelerator, just go to Runtime -> Change runtime type -> Hardware accelerator -> GPU Pytorch — Tensors Numpy based operations are not optimized to utilize GPUs to accelerate its numerical computations. For modern deep neural networks, GPUs often provide speedups of . So, unfortunately, numpy won’t be enough for modern deep learning. This where Pytorch introduces the concept of . A Pytorch Tensor is conceptually identical to an n-dimensional numpy array. Unlike the numpy, PyTorch Tensors can utilize GPUs to accelerate their numeric computations 50x or greater Tensor Let’s see how you can create a Pytorch Tensor. First, we will import the required libraries. Remember that torch, numpy and matplotlib are pre-installed in Colab’s virtual machine. torch numpy matplotlib.pyplot plt import import import as The default tensor type in PyTorch is a float tensor defined as . We can create tensors by using the inbuilt functions present inside the package. torch.FloatTensor torch ## creating a tensor rows and columns consisting ones 
>> x = torch.ones( , ) 
>> print(x) tensor([[ , ], [ , ], [ , ]]) 
## creating a tensor rows and columns consisting zeros 
>> x = torch.zeros( , ) 
>> print(x) tensor([[ , ], [ , ], [ , ]]) of 3 2 of 3 2 1. 1. 1. 1. 1. 1. of 3 2 of 3 2 0. 0. 0. 0. 0. 0. Creating a tensor by random initialization To increase the reproducibility, we often set the random seed to a specific value first. 
>> torch.manual_seed( ) #generating tensor randomly 
>> x = torch.rand( , ) 
>> print(x) #generating tensor randomly normal distribution 
>> x = torch.randn( , ) 
>> print(x) 2 3 2 from 3 3 Simple Tensor Operations Slicing of Tensors You can slice PyTorch tensors the same way you slice ndarrays #create a tensor 
>> x = torch.tensor([[ , ], [ , ], [ , ]]) 
>> print(x[:, ]) # Every row, only the last column >> print(x[ , :]) # Every column first row >> y = x[ , ] # take the element first row and first column and create a another tensor >> print(y) 1 2 3 4 5 6 1 0 in 1 1 in Reshape Tensor Reshape a Tensor to a different shape >> x = torch.tensor([[ , ], [ , ], [ , ]]) #( rows and columns) 
>> y = x.view( , ) #reshaping to rows and columns 1 2 3 4 5 6 3 2 2 3 2 3 Use of to reshape the tensors -1 indicates that the shape will be inferred from previous dimensions. In the below code snippet will result in a tensor of shape 6x1 because we have fixed the size of rows to be 6, Pytorch will now infer the best possible dimension for the column such that it will be able to accommodate all the values present in the tensor. -1 x.view(6,-1) >> x = torch.tensor([[ , ], [ , ], [ , ]]) #( rows and columns) 
>> y = x.view( , ) #y shape will be x1 1 2 3 4 5 6 3 2 6 -1 6 Mathematical Operations #Create two tensors 
>> x = torch.ones([ , ]) 
>> y = torch.ones([ , ]) #adding two tensors 
>> z = x + y #method >> z = torch.add(x,y) #method #subtracting two tensors 
>> z = x - y #method >> torch.sub(x,y) #method 3 2 3 2 1 2 1 2 In-place Operations In Pytorch all operations on the tensor that operate in-place on it will have an postfix. For example, is the out-of-place version, and is the in-place version. _ add add_ >> y.add_(x) #tensor y added x and result will be stored y with in Pytorch to Numpy Bridge Converting a Pytorch tensor to a numpy ndarray is very useful sometimes. By using . on a tensor, we can easily convert tensor to ndarray. numpy() >> x = torch.linspace( , , steps = ) #creating a tensor using linspace 
>> x_np = x.numpy() #convert tensor to numpy 
>> print(type(x), type(x_np)) 
< 0 1 5 ' . '> < ' . '> class torch Tensor class numpy ndarray To convert numpy ndarray to pytorch tensor, we can use to convert ndarray to tensor. .from_numpy() >> a = np.random.randn( ) #generate a random numpy array 
>> a_pt = torch.from_numpy(a) #convert numpy array to a tensor 
>> print(type(a), type(a_pt)) 
< 5 ' . '> < ' . '> class numpy ndarray class torch Tensor During the conversion, Pytorch tensor and numpy ndarray will share their underlying memory locations and changing one will change the other. CUDA Support To check how many CUDA supported GPU’s are connected to the machine, you can use the code snippet below. If you are executing the code in Colab you will get 1, that means that the Colab virtual machine is connected to one GPU. is used to set up and run CUDA operations. It keeps track of the currently selected GPU. >> print(torch.cuda.device_count()) 1 If you want to get the name of the GPU Card connected to the machine: >> print(torch.cuda.get_device_name( )) 
Tesla T4 0 The important thing to note is that we can reference this CUDA supported GPU card to a variable and use this variable for any Pytorch Operations. All CUDA tensors you allocate will be created on that device. The selected GPU device can be changed with a context manager. torch.cuda.device #Assign cuda GPU located at location to a variable 
>> cuda0 = torch.device( ) #Performing the addition on GPU 
>> a = torch.ones( , , device=cuda0) #creating a tensor on GPU >> b = torch.ones( , , device=cuda0) #creating a tensor on GPU >> c = a + b 
>> print(c) 
tensor([[ , ], [ , ], [ , ]], device= ) '0' 'cuda:0' 3 2 'a' 3 2 'b' 2. 2. 2. 2. 2. 2. 'cuda:0' As you can see from the above code snippet the tensors are created on GPU and any operation you do on these tensors will be done on GPU. If you want to move the result to CPU you just have to do .cpu() #moving the result to cpu 
>> c = c.cpu() 
>> print(c) 
tensor([[ , ], [ , ], [ , ]]) 2. 2. 2. 2. 2. 2. Automatic Differentiation In this section, we will discuss the important package called automatic differentiation or autograd in Pytorch. The autograd package gives us the ability to perform automatic differentiation or automatic gradient computation for all operations on tensors. It is a define-by-run framework, which means that your back-propagation is defined by how your code is run. Let’s see how to perform automatic differentiation by using a simple example. First, we create a tensor with parameter set to True because we want to track all the operations performing on that tensor. requires_grad #create a tensor requires_grad = True 
>> x = torch.ones([ , ], requires_grad = True) 
>> print(x) 
tensor([[ , ], [ , ], [ , ]], requires_grad=True) with 3 2 1. 1. 1. 1. 1. 1. Perform a simple tensor addition operation. >> y = x + #tensor addition 
>> print(y) #check the result 
tensor([[ , ], [ , ], [ , ]], grad_fn= 5 6. 6. 6. 6. 6. 6. ) < > AddBackward0 Because y was created as a result of an operation on x, so it has a grad_fn. Perform more operations on y and create a new tensor z. >> z = y*y + >> print(z) 
tensor([[ , ], [ , ], [ , ]], grad_fn= <SumBackward0>) 1 37. 37. 37. 37. 37. 37. ) >> t = torch.sum(z) #adding all the values in z 
>> print(t) 
tensor(222., grad_fn= < > AddBackward0 Back-Propagation To perform back-propagation, you can just call t.backward() >> t.backward() 
#peform backpropagation but pytorch will not print any output. Print gradients . d(t)/dx >> print(x.grad) 
tensor([[ , ], [ , ], [ , ]]) 12. 12. 12. 12. 12. 12. will give you the partial derivative of with respect to . If you are able to figure out how we got a tensor with all the values equal to 12, then you have understood the automatic differentiation. If not don't worry just follow along, when we execute we are calculating the partial derivate of t with respect to . Remember that t is a function of , which in turn is a function of . x.grad t x t.backward() x z x d(t)/dx = 2y * 1 at x = 1 and y = 6, where y = x + 5 The important point to note is that the value of the derivative is calculated at the point where we initialized the tensor x. Since we initialized x at a value equal to one, we get an output tensor with all the values equal to 12. The entire code discussed in the article is present in the Kaggle Kernel. Feel free to fork it or download it. . The best part is that you can directly run the code in Kaggle Kernel once you fork it, don’t need to worry about installing the packages GettingStartedWithPytorch_GPU If Colab is your jam, click to execute the code directly and get your hands dirty. here Niranjankumar-c/DeepLearning-PadhAI Conclusion In this post, we briefly looked at the Pytorch & Google Colab and we also saw how to enable GPU hardware accelerator in Colab. Then we have seen how to create tensors in Pytorch and perform some basic operations on those tensors by utilizing CUDA supported GPU. After that, we discussed the Pytorch package which gives us the ability to perform automatic gradient computation on tensors by taking a simple example. If you have any issues or doubts while implementing the above code, feel free to ask them in the comment section below or send me a message in citing this article. autograd LinkedIn Learn More If you want to learn more about Data Science, Machine Learning. Check out the and by Abhishek and Pukhraj from . One of the good points about these courses is that they teach in both Python and R, so it’s your choice. Machine Learning Basics Advanced Machine Learning Starttechacademy Recommended Reading Deep Learning Best Practices: Activation Functions & Weight Initialization Methods — Part 1 Demystifying Different Variants of Gradient Descent Optimization Algorithm In my next post, we will discuss how to implement the feedforward neural network using Pytorch (nn.Functional, nn.Parameters). So make sure you follow me on medium to get notified as soon as it drops. Until then Peace :) NK. Author Bio is Retail Risk Analyst Intern at HSBC Analytics division. He is passionate about Deep learning and Artificial Intelligence. He is one of the top writers at in . You can find all of Niranjan’s blog . You can connect with Niranjan on , and to stay up to date with his latest blog posts. Niranjan Kumar Medium Artificial Intelligence here LinkedIn Twitter GitHub . I am looking for opportunities either full-time or freelance projects, in the field of Machine Learning and Deep Learning. If there are any relevant opportunities, feel free to drop me a message on LinkedIn or you can reach me through email as well. I would love to discuss Originally published at https://www.marktechpost.com on June 9, 2019.

Getting Started With Pytorch In Google Collab With Free GPU

this story is a continuation to the series on how to easily build an abstractive text summarizer , (check out for this series) , today we would go through how you would be able to build , so we would through representing words to our summarizer github repo a summarizer able to understand words my goal in this series to present the latest novel ways of abstractive text summarization in a simple way , (you can check ) from my overview blog corner stone method of using seq2seq models with attention to using pointer generator to using reinforcement learning with deep learning we would use google colab , so you won’t have to use a powerful computer , nor would you have to download data to your device , as we would connect google drive to google colab to have a fully integrated deep learning experience (you can check ) my overview on working on free deep learning ecosystem platforms All code can be found online through my github repo This tutorial has been based over the work of , they have truly made great work on simplifying the needed work to apply summarization using tensorflow, I have built over their code , to convert it to a python notebook to work on google colab , I truly admire their work https://github.com/dongjun-Lee/text-summarization-tensorflow so lets begin !! 1- Setup 1-A To begin we first create a google colab notebook 1- go to https://colab.research.google.com 2- select Google Drive Tab (to save your new google colab to google drive) 3- select (you can also select python 2 notebook) New Python 3 Notebook a blank notebook would be created to your google drive , it would look like this You can change the runtime of your notebook from selecting the runtime button in the top menu , to change which python version you are using choose a hardware accelerator from ( GPU , TPU ) 1-A-A or you can clone the code directly from my github repo go to , but this time we would select github tab https://colab.research.google.com then we just paste the this , and click upload link 1-B Now after we are have created our google colab , lets connect to google drive in the newly created notebook , add a new code cell then paste this code in it # https://stackoverflow.com/questions/47744131/colaboratory-can-i-access-to-my-google-drive-folder-and-file !apt-get install -y -qq software-properties-common python-software-properties module-init-tools!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null!apt-get update -qq 2>&1 > /dev/null!apt-get -y install -qq google-drive-ocamlfuse fusefrom google.colab import authauth.authenticate_user()from oauth2client.client import GoogleCredentialscreds = GoogleCredentials.get_application_default()import getpass!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URLvcode = getpass.getpass()!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} !mkdir -p drive!google-drive-ocamlfuse drive this would connect to your drive , and create a folder that your notebook can access your google drive from It would ask you for access to your drive , just click on the link , and copy the access token , it would ask this twice after writing this code , you run the code by clicking on the cell (shift enter) or by clicking the play button on the top of your code cell then you can simply access any file by its path in form of path = "drive/test.txt" 1-C Now Lets get the data that we would work on our data set that we would work on is in form of and their news headlines . The input would be and the output needed would be its summary or in this case would be the news content headline There are 2 popular dataset for this task Amazon Product Reviews CNN /Daily news dataset (which we would use in our case) you the data , you can just , it would just take some seconds not more. don’t have to download copy it to your google drive Here is the for the folder containing the data . Link Here we would use , which enables you to easily copy files between different google drives Copy, URL to Google Drive first you would paste the above Link paste your link , name it , then save to google drive then you simply click on Save,Copy to Google Drive (after autentication your google drive) after authenticating , you just click save to google drive Now after setup process , we can start our work , so lets Begin !! 2- Dependencies and paths 2-a First Lets install needed dependencies in google colab you are able to install using pip , by simply !pip, in every code section you simply click on and then just start writing your code !pip install gensim!pip install wget import nltknltk.download('punkt') 2-b then Lets set needed Dependencies from nltk.tokenize import word_tokenizeimport reimport collectionsimport pickleimport numpy as npfrom gensim.models.keyedvectors import KeyedVectorsfrom gensim.test.utils import get_tmpfilefrom gensim.scripts.glove2word2vec import glove2word2vec 2-c Then lets define where the data can be found default_path = "drive/Colab Notebooks/Model 2/" #default path for the folder inside google drive **#path for training text (article)**train_article_path = default_path + "sumdata/train/train.article.txt" **#path for training text output (headline)**train_title_path   = default_path + "sumdata/train/train.title.txt" **#path for validation text (article)**valid_article_path = default_path + "sumdata/train/valid.article.filter.txt" **#path for validation text output(headline)**valid_title_path   = default_path + "sumdata/train/valid.title.filter.txt" 3- Building A Dictionary for the text summarization to work , you must represent your words in a dictionary format assume we have an article like five-time world champion michelle kwan withdrew from the # us figure skating championships on wednesday , but will petition us skating officials for the chance to compete at the # turin olympics # each word would have a representation in a dict and we would also need the reverse operation also , like to apply this we would need some helper functions , like 3-A Simple cleaning data function the goal of this function would be a simple cleaning of data , just by replacing some unneeded characters with # def clean_str(sentence):sentence = re.sub("[#.]+", "#", sentence)return sentence this substitution of characters is rather simple , you can of course add multiple substitution steps 3-B Function that actually return text and apply the above cleaning function def get_text_list(data_path, toy):with open (data_path, "r", encoding="utf-8") as f:if not toy:return [clean_str(x.strip()) for x in f.readlines()][:200000]else:return [clean_str(x.strip()) for x in f.readlines()][:50] this function would be called for mltiple cases if you need to load training data or test data or if you just need a sample of any of the above by simply setting toy = True 3-C Now lets Build the function that would actually create the needed dictionary here you would see that we add 4 built-in words , these are essential for the seq2seq algorithim , they are <padding> this would be used to make the sequences of same length <unk> this would be used to identify that the word is not found inside the dict <s> this would be used to identify the beingin of a sentence </s> this would be used to identify the end of a sentence copy the code , as here the padding is incorrect due to the editor of medium from github def build_dict(step, toy=False):if step == "train": train_article_list = get_text_list(train_article_path, toy)train_title_list = get_text_list(train_title_path, toy) #First lets load the training data words = list()for sentence in train_article_list + train_title_list:for word in word_tokenize(sentence):words.append(word) #then lets collect all words from the training data#by simply tokenizing each text sample to its words#here we would use the built-in function imported from nltk toolkit#which simply return a list of words from a sentence word_counter = collections.Counter(words).most_common() word_dict = dict()word_dict["<padding>"] = 0word_dict["<unk>"] = 1word_dict["<s>"] = 2word_dict["</s>"] = 3 #we would only select the most common words #first lets set the 4 built-in words for word, _ in word_counter:word_dict[word] = len(word_dict) #then lets build our dict , by simply looping over word_co with open(default_path + "word_dict.pickle", "wb") as f:pickle.dump(word_dict, f) #then lets save this to a pickle #all of the above was for the training step#when you are in the validation you can simply load the pickles that#you have just saved elif step == "valid":with open(default_path + "word_dict.pickle", "rb") as f:word_dict = pickle.load(f) #for both of the 2 cases (training , or validation)#we would create a reversed dict reversed_dict = dict(zip(word_dict.values(), word_dict.keys())) #then we would simply for the 2 cases (training , or validation)#define a max len for article and for the summary article_max_len = 50summary_max_len = 15return word_dict, reversed_dict, article_max_len, summary_max_len 4- Now Lets Build Our Dataset After building the dict for our data , we would begin to build the actual dataset that would be used in our algorithm Using the above example of an article , five-time world champion michelle kwan withdrew from the # us figure skating championships on wednesday , but will petition us skating officials for the chance to compete at the # turin olympics # the algorthim would need this to be represented in which is simply getting the collection of word dict for the words in the given sentence same would occur on the test data def build_dataset(step, word_dict, article_max_len, summary_max_len, toy=False):**#---case of train#---we would load both (article , headline) for training**    if step == "train":article_list = get_text_list(train_article_path, toy)title_list = get_text_list(train_title_path, toy)**#---case of valid#---we only load articles**    elif step == "valid":article_list = get_text_list(valid_article_path, toy)**#---if step is neither (train nor valid) raise error**    else:raise NotImplementedError**#---(for each aricle) get list of words#--- so now x (article) contains list of words**    x = [word_tokenize(d) for d in article_list] x = [[word_dict.get(w, word_dict["<unk>"]) for w in d] for d in x] #---(for each aricle) get index of word from word_dict for each article#---if not found , use "<unk>" tokken#---so now we have our train dataset **#---(for each aricle) limit x to article_max_len**    x = [d[:article_max_len] for d in x] **#---(for each aricle) if x was less than article_max_len#--- pad the x by using "<padding>" tokken**    x = [d + (article_max_len - len(d)) * [word_dict["<padding>"]] for d in x] if step == "valid":  
    return x  
else: **#-------if step = "train"#-------we must do the same steps on headline#-------but here we don't use the concept of padding**        y = [word_tokenize(d) for d in title_list]y = [[word_dict.get(w, word_dict["<unk>"]) for w in d] for d in y]y = [d[:(summary_max_len - 1)] for d in y]return x, y so lets simply call both (build dict and build dataset) print("Building dictionary...")word_dict, reversed_dict, article_max_len, summary_max_len = build_dict("train", False) print("Loading training dataset...")train_x, train_y = build_dataset("train", word_dict, article_max_len, summary_max_len, False) 5- Word Embeddings But we can’t yet feed the our neural network with a list containing the indexes of words , as it would understand them . We need to represent the word itself in a format that our neural net would understand , and here comes the concept of word embeddings it is a simple concept , that replaces each word in your dict with a list of numbers , (in our case we would model each word with a 300 float number list) There are already trained models that have been trained over millions of text to correctly model the words , once you are able to correctly model the words , your neural net would be able to truly understand the text within the article . A very well known test to identify how well the algorithm understand text after using word embeddings , is applying word similarity on a given word as you can see , the output tells us that the model would now be capable of understanding the relations between words , which is an extremely important factor in the success of out neural net 5-A lets get the trained model for our work there is a very well known pretrained model called provided by stanford , you can download it from Glove pre-trained vectors https://nlp.stanford.edu/projects/glove/ or you can simply copy it from my google drive like i have explained before , here is the for the glove vectors in a pickle format link 5-B Build a function to get an array of word embeddings def get_init_embedding(reversed_dict, embedding_size): print("Loading Glove vectors...") **#---Load glove model which is in a pickle format**    with open( default_path + "glove/model_glove_300.pkl", 'rb') as handle:word_vectors = pickle.load(handle) **#---Loop through all words within the reversed_dict**    used_words = 0word_vec_list = list()for _, word in sorted(reversed_dict.items()):try: word_vec = word_vectors.word_vec(word)used_words += 1except KeyError: word_vec = np.zeros([embedding_size], dtype=np.float32) #to generate for <padding> and <unk> #-----------if the word i found in the dict ,#-----------save its value #-----------else , generate an array of zeros#-----------of length = embedding_size#-----------which in this case would be 300#-----------this is the case also for <padding> and <unk>#-----------where <s>, </s> token would be zeros#-----------like seen below word_vec_list.append(word_vec) #-------add it to the array#-------remember that we are looping in sorted reversed_dict#-------so the index of the element inside word_vec_list#-------would be the same as  index of word#-------no need of a dict , an array is sufficient **#---just print out the percentage of knwon words**    print("words found in glove percentage = " + str((used_words/len(word_vec_list))*100) ) **#----Assign random vector to <s>, </s> token**    word_vec_list[2] = np.random.normal(0, 1, embedding_size)word_vec_list[3] = np.random.normal(0, 1, embedding_size) **#----then return the array**return np.array(word_vec_list) to call the function we simply call word_embedding = get_init_embedding(reversed_dict, 300) To sum it all UP so we can say that we have now correctly represented the text for our task of text summarization so to sum it all up , we have build the code to by simply calling word_dict, reversed_dict, article_max_len, summary_max_len = build_dict("train", False) train_x, train_y = build_dataset("train", word_dict, article_max_len, summary_max_len, False) word_embedding = get_init_embedding(reversed_dict, 300) the coming steps in the coming tutorial if GOD wills it , we would go through how to build the model itself , we would build a seq2seq encoder decoder model using LSTM , we would go through the very details of building such a model using tensorflow , this would be the corner stone for the next tutorials in the series , that would go through the latest approaches for this problem from using pointer generator model using reinforcement learning with deep learning don’t forget to clone the code for this tutorial from my repo and you can take a look on the talking about an overview on text summarization previous tutorial you can also check this talking about the eco system of a free deep learning platform blog I truly hope you have enjoyed this tutorial , i am waiting for your feedback , and i am waiting for you in the next tutorial if GOD wills it

Abstractive Text Summarization (tutorial  2) , Text Representation made very easy

Python and Finance: How to Level Up Your Spreadsheets

The journey on the unsinkable — what AI can learn from the disaster Have you ever thought of being one of the passengers on the Titanic in 1912? Would you survive? Would your children survive? What would happen if you were in the upper deck? We are often told that women and children were given priority to the lifeboats, but I was not satisfied with this short, qualitative answer. With the advent of , we are able to dig deeper. We can spot patterns and infer what would have happened to you and your family should you all were onboard. machine learning Curiosity is the wick in the candle of learning. — William Arthur Ward TL;DR I developed a machine learning model that does so, and deployed it as a . Check out and complete the quiz to find out your chance of survival. Web App http://18.212.193.40:8888/ Mmm… borderline for me… Table of Contents Introduction Data Machine Learning Web App Deployment The Takeaway Originally published at edenau.github.io . Introduction Motivation For data scientists, is arguably one of the most widely used datasets in the field of machine learning, along with , etc.. For some non-data scientists, machine learning is a black box that does magic; but for some sceptics who ‘learn from the past’ (and inadvertently human-trained a classification model in their head), they foresee that Titanic Kaggle dataset MNIST hand-written digit Iris flower AI is the new dot-com. Photo by on Marc Sendra Martorell Unsplash One of my professors argued that machine learning-driven time-series forecasting model would not perform with flying colours, as it is just . looking in the rearview mirror However, the omnipresence of machine learning models proves their values to our society, and the fact that they have already brought a lot of convenience to us is undeniable — Google auto-completes email sentences which really saves me a lot of time. I hope I can share my experience, help spread the knowledge, and demonstrate how new technologies can achieve things that we could not in the last century. AI is the new electricity. — Andrew Ng PS: machine learning is a subset of artificial intelligence, in case you are confused. What is Kaggle? Kaggle is a platform for data scientists to share data, exchange thoughts, and compete in predictive analytics problems. There are many high-quality datasets that are freely accessible on Kaggle. Some Kaggle competitions even offer prize money, and they attract a lot of famous machine learning practitioners to participate. People often think that Kaggle is not for beginners, or it has a very steep learning curve. They are not wrong. But they do offer challenges for people who are ‘getting started’. As a (junior) data scientist, I could not resist searching for interesting datasets to start my journey on Kaggle. And I bumped into the Titanic. Kaggle is an Airbnb for data scientists — this is where they spend their nights and weekends. — Zeeshan-ul-hassan Usmani Overview Here goes the overview of the technical bit. I first found the dataset on Kaggle and decided to work on it and analyze it with . I used it to develop and train an ensemble of classifiers using that would predict one’s chances of survival. I then saved it by and deployed it as a Web App on localhost using . Finally, I leveraged AWS free tier (available for 12 months) to cloud-host it. Python scikit-learn pickle Flask There are many tutorials online that focus on how to code and develop a machine learning model in Python and other languages. Therefore, I will only explain my work in a relatively qualitative manner with graphs and results, instead of bombarding you with codes. If you really want to go through my codes, they are available on . GitHub Tools I decided to use Python since it is the most popular programming language for machine learning with numerous libraries. And I don’t know R. And no one uses MATLAB for machine learning. Instead of my local machine, I went for so that I could work cross-platform without any hassle. Google Colab Sit tight, here we go! Data Data Inspection Let’s import the data into a . It consists of passenger ID, survival, ticket class, name, sex, age, number of siblings and spouses onboard, number of parents and children onboard, ticket number, passenger fare, cabin number, and port of embarkation. DataFrame First 5 rows of data What immediately came to my mind was the following points: is key (unique), PassengerId is the target that we would like to infer, Survived might not help but their titles might, Name is a mess, and Ticket there are missing data labelled as . NaN I decided to drop variable for now for simplicity. It could possibly be holding useful information, but it would require extensive feature engineering to extract them. Ticket We should start with the easiest, and take it from there. The ratio of missing data On the other hand, let’s take a closer look at the missing data. There are a few missing entries in variables and , which should be able to be inferred by other variables. Around 20% passenger ages were not recorded. This might pose a problem to us, since is likely to be one of the key predictors in the dataset. ‘Women and children first’ was a code of conduct back then, and reports suggested that they were indeed saved first. There are >77% of missing entries in , which is unlikely to be very helpful, and let’s drop it for now. Embarked Fare Age Cabin Data Visualization Pair plot (not shown) is usually my go-to at the beginning of a data visualization task, as it is usually helpful, and it has a high information-to-lines-of-code ratio. One single line of gives you n² plots (technically n(n+1)/2 distinct plots), where n represents the number of variables. It gives you a basic understanding of the relationship between every pair of variables, and the distribution of each variable itself. Let’s dive into different variables. seaborn.pairplot() We first inspect the relationship of target variable with each predictor one by one. By a simple count plot , we found that most people belonged to the third class, which wasn’t surprising; and in general they had a lower probability of survival. Even with this single predictor, , we could infer that a first-class passenger would be more likely to survive, while this would be unlikely for a third-class passenger. Survived seaborn.countplot() given everything else unknown Meanwhile, women and children were more likely to survive, which aligned with the aforementioned theory of . First-class young female passengers would now be the ones with the highest chance of survival, if we only examine variable , , and . ‘women and children first’ Pclass Sex Age Nevertheless, it might be harder to interpret the density plot of passenger fare. For both ‘survived’ and ‘not survived’ classes, they span over a wide range of fare, with the ‘not survived’ class having a smaller mean and variance. Note that there is a funny tail in the ‘survived’ class, which corresponds to three people getting their first-class tickets with $512 each (no idea what currency the dataset was referring to). They all got onboard at the port of Cherbourg, and all of them survived. seaborn.kdeplot() On the other hand, the port of embarkation seems to also play a role in determining who would survive. Most people embarked at the port of Southampton — the first stop of the journey, and they had the lowest survival rate. Maybe they were assigned to cabins further away from exits, or spending more time on a cruise would make people relaxed or tired. No one knows. Or maybe it’s just indirectly caused by some third variable — say maybe there were fewer women/children/first-class passengers that got onboard at the first port. This plot does not provide such information. Further investigation is required, . and is left as an exercise for the reader (Nani?) If you are a fan of tables instead of plots, we can also visualize the data by and take the mean for each class. However, I don’t think there is a clear pattern shown in the table of below. pandas.DataFrame.groupby() Parch Correlation matrix generated by illustrates the strength of correlation between any two variables. As you can see, has the highest magnitude of correlation with , whereas guess what, and are highly correlated. and do not seem to play a big role in predicting one’s survival chance, although instinct suggests otherwise. Family members onboard might be your helping hands in escaping from the sinking ship, or they could also be a burden (not from my personal experience, in case you are reading). More on that later in . seaborn.heatmap() Sex Survived Fare Pclass SibSp Parch Feature Engineering Missing Data Imputation We have found earlier in that there were missing data entries. For instance, we seem to have no clue how much did a 60-year-old Thomas Storey pay for his ticket. Instinct tells us that ticket fare hugely depends on ticket class and port of embarkation (sex might also be a factor in the early 20th century), We can cross-check with the correlation matrix above. Therefore, we will just take the mean (or median if you want) of third-class fare at Southampton. This is just an educated guess and is probably wrong, but it is good enough. Bear in mind that it is impossible to have noiseless data, and machine learning models are (to different extents) robust against noise. Go digging in historical archives is not worth it. Data Inspection There were also two women whom we had no idea where they got on the ship. This should be strongly correlated with ticket class and fare. As they both paid 80 dollars for a first-class seat, I would bet my money on Cherbourg (C in the plots). If there are only a few missing entries in a particular variable, we can use the tricks above to make educated guesses by essentially taking the value. Nonetheless, it would be really dangerous to do the same thing if we have more missing data, like in where about 20% of them are missing. maximum likelihood Age We can no longer make educated guesses by inspection. Since we dropped variable , and all other missing entries are filled in, we can leverage all other variables to infer the missing by random forest regressor. There are 80% of ‘training’ data to infer the remaining 20%. Cabin Age Machine Learning Feature Engineering As suggested in , passengers’ name would probably not helpful in our case, since they are all distinct and, you know what, being called wouldn’t make me less likely to survive. But we could extract the titles of their names. Data Inspection Eden While most of them had titles of ‘Mr’, ‘Mrs’, and ‘Miss’, there were quite a number of less frequent titles — ‘Dr’, ‘The Reverend’, ‘Colonel’ etc., some of them only appeared once, such as ‘Lady’, ‘Doña’, ‘Captain’ etc.. Their rare appearance would not help much in model training. In order to find patterns with data science, you need data. One datum point has no patterns whatsoever. Let’s just categorize all those relatively rare titles as ‘Rare’. Categorical data requires extra care before model training. Classifiers simply cannot process string inputs like ‘Mr’, ‘Southampton’ etc.. While we can map them to integers, say (‘Mr’, ‘Miss’, ‘Mrs’, ‘Rare’) → (1, 2, 3, 4), there should be no concept of ordering amongst titles. . In order not to mislead machines and accidentally construct a sexist AI, we should one-hot-encode them. They become: Being a Dr does not make you superior ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0, 0, 0, 1) ) On the other hand, I decided to add two more variables — and . Adding makes more sense since the whole family would have stayed together on the cruise. You wouldn’t have a moment with your partner but abandon your parents, would you? Besides, being alone might be one of the crucial factors. You could be more likely to make reckless decisions, or you could be more flexible without taking care of your family. Experiments (adding variables one at a time) suggested that their addition did improve the overall predictability. FamilySize IsAlone FamilySize = SibSp + Parch + 1 Model Evaluation Codes are available on . GitHub Before picking and tuning any classifiers, it is VERY important to standardize the data. Variables measured at various scales would screw things up. covers a range between 0 and 512, whereas is binary (in our dataset) — 0 or 1. Fare Sex You wouldn’t want to weigh **_Fare_** more than **_Sex_** . I have some thoughts (and guesses) on which classifiers would perform well enough, often by experience. I personally prefer Random Forest as it usually guarantees good enough results. But let’s just try out all the classifiers we know — SVM, KNN, AdaBoost, you name it, and they were all tuned by grid search. XGBoost stands out eventually with an 87% test accuracy, but that does not mean it would perform the best in inferring unknown data subsets. To increase the robustness of our classifier, an ensemble of classifiers was trained and final results were obtained by . It is vital to embed models with different strengths into the ensemble, otherwise there is no point building an ensemble model at the expense of computation time. with different natures majority voting Finally, I submitted it to Kaggle and achieved around accuracy. Not bad. There is always room for improvement. For instance, there is surely some useful information hidden in and , but we dropped them for simplicity. We could also create more features, e.g. a binary class that is 1 if or 0 otherwise. 80% Cabin Ticket Underage Age < 18 But we will move on for now. Saving and Loading Model I am not satisfied with just a trained machine learning model. I want it to be accessible by everyone (sorry for people who don’t have internet access). Therefore, we have to save the model and deploy it elsewhere, and this can be done by library. Parameters and in function represent write access and read-only in binary mode respectively. pickle 'wb' 'rb' open() pickle.dump(<model>, open(<file_path>, 'wb'))pickle.load(open(<file_path>, 'rb')) Photo by on Jonathan Pielmayer Unsplash Web App Deployment Web Framework is an easy-to-use web framework in Python. I only had little prior experience in building websites (HTML in primary school, last year, Wix doesn’t count), and I found it straightforward and simple. The simplest thing you can do is: Flask GitHub pages from flask import Flaskapp = Flask(__name__) .route("/")def hello():return "<h1>Write something here.</h1>" @app app.run(host='0.0.0.0', port=60000) And voilà! You can browse it in your localhost. What else do we need? We want people to fill in a form to collect the data required and pass it to the machine learning model (and not sell it, Mark). The model would have an output, which we will redirect users to that page. We will use to build a form in Python, and a single form is defined by a class, which looks like the following: WTForms from wtforms import Form, TextField, validators, SubmitField,  DecimalField, IntegerField, SelectField class ReusableForm(Form): sex = SelectField('Sex:',choices=[('1', 'Male'), ('0', 'Female') ],validators=[validators.InputRequired()]) fare = DecimalField('Passenger Fare:',default=33,places=1,validators=[validators.InputRequired(),validators.NumberRange(min=0,max=512, message='Fare must be between 0 and 512')\]) submit = SubmitField('Predict') I found an HTML template from and built on top of it. Will Koehrsen You look at the people who have been there before you, the people who have dealt with this before, the people who have built the spaghetti code, and you thank them very much for making it open source, , and you take what they have made and you put it in your program, and you never ever look at it again. — Tom Scott and you give them credit No way I delve into a CSS spaghetti. Cloud Hosting Now that the webpage can be viewed via my localhost, and everything works fine. The last step would be hosting it online. Unfortunately, I only used one site hosting service before, GitHub Pages, which only hosts static sites. There are 3 major cloud hosting services now — AWS, GCP, and Azure. AWS is by far the most popular one, so I went for its 12-month free tier. They had lots of tutorials and documentation (and customer services), and it is quite easy to follow. Photo by on Nathan Anderson Unsplash I connected to the Linux server instance with my private key, migrated my repository to the server, ran my script, and it worked! The only problem was system timeout when it was idle for too long. By some , turns out it can be fixed by starting a screen session, which we run the scripts inside the session. googling [ec2-user@ip-<SOME_IP> ~]$ screen When you want to check the status next time, use to resume your screen session. You can exit the session by followed by . screen -r Ctrl+A D And now, it is running 24/7 non-stop! Check it out at . http://18.212.193.40:8888 The Takeaway Endnotes To sceptics: AI is not false hope. They are backed by sophisticated statistical theories. They are hard works. They are keys to revolutionize industries. To new learners: Data science is not just about training and tuning neural networks. It requires curiosity to drive us, knowledge to kick-start, energy to clean data, experience (i.e. trial and error) to engineer features, patience to train models, maturity to handle failure, and wisdom to explain. To technology enthusiasts: AI is no magic. There is nothing new under the sun. Photo by on Antonio Lainez Unsplash To future me: Doing data science is one thing, writing it is a different story. — current me Related Articles If you want to start a machine learning project now, check out the article below for starters: _Start training your neural networks with free GPUs today_towardsdatascience.com Quick guide to run your Python scripts on Google Colaboratory If you want to check out what else we can do using Python, these articles might help: _Exploring data visualization tools in Python_towardsdatascience.com Visualizing bike mobility in London using interactive maps and animations _Exploring climate-related data manipulation tools in Python_towardsdatascience.com Handling NetCDF files using XArray for absolute beginners Remarks If you want to contribute in any means, feel free to drop me an or find me on . As you can see, this project would definitely benefit from getting a ! email Twitter domain name Originally published at edenau.github.io .

Would You Survive the Titanic?

is a fantastic tool for data exploration. It . While is great for data science, I’m going to demonstrate the use of for a completely different use case: or simply put, . Jupyter Notebook combines markdown text, executable code, and output all inside a single document served in a browser Jupyter Notebook DevOps Runbook a way to respond quickly to system outages Problem Imagine you are spending an evening with your loved one and suddenly see a flurry of slack/pager alerts about your API latency climbing up. It’s all downhill from there. You get online and check all usual suspects: . You jump from terminal to AWS console to NewRelic to conference call and what not. Let’s just say the whole experience is stressful until you find and fix the issue. recent deployments, dependent services, load balancer, incoming traffic, database and so on More matured organisations maintain runbooks for incident response. Runbook outlines the steps to be followed and takes the guess work out of debugging. First, let’s see some challenges with current form of runbooks: You need to manually execute each step, no automation Unless very well written, there can be ambiguity/confusion in following instructions It’s quite an effort to get everybody onboard to keep the runbooks up to date Solution To tackle some of these problems I am proposing use of Jupyter Notebooks for writing runbooks. Here’s how your API latency debugging session might look like inside a Notebook environment (please watch this video in full screen mode, original if needed). youtube link Incident response with Jupyter Notebook You can . pull in graphs, check deployment times, rollback changes, run SQL queries, shell scripts, SSH all from within Notebook Benefits Here are some benefits of maintaining your runbooks in executable Notebook format. Less confusion. Code is much more deterministic than instructions written in English. Reduces the incident time & impact. On-call responds faster with code required to investigate/fix an issue at her fingertip. Automate at your own pace. Since Notebook supports markdown, it’s possible to just import existing runbooks as is & automate a few steps every sprint. Better collaboration. It provides a first class platform for sharing all the tribal knowledge and local scripts that developers keep to combat an issue. There’s real power when we combine individual steps to build more complex logic. Following is possible today. It’s a step towards self healing system. Challenges Executable Notebook format is promising but here are some challenges with current Jupyter implementation. Typical Jupyter installation is single user local setup requiring the Jupyter server to be running locally. There’s no easy way to share Notebooks. could have been a good choice but it’s hosted on Google servers. The Notebook server needs to be self hosted for the code to have access to all the infrastructure within our VPC. Google Colaboratory Any infrastructure code will require credentials, ssh keys etc. We need a way to share them safely & not just stick it everywhere in the Notebook code snippets. I’m building , a platform that tackles these challenges & provides an easy way to write and share executable runbooks within team. provide a complete overview of Nurtch capabilities and how-to’s. what do you think of this approach to managing runbooks. Nurtch Docs Let me know View all posts by Amit Rathi Originally published at blog.amirathi.com on March 27, 2018.

Simplify DevOps with Jupyter Notebook

TL;DR. After reading this post, you will learn how to run state of the art object detection and segmentation on a video file Fast . Even on an old laptop with an integrated graphics card, old CPU, and only 2G of RAM. Run like Fast and Furious So here is the catch. This will only work if you have an internet connection and own a Google Gmail account. Since you are reading this, it is very likely you are already qualified. All the code in the post runs entirely in the cloud. Thanks to Google’s Colaboratory a.k.a. Google Colab! I am going to show you how to run our code on Colab with a server-grade CPU , > 10 GB of RAM and a powerful GPU for FREE! Yes, you hear me right. Using Google Colab with GPU enabled Colab was build to facilitate machine learning professionals collaborating with each other more seamlessly. I have shared my , click to open it. Python notebook for this post Log in to your Google Gmail account on the upper right corner if you haven’t done so. It will ask you to open it with Colab at the top of the screen. Then you are going to make a copy so you can edit it. Now you should be able to click on the “ ” menu button to choose the version of Python and whether to use GPU/CPU to accelerate the computation. Runtime The environment is all set. So easy isn’t it? No hassles like installing Cuda and cudnn to make GPU working on a local machine. Run this code to confirm TensorFlow can see the GPU. It outputs: Found GPU at: /device:GPU:0 Great, We are good to go! If you are curious about what the GPU model you are using. It is a Nvidia Tesla K80 with 24G of memory. Quite powerful. Run this code to find out yourself. You will see the 24G graphics memory does help later. It makes possible to process more frames at a time to accelerate the video processing. Mask R-CNN Demo The demo is based on the . It is an implementation of on Keras+TensorFlow. It not only generates the bounding box for a detected object but also generate a mask over the object area. Mask R-CNN GitHub repo Mask R-CNN Install Dependencies and run Demo Mask R-CNN has some dependencies to install before we can run the demo. Colab allows you to install Python packages through , and general Linux package/library through . pip apt-get In case you don’t know yet. Your current instance of Google Colab is running on an Ubuntu virtual machine. You can run almost every Linux command you normally do on a Linux machine. Mask R-CNN depends on , we are installing it with the following cell. pycocotools It clones the coco repository from GitHub. Install build dependencies. Finally, build and install the coco API library. All this happens in the cloud virtual machine, and quite fast. We are now ready to clone the Mask_RCNN repo from GitHub and cd into the directory. Notice how we change directory with Python script instead of running a shell ‘cd’ command. Since we are running Python in current notebook. Now you should be able to run the Mask R-CNN demo on colab like you would on a local machine. So go ahead and run it in your Colab notebook. So far those sample images came from the GitHub repo. But how do you predict with custom images? Predict with Custom Images In order to upload a image to Colab notebook, there are three options that I think of. 1. Use a Free image hosting provider like the . imgbb 2.Create a GitHub repo, then download the image link from colab. After uploading images by either of those two options, you will get a link to the image. Which can be downloaded to your colab VM with Linux command. It downloads one image to the ./images folder. wget !wget -P ./images https://preview.ibb.co/cubifS/sh_expo.jpg The first two options will be ideal if you just want to upload 1 or 2 images and don’t care other people on the internet also be able to see it given the link. 3. Use Google Drive The option is ideal if you have private images/videos/other files to be uploaded to colab. Run this block to authenticate the VM to connect to your Google Drive. It will ask for two verification code during the run. Then execute this cell to mount the Drive to the directory ‘ ’ drive You can now access your Google drive content in directory ./drive !ls drive/ Hope you are having fun so far, why not try this on a video file? Processing Videos Processing a video file will take three steps. 1. Video to images frames. 2. Process images 3. Turn processed images to output videos. In our previous demo, we ask the model to process just one image at a time. As configured in the . IMAGES_PER_GPU If we are going to process the whole video one frame at a time, it will take a long time. So instead we are going to leverage GPU to process multiple frames in parallel. The pipeline of Mask R-CNN is quite computationally intensive and takes a lot of GPU memory. I find the Tesla K80 GPU on Colab with 24G of memory can safely process 3 images at a time. If you go beyond that the notebook might crash in the middle of processing the video. So in the code below, we set the to 3 and use library to stage 3 images at a time before processing them with the model. batch_size cv2 After running this code, you should now have all processed image files in one folder . ./videos/save The next step is easy, we just need to generate the new video from those images. We are going to use cv2’s to accomplish this. VideoWriter But two things you want to make sure: 1. The frames need to be ordered in the same way as they are extracted from the original video. (Or backward if you prefer to watch the video that way) 2.The frame rate matches the original video. You can use the following code to check the frame rate of a video or just open the file property. Finally here is the code to generate the video from processed image frames. If you have gone this far, the processed video should now be ready to be downloaded to your local machine. Free free to try your favorite video clip. Maybe intentionally decrease the frame rate when reconstructing the video to watch it in slow motion. Summary and Further reading In the post, we walked through how to run your model on Google Colab with GPU acceleration. You have learned how to do object detection and segmentation on a video. Thanks to the powerful GPU on Colab, made it possible to process multiple frames in parallel to speed up the process. Further reading If you want to learn more about the technology behind the object detection and segmentation algorithm. Here is the original goes through the detail of the model. paper of Mask R-CNN Or if you just get started with objection detection, check out my goes through important basics shared between many models. object detection/localization guide series Here again , and for your convenience. the Python notebook for this post GitHub repo Like this article? Consider and share with more people. 👏 Share on Twitter Share on Facebook Originally published at www.dlology.com . For more deep learning experiences like this.

How to run Object Detection and Segmentation on a Video Fast for Free

Sharing some of the latest research, announcements, and resources on deep learning. By Isaac Madan ( email ) Continuing our series of deep learning updates, we pulled together some of the awesome resources that have emerged since our last post. In case you missed it, you can find all past updates . As always, this list is not comprehensive, so if there’s something we should add, or if you’re interested in discussing this area further. If you’re a machine learning practitioner or student, join our Talent Network to get exposed to awesome ML opportunities. here let us know here Research & Announcements by Google Research. Google releases its own cloud notebook platform — try it. Colaboratory Colaboratory is a data analysis tool that combines text, code, and code outputs into a single collaborative document. by Google. Original paper . OpenFermion: The Electronic Structure Package For Quantum Computers OpenFermion is an open source effort for compiling and analyzing quantum algorithms to simulate fermionic systems, including quantum chemistry. Among other functionalities, the current version features data structures and tools for obtaining and manipulating representations of fermionic and qubit Hamiltonians. here by Mills of University of Ontario. Deep learning and the Schrödinger equation et al We have trained a deep (convolutional) neural network to predict the ground-state energy of an electron in four classes of confining two-dimensional electrostatic potentials. . NVIDIA open sources its deep learning chip architecture to broaden its adoption as an IoT standard. NVIDIA Deep Learning Accelerator The NVIDIA Deep Learning Accelerator (NVDLA) is a free and open architecture that promotes a standard way to design deep learning inference accelerators. . (Wikipedia). Watch first place winner Ning6A1 by BengKiat Ng solve the maze (Youtube video). Read more about the contest via this blog post. Micromouse contest first place video Micromouse is an event where small robot mice solve a 16x16 maze here by Google. Google announces Neural Networks API for Android which executes against machine learning models on the device, bringing more AI to the edge. Neural Networks API The Android Neural Networks API (NNAPI) is an Android C API designed for running computationally intensive operations for machine learning on mobile devices. by Creswell of Imperial College London. Generative Adversarial Networks: An Overview et al The aim of this review paper is to provide an overview of GANs for the signal processing community, drawing on familiar analogies and concepts where possible. In addition to identifying different methods for training and constructing GANs, we also point to remaining challenges in their theory and application. by Vertex AI. Open source portable deep learning engine. Announcing PlaidML: Open Source Deep Learning for Every Platform Our mission is to make deep learning accessible to every person on every device. Resources, Tutorials & Data by Andreas Jansson and Ben Firshman. A handy tool that renders Arxiv academic papers as easy to read web pages so you don’t have to read the PDF versions that is typical of most ML papers. Arxiv Vanity by Alena Kruchkova. Great series of lecture videos that follow the Deep Learning book by Goodfellow . Original book . Video lectures accompanying book Deep Learning et al here by Adrian Rosebrock. Tutorial demonstrating near real-time object detection via a Rasberry Pi. Raspberry Pi: Deep learning object detection with OpenCV by Data4Bio. Thorough and understandable explanation of Principal Component Analysis (Youtube video). Dimensionality Reduction: Principal Components Analysis, Part 1 by ClearBrain. Explaining Your Machine Learning Model (or 5 Ways to Assess Feature Importance) Knowing which features, inputs, or variables in a model are influencing its effectiveness is valuable to improving its actionability. Assessing feature importance though is not straightforward. Below we outline five ways of addressing feature importance, with a focus on logistic regression models for simplicity. by Sebastian Ruder. Word embeddings in 2017: Trends and future directions Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers (Wikipedia). This post will focus on the deficiencies of word embeddings and how recent approaches have tried to resolve them. by Coursera. Course just started October 23. How to Win a Data Science Competition: Learn from Top Kagglers In this course, you will learn to analyse and solve competitively such predictive modelling tasks. By . Isaac is an investor at Venrock ( ). If you’re interested in deep learning or there are resources I should share in a future newsletter, I’d love to hear from you. If you’re a machine learning practitioner or student, join our Talent Network to get exposed to awesome ML opportunities. Isaac Madan email here is a newsletter of entrepreneurial ideas & perspectives by investors, operators, and influencers. Requests for Startups ** **❤” Please tap or click “︎ to help to promote this piece to others.

Up to Speed on AI & Deep Learning: October Update, Part 3

Welcome to Hacker Noon Writing Prompts! See all other stories that follow this template . here Which of the following topics are you most excited by? Artificial intelligence Why does it excite you? AIs are becoming entrenched in everyday business operations, serving as cognitive collaborators to employees -supporting creative tasks, generating new ideas, and tackling previously untenable innovations. AI as a service will affect our life on every level. It will solve problems we face, know before we get sick, shop for us and so much more. In combination with metaverse and web3, we will see a new digital landscape pop up before our eyes without even realizing it. What are the trends in this area or products in this area that you are the fondest of? Why? Well, in healthcare I would say because it will be more preventive work that warns in advance when something is not right. What are the positive impacts they can have on society? It can remove mundane tasks for us so we can focus more on what we want. What are the negative impacts they can have on society? It can be dominating everything and remove what it means to be human. We see already AI in art, music, and other creative areas which we have done in the past for enjoyment. Will it be in the future that “ was this done by AI or a human?” What are your predictions on how these technologies will evolve? I believe that AI will continue to evolve and become more sophisticated. I think that it will become better at completing more complex tasks and becoming more efficient. Additionally, I think that AI will become more accessible to people and be used in more everyday applications. What are your predictions on the ethics of future societies in relation to these technologies? As AI becomes more advanced, it will become increasingly difficult for humans to make ethical decisions without taking into account the opinion of AI. This could lead to a situation where humans are no longer in control of their own ethical decision-making and instead defer to AI on ethical issues. Developers will have to partner with an AI to produce code. Doctors have to partner with an AI to determine the right diagnoses. [Please keep this CTA when you submit]: Would you like to take a stab at answering some of these questions? The link for the template is , just start writing! Interested in what others had to say in their answers? Click . Interested in reading the content from all of our writing prompts? Click . HERE HERE HERE

How AI Serves as a Cognitive Collaborator to Employees

This blog is derived from the webinar, “Accelerate Releases Through Code Ownership with Collaborator and Bugsnag”, focused on establishing a culture of code ownership and its benefits through the lens of the SmartBear tools Collaborator and Bugsnag. Taking a line from the SmartBear 2021 Annual State of Software Quality Report: "Quality is top of the mind for every individual and every team. Every developer has felt the pain of software releases that did not go well leading to customer calls and late nights. Quality software remains the insurance policy that continues to pay out." Engineering teams must constantly catch and remediate defects in order to ensure stable, high-quality applications. Testing and monitoring methods are typically applied throughout the entire software development lifecycle (SDLC) - from initial development, through testing and into production. With each added stage in the process, the relative cost of the issue increases. By the time an issue proceeds undetected into production, the reputation of the brand is on the line. One effective strategy to streamline bug identification and resolution through the SDLC is to leverage the practice of code ownership. In this blog we discuss , and how they drive a sense of clear responsibility, a factor proven to correlate with overall quality and efficiency in resolving issues. General code ownership principles by providing tools to help squads or code owners make decisions best suited to their parts of an application. How Bugsnag and Collaborator natively support and enhance this approach What Is Code Ownership? Code ownership is the idea that individual developers or teams are responsible for the features they create and deploy. For example, suppose that an ecommerce application has two teams responsible for user conversion and ecommerce. If there’s an error in the shopping cart, the ecommerce team owns the code and would be responsible for a fix. [source codeownership blog] We see many teams adopting what is referred to as code stewardship, or weak code ownership, where teams of engineers take general responsibility over building, testing and maintaining certain parts of the application. Benefits of Weak Code Ownership: : “The proportion of reviewers without expertise shares a strong, increasing relationship with the likelihood of having post-release defects.” [Code Ownership and Review Study]. Naturally, developers familiar with a certain part of an application should be responsible for other processes around it, such as reviewing code changes. Developers who have contributed to a module or function may have a deeper understanding of specific dependencies, data requirements or other tricky circumstances that need to be met when new code is added, and can catch more issues or incorrect assumptions if they have navigated this themselves. Minimizing Defects : By allocating code review and error monitoring responsibilities to a module’s owners, these codeowners can review code changes more efficiently as they are often already up to speed on their module’s functionality and context. Efficiency A culture of ownership encourages developers to take initiative as the subject matter expert of their code and actively participate in the relevant technical decisions. Granting teams the ability to monitor their own parts of the code, triage, prioritize and resolve their own issues, and review code changes with their own requirements brings the flexibility to tailor decisions around the functionality they understand. This fosters a healthier development environment. Culture: One example is tying error budgets to individual squads rather than defining this on the organization level. By doing so, teams can define their own error budgets and goals for their modules, which may bring more precision into those estimates on a company level. Challenges Below are some challenges we hear from teams implementing code ownership principles around their existing structures: How do we make sure teams get alerted of stability issues in the parts of the code they own? How do we align teams around which bugs to focus on? How do we make sure nothing slips through the cracks if there are gaps between what each team owns, or if things come through that no one team is specifically responsible for? How can we make this work within our broader ecosystem of tools, like JIRA, to minimize duplicating effort keeping things organized between all these data sources? How Do Collaborator and Bugsnag Fit in? helps development teams identify stability issues in every release and provides actionable insights to prioritize and fix those with the greatest impact. Bugsnag is an all-encompassing term for our many integrations with tools like Slack, JIRA and PagerDuty, as well as the intelligent logic layered on top to trigger and route alerts to the right people at the right times. Bugsnag's Alerting and Workflow Engine Bugsnag can be configured to alert the team who built the module or the integration when there are spikes in the errors coming from that code. The alert triggers can be tuned per module to ensure they are meaningful and align with that module’s behavior. For example, if one team is responsible for an app’s integrations with third party services, they may want to utilize spike detection to get alerted of potential outages. Another team might want alerts for every new error if they are responsible for rolling out a new feature. Teams should aim to minimize alert fatigue by adjusting alerting processes to follow code ownership boundaries. This helps to avoid high volumes of alerts firing off to one channel, where it is unclear who is responsible for what. : Engineering teams typically have pre-existing workflows built into their toolsets to support their squads, such as JIRA boards. Our integrations allow you to keep in sync with issue trackers and notification platforms in order to minimize cross management. Issue tracker integrations Our platform incorporates - teams can build a triaging process and know at any given time what the plan is for each error, so nothing gets lost in a high volume of data even when there may be gaps in coverage of errors. state tracking : A great example of an engineering organization that leveraged Bugsnag’s features to transform the way they triage and resolve issues around a code ownership model is found here: Customer story https://www.bugsnag.com/customers/gusto adds a level of accountability with clear, documented review processes and cross-functional alignment to catch errors early in your SDLC. Collaborator : Designed to fit into the same workflows as tools like GitHub and JIRA, Collaborator makes code review simpler, puts processes in place to give developers ownership of their code and facilitates bringing in peers in for additional approval of code shipped to production. Integrations with tools : Regardless of integration, a team’s workflow for review is the same for code artifacts as it is for documentation review, as it is for Simulink model review. Enforce process with a uniform workflow : For most integrations, Collaborator provides a gating mechanism to ensure artifact review before any release is approved, further shifting defect discovery earlier and enabling the release of the highest quality code possible. Provides review enforcement : 82% of teams satisfied with their code review process have specific guidelines on how they should be performed. [source] To create clear guidelines that ensure everyone on your team understands expectations, Collaborator lets you build custom checklists in review templates so participants with different roles and responsibilities can easily see what’s expected of them on each project. Custom rules to be set up per team Shift Left in the SDLC The vast majority of defects are typically introduced in the coding/development phase of the SDLC, but are mostly undetected until later phases, such as pre-production testing (unit tests, functional tests and system tests) and production. As seen below, an increasing number of bugs are typically found throughout the testing phases into release. The diagram highlights that the cost to repair a bug in a staging or production release is exponentially higher than a bug found during coding or early testing phases. This is in part because mean time to resolution is generally increased in later stages, and revenue or brand reputation can also take a hit when a bug is presented to your user base. Given there is a direct relationship between the number of code reviewers and the quality of the code post-review, the goal becomes shifting error identification and resolution earlier towards the coding phase through more robust peer reviews, supported by a tool like Collaborator. This in turn shifts the “percentage defects found” curve to the left, and lowers the cost associated with remediating those repairs, from developer’s time, user experience and brand loyalty perspectives. An ROI tool can be found on the Collaborator page, where you can drop in some basic figures for cost of developers, and quickly understand the advantages that early code review gives you as a return on investment. https://smartbear.com/product/collaborator/roi-calculator/ How Collaborator and Bugsnag Close the Loop Collaborator and Bugsnag in tandem allow teams to minimize the errors released into production with more thorough code reviews, while also providing robust monitoring and alert routing for those errors that do ultimately make it to your end users. We can see an ideal workflow below, illustrating the following: Customizable review and approval workflow completed in Collaborator by module owners to ensure code changes meet criteria. Approval of PR results in automatic merging of repositories, effectively applying the new changes to the codebase and moving changes forward for release. Once the newest release is in production, Bugsnag will begin to process events in real time from users that have adopted the new version. The Release dashboard will pick up the new version and monitor its live stability (crash free session and user rates). As new exceptions occur on user’s devices, Bugsnag will leverage information about where in the code the exception occurred and alert the right team to new errors or spikes in known errors in their modules, via our integrations. Once receiving an alert, developers leverage the information in the Bugsnag dashboard to triage, diagnose and prioritize the error. When the bug is ultimately addressed and fixed by the squad, this will result in some commits and a PR into GitHub, which will then kick off a new squad-specific review process via Collaborator to review and release the fix. To view live demonstrations of this feedback loop in the context of both tools, take a look at our full Webinar . here Additional Resources Bugsnag Code ownership Blog SmartBear Annual Report Code Ownership and Review Study

Use Collaborator and Bugsnag to Set Up a Code Ownership Loop

Web3 and DeFi are hot and trendy topics these days. However, many people don't know what they actually mean. And this is not because they are too lazy to check in with Google. There's just way too much hype and fluffy marketing out there that can make any crypto-beginner feel like their heads will explode with all this groundbreaking stuff. Also, some companies have habits to promote themselves through trends instead of substance. It is the same as with "organic/nano/bio/carbon-free" stickers, in fact advertising only words and phrases often used on products or services that have nothing to do with them.
The same goes for Web3 and DeFi when they are used like ultimate certificates verifying disruptive and decentralizing potential of online projects, even when there is none. Take Meta metaverse as an example. At the end of the day, a user is left with confusion: How to draw the line between Web3 and non-Web3 projects? Why do some have the Web3 label, while they are developed by the same tech giants that have been controlling much of the present Internet (Web2) for years and don't seem to change anything about their centralized governance model and data storage policy? So with all the hype going around, the distinctions between Web3, DeFi, and ordinary mortal online business sometimes can be blurred. We are here to fix it. Let's dive in. What does Web3 actually mean? Probably you might have met Web3's definition as a new-gen Internet that will give users more freedom of interaction. Well, it's pretty popular to define Web3 through some core human values. Web3 will make you free and independent, they say, like it's a new blueprint for the Liberator, but blockchain-powered. (Lol) Let's cut the fluff and see what Web3 really is. As long as there's no firm official definition for it yet, we can allow some freestyle with explanations. Let's look at Web3 from narrow and broad perspectives. Web3, in the narrow sense, refers to a new vision of the Internet: decentralized, blockchain-powered, and empowering p2p transactions. This cutting-edge Web aims to overcome the disadvantages of today's Internet, i.e., Web2, that creates a plethora of limitations and risks for users. One of the most negative aspects of Web2 is that it is, in fact, monopolized and controlled by the cartel of upper-echelon tech giants like Google, Facebook, Amazon, etc. The cartel has an unlimited capacity to dictate to users the rules of their interactions and what content they are allowed to publish, what fees they have to pay, what data to provide, etc. Web2 users have no other option but to look up at a tech giant intermediary and interact according to its "accept-or-die" consensus algorithm. That doesn't seem fair. At the same time, the Web2 cartel has complete discretion over users' data – they can sell it, disclose it to third parties, or use it to manipulate consumers' perception for advertising purposes. Yes, history already knows of such cases. Web 3.0 is tasked with the mission of getting done with this imbalance of power and breaking users’ dependency on tech giants. Instead, Web3 offers a new environment, where users will be enabled to interact peer-2-peer without the need to follow 3rd party’s rules and remain in total control over their data and identity. Here is an excellent , one of the pioneers and leaders of the Web 3.0 movement, explaining this vision of a new form of users’ interactions: quote by Gavin Wood “Web 3.0, or as might be termed the “post-Snowden” web, is a reimagination of the sorts of things that we already use the Web for, but with a fundamentally different model for the interactions between parties. Information that we assume to be public we publish. Information that we assume to be agreed, we place on a consensus-ledger. Information that we assume to be private, we keep secret and never reveal”. So let’s say that tomorrow, your familiar Web would look radically different: new browsers, UX, different ways of communication, no need to adopt third-party terms and conditions that are never in your favor, and no risk of being blocked by a platform’s algorithm. Only you and your peers would set the rules, drive interactions and get rewards. No helpers. No need for anyone’s permission and consent. The technology behind Web3 Here are the key technological fundamentals that would make Web3 a reality: Decentralized blockchain protocols Public, interconnected (interoperable) decentralized blockchain protocols are the foundation of Web 3. The main job of the protocols is to ensure trusted transactions between users, strong privacy in their interactions, as well as distributed and censorship-resistant data storage. Smart contracts They are the key facilitators of blockchain protocols’ work. Smart contracts are programs that automatically self-execute upon certain conditions being met. They are tamper-resistant, which means excluding the risk of any interference or modification of their terms in any way. In Web 3, among other things, they also drive such vital elements of its ecosystem, like DAOs and distributed data storage. DAOs A DAO, or “Decentralized Autonomous Organization,” is a community-led online entity with no central authority. Unlike traditional companies, DAOs eliminate the hierarchy of corporate management bodies and offer a decentralized governance model. Voting, decision making, distribution of dividends, execution of the agreed-upon decisions, and other business-related matters are regulated by the rules laid down in blockchain-based smart contracts and performed by them automatically. This is done to exclude any risk of human error or manipulation with votes. Neither a sleazy CEO nor pushy shareholders would have any chance to manipulate the entity’s work to their benefit. The point is that it’s safer to rely on algorithms rather than on humanity’s goodwill. We all know man is weak. Distributed data storage What does it mean? The vision is to ensure that users’ information is stored sliced to multiple independent network nodes instead of one server that is not under users’ control. This distributed approach will procure high-quality data protection, as well as enable users to manage their data, encrypt and keep it private, authorize access to data, and backup the storage. Crypto payment networks It would be nothing new to say that crypto pioneered the principles of decentralization and anonymity, which are now at the core of Web 3.0. Contrary to fiat payments, which immediately bring nation-state borders along with regulatory complexities and limitations, crypto has offered an alternative financial paradigm — peer-to-peer trustless and anonymous transactions with no intermediary involved. Web3, in the broad sense, means a better framework for sharing economy. The concept of sharing economy (or shared economy) existed before Web3.  In general words, it stands for an economic model defined as a (P2P) based activity of acquiring, providing, or sharing access to goods and services that is often facilitated by a community-based online platform. peer-to-peer This sounds pretty similar to what Web3 offers. Uber and Airbnb are the most prominent examples of Web2 sharing economy projects. But the problem here is that the way they operate is far from being in line with sharing economy philosophy. In fact, these projects act as intermediaries in what are supposedly peer-to-peer transactions leaving participants with no option but to follow the terms and conditions on offer. Also, the distribution of revenue generated on such platforms among participants quite often doesn’t work for their best interests. Clearly, the term “sharing economy” has been misappropriated in these examples and would be better replaced with a more accurate term such as "pseudo-sharing." Contrary to this, the Web3 sharing economy offers a real peer-to-peer environment wherein users are able to define their own rules of doing business and interact directly with each other without having to share their revenue with any third party. Today the most developed sector of the Web3 sharing economy is decentralized finance or "DeFi" for short. By February 2022, DeFi’s has reached $93 billion. total market cap DeFi as a sector of Web3 economy DeFi is a blockchain-based concept of an alternative financial system aimed to replace the traditional. The goal of DeFi is to provide an open, permissionless, and tamper-proof financial infrastructure, which would allow easier access to the same financial services like payments, lending, insurance, borrowing, trading, etc., as well as inventing new ones without limitations of multiple cross-border inconsistent regulations and bureaucracy. As a constituent component of the Web3 economy, DeFi sticks to the same : ideology trustless  p2p transactions in the absence of a middleman, powered by blockchain technology, and user’s full control over their funds and personal data. With the use of open-source code and great availability for developer tools in DeFi, participants can experiment with more financial instruments. They can also upgrade old ones and create new products without seeking authorization from regulatory watchdogs. When comparing DeFi to traditional financial systems, they differ primarily on the following points: In DeFi,  a public blockchain acts as the trust source, governing all operations in the financial sector. In contrast, in traditional finance, trust comes from regulatory and licensing activities of authorized state bodies (like security exchange commissions, central banks, etc.). Source of trust. . DeFi has no entry barriers. Its participants are only expected to understand how DeFi works and how to deal with it. No official certification is needed. Entry barriers .  In DeFi, anybody with programming skills can take part in building financial services and tools on top of public blockchains. Unlike traditional finance, in DeFi, newly created products do not require being approved or licensed by a regulator. Freedom of financial engineering One of the most important goals of DeFi is to enable seamless cross-border transactions at a low cost. To achieve this, DeFi projects apply a . This means that blockchain protocols (Layer 1) are paired up with Layer-2 scaling solutions called “state channels”, allowing users to transact with one another directly outside of the blockchain (i.e., off-chain) and minimize their on-chain operations only to the specific necessary sequences. Speed and cost of transactions. two-layer concept So wrapping it up, DeFi seeks to revolutionize the financial sector by leveraging a set of progressive technology,  offering extra functionality in addition to reducing operational risks. How do Web3 and DeFi impact each other? Web3 sets both the technological and economic frameworks for the DeFi sector. As the , Web3 enables technology in finance secure seamless and permissionless cross-border transactions at low cost and without bureaucratic handicaps transparency of user interactions and tamperproof records of transaction facts (like price, the total volume of debt, etc.) privacy and full control of financial data trustless governance model based on smart contracts, eliminating  any risk of 3rd party manipulations and interference, and absence of legal paperwork and requirements of regulatory/supervisory bodies. As the , Web3 economic framework democratizes access to finance so that everyone can explore it due to the absence of any qualification requirements, allows one to creatively engineer an unlimited variety of  income sources eliminates dependence on the banks globalizes finance and, as a result, changes the whole macroeconomic paradigm. Key Takeaway As Web3 evolves, so does the DeFi sector. Combo of them both is one of the most incredible examples demonstrating how cutting-edge technology can revolutionize the existing order of things and unlock a unique path towards opportunities that were never before imaginable. Want to learn more about Web3 and DeFi? | | | Yellow Network website OpenDAX v4 stack GitHub Twitter Telegram Stay tuned as Yellow unveils the developer tools behind Yellow Network, brokerage nodes stack, and community liquidity mining software!

Stories, News & Mentions about COLABORATOR

Colaborator’s Stories on HackerNoon