Next Glass

nextglass

As someone who loves cutting-edge technology, I choose to build my first SaaS with a modern tech stack. With the rise of and architecture, I created with Next JS static generation for the frontend and the Node.js backend deployed to AWS. JAMStack serverless PostMage Because I'm a solo full-stack developer, my time and resources are extremely limited. In this article, I'll share all the technologies I use to : from programming language to development tools. You'll find how I overcome this challenge to build a SaaS as a solo developer. build my SaaS product Hope my story gives you the inspiration to create your SaaS products. TypeScript Everywhere For building my SaaS, I wrote every line of code in . Yes, all the code: Frontend, Backend and also, Infrastructure as code in TypeScript. TypeScript The whole project only uses one and unique programming language. No time to learn new languages and save time by making the code easy to maintain. Why did I choose ? It makes the development much more pleasant with strongly-typed and has better integration to IDE. So, if you are still a JavaScript developer, you should give it a try. TypeScript Frontend framework For the frontend, I use . It's a to build a complex application. The good news, Next JS supports TypeScript out-of-the-box. Next.js React framework I use styling the React components. As a developer, you usually build an ugly interface. With Tailwind CSS, you can have now build a not so ugly interface even if you aren't a designer. Tailwind CSS As a true believer of JAMStack, I have previously taken some time to try Jekyll, Hexo and 11ty for different projects. I choose to build my SaaS in . So, at build time, all the pages are generated and pre-rendered. Perfect for SEO, cheap hosting, fast, secure, and highly scalable. static generated mode using Next JS Static hosting I use as a hosting service for the frontend, it's a brand new alternative to Netlify or Vercel. Cloudflare has announced it in December 2020 in beta and released it to the public in April 2021. Cloudflare Pages There is some small missing feature (nothing big) in Pages. Until the Cloudflare team solve it, I've found temporary workarounds. So, it isn't a big deal. The good thing about is its generous free tier: unlimited bandwidth (Vercel and Netlify are limited to 100GB per month) and you can set up a password-protected website for free (not included for free in Vercel or Netlify). Cloudflare Page Serverless REST API On the backend side, I've built a REST API with and . To support TypeScript in , I use a plugin. Express.js needs another plugin to work with Serverless Framework named . Express.js Serverless Framework Serverless Framework serverless-bundle serverless-http For better developer experience, I've also used two other plugins: and . The first plugin is to support dotenv files and the second one is to run Serverless Framework on your local computer. serverless-dotenv-plugin serverless-offline As a solo developer, I choose serverless architecture for making my life easier with . No need to become a DevOps engineer: no need to SSH, make OS updates, configure proxy/webserver/load balancer/firewall, etc. easy deployment, low maintenance, and scalable backend Authentication The REST API is protected by the . It's AWS built-in feature to secure any AWS resources, in our case, API gateway and AWS lambda. It denies the API invocation when the user isn't connected to the SaaS application. So, when it's protected, external actors won't be able to invoke your resource. IAM authentication Because the API is deployed to AWS, I choose to use for authentication. The good thing is that Cognito saves a lot of time by providing everything you need to implement authentication for your SaaS. You get access without any effort to (Facebook, Google, Apple and Amazon). AWS Cognito Email authentication and Social sign-in The connection between AWS Cognito and React frontend is done through . Amplify provides React components and code for making your frontend integration to AWS easier and faster. AWS Amplify NoSQL Database Major and well-known databases like PostgreSQL and MySQL don't fit very well in Serverless architecture. Due to the nature of serverless, it can create a lot of connections to the database and exhaust the database connection limit. On most providers, even if you don't have any traffic on your SaaS, you still need to pay your DB instance. On the opposite, when your application starts to grow, your database can quickly become the bottleneck. As a solo full-stack developer, I wanted something extremely easy to manage and 100% compatible with serverless. So, I choose as a primary database. DynamoDB is a NoSQL database fully managed by AWS and I use it to store user states. They almost handle everything and I just need to focus on my code. DynamoDB Infrastructure as code As you can see, I use several AWS services for my SaaS app. It's extremely painful to set up manually cloud resources in each environment (development, staging or production) and hard to maintain consistency between them. AWS gives developers access to where you can define your cloud resources in . In one command, you can deploy to your AWS account and get everything provisioned. AWS CDK TypeScript Deployment Like many developers, I use for the version control of my code. Many modern hosting services like Vercel and Netlify, automatically build and deploy your code at each commit. If you work with Git branches, you can also preview the results without pushing them to production. Git and GitHub Cloudflare pages For the backend and the infrastructure, I use a third-party service named to deploy automatically at each commit. Like the frontend, it also builds and deploys the backend resources on AWS. Seed.run DNS and CDN As you can doubt, I use for DNS and CDN without any surprise ;) Cloudflare Pages automatically deploy your code in the Cloudflare network, I only need to point my domain to Cloudflare DNS server and they handle the rest. Using Cloudflare, you get plenty of security features like a firewall and a DDoS protection for your SaaS products. Cloudflare Error tracking I use as the error tracking solution. It automatically reports when something goes wrong with useful information like stack trace, breadcrumbs (a trail of events that happened before an issue), browser information, OS information, etc. It makes debugging in production much easier with enriched data: Sentry is only set up for the frontend and not for the REST API, I keep using the native solution. Indeed, with AWS lambda creates a lot of overhead and the setup wasn't straightforward. In the next section, you'll find the solution I use for error tracking in the backend. Sentry Sentry Logging, monitoring and alert AWS Lambda automatically sends logs to , so no need to use Sentry. Here is an example of logs stored in : AWS CloudWatch CloudWatch You also get access to your lambda metrics. Perfect to understand how your serverless functions behave and detect if there are any errors. I also use to have additional information for my logging and monitoring. The interface is easier to use compared to Cloudwatch: Lumigo You can also enable tracing in where you can visualize your AWS service and external API calls. It makes your debugging session easier by letting you know if there is an error in your code or it's from an external service. Lumigo Payment and subscription The last piece of a SaaS and the most important thing for a business is to accept payment. Accepting a one-time payment is hard but, the task for recurrent payment is much complex. Unfortunately, for a SaaS business, we need to handle the second case. Your customers need to choose the plan and enter their personal information when they subscribe for the first time. After that, your users should have a self-service portal where they can manage their plan: upgrade, downgrade, cancel, pause, resume their subscription plan. They sometimes also need to update their personal information. And, they also need access to their invoice history when needed. can manage everything I mention in this section, it hides all these complexities and makes the integration to payment easier. Stripe Conclusion It took me 5 months of development to build this full-stack React SaaS template. Instead of focusing on my business, I was solving these technical details. Building the first version of your SaaS should only take 1 month and not 5. By going through this long journey, I've learned so many things and I've made tons of mistakes. I hope others developers won't do the same mistakes, so I build . Nextless JS, React Boilerplate for SaaS products With Nextless.js, you get everything I mentioned in this article without you writing any line of code. Save you time, focus on things that matter and launch your SaaS faster.

The Tech Stack of a Solo-Developer to build a SaaS With React and AWS

Sentiment Analysis using Deep Learning techniques with India Elections 2019 — A Case study

Technological evolution has changed the landscape, everything which we feel and hear today is revolving around some of the modern technology. This technology involves Artificial Intelligence, big data, cloud computing, data science, and much more, which has changed the landscape to a great extent. To integrate this technology, many of the IT professionals are finding and implementing the trajectory of today's modern technologies. Big Data is one such technology that has gained high momentum during the last few years. And when it comes to big data, then the very first term which strikes in an individual's mind is Hadoop. Not a single big data processing tool has bagged more success and popularity than Hadoop, this open-source tool was launched by Apache Software Foundation on April 1, 2006, and has gained huge popularity among the users in a short duration of time. This field is an ever-evolving field that is evolving continuously, and many features are added to its ecosystem. ( ) Source Explore to Know Which Basic Skills You Must Poses For Learning Hadoop When it comes to learning Hadoop, then one of the basic questions which strike in every individual mind is how they can start learning Hadoop? They do not exactly know which skills they must pose to become a professional with the data handling process. Before any of the individuals start learning Hadoop, they must collect all the basic knowledge related to it in detail. They must get the answer for various questions like, Why do they want to learn Hadoop? Whether they want to learn it because others are running on the track? Will it help them in the long run? There are many more questions which any of the individuals need to consider, they must also make sure that they evaluate some of the to find the value of this advanced tool. More than 91% of business entrepreneurs rely on customer data to make perfect decisions for their business. Moreover, they believe that these data can help them in boosting their business success and growth to a great extent. market statistics Marketing strategy is changing continually with the changing trend, there is a surge in all sectors in data generation, it has increased by 90% during the last few decades. Worldwide revenues for business analytics (BDA) and big data solutions are estimated to reach around $189.1 billion this year, this measures an increase of 12.0% from the previous year. A new update from International Data Corporation (IDC) represents that BDA revenues are going to maintain this growth from 2018 to 2022, the compound annual growth rate (CAGR) reach to 13.2%. By 2022, the IDC expects that the BDA revenue is going to reach around $274.3 billion. The gap between the ongoing demand is expected to grow high hence it becomes mandatory to develop the big data skills. Hence, there is an ongoing opportunity that big data has opened for Hadoop professionals, indeed. ( ) Source Big Data Introduction Big Data can be termed as a collection of huge data that has been collected by exploring various platforms. It describes a collection of data that is collected in huge amounts and is growing exponentially with the passage of time. This type of data collection is complex; hence any of the individuals need to make use of the tools which allow them to process and store the data accurately and more efficiently. Big data can be classified into mainly three categories, this category includes: Structured. Unstructured. Semi-structured. Structured Data If any of the individuals who want to access, process, and store the data can follow a fixed format, this can be termed as 'structured' data. Computer science has bagged huge success in developing fields over a period of time, hence it becomes much easier for the and developers to work with any kind of data, they can even derive value out of it. But in today's world, as the size of data is increasing with each passing day, it becomes quite tougher for any of the individuals to handle the data with this format. angularjs development Example of Structured Data: Unstructured Data Any data with an unknown structure can be termed as unstructured data. The size of data is comparatively very huge when it comes to unstructured data, it poses multiple challenges and helps in deriving value from the same. A heterogeneous data source is one of the best examples of unstructured data. It simply contains a mixture of simple text, videos, images, and much more. Example of Unstructured Data: Semi-Structured This data type contains both types of data structured and unstructured data. Semi-structured data is organized well in the format; hence it becomes easy for individuals to work with the data more accurately and efficiently than ever before. The XML file is the best example of semi-structured data. Examples of Semi-structured Data: As the data is increasing with each passing day and estimated to reach around 78 yottabytes, thus it becomes vital for each and every individual to learn and understand how they can handle the data more accurately and efficiently in a systematic manner. Many tools and techniques are developed, which can help them to work with structured and unstructured data more accurately than ever before. What Actually Hadoop is? It is an open-source framework that anyone can use as per their convenience. Any of the individuals can make the needed change in the source code as per their requirement. Apache Hadoop poses the capabilities to process and store huge amounts of data more efficiently. It is used for data analyzing, governance, storing, accessing, processing, and also for performing various actions on the data. Most of the businesses that have to deal with a huge amount of data on a regular basis make use of Hadoop. It is used to process the large cluster of commodity hardware, which is referred to as a group of a system which is possessing connections with multiple nodes. Hadoop provides numerous features to the users, let's explore to know which are those features that we can leverage when considering using this framework. Features of Hadoop. Scalability. Cost-Effective. Distribution of Data. Parallel Processing of Data. Supports Large Clusters of Nodes. Automatic Failover Management. Supports Heterogeneous Clusters. The major advantage of Hadoop is scalability, which it offers to the users. It can be used to scale up to more than 200 PB and 4500 server clusters of data. Distributive ways are used to handle stored data, this helps in avoiding the overloading of data in the process. Hadoop Ecosystem: Explore to Know What it Consists of The Hadoop ecosystem consists of numerous things which help the users to perform all the tasks more accurately and efficiently than ever before. Let's explore the ecosystem of the Hadoop system to know how it benefits the users in the most accurate way. Hadoop Distributed File System (HDFS) HDFS performs the most important role when it comes to using the Hadoop framework. When it comes to storing and distributing data, then it uses nodes that are present in the cluster. The whole process results in reducing the time which is required for storing the data into the disk. Explore the Architecture of HDFS to know how it works to provide systematic output. MapReduce Hadoop MapReduce processes the huge amount of data that is stored in a cluster by HDFS. It allows parallel processing on the stored data and resolves the issues via the massive scalability in a cluster.  It provides the high cost of processing standards to the users who choose using it. After collecting the task, the cluster reduces the data to provide users with appropriate results and sends the output back to the Hadoop server. Code for MapReduce Framework hadoop; java.util.*; java.io.IOException; java.io.IOException; org.apache.hadoop.fs.Path; org.apache.hadoop.conf.*; org.apache.hadoop.io.*; org.apache.hadoop.mapred.*; org.apache.hadoop.util.*; { { {
     	String line = value.toString();
     	String last token = ;
     	StringTokenizer s = StringTokenizer(line, );
     	String year = s.nextToken(); (s.hasMoreTokens()) {
        	lasttoken = s.nextToken();
     	} avgprice = Integer.parseInt(last token);
     	output.collect( Text(year), IntWritable(avgprice));
  	}
   } { { max avg = ; val = Integer.MIN_VALUE; (values.hasNext()) { ((val = values.next().get())>max avg) {
           	output.collect(key, IntWritable(val));
        	}
     	}
  	}
   } {
  	JobConf conf = JobConf(ProcessUnits.class);
 	 
  	conf.setJobName( );
  	conf.setOutputKeyClass(Text.class);
  	conf.setOutputValueClass(IntWritable.class);
  	conf.setMapperClass(E_EMapper.class);
  	conf.setCombinerClass(E_EReduce.class);
  	conf.setReducerClass(E_EReduce.class);
  	conf.setInputFormat(TextInputFormat.class);
  	conf.setOutputFormat(TextOutputFormat.class);
 	 
  	FileInputFormat.setInputPaths(conf, Path(args[ ]));
  	FileOutputFormat.setOutputPath(conf, Path(args[ ]));
 	 
  	JobClient.runJob(conf);
   }
} package import import import import import import import import public class ProcessUnits //Mapper class public static < ,/* */ ,            	/* */ ,            	/* */ >    	/* */ class E_EMapper extends MapReduceBase implements Mapper LongWritable Input key Type Text Input value Type Text Output key Type IntWritable Output value Type //Map function IOException public void map (LongWritable key, Text value,
  	OutputCollector<Text, IntWritable> output,   
 	 
  	Reporter reporter) throws null new "\t" while int new new //Reducer class public static < , , , > class E_EReduce extends MapReduceBase implements Reducer Text IntWritable Text IntWritable //Reduce function IOException public void reduce ( Text key, Iterator <IntWritable> values,
  	OutputCollector<Text, IntWritable> output, Reporter reporter) throws int 30 int while if new //Main function Exception public static void main (String args[ ]) 2 throws new "max_eletricityunits" new 0 new 1 Apache Pig This high-level scripting language is used to write the data analysis programs for a large number of data sets in the Hadoop cluster. Apache Pig allows programmers to enable developers to generate query execution, it allows them to analyze a great amount of data on a regular basis. Pig Latin is the scripting language, which is considered as one of the vital parts of the Pig. Apache HBase Apache HBase supports the working of HDFS and is considered to be a column-oriented database that processes on a real-time basis. It performs an effective process on a huge amount of a database, this includes a file containing a large number of columns and rows. Users who aim to manage region servers more accurately can consider using Apache HBase, it helps them to use all the master notes more efficiently than ever before. Apache Hive Combo of Apache Hive and SQL-like interface allows users to square the data from HDFS. The new version of the SQL language of Hive is popularly known as HiveQL. Users can install a Hive from a Stable Release, they can start from the initial phase by downloading recent stable releases of Hive from Apache download mirrors. Once after completing the installation process, make sure that you unpack the tarball and create the subdirectory named hive-x.y.z: $ tar -xzvf hive-x.y.z.tar.gz After installing the Apache Hive users, let's Set the environment variable to point to the installation directory, consider the simple code to install the directory. $ cd hive-x.y.z $ export HIVE_HOME={{pwd}} Now add $HIVE_HOME to any of the PATH which you desire using the following code: $ export PATH=$HIVE_HOME/bin:$PATH Compile Hive on a Master Once you complete the development task, it's time to compile the same for the master branch, follow the code below to do the task accurately. Code to Compile Apache Hive $ git clone https://git-wip-us.apache.org/repos/asf/hive.git $ cd hive $ mvn clean package -Pdist [-DskipTests -Dmaven.javadoc.skip=true] $ cd packaging/target/apache-hive-{version}-SNAPSHOT-bin/apache-hive-{version}-SNAPSHOT-bin $ ls LICENSE NOTICE README.txt RELEASE_NOTES.txt bin/ (all the shell scripts) lib/ (required jar files) conf/ (configuration files) examples/ (sample input and query files) hcatalog / (hcatalog installation) scripts / (upgrade scripts for hive-metastore) Here the version is referred to as the current Hive version. If any of the users is crafting Hive source with the help of Maven (mvn), then it can be referred to the directory <install-dir> for the page. This can help an individual to know whether they have installed the Apache Hive perfectly or not. Apache Sqoop Apache Sqoop enables users to import and export, users can transfer any kind and size of data from Hadoop to any other relational database management systems. It also allows them to transfer a large amount of data more quickly than ever before. Apache Sqoop is a connector based architecture that establishes connectivity with all external systems. Apache Zookeeper The main role of Apache Zookeeper is to manage the coordination between all the applications or software which are listed above. It looks after the functioning of all the rest software in the Hadoop ecosystem and makes their work much easier than ever before. Concluding Lines Learning any technology is the initial journey that leads an individual on the path of success. It opens a huge scope and opportunities for the learners. Hence, motivate and persist yourself to indulge in the world of today's challenging technology. Having a positive learning attitude can help professionals and developers to stay up to date with modern technologies and present market trends. Start your learning journey with Hadoop and leverage various benefits that this advanced framework offers to you.

Hadoop for Hoops: Explore the Whole Ecosystem and to Know How It Really Works

Text classification is task of categorising text according to its content. It is the fundamental problem in the field of Natural Language Processing(NLP). More general applications of text classifications are in email spam detection, sentiment analysis and topic labelling etc. For this post we restrict our text classification to only two classes also know as binary classification e.g into positive or negative text. For integrating text classification in iOS we will be using a pre-trained model on IMDB database. We will be using as out underlaying machine learning inference engine. tensorflowlite In its simplest form our app will take a sentence as input from user, feeds into classification client, that will classify text as either positive or negative depending on text content. Lets name our classification client as . Here is an overview of the client class TextClassificationClient { ?(modelFileInfo: , labelsFileInfo: , vocabFileInfo: ) { }
} final class TextClassificationClient // other stuff init FileInfo FileInfo FileInfo // we will initialise the tensorflowlite here The initialiser takes three parameters of type FileInfo which is a tuple defined below modelFileInfo = (name: , let FileInfo "text_classification" : " ") = ( : " ", : " ") = ( : " ", : " ") extension tflite let labelsFileInfo FileInfo name labels extension txt let vocabFileInfo FileInfo name vocab extension txt Model file is the trained model file which tensorflowlite takes to perform inference. Labels file is plain text file consisting of classes of classification e.g Positive and Negative. Vocab is again a plain text file containing the collection of some words and their corresponding embedding, these words were used during model training and also will be used during inference. Model training is outside of scope of this post and as I mentioned in start we will use a pre-trained IMDB model from . For detail overview about model training please see . here here In order to use tensorflowlite in iOS app we have to integrate it. For this purpose we will be using cocoapods. is the most commonly used dependency management tool used for managing third party dependencies in iOS apps. Cocopods Add pod ‘ ’ in your project pod file and run pod install to install and integrate tensorflowlite in iOS app. TensorFlowLiteSwift Let's get back to our class. In this class we have a method classify that will actually classify the text. Below is the implementation of this method TextClassificationClient -> [ ] { input = tokenizeInputText(text: text) data = (copyingBufferOf: input[ ]) { interpreter.copy(data, toInputAt: ) interpreter.invoke() outputTensor = interpreter.output(at: ) outputTensor.dataType == .float32 { outputArray = [ ](unsafeData: outputTensor.data) ?? [] output = [ ]() (index, label) labels.enumerated() {
        output.append( (id: , title: label, confidence: outputArray[index]))
      }
      output. (by: >) output
    }
  } { (error)
  } []
} func classify (text: String) Result let let Data 0 do try 0 try let try 0 if let Float var Result for in Result "" sort return catch print return It takes a string as parameter and return an array of Result type. The first line takes string and converts it into tokens in order to feed it into inference engine(interpreter instance variable in this case). We run interpreter on tokenised text get the output from interpreter and return it. The Result type is defined as below { id: title: confidence: } struct Result let String let String let Float Below is the screenshot of an inference The complete working code of app can be found at Github repository . here

Text Classification in iOS using tensorflowlite [A How-To Guide]

Advanced NLP techniques for deep learning With the problem of Image Classification is more or less solved by Deep learning, For those who don’t know, Text classification is a common task in natural language processing, which transforms a sequence of a text of indefinite length into a category of text. How could you use that? . Text Classification is the next new developing theme in deep learning To find the sentiment of a review. Find toxic comments on a platform like Facebook Find Insincere questions on Quora. A current ongoing competition on Kaggle Find fake reviews on websites Will a text advert get clicked or not And much more. The whole internet is filled with text and to categorize that information algorithmically will only give us incremental benefits, to say the least in the field of AI. Here I am going to use the data from Quora’s Insincere questions to talk about the different models that people are building and sharing to perform this task. Obviously, these standalone models are not going to put you on the top of the leaderboard, yet I hope that this ensuing discussion would be helpful for people who want to learn more about text classification. This is going to be a long post in that regard. As a side note: if you want to know more about NLP, I would like to recommend this awesome course on in the . This course covers a wide range of tasks in Natural Language Processing from basic to advanced: sentiment analysis, summarization, dialogue state tracking, to name a few. Natural Language Processing Advanced machine learning specialization So let me try to go through some of the models which people are using to perform text classification and try to provide a brief intuition for them. 1. TextCNN: The idea of using a CNN to classify text was first presented in the paper by Yoon Kim. Instead of image pixels, the input to the task is sentences or documents represented as a matrix. Each row of the matrix corresponds to one word vector. That is, each row is word-vector that represents a word. Thus a sequence of max length 70 gives us an image of 70(max sequence length)x300(embedding size) Convolutional Neural Networks for Sentence Classification Now for some intuition. While for an image we move our conv filter horizontally also since here we have fixed our kernel size to filter_size x embed_size i.e. (3,300) we are just going to move down for the convolution taking look at three words at once since our filter size is 3 in this case. Also one can think of filter sizes as unigrams, bigrams, trigrams etc. Since we are looking at a context window of 1,2,3, and 5 words respectively. Here is the text classification network coded in Keras: # def model_cnn(embedding_matrix):filter_sizes = [1,2,3,5]num_filters = 36 https://www.kaggle.com/yekenot/2dcnn-textclassifier inp = Input(shape=(maxlen,))x = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)x = Reshape((maxlen, embed_size, 1))(x) maxpool_pool = []for i in range(len(filter_sizes)):conv = Conv2D(num_filters, kernel_size=(filter_sizes[i], embed_size),kernel_initializer='he_normal', activation='elu')(x)maxpool_pool.append(MaxPool2D(pool_size=(maxlen - filter_sizes[i] + 1, 1))(conv)) z = Concatenate(axis=1)(maxpool_pool)z = Flatten()(z)z = Dropout(0.1)(z) outp = Dense(1, activation="sigmoid")(z) model = Model(inputs=inp, outputs=outp)model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) return model I have written a simplified and well-commented code to run this network(taking input from a lot of other kernels) on a for this competition. Do take a look there to learn the preprocessing steps and the word to vec embeddings usage in this model. You will learn something. Please do upvote the kernel if you find it helpful. This kernel scored around 0.661 on the public leaderboard. kaggle kernel 2. BiDirectional RNN(LSTM/GRU): TextCNN takes care of a lot of things. For example, it takes care of words in close range. It is able to see “new york” together. But it still can’t take care of all the context provided in a particular text sequence. It still does not learn the seem to learn the sequential structure of the data, where every word is dependent on the previous word. Or a word in the previous sentence. RNN help us with that. They are able to remember previous information using hidden states and connect it to the current task. Long Short Term Memory networks (LSTM) are a subclass of RNN, specialized in remembering information for a long period of time. Moreover, the Bidirectional LSTM keeps the contextual information in both directions which is pretty useful in text classification task (But won’t work for a time series prediction task). For a most simplistic explanation of Bidirectional RNN, think of RNN cell as taking as input a hidden state(a vector) and the word vector and giving out an output vector and the next hidden state. Hidden state, Word vector ->(RNN Cell) -> Output Vector , Next Hidden state For a sequence of length 4 like ‘you will never believe’, The RNN cell will give 4 output vectors. Which can be concatenated and then used as part of a dense feedforward architecture. In the Bidirectional RNN, the only change is that we read the text in the normal fashion as well in reverse. So we stack two RNNs in parallel and hence we get 8 output vectors to append. Once we get the output vectors we send them through a series of dense layers and finally a softmax layer to build a text classifier. Due to the limitations of RNNs like not remembering long term dependencies, in practice, we almost always use LSTM/GRU to model long term dependencies. In such a case you can just think of the RNN cell being replaced by an LSTM cell or a GRU cell in the above figure. An example model is provided below. You can use CuDNNGRU interchangeably with CuDNNLSTM, when you build models. # BiDirectional LSTMdef model_lstm_du(embedding_matrix):inp = Input(shape=(maxlen,))x = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)'''Here 64 is the size(dim) of the hidden state vector as well as the output vector. Keeping return_sequence we want the output for the entire sequence. So what is the dimension of output for this layer?64*70(maxlen)*2(bidirection concat)CuDNNLSTM is fast implementation of LSTM layer in Keras which only runs on GPU'''x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)avg_pool = GlobalAveragePooling1D()(x)max_pool = GlobalMaxPooling1D()(x)conc = concatenate([avg_pool, max_pool])conc = Dense(64, activation="relu")(conc)conc = Dropout(0.1)(conc)outp = Dense(1, activation="sigmoid")(conc)model = Model(inputs=inp, outputs=outp)model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])return model I have written a simplified and well-commented code to run this network(taking input from a lot of other kernels) on a for this competition. Do take a look there to learn the preprocessing steps and the word to vec embeddings usage in this model. You will learn something. Please do upvote the kernel if you find it helpful. This kernel scored around 0.671 on the public leaderboard. kaggle kernel 3. Attention Models The concept of Attention is relatively new as it comes from paper written jointly by CMU and Microsoft guys in 2016. Hierarchical Attention Networks for Document Classification So in the past, we used to find features from the text by doing a keyword extraction. Some words are more helpful in determining the category of a text than others. But in this method we sort of lost the sequential structure of the text. With LSTM and deep learning methods while we are able to take care of the sequence structure we lose the ability to give higher weight to more important words. Can we have the best of both worlds? And that is attention for you. In the author’s words: Not all words contribute equally to the representation of the sentence meaning. Hence, we introduce attention mechanism to extract such words that are important to the meaning of the sentence and aggregate the representation of those informative words to form a sentence vector In essense we want to create scores for every word in the text, which is the attention similarity score for a word. To do this we start with a weight matrix(W), a bias vector(b) and a context vector u. All of them will be learned by the optimization algorithm. Then there are a series of mathematical operations. See the figure for more clarification. We can think of u1 as non-linearity on RNN word output. After that v1 is a dot product of u1 with a context vector u raised to an exponentiation. From an intuition viewpoint, the value of v1 will be high if u and u1 are similar. Since we want the sum of scores to be 1, we divide v by the sum of v’s to get the Final Scores,s These final scores are then multiplied by RNN output for words to weight them according to their importance. After which the outputs are summed and sent through dense layers and softmax for the task of text classification. def dot_product(x, kernel):"""Wrapper for dot product operation, in order to be compatible with bothTheano and TensorflowArgs:x (): inputkernel (): weightsReturns:"""if K.backend() == 'tensorflow':return K.squeeze(K.dot(x, K.expand_dims(kernel)), axis=-1)else:return K.dot(x, kernel) class AttentionWithContext(Layer):"""Attention operation, with a context/query vector, for temporal data.Supports Masking.Follows the work of Yang et al. [ ]"Hierarchical Attention Networks for Document Classification"by using a context vector to assist the attention# Input shape3D tensor with shape: `(samples, steps, features)`.# Output shape2D tensor with shape: `(samples, features)`.How to use:Just put it on top of an RNN Layer (GRU/LSTM/SimpleRNN) with return_sequences=True.The dimensions are inferred based on the output shape of the RNN.Note: The layer has been tested with Keras 2.0.6Example:model.add(LSTM(64, return_sequences=True))model.add(AttentionWithContext())# next add a Dense layer (for classification/regression) or whatever...""" https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf def __init__(self,W_regularizer=None, u_regularizer=None, b_regularizer=None,W_constraint=None, u_constraint=None, b_constraint=None,bias=True, **kwargs): self.supports_masking = Trueself.init = initializers.get('glorot_uniform') self.W_regularizer = regularizers.get(W_regularizer)self.u_regularizer = regularizers.get(u_regularizer)self.b_regularizer = regularizers.get(b_regularizer) self.W_constraint = constraints.get(W_constraint)self.u_constraint = constraints.get(u_constraint)self.b_constraint = constraints.get(b_constraint) self.bias = biassuper(AttentionWithContext, self).__init__(**kwargs) def build(self, input_shape):assert len(input_shape) == 3 self.W = self.add_weight((input_shape[-1], input_shape[-1],),initializer=self.init,name='{}_W'.format(self.name),regularizer=self.W_regularizer,constraint=self.W_constraint)if self.bias:self.b = self.add_weight((input_shape[-1],),initializer='zero',name='{}_b'.format(self.name),regularizer=self.b_regularizer,constraint=self.b_constraint) self.u = self.add_weight((input_shape[-1],),initializer=self.init,name='{}_u'.format(self.name),regularizer=self.u_regularizer,constraint=self.u_constraint) super(AttentionWithContext, self).build(input_shape) def compute_mask(self, input, input_mask=None):# do not pass the mask to the next layersreturn None def call(self, x, mask=None):uit = dot_product(x, self.W) if self.bias:uit += self.b uit = K.tanh(uit)ait = dot_product(uit, self.u) a = K.exp(ait) # apply mask after the exp. will be re-normalized nextif mask is not None:# Cast the mask to floatX to avoid float64 upcasting in theanoa *= K.cast(mask, K.floatx()) # in some cases especially in the early stages of training the sum may be almost zero# and this results in NaN's. A workaround is to add a very small positive number ε to the sum.# a /= K.cast(K.sum(a, axis=1, keepdims=True), K.floatx())a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx()) a = K.expand_dims(a)weighted_input = x * areturn K.sum(weighted_input, axis=1) def compute_output_shape(self, input_shape):return input_shape[0], input_shape[-1] def model_lstm_atten(embedding_matrix):inp = Input(shape=(maxlen,))x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)x = AttentionWithContext()(x)x = Dense(64, activation="relu")(x)x = Dense(1, activation="sigmoid")(x)model = Model(inputs=inp, outputs=x)model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])return model I have written a simplified and well-commented code to run this network(taking input from a lot of other kernels) on a for this competition. Do take a look there to learn the preprocessing steps and the word to vec embeddings usage in this model. You will learn something. Please do upvote the kernel if you find it helpful. This kernel scored around 0.682 on the public leaderboard. kaggle kernel Hope that Helps! Do check out the kernels for all the networks and see the comments too. I will try to write a part 2 of this post where I would like to talk about capsule networks and more techniques as they get used in this competition. Here are the kernel links again: , , TextCNN BiLSTM/GRU Attention Do upvote the kernels if you find them helpful. References: CNN for NLP https://en.diveintodeeplearning.org/d2l-en.pdf https://gist.github.com/cbaziotis/7ef97ccf71cbc14366835198c09809d2 http://univagora.ro/jour/index.php/ijccc/article/view/3142 Shujian’s kernel on Kaggle Originally published at mlwhiz.com on December 17, 2018.

What Kagglers are using for Text Classification

This all started when I was asked to speak at an forum in July. It represented a great opportunity for me to talk about to a new audience. But there was a problem. I don’t know anything about financial technology! AI FinTech Machine Box My first thought was, “Google machine learning use cases in fintech”. So I did. The results were mostly about anomaly detection and fraud prevention. Great use cases for machine learning, but it is a bit of a solved problem. Given that this was a forum on AI in financial technology, I figured there would already be lots of talks from experts in anomaly detection. One of Machine Box’s using only a single photo of a face to train upon. We already have a couple of customers using to verify people, so I figured I’d draw a link between that and securing in-person credit card transactions. key use cases is face detection Facebox https://www.webroi.ca/blog/tag/boring-industry/ BOOORRINNNNGG! Yes, face recognition can be used to secure transactions, and I did end up talking about how you could accomplish this using and other methods, but I didn’t think it really spoke to the point of simplifying machine learning with better tools, which is what we’re all about. Facebox , then you know that I frequently use as a prime example of how not to use machine learning. The stock market is a highly complex, multi-dimensional monstrosity of complexity and interdependencies. Not a good use case to try machine learning on. If you follow my posts predicting the stock market But… what if you predict the stock market with machine learning? could The first step in tackling something like this is to simplify the problem as much as possible. I decided to make it a two-class problem; given some input, the market either goes up or down. And I limited to the Dow Jones Industrial Average. the market What would make a good input? I decided, somewhat arbitrarily, upon news headlines as the input. I would train a on as many news headlines for a given day using natural language processing, sorting them into one of two classes; the market went up after the headlines, or, the market stayed the same or went down. classification model Now comes the most difficult part; gathering the data. VERY fortunately, a quick Google search revealed excellent dataset. It is a giant table of news headlines, labeled with their the Dow Jones’ performance that day. this So, 5 minutes into the process, I had a glorious dataset and a plan. Next came the execution. Because my developer skills are extremely limited, I decided to make life easy on myself and use , which iterates through folders looking for labeled data, and then trains automatically. But in order to run it on files and folders, I had to first convert the dataset into many tiny text files containing the text of the headlines, and put them into folders labeled either 0 or 1, indicating upward or downward movement in the Dow Jones. This is the script I used to perform that probably unnecessary task for real developers. this tool Classificationbox Modifying this script for the dataset took about 10 minutes. Running it took less than 30 seconds. The next step is to run on the folder with all the data in it. This will train with a random selection of 80% of the data. It will then verify the model with the remaining 20%. Textclass Classificationbox The result; 54% accuracy. Normally, an accuracy that low means your model isn’t useful. You need something like 80% to get to a place where the model starts to make sense for use in the real word. But when I told a room full of financial people that the model had a 54% accuracy, I expected a chuckle, instead, I got very straight faces. A few seconds later, someone said, somewhat under their breath, “You could sell 4%”. only Perhaps. But my conclusion was that news headlines can’t predict the Dow Jones, at least, with the dataset I had. I highly recommend you give it a try and see what results you get. Maybe I made an error in my script!

I spent 20 minutes trying to predict the stock market with AI — these are my results

Machine Learning Practitioners have different personalities. While some of them are “I am an expert in X and X can train on any type of data”, where X = some algorithm, some others are “Right tool for the right job people”. A lot of them also subscribe to “Jack of all trades. Master of one” strategy, where they have one area of deep expertise and know slightly about different fields of Machine Learning. That said, no one can deny the fact that as practicing Data Scientists, we will have to know basics of some common machine learning algorithms, which would help us engage with a new-domain problem we come across. This is a whirlwind tour of common machine learning algorithms and quick resources about them which can help you get started on them. 1. Principal Component Analysis(PCA)/SVD PCA is an unsupervised method to understand global properties of a dataset consisting of vectors. Covariance Matrix of data points is analyzed here to understand what dimensions(mostly)/ data points (sometimes) are more important (ie have high variance amongst themselves, but low covariance with others). One way to think of top PCs of a matrix is to think of its eigenvectors with highest eigenvalues. SVD is essentially a way to calculate ordered components too, but you don’t need to get the covariance matrix of points to get it. This Algorithm helps one fight curse of dimensionality by getting datapoints with reduced dimensions. Libraries: https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.svd.html http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html Introductory Tutorial: https://arxiv.org/pdf/1404.1100.pdf 2a. Least Squares and Polynomial Fitting Remember your Numerical Analysis code in college, where you used to fit lines and curves to points to get an equation. You can use them to fit curves in Machine Learning for very small datasets with low dimensions. (For large data or datasets with many dimensions, you might just end up terribly overfitting, so don’t bother). OLS has a closed form solution, so you don’t need to use complex optimization techniques. As is obvious, use this algorithm to fit simple curves / regression Libraries: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.polyfit.html Introductory Tutorial: https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/linear_regression.pdf 2b. Constrained Linear Regression Least Squares can get confused with outliers, spurious fields and noise in data. We thus need constraints to decrease the variance of the line we fit on a dataset. The right method to do it is to fit a linear regression model which will ensure that the weights do not misbehave. Models can have L1 norm (LASSO) or L2 (Ridge Regression) or both (elastic regression). Mean Squared Loss is optimized. Use these algorithms to fit regression lines with constraints, avoiding overfitting and masking noise dimensions from model. Libraries: http://scikit-learn.org/stable/modules/linear_model.html Introductory Tutorial(s): https://www.youtube.com/watch?v=5asL5Eq2x0A https://www.youtube.com/watch?v=jbwSCwoT51M 3. K means Clustering Everyone’s favorite unsupervised clustering algorithm. Given a set of data points in form of vectors, we can make clusters of points based on distances between them. It’s an Expectation Maximization algorithm that iteratively moves the centers of clusters and then clubs points with each cluster centers. The input the algorithm has taken is the number of clusters which are to be generated and the number of iterations in which it will try to converge clusters. As is obvious from the name, you can use this algorithm to create K clusters in dataset Library: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html Introductory Tutorial(s): https://www.youtube.com/watch?v=hDmNF9JG3lo https://www.datascience.com/blog/k-means-clustering 4. Logistic Regression Logistic Regression is constrained Linear Regression with a nonlinearity (sigmoid function is used mostly or you can use tanh too) application after weights are applied, hence restricting the outputs close to +/- classes (which is 1 and 0 in case of sigmoid). Cross-Entropy Loss functions are optimized using Gradient Descent. A note to beginners: Logistic Regression is used for classification, not regression. You can also think of Logistic regression as a one layered Neural Network. Logistic Regression is trained using optimization methods like Gradient Descent or L-BFGS. NLP people will often use it with the name of Maximum Entropy Classifier. This is what a Sigmoid looks like: Use LR to train simple, but very robust classifiers. Library: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html Introductory Tutorial(s): https://www.youtube.com/watch?v=-la3q9d7AKQ 5. SVM (Support Vector Machines) SVMs are linear models like Linear/ Logistic Regression, the difference is that they have different margin-based loss function (The derivation of Support Vectors is one of the most beautiful mathematical results I have seen along with eigenvalue calculation). You can optimize the loss function using optimization methods like L-BFGS or even SGD. Another innovation in SVMs is the usage of kernels on data to feature engineer. If you have good domain insight, you can replace the good-old RBF kernel with smarter ones and profit. One unique thing that SVMs can do is learn one class classifiers. SVMs can used to Train a classifier (even regressors) Library: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html Introductory Tutorial(s): https://www.youtube.com/watch?v=eHsErlPJWUU SGD based training of both Logistic Regression and SVMs are found in SKLearn’s , which I often use as it lets me check both LR and SVM with a common interface. You can also train it on >RAM sized datasets using mini batches. Note: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html 6. Feedforward Neural Networks These are basically multilayered Logistic Regression classifiers. Many layers of weights separated by non-linearities (sigmoid, tanh, relu + softmax and the cool new selu). Another popular name for them is Multi-Layered Perceptrons. FFNNs can be used for classification and unsupervised feature learning as autoencoders. Multi-Layered perceptron FFNN as an autoencoder FFNNs can be used to train a classifier or extract features as autoencoders Libraries: http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html https://github.com/keras-team/keras/blob/master/examples/reuters_mlp_relu_vs_selu.py Introductory Tutorial(s): http://www.deeplearningbook.org/contents/mlp.html http://www.deeplearningbook.org/contents/autoencoders.html http://www.deeplearningbook.org/contents/representation.html 7. Convolutional Neural Networks (Convnets) Almost any state of the art Vision based Machine Learning result in the world today has been achieved using Convolutional Neural Networks. They can be used for Image classification, Object Detection or even segmentation of images. Invented by Yann Lecun in late 80s-early 90s, Convnets feature convolutional layers which act as hierarchical feature extractors. You can use them in text too (and even graphs). Use convnets for state of the art image and text classification, object detection, image segmentation. Libraries: https://developer.nvidia.com/digits https://github.com/kuangliu/torchcv https://github.com/chainer/chainercv https://keras.io/applications/ Introductory Tutorial(s): http://cs231n.github.io/ https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/ 8. Recurrent Neural Networks (RNNs): RNNs model sequences by applying the same set of weights recursively on the aggregator state at a time t and input at a time t (Given a sequence has inputs at times 0..t..T, and have a hidden state at each time t which is output from t-1 step of RNN). Pure RNNs are rarely used now but its counterparts like LSTMs and GRUs are state of the art in most sequence modeling tasks. RNN (If here is a densely connected unit and a nonlinearity, nowadays f is generally LSTMs or GRUs ). LSTM unit which is used instead of a plain dense layer in a pure RNN. Use RNNs for any sequence modelling task specially text classification, machine translation, language modelling Library: (Many cool NLP research papers from Google are here) https://github.com/tensorflow/models https://github.com/wabyking/TextClassificationBenchmark http://opennmt.net/ Introductory Tutorial(s): http://cs224d.stanford.edu/ http://www.wildml.com/category/neural-networks/recurrent-neural-networks/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 9. Conditional Random Fields (CRFs) CRFs are probably the most frequently used models from the family of Probabilitic Graphical Models (PGMs). They are used for sequence modeling like RNNs and can be used in combination with RNNs too. Before Neural Machine Translation systems came in CRFs were the state of the art and in many sequence tagging tasks with small datasets, they will still learn better than RNNs which require a larger amount of data to generalize. They can also be used in other structured prediction tasks like Image Segmentation etc. CRF models each element of the sequence (say a sentence) such that neighbors affect a label of a component in a sequence instead of all labels being independent of each other. Use CRFs to tag sequences (in Text, Image, Time Series, DNA etc.) Library: https://sklearn-crfsuite.readthedocs.io/en/latest/ Introductory Tutorial(s): http://blog.echen.me/2012/01/03/introduction-to-conditional-random-fields/ 7 part lecture series by Hugo Larochelle on Youtube: https://www.youtube.com/watch?v=GF3iSJkgPbA 10. Decision Trees Let’s say I am given an Excel sheet with data about various fruits and I have to tell which look like Apples. What I will do is ask a question “Which fruits are red and round ?” and divide all fruits which answer yes and no to the question. Now, All Red and Round fruits might not be apples and all apples won’t be red and round. So I will ask a question “Which fruits have red or yellow color hints on them? ” on red and round fruits and will ask “Which fruits are green and round ?” on not red and round fruits. Based on these questions I can tell with considerable accuracy which are apples. This cascade of questions is what a decision tree is. However, this is a decision tree based on my intuition. Intuition cannot work on high dimensional and complex data. We have to come up with the cascade of questions automatically by looking at tagged data. That is what Machine Learning based decision trees do. Earlier versions like CART trees were once used for simple data, but with bigger and larger dataset, the bias-variance tradeoff needs to solved with better algorithms. The two common decision trees algorithms used nowadays are Random Forests (which build different classifiers on a random subset of attributes and combine them for output) and Boosting Trees (which train a cascade of trees one on top of others, correcting the mistakes of ones below them). Decision Trees can be used to classify datapoints (and even regression) Libraries http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html http://xgboost.readthedocs.io/en/latest/ https://catboost.yandex/ Introductory Tutorial: http://xgboost.readthedocs.io/en/latest/model.html https://arxiv.org/abs/1511.05741 https://arxiv.org/abs/1407.7502 http://education.parrotprediction.teachable.com/p/practical-xgboost-in-python TD Algorithms (Good To Have) If you are still wondering how can any of the above methods solve tasks like defeating Go world champion like DeepMind did, they cannot. All the 10 type of algorithms we talked about before this was Pattern Recognition, not strategy learners. To learn strategy to solve a multi-step problem like winning a game of chess or playing Atari console, we need to let an agent-free in the world and learn from the rewards/penalties it faces. This type of Machine Learning is called Reinforcement Learning. A lot (not all) of recent successes in the field is a result of combining perception abilities of a convent or a LSTM to a set of algorithms called Temporal Difference Learning. These include Q-Learning, SARSA and some other variants. These algorithms are a smart play on Bellman’s equations to get a loss function that can be trained with rewards an agent gets from the environment. These algorithms are used to automatically play games mostly :D, also other applications in language generation and object detection. Libraries: https://github.com/keras-rl/keras-rl https://github.com/tensorflow/minigo Introductory Tutorial(s): Grab the free Sutton and Barto book: https://web2.qatar.cmu.edu/~gdicaro/15381/additional/SuttonBarto-RL-5Nov17.pdf Watch David Silver course: https://www.youtube.com/watch?v=2pWv7GOvuf0 These are the 10 machine learning algorithms which you can learn to become a data scientist. You can also read about machine learning libraries . here We hope you liked the article. Please for a free ParallelDots account to start your AI journey. You can also check demo’s of our APIs . Sign Up here Read the original article . here

Ten Machine Learning Algorithms You Should Know to Become a Data Scientist

Building good end to end tests is hard. Having good end to end tests is wonderful. How do you get from the former to the latter? Last week I introduced a simple written in . Half way through writing that code I decided to stop and add end to end tests — so they could help me make faster progress. It worked, and saved time overall. chat server and client Dart Testing can be a big time sink; but I was sure it would work, because I’d used the same pattern before in similar code. Now I’m here to share that pattern with you. For those following along at home, here’s how to get and run the tests. You’ll need and , then: git dart git clone cd built_value.dartgit checkout tags/v0.4.3-articlecd chat_examplepub getpub run test https://github.com/google/built_value.dart.git If you run this you’ll see it’s ridiculously fast_._ On my machine it takes around one second — total — to run all 13 end to end tests. Further, if you examine , you’ll find they’re incredibly easy to write. Here’s a that private messages are private: one of the tests test group('tell', () {test('goes to a single user', () async {var alice = environment.newUser()..type('/login Alice letmein');var bob = environment.newUser()..type('/login Bob letmein');var eve = environment.newUser(); alice.type('/tell Bob Hi there.');  
bob.expectMatch(r'Alice \$private\$: Hi there.');  
eve.expectNoMatch('Hi there.'); });}); It’s testing both the client code — the code that runs in your browser — and the server. We create three users: Alice, Bob and Eve. Alice and Bob log in, then Alice sends a “tell”, a direct message, to Bob. The test passes if Bob can see the message and Eve can’t. Dart Everywhere The key to how the chat example is tested is that both the client code and the server code are written in Dart. This means that it’s perfectly possible to run the client code on the server, or the server code on the client — with the exception of VM-only libraries like “dart:io” and browser-only libraries like “dart:html”. So what I did was to client-only code. That means that “dart:html”-backed classes are split into an interface and separate implementation. For example, here’s the HTML display: fake out /// Chat window main text display.abstract class Display {/// Adds [text] to the display, coloured to indicate a local/// command.void addLocal(String text); /// Adds [text] to the display.void add(String text);} Then, to load the client code in a VM, I supply a fake implementation: /// Fake [Display] that stores added text.class FakeDisplay implements Display {List<String> text = <String>[]; @overridevoid add(String text) {this.text.add(text);} @overridevoid addLocal(String text) {}} Now I can run the server and client code in the same VM. So far so good. But they need to be able to talk to each other. The same approach is used; connection classes are split out into interfaces: /// Two-way connection between client and server; the client.abstract class ClientConnection {Stream<String> get dataFromServer; void sendToServer(String string);} And then for testing, fake implementations are used that plug the whole thing together. A maintains an instance of the server code and creates connected to it. test environment test users And that’s it. Now, you might argue that isn’t really end to end testing. To do that we’d need to use something like . Such tests have their place, particularly when integrating with complex systems. They’re necessary for shipping rock solid software. But they’re not the best way to boost developer velocity. WebDriver For that you need fast, easy to write tests that, like these, test the code you’re working on. For some code, unit tests are ideal; but to really understand things at a high level you want tests that interact with the whole application . They’re “end to end” — but only for the parts you care most about. just as a user A Bit More Testing While working on this article I noticed a bug in the chat code: not all commands are correctly echoed locally. That is, some commands just “disappear” when typed. It’s not surprising: I don’t have any tests for this! So I decided to add some. First, the “FakeDisplay” needs to record locally echoed text too: class FakeDisplay implements Display {List<String> text = <String>[];List<String> localText = <String>[]; @overridevoid add(String text) {this.text.add(text);} @overridevoid addLocal(String text) {this.localText.add(text);}} Then we need to be able to assert on it, so “TestUser” gets a new method: /// Checks local text for this user.void expectLocalMatch(Pattern pattern) {expect(_display.localText, anyElement(matches(pattern)));} And that’s it, I’m ready to add tests like this one: test('away echoes locally', () async {environment.newUser()..type('/away Not here.')..expectLocalMatch(r'/away Not here\.');}); Bug caught, and fixed. Here’s the full . I added seven new end to end tests, bringing the total to 20, and fixed two bugs as a result. pull request The tests now all run in about 1.1 seconds. I find that acceptable. Happy Holidays I’ve now written in depth about everything from my . Fortunately, it’s a good time to take a break. You can expect some more articles from me in 2017 — as soon as I have something new to write about. In the meantime, if there’s something you’d particularly like to me to cover, just let me know. talk at the Dart summit (video) is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

End to End Testing in One Short Second with Dart

Document or text classification is one of the predominant tasks in Natural language processing. It has many applications including news type classification, spam filtering, toxic comment identification, etc. In big organizations the datasets are large and training deep learning text classification models from scratch is a feasible solution but for the majority of real-life problems your dataset is small and if you want to build your machine learning model you need to be smart. In this article (originally posted by on the ), I will talk about pragmatic approaches towards text representation which make document classification on small datasets doable. Shahul ES neptune.ai/blog Text Classification 101 The text classification workflow begins by cleaning and preparing the corpus out of the dataset. Then this corpus is represented by any of the different text representation methods which are then followed by modeling. In this article, step of this pipeline. we will focus on the “Text Representation” Example text classification dataset We will use the data from kaggle competition. Here, the Real or Not? NLP with disaster tweets task is to predict which tweets are about real disasters and which ones are not. If you want to follow the article step-by-step you may want to install all the libraries that I used for the analysis. Let’s take a look at our data, pandas pd

tweet= pd.read_csv( )
test=pd.read_csv( )

tweet.head( ) import as '../input/nlp-getting-started/train.csv' '../input/nlp-getting-started/test.csv' 3 The data contains of id, keyword, location, text, and target which is binary. We will only consider the tweets to predict the target. print( .format(tweet.shape[ ],tweet.shape[ ]))
print( .format(test.shape[ ],test.shape[ ])) 'There are {} rows and {} columns in train' 0 1 'There are {} rows and {} columns in test' 0 1 The training dataset has less than 8000 tweets. That, combined with the fact that tweets are 280 characters tops make it a tricky, small(ish) dataset. Text data preparation Before we get into any NLP task, we need to do some data preprocessing and basic cleaning. It is not a focus of this article but if you want to read more about this step check out this . article In short, we will: : the process by which sentences are converted to a list of tokens or words. Tokenize : drop words like ‘a’ or ‘the’ Remove stopwords : reduce the inflectional forms of each word into a common base or root ( . Lemmatize “studies”, “studying” -> “study”) new_corpus=[]
	
	lem=WordNetLemmatizer() text df[ ]:
    	words=[w w word_tokenize(text) (w stop)]
  	 
    	words=[lem.lemmatize(w) w words]
  	 
    	new_corpus.append(words) new_corpus

corpus=preprocess_news(df) : def preprocess_news (df) '''Function to preprocess and create corpus''' for in "question_text" for in if not in for in return Now, let’s see how to represent this corpus so that we can feed this into any machine learning algorithm. Text Representation Text cannot be used directly as input to a machine learning model but needs to be represented in the numeric format first. This is known as text representation. Countvectorizer Countvectorizer provides an easy method to vectorize and represent a collection of text documents. It tokenizes the input text and builds a vocabulary of known words and then represents the documents using this vocabulary. Let’s understand it by using an example, text = [ ] vectorizer = CountVectorizer() vectorizer.fit(text) print(vectorizer.vocabulary_) vector = vectorizer.transform(text) print(vector.shape)
print(type(vector))
print(vector.toarray()) "She sells seashells in the seashore" # create the transform # tokenize and build vocab # summarize # encode document # summarize encoded vector You can see that the Coutvectorizer has built a vocabulary out of the given text and then represented the words using a . We can try and transfer another text using this vocabulary and observe the output to get a better understanding. numpy sparse matrix vector=vectorizer.transform([ ])
vector.toarray() "I sell seashells in the seashore" You can see that: The index positions 3 and 4 have zeroes meaning that these two words are not present in our vocabulary and all other positions have 1 meaning that these words are present in our vocabulary. The corresponding words missing from the vocabulary are “sells” and “she”. Now that you understand how Coutvectorizer works, we can fit and transform our corpus using it. vec=CountVectorizer(max_df= ,max_features= )
vec.fit(df.question_text.values)
vector=vec.transform(df.question_text.values) 10 10000 You should know that Countvectorizer has a few important parameters that you should adjust to your problem: : build a vocabulary that only considers the top n tokens ordered by term frequency across the corpus. max_features : When building the vocabulary ignore terms that have a token frequency strictly lower than the given threshold min_df : When building the vocabulary ignore terms that have a token frequency strictly higher than the given threshold. max_df What usually helps with selecting reasonable values (or ranges for hyperparameter optimization methods) is good exploratory data analysis. Check out my other article to read about it. TfidfVectorizer One issue with Countvectorizer is that common words like “the” will appear many times (unless you remove them at the preprocessing stage) and these words are not actually important. One popular alternative is Tfidfvectorizer. It is an acronym for Term frequency-inverse document frequency. : This summarizes how often a given word appears within a document. Term Frequency : This downscales words that appear a lot across documents. Inverse Document Frequency Let’s look at an example: sklearn.feature_extraction.text TfidfVectorizer text = [ , , ] vectorizer = TfidfVectorizer() vectorizer.fit(text) print(vectorizer.vocabulary_)
print(vectorizer.idf_) vector = vectorizer.transform([text[ ]]) print(vector.shape)
print(vector.toarray()) from import # list of text documents "She sells seashells by the seashore" "The sea." "The seashore" # create the transform # tokenize and build vocab # summarize # encode document 0 # summarize encoded vector The vocabulary again consists of 6 words and the inverse document frequency is calculated for each word, assigning the lowest score to “the” which occurred 4 times. Then the scores are normalized between 0 and 1 and this text representation can be used as input into any machine learning model. Word2vec The big issue with the above approaches is that the context of the word is lost when representing it. Word embeddings provide a much better representation of the words in NLP by encoding some context information. It provides a mapping from a word to a corresponding n-dimensional vector. Word2Vec was developed at Google by Tomas Mikolov, et al. and uses a to learn word embeddings. The vectors are learned by understanding the context in which the word occurs. shallow neural network Specifically, it looks at co-occurring words. Given below is the co-occurrence matrix for the sentence “The cat sat on the mat”. Word2vec is composed of two different models: (CBOW) model can be thought of as learning word embeddings by training a model to Continuous Bag of Words predict a word given its context. model is the opposite, learning word embeddings by training a model to Skip-Gram predict context given a word. The basic idea of word embedding is words that occur in similar context tend to be closer to each other in vector space. Let’s check how to implement word2vec in python. gensim gensim.models Word2Vec

model = gensim.models.Word2Vec(corpus, 
                               min_count = , size = , window = ) import from import 1 100 5 Now you have created your word2vec model, some of the important parameters that you can actually change and observe the differences are, : this indicates the embedding size of the resulting vector for each word. size : When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold min_count : The number of words surrounding the word is considered when building the representation. Also known as the window size. window In this article, we focus on pragmatic approaches for small datasets and we will use pre-trained word vectors instead of training vectors from our corpus. This method is guaranteed to yield better performance. First, you will have to download the trained vectors from . Then you can load the vectors using gensim. here gensim.models.KeyedVectors load_word2vec_format word2vecDict = load_word2vec_format( ,
        binary= , unicode_errors= )
    embeddings_index = dict() word word2vecDict.wv.vocab:
        embeddings_index[word] = word2vecDict.word_vec(word) embeddings_index from import : def load_word2vec () '../input/word2vec-google/GoogleNews-vectors-negative300.bin' True 'ignore' for in return Let’s check the embedding, w2v_model=load_word2vec()
w2v_model[ ].shape 'London' You can see that the word is represented using a 300-dimensional vector. So every word in your corpus can be represented like this and this embedding matrix is used to train your model. FastText Now, let’s learn about fastText which is an extremely useful module available in . FastText has been developed by Facebook and yields great performance and speed in text classification tasks. gensim It supports both Continuous Bag of Words and Skip-Gram models. The main difference between previous models and FastText is that it breaks the word in several n-grams. Let’s take the word orange for example. The trigrams of word orange are,org,ran,ang,nge(ignoring the starting and ending boundaries of the word). The word Rare words or typos can now be properly represented since it is highly likely that some of their n-grams also appears in other words. embedding vector (text representation)for orange will be the sum of these n-grams. For example, for a word like , which might never have been in any corpus, gensim might return any two of the following solutions: a zero vector or a random vector with low magnitude. stupedofantabulouslyfantastic , however, . In this particular case, the final vector might be closer to the vectors of fantastic and fantabulous. FastText can produce better vectors by breaking the word into chunks and using the vectors for those chunks to create a final vector for the word Again, we will use a pre-trained model rather than training our own word embeddings. For this, you can download pre-trained vectors from . here Each line of this file contains a word and it’s a corresponding n-dimensional vector. We will create a dictionary using this file for mapping each word to its vector representation. gensim.models FastText print( )
	embeddings_index = {}
	f = open( ,encoding= ) line tqdm(f):
    	values = line.strip().rsplit( )
    	word = values[ ]
    	coefs = np.asarray(values[ :], dtype= )
    	embeddings_index[word] = coefs
	f.close()
	print( % len(embeddings_index)) embeddings_index

embeddings_index=load_fastext() from import : def load_fasttext () 'loading word embeddings...' '../input/fasttext/wiki.simple.vec' 'utf-8' for in ' ' 0 1 'float32' 'found %s word vectors' return Let’s check the embedding for a word, embeddings_index[ ].shape 'london' GloVe (Global vectors for word representation) GloVe stands for global vectors for word representation. It is an unsupervised learning algorithm developed by Stanford. The basic idea of GloVe is to derive a semantic relationship between words using a co-occurrence matrix. The idea is very similar to word2vec but there are slight differences. Go here to read more. For this, we will use which are trained on large corpora. This is guaranteed to perform better in almost any situation.You can download it from . pre-trained glove vectors here After downloading we can load our pre-trained word model. Before that, you should understand the format in which it is made available. Each line contains a word and its corresponding n-dimensional vector representation. Like this, So, to use this you should first prepare a dictionary that contains the mapping between word and corresponding vector. This can be called an embedding dictionary. Let’s create one for our purpose. embedding_dict = {}
    path = open(path, ) f: line f:
            values = line.split()
            word = values[ ]
            vectors = np.asarray(values[ :], )
            embedding_dict[word] = vectors
    f.close() embedding_dict


embeddings_index = load_glove() : def load_glove () '../input/glove-global-vectors-for-word-representation/glove.6B.100d.txt' with 'r' as for in 0 1 'float32' return Now, we have a dictionary containing every word in the glove pre-trained vectors and their corresponding vector in a dictionary. Let’s check the embedding for a word. embeddings_index[ ].shape 'london' Till now we were dealing with representing words and these techniques are most useful for word-level operations. Sometimes we need to explore operations. These encoders are called sentence encoders. sentence level A good sentence encoder is expected to encode sentences in such a way that the vectors of similar sentences have a minimal distance between them in the vector space. For example, It is sunny today It is rainy today It is cloudy today. These sentences will be encoded and represented so that they are close to each other in the vector space. Let’s move on and check how to implement universal sentence encoder and find similar sentences using it. You can download the pertained vectors from . here We will load the module using the TensorFlow hub. module_url = embed = hub.load(module_url) "../input/universalsentenceencoderlarge4" # Import the Universal Sentence Encoder's TF Hub module Next, we will create the embedding for each sentence in our list. sentence_list=df.question_text.values.tolist()
sentence_emb=embed(sentence_list)[ ].numpy() 'outputs' Here is an article to read more about universal sentence encoder. Elmo, BERT, and others. When using any of the above embedding methods one thing we forget about is the context in which the word was used. This is one of the main drawbacks of such word representation models. For example, the word word “stick” will be represented using the same vector independent of the context in which it was used which doesn’t make much sense. With the recent developments in the field of NLP and models like BERT (bidirectional encoder representation from transformers), this has been made possible. Here is an article to read more. Text Classification In this section, we will prepare the embedding matrix which is passed to the Keras Embedding layer to learn text representations. You can use the same steps to prepare the corpus for any word-level embedding methods. Let’s create a word index and fix a maximum sentence length, pad each sentence in our corpus using and . Keras Tokenizer pad_sequences MAX_LEN= tokenizer_obj=Tokenizer()
tokenizer_obj.fit_on_texts(corpus)
sequences=tokenizer_obj.texts_to_sequences(corpus)

tweet_pad=pad_sequences(sequences,
                        maxlen=MAX_LEN,
                        truncating= ,
                        padding= ) 50 'post' 'post' Let’s check the number of unique words in our corpus, word_index=tokenizer_obj.word_index
print( ,len(word_index)) 'Number of unique words:' Using this word index dictionary and embedding dictionary you can create an for our corpus. This embedding matrix is passed on to the of the neural network to learn word representations. embedding matrix embedding layer num_words = len(word_index)
    embedding_matrix = np.zeros((num_words, emb_size)) word, i tqdm(word_index.items()): i > num_words: emb_vec = embedding_dict.get(word) emb_vec :
        embedding_matrix[i] = emb_vec embedding_matrix : def prepare_matrix (embedding_dict, emb_size= ) 300 for in if continue if is not None return We can define our neural network and pass this embedding index to the Embedding layer of the network. We pass the vectors onto the Embedding layer and set to prevent the weights from being updated. trainable=False inp = Input(shape=(MAX_LEN,))

    x = Embedding(num_words, embedding_matrix.shape[ ], weights=[embedding_matrix],
                  trainable= )(inp)

    x = Bidirectional(
        LSTM( , return_sequences= , name= , 
             dropout= , recurrent_dropout= ))(x)

    x = GlobalAveragePool1D()(x)
    x = Dense( , activation= )(x)
    model = Model(inputs=inp, outputs=x)

    model.compile(loss= ,
                  optimizer= ,
                  metrics=[ ]) model : def new_model (embedding_matrix) 1 False 60 True 'lstm_layer' 0.1 0.1 1 "sigmoid" 'binary_crossentropy' 'adam' 'accuracy' return For example, to run the model using word2vec embeddings, embeddings_index=load_word2vec()
embedding_matrix=prepare_matrix(embeddings_index)
model=new_model(embedding_matrix)

history=model.fit(X_train,y_train,
                  batch_size= ,
                  epochs= ,
                  validation_data=(X_test,y_test),
                  verbose= ) 8 5 2 You can call your desired type of embeddings and follow the same steps to implement any of them. Comparison So which text classification method worked best in our example problem? You can use Neptune to compare the performance of our model using different embeddings by simply setting up an experiment. Glove embeddings performed a little better in test sets when compared to the other two embeddings. You may be able to get better results by doing extensive cleaning on the data and tuning the model. You can here if you want to. explore experiments Final Thoughts In this article, we discussed and implemented different feature representation methods for text classification that you can use for smaller datasets. Hopefully, you will find them useful in your projects. This article was originally posted on the Neptune blog. If you liked it, you may like it there :) You can also find me tweeting @Neptune_a i or posting on LinkedIn about ML and Data Science stuff.

Document Classification Process: 7 Pragmatic Approaches For Small Datasets

Introduction Natural Language Processing. Machine Learning. Deep Learning. Transformers. LSTMs. All these seem jargon to you or never got around the thousands of lines of code and 'tricks' to use them ? Then this post is for you. The field of Machine Learning and Natural Language Understanding has seen rapid development in the past couple of years, thanks in no part to Deep Learning. Current AI has shown amazing leaps in performance on tasks with limited data with the help of Transfer Learning. For those who don't know what that is... It is basically a magical tool that allows anyone to take existing AI models and train them for their own data, however small the dataset maybe. Sounds good, right ? The harsh truth is that getting these models to work requires a substantial knowledge of coding, machine learning and deep learning. And even if you have the prerequisite  knowledge, it can still be a very daunting task. Extremely daunting. Introducing aasaan.ai - A no code platform that allows anyone, with no experience in machine learning and coding, to build, train and deploy data classifiers. So How does Aasaan work ? We will be using the Yelp Sentiment Classification data. However, we will not be using the labels that come with the dataset. The idea being that in real life, you will not be having labelled dataset and you may want to have custom labels. So let us begin! First step: Go to Aasaan.ai Upload your dataset to aasaan. The first step is, obviously, uploading the dataset to aasaan. At the moment, the platform only supports csv files but we plan to add many more formats in the future. For this example, you can download . But wait! You are in luck. This dataset is included as a sample dataset. Yelp Reviews Dataset Once you upload the csv, aasaan shows you an overview of the dataset. Now , select the column with the text that you wish to classify. At this point, you have the option to use the labels in your dataset (if you have those). If you have selected the labels, proceed to the section where we train the networks. Otherwise, follow the tutorial. Add Labels for your dataset. As I mentioned before, in a real world scenario, you will most probably not have access to labelled data. Or maybe, you want to have custom labels. (We do support for scenarios where your data is labelled) At this stage, aasaan asks for all the labels/categories/classes that you wish to classify your data into. For instance, in this example, we will be using the standard sentiment labels - Positive and Negative. For the Yelp Sentiment Classification dataset, we will be adding the and labels. Positive Negative Labelling the dataset. To train the model, you need to label the data. Fret not! Just label 50 examples to start with. Once, you have labelled 50 examples using the added labels, we can move forward to the exciting bit! After a few minutes of labelling . . . . Train the model. Now relax. Wait for a few minutes while aasaan trains for you. Evaluate the model. For the yelp dataset, the text from the test set is already preloaded. Just press predict or on the right arrow for the next text. For using the API or CSV prediction tab, you need to join our waiting list. It is free and we would love to have your feedback. Not too shabby for something made in a few minutes. Such is the power of deep learning and transfer learning now you can leverage it too. Without breaking a sweat. Interested? We have successfully built, trained and deployed a data classifier without - Writing a single line of code Configuring any gpu Looking at daunting documentations Reading and understanding machine learning papers. Visit us at aasaanai to try for yourself and be an early beta tester and join our waiting list!

Stories, News & Mentions about NEXTGLASS

Nextglass’s Stories on HackerNoon