Unsupervised has created a new form of AI that can scour the data that your business has — no matter how messy, disparate, unstructured, or complex — and find the insights that help you optimize the key parts of your business

Unsupervised

In this article, I'm going to discuss the topic of AI adoption. I'm going to talk about a couple of ways that organizations and companies can adopt AI more easily. More specifically, I'm going to talk about two areas of machine learning which I believe can be very easily integrated into existing systems and infrastructures, independent of the industry or the company, and can very quickly add value. Risks Behind AI Adoption One of the challenges behind AI adoption is that there are significant risks and this makes many companies hesitant to adopt AI. One of the major risks is obviously the risk of hiring. If you hire the wrong person or people, then you will have incurred a huge waste of time and money. Another issue is that a company might not be ready for the adoption of AI in terms of the culture. Unless the senior executives buy into AI, but also the middle layers, then it's very unlikely that AI can work well. And this is one of the topics that I've discussed and talked about a lot in my work, also in my book . And unless a company has the right culture for AI, it's very likely going to fail AI adoption. So that being said, companies might be hesitant to adopt AI, and then this creates a culture of fear and the problem perpetuates itself. The Decision Makers Handbook to Data Science Some other risks are obviously the risks of breaking things, right? So, if you're a company that has been going on for 20 or 30 years, maybe you're running some legacy systems and rebuilding these systems in a way that they can integrate with AI. Whether we're talking about a product or developing another platform internally in one way or another, there might not be a huge incentive to make these changes because you're afraid you must break things. And this is why, if an organization is very defensive around the adoption of AI, it's very important to start with some which are easy to understand, they're easy to implement, and the management, upper management, and mid-level management can easily understand and they support. use cases That's why, unfortunately, many AI initiatives fail; whether we're talking about products or services because, sometimes, service providers or product developers pitch ideas. Companies that might seem a little bit too far out there. Maybe the valuation is not clear, or maybe adopting these technologies would require a significant change in the culture or the infrastructure layer of a company. Two Types of Technologies There are two types of technologies that I believe can very easily add value without much effort. And these two technologies are: Anomaly Detection The term anomaly detection refers to a class of methods and algorithms in machine learning which are used to identify points that are, in a sense, unusual. These are points that do not seem to follow the regular patterns of behavior. And the thing with anomaly detection is that it can be very useful in contexts like fraud but can also be used in other domains like manufacturing. You can use anomaly detection to detect whether some machine is not behaving as it should and then act against this. Maybe this machine is about to break down, or similarly in retail, maybe you want to use anomaly detection to detect spikes in demand or to detect issues with the supply chain. The benefits of anomaly detection are that it doesn't really require that much to integrate with existing systems. In many cases, you can just take data as they are and it's very easy to just feed them into algorithms. And the thing is those anomaly detection algorithms, require fine-tuning, but they do not require training. So, one of the main roadblocks to using AI, in many cases, is the absence of what we call labeled data. So, these are data points that have a target variable. So, if we're talking, for example, about images, then labeled data is data that has been labeled as “this is an image of a human”, “this is an image of an animal”, et cetera. So, anomaly detection doesn't require labeled data, which means that it is much easier and faster to roll out than . Obviously, supervised learning is still king in terms of the applications of machine learning, but even if it's king, it's not always feasible to use it. supervised learning That's why anomaly detection can be so powerful if we're talking about early AI adoption Unsupervised Learning On a similar note, the second family of techniques and methods that I’d like to discuss is . Anomaly detection can also be seen as a subset of unsupervised learning. unsupervised learning But unsupervised learning also includes the topics and methods like . And clustering is one of the most famous cases and applications of data science. clustering So, for example, user segmentation is a very famous application, or pretty much any kind of segmentation. And I believe that the combination of dashboards, which can help an organization extract useful insight, and the addition of some basic unsupervised learning and clustering on this dashboard can be a very good way to educate the management of the uses of AI and help them better understand how AI can add value in an organization. Again, the benefit of using unsupervised learning is that you don't really require labeled data, you only need data that is structured in one way or another. But obviously, in some cases, you can also deal with structured data like text, as long as it's digitized. But the important thing is, much like anomaly detection, you don't really need to spend time leveling cases. So that's very powerful because you can get a useful system and, at the same, time you can get decision-makers within a company. They can get used to AI. And this is going to obviously help the gradual adoption and improve the understanding of what AI can do and how it can be used. Therefore, if a company is quite risk averse in that sense, I think that using these techniques, these methods can be a very good way to start with AI adoption. Conclusion - The Best Option for AI Adoption There are many other ways to do AI adoption. And in this article, I'm just referring to two methods that can be used by risk-averse organizations. Obviously, if an organization wants to move faster, then the best option by far is to go for supervised learning.

Ai Adoption: What You Need to Know About Anomaly Detection and Unsupervised Learning

As the American Society of Quality , many organizations have quality-related costs of up to 40% of their total production revenue. A large part of this cost comes from the inefficiency of manual inspection, which is the most common way to provide quality control in manufacturing. reports The application of artificial intelligence for quality control automation presents a more productive and accurate way of doing a visual inspection of production lines. However, traditional machine learning methods present several limitations to how we can train and utilize models for defect detection. So in this article, we’ll discuss the advantages of unsupervised learning for defect detection, and elaborate on the approaches MobiDev uses in our practical experience. What is AI defect detection and where it is used? is based on computer vision that provides capabilities for automating the whole process using machine learning algorithms. Defect detection models are trained to visually examine items that pass the production line and recognize anomalies on their surface, and spot inconsistencies in dimensions, shape, or color. AI defect detection AI quality inspection The output depends on what the model is trained to do, but in the case of defect detection, the flow typically looks like this: https://www.youtube.com/watch?v=UY6xbrcViVw Applied to quality control processes, defect detection AI is efficient at inspecting large production lines and spotting faults even on the smallest parts of a final product. This relates to a large spectrum of manufactured products that may contain surface defects of different natures. Intel describes a case of implementing computer vision to automate tire quality inspection. As is stated in the , the accuracy of quality control rose up from 90% to 99% percent, while approximately $49,000 in labor costs were cut by the production line. report But such systems are not bound to stationary hardware in the factory. For instance, drones with cameras can be used for inspecting or other outdoor surfaces, which significantly decreases the time required for covering large city areas. pavement defects The pharmacy industry also benefits from inspecting manufacturing lines for different products. For instance, Orobix applies defect detection to with a specific type of camera that can be used by an untrained human operator. The same principle is applied to inspecting defects, like cracks and air blobs caught in the glass. drug manufacturing pharmaceutical glass Such examples can be found in the food industry, textile, electronics, heavy manufacturing, and other branches. But, there are some specific problems in how we can approach defect detection algorithms with traditional machine learning. As manufacturers inspect thousands of products daily, it becomes difficult to collect sample data for training, and also to label it. This is where unsupervised learning comes into play. What is unsupervised learning? The majority of machine learning applications rely on methods. Supervised learning entails that we provide ground truth information to the model by manually labeling collected data. supervised machine learning In terms of a production line, collecting and labeling data can be impossible, since there is no way we can gather all the variations of cracks or dents on a product to ensure accurate detection by the model. Here we face four problems: difficulties with obtaining a large amount of anomalous data possibility of a very small difference between a normal and an anomalous sample a significant difference between two anomalous samples inability to know in advance the type and number of anomalies algorithms allow you to find patterns in a data set without pre-labeled results and discover the underlying structure of the data where it is impossible to train the algorithm the way you normally would. Unsupervised machine learning Unlike supervised learning, the flow of training becomes less labor intensive, as we expect the model to discover patterns in data with a higher threshold for variations. Anomaly detection reveals previously unseen rare objects or events without any prior knowledge about them. The only available information is that the percentage of anomalies in the dataset is small. Concerning defect detection, this helps to resolve the problem with data labeling and collecting huge amounts of samples. So let’s see how unsupervised learning methods can be used for defect detection model training. How does unsupervised learning apply to defect detection? Defect detection relates to the problem of anomaly detection in machine learning. While we don’t rely on labeling, there are other approaches in unsupervised learning that aim at grouping data and providing hints to the model. is grouping unlabeled examples according to similarity.  Clustering is widely used for recommender engines, market or customer segmentation, social network analysis, or search result clustering. Clustering aims to observe frequently occurring patterns, correlations, or associations from datasets. Association mining are designed to model the distribution probability with latent variables. It is mainly used for data preprocessing, reduction of features in a dataset, or decomposing the dataset into multiple components based on features. Latent variable models Uncovered patterns with unsupervised learning can be used for implementing traditional machine learning models. For instance, we might apply clustering to the available data, and then use those clusters as a training dataset for supervised learning models. Concrete Crack Detection with Unsupervised ML Having a vast experience in machine learning, we’ve conducted an experiment using the . The goal was to create a model capable of recognizing images with defects and normal ones using unsupervised learning. Concrete Crack dataset Additionally, the study checks how the number of defect images affects certain algorithms used in this project. In the use case we selected, we assume that image labels cannot be known in advance during training. Only a test dataset is labeled in order to verify the quality of model prediction, as training occurs by an unsupervised approach. So here we used five different approaches to get classification results from an unsupervised learning model. CLUSTERING Since we don’t have any labeled ground truth data, grouping unlabeled examples are done with clustering. In our case, there are two clusters of images we need to single out from the dataset. This was performed with a pre-trained VGG16 convolutional neural network for feature extraction and K-means for clustering. What clustering did here, is group images with and without cracks based on their visual similarities. In a nutshell, clustering looks something like this: Clustering methods are easy to implement and usually are considered as a baseline approach for further deep learning modeling. BIRCH CLUSTERING Upon this approach, images were clustered based on visual similarity with a pre-trained ResNet50 neural network for feature extraction and for clustering. This algorithm constructs a tree data structure with the cluster centroids being read off the leaf. Birch It is a memory-efficient, online-learning algorithm. The results of clustering were visualized with : Principal Component Analysis As we can see, birch clustering shows a pretty good distribution of classes, even on points where the sample is quite far from its centroid. CUSTOM CONVOLUTIONAL AUTOENCODER Custom convolutional autoencoder contains two blocks: encoder and decoder. It helps to obtain features in the encoder part and reconstruct images from them in the decoder part. As we don’t have labels for network training, we need to choose another approach to obtain classes – for example, an adaptively selectable threshold. The purpose of an adaptively selectable threshold is to divide two distributions (crack-free images and images with cracks) as precisely as possible: DCGAN DCGAN generates images from z-space with the help of adversarial loss (BCALoss). Finally, we have three losses – generator loss, discriminator loss, and MSE loss (to compare generated images and ground truth). We can build our classification on the same approach as in custom autoencoder – through comparison between losses on images with cracks and without with the help of an adaptive selectable threshold. For threshold, it will be appropriate to use discriminator loss or MSE loss depending on their distributions. GANOMALY uses the conditional GAN approach to train a Generator to produce images of the normal data. During inference, when an anomalous image is passed, it is not able to capture the data correctly. GANomaly It leads to poor reconstruction for defected images and good reconstruction for normal ones and gives the anomaly score. How to approach unsupervised anomaly detection Perhaps, the most beneficial side of unsupervised learning techniques is that we can avoid gathering tremendous amounts of sample data, and labeling it for training. Applying unsupervised learning techniques to derive data patterns, we’re not limited to which model can be used for actual classification and defect detection. However, unsupervised learning models are suited better for segmenting existing data into classes, since it’s quite difficult to check the model prediction accuracy, especially without a labeled dataset. Written by , AI/ML Team Leader at . Viktoriia Akhremenko MobiDev The full article was originally published and is based on MobiDev technology research. here

How Unsupervised Learning Can Help in Defect Detection & Quality Control in Manufacturing

There won’t be any business insights if the data quality is poor. When preparing data, I often go through many different approaches to reach a level of quality of data that can provide accurate results. In this article, I describe how unsupervised ML can help in data preparation for machine learning projects and how it helps to get more accurate business insights. What’s Wrong with Traditional Data Preparation Approaches? For accurate predictions, the data must not only be properly labeled, de-deputed, broad, consistent, etc. The point is that the machine learning model should process the “right” data. It is not entirely clear what are the criteria of the “right” data. When data scientists prepare the data, they often require a domain expert assisting in data categorization and labeling. If taking into account the majority of machine learning projects, there are often no domain experts available for such tasks. Here, the correctness of data may fail due to the wrong understanding of the final output expected from the ML model, incorrect categorization of data, and human error. In such a case, the ML model might output the wrong result at the very beginning and cause further errors in the future. One more question is whether the expected result is really something that might add value when optimizing business processes. What if the initial hypothesis is wrong? Here I mean cases when there is some underlying structure of data that neither product owner nor data scientists know about. Anyhow, data scientists spend up to 80% of their time cleaning the gathered data before training the ML model, which is not a guarantee of the entire absence of errors and bias. That is why it is often difficult to reach the ideal data quality and meet all the data standard requirements. And that’s how we came to unsupervised machine learning approaches. What is Unsupervised Machine Learning? Before reviewing unsupervised machine learning, let’s define what supervised learning is. According to , supervised learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from labeled training data consisting of a set of training examples. These labeled training data is useful for the ML model since then it differentiates data categories more accurately. The measure for the model’s performance, in this case, is defined from the beginning, and it is clear what can be considered as an “accurate” output. The possible consequences of such an approach are described in the previous paragraph. Wikipedia In unsupervised machine learning, there are no data labels outlined, and we cannot measure the performance of an algorithm. The algorithm detects a deep structure of data on its own and distributes a dataset into different categories. The algorithm is commonly chosen according to business goals, and if performed correctly, the business solution is tenfold more powerful than a supervised learning-based one. The unsupervised ML algorithm does not have predefined tasks and may detect hidden data patterns that we may not have known about. Also, it’s better to use unsupervised ML when aiming to find new patterns in future data, thus, optimizing the ML-based solution in advance. Unsupervised ML Approaches to Define the Underlying Structure of Data Unsupervised approaches are mostly focused on the finding of the underlying data structure. Here I describe the most often used approaches. DIMENSIONALITY REDUCTION ALGORITHM Dimensionality reduction is the ML method used to identify patterns in data and deal with computationally extensive problems. This method includes a set of algorithms aimed to reduce the number of input variables in a dataset by projecting the original high-dimensional input data to a low-dimensional space. Dimensionality reduction is helpful when working with visual and audio data involving speech, video, images, or text, and also when simplifying datasets in order to better fit a predictive model. There are many examples of dimensionality reduction algorithms, but I’d like to emphasize UMAP, which we found useful when working on one of the machine learning projects. algorithm allows the construction of a high dimensional graph representation of data and further optimization of a low-dimensional graph to be as structurally similar as possible. UMAP (Uniform Manifold Approximation and Projection) The main feature of this algorithm is the nonlinear representation of data. Compared to other dimensionality reduction algorithms, it is good at scaling dimensionality and size of a dataset and fast projection. For , UMAP can project the 784-dimensional, 70,000-point MNIST dataset in less than 3 minutes, compared to 45 minutes for scikit-learn’s t-SNE implementation. example UMAP projection to various toy datasets, powered by umap-js Source DATA CLUSTERING Clustering allows defining a structure in a set of unlabeled data. Simply put, it organizes data into groups (clusters), basing on its similarity and dissimilarity. This approach is applicable for market and customer segmentation, recommendation engines, document classification, fraud identification, and other cases. The goal of clustering is to detect distinct groups in an unlabeled dataset, where the users are expected to determine the criteria of what is a “correct” cluster so that clustering results meet their expectations. The example of Clustering algorithm in action There are many clustering algorithms, including KNN, K-means, Gaussian Mixture Model, and others, but here I’d like to emphasize two algorithms that have proven their high performance in our practice: DBSCAN and HDBSCAN. algorithm takes all instances that are close to each other and groups them together, based on a distance measurement and a minimum number of instances specified by a data science engineer. Since the distances within multiple clusters are specified, clusters will contain those instances which are most densely located. If some instance is located out of the specified distance, and not belongs to any cluster, it will be labeled as an outlier. DBSCAN (density-based spatial clustering of applications with noise) DBSCAN clustering visualization: Smiley Face Source This algorithm is good at finding those data structures and associations that can be useful to define patterns and predict trends. For example, when developing recommendation engines, the DBSCAN algorithm can be applied to customer databases in order to find the most demanded products of specific users. converts the DBSCAN into a hierarchical manner and extracts a flat clustering based on the stability of clusters. HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm HDBSCAN clustering algorithm Source Firstly, the HDBSCAN algorithm changes the space according to the density or sparsity, creates the minimum tree of the distance weighted graph, and constructs a cluster hierarchy of connected components. Based on the minimum cluster size, the algorithm condenses the cluster hierarchy and, finally, extracts the stable clusters from the condensed tree. ANOMALY DETECTION ALGORITHM The anomaly detection algorithm is not independent itself. It often goes along with the dimensionality reduction and clustering algorithms. By using the dimensionality reduction algorithm as a pre-stage for anomaly detection, we, firstly, transform high-dimensional space into a lower-dimensional one. Then we can figure out the density of the major data points in this lower-dimensional space, which may be identified as “normal.” Those data points located far away from the “normal” space are outliers or “anomalies.” The clustering algorithm is often used to improve the analysis of anomalies. It allows grouping of similar anomalies and further manual categorization based on their behavior types. By following such a process, we can use unsupervised ML algorithms to detect anomalies, cluster them, provide labels to each cluster manually; thus, getting business insights we couldn’t have predicted. A good example of anomaly detection is spam filtering, where the ML algorithm analyzes all incoming messages, clusters them, and detects spam messages as outliers. It is also extensively used for fraud detection in financial, insurance, IT, and other spheres. The example of Anomaly Detection algorithm Source This algorithm is also a part of ML-based predictive maintenance systems in manufacturing. By detecting anomalies in the system behavior, this algorithm allows the prediction of equipment failures before they occur. ASSOCIATION MINING ALGORITHM Association mining is the unsupervised ML algorithm used to identify hidden relationships in large datasets which frequently occur together. Compared to most ML algorithms that process numeric data, association mining can deal with non-numeric, categorical data, which means that it requires more actions than simple counting. This algorithm is commonly used to identify patterns and associations in transactional, relational, or any similar database. For example, it is possible to build the ML algorithm, which will analyze the market basket through the processing of data from barcode scanners, and define goods purchased together. In the sphere, the association mining may be for setting a correct diagnosis. By processing symptom characteristics and various illness factors and defining associations, we may predict disease probability. Moreover, if adding new symptoms in the future, finding relationships between the new data and diseases is possible. healthcare In the public sector, the association mining algorithm can be used for census data processing. As a result, it may help to optimize the planning of public services and businesses such as transport, education, health, setting up new shopping malls, and factories. Unsupervised ML Business Cases Let’s review a couple of real-life business cases where we used unsupervised ML. CLOUD COMPUTING. HARDWARE PERFORMANCE COMPARISON One of our projects involved the usage of unsupervised ML algorithms to compare the performance of virtual machines. Since the project area was a bit tough for understanding, we decided not to use any labels because they might be subjective, but rather build an independent system; thus, we did the following: Created the virtual machine on the cloud Benchmarked it via running different tests in order to measure the performance of VM Collected about 2000 characteristics as raw data Analyzed the gathered raw data and extracted the most valuable benchmarks Compressed benchmarks into several coefficients (parallelization, one core, stability, database, RAM) Calculated the custom coefficient as a balance between the performance and price Selected the best instance type based on their characteristics and price The process described above is a good example of dimensionality reduction technique since we didn’t include all the characteristics to the system but the collapsed information which could be considered as “representative” in terms of results. HUMAN LABELING EVALUATION AND AUTOMATION The project involved the evaluation of human labels, which is now a tough task in data science. We had to find the best ways to evaluate the quality of human labels. Having the data about individuals and legal entities, we should divide it into three classes, analyze it, find specific labels, and predict labeling accuracy. We set the human action label based on specific characteristics, constructed the classifier, and evaluated the quality of these labels. As you can see from the graph above, it’s almost impossible to divide classes without users as a variable because they are too close to each other according to the characteristics and their labels. But let’s take a look at this chart with the data where we included users as a variable, where the final human action labels were dependent on the person. By adding a person into that algorithm, we could achieve the best results. The important thing here is that the data should be independent, which means that it should not have relationships with the person interested in labeling. Thereby, we concluded that there was no proper way to proceed with classification tasks, and it’s better to use raw data. In conclusion, I would say that there is no perfect path for any business case. And unsupervised machine learning is just a tool for getting expected results. It will work well if you are sure that it meets your business requirements. Highlights: Unsupervised machine learning still requires a good quality of data to be processed, although not labeled. A proper data preparation approach drives “right” business insights by improving their accuracy. Unsupervised machine learning algorithms should be chosen depending on a particular business case, not on the popularity of a specific approach and . trends The goal of and is to tackle business problems, but not Data Science ones. Data Science machine learning consulting Written by Liudmyla Taranenko, Data Science Engineer at MobiDev

How To Improve Data Quality When With Unsupervised Machine Learning

Essential Guide to Clustering In Unsupervised Learning

What is hierarchical clustering in unsupervised learning? Data in itself has no value, it actually finds its expression when it is processed right, for the right purpose using the right tools. So when it comes to understanding the data it becomes extremely important that we are not only looking to extract obvious insights but also to identify the hidden patterns which may not be easy to find just by exploratory data analysis. To make intelligent predictions, identifying patterns and make effective recommendations our data need to be segregated into meaningful clusters. This stream of machine learning where we do not rely on a labeled data set which has a target variable already defined instead rely upon clustering the data sets into groups and try to make predictions about the behavior is called unsupervised learning. Unsupervised learning collaborates with supervised machine learning to make our model robust and reliable. So today we will look into unsupervised learning techniques, we will go into details of - What is Unsupervised Learning? - Types Of Unsupervised Learning - Understanding clustering & its types - Hands-on on K-Means & hierarchical clustering Let’s start this journey of learning by understanding unsupervised learning. What Is Unsupervised Learning? It’s a learning process where our machine has the unlabeled data set to be processed and it has to find the unknown/unseen pattern in the given data set. It’s is a kind of self-organized Hebbian learning that helps the machine to find previously unknown patterns in data set without pre-existing labels. src Here in the pic, you can see that the first robot has got the data set where the required result to be tested & predicted is known to him. This is what we call with labeled data. Whereas in the second case our robot is speaking about how it doesn’t have any clue about what is an outcome he is trying to achieve with the given data set. Here there is no pre-labeled data set fed to the robot, this is what we call supervised learning Unsupervised learning. Unlike supervised learning where we have labeled data that needs to be classified or predicted, here we don’t have any target variable which is used to measure the model outcome. Our machine self-learn from the unlabeled data set & and allows modeling probability densities of the given input. But the question is how does our machine learn in this scenario? Well, here what we do is that we group the given data set where every group has some kind of pattern or common behavior amongst them. Let’s get into the depth of it by understanding what is clustering & types of unsupervised learning. Unsupervised Learning Type: Two major types of unsupervised learning methodology are: 1. Principal Component Analysis 2. Cluster Analysis(Clustering) A. Principal Component Analysis: Principal Component Analysis (PCA) is a statistical procedure that orthogonally transforms the original n coordinates of a data set into a new set of n coordinates called principal components. As a result of the transformation, the first principal component has the largest possible variance; each succeeding component has the highest possible variance under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding components. B. Cluster Analysis(Clustering): Clustering is a methodology where we group or segment the given datasets where each cluster/group of data has shared attributes to extrapolate algorithmic relationships. In this machine learning technique, the cluster of the data is not labeled, classified or categorized. The clustering of data is done based on the similarity of the feature. Instead of responding to feedback, cluster analysis identifies commonalities in the data and reacts based on the presence or absence of such commonalities in each new piece of data. As per Wiki, or is the task of grouping a set of objects in such a way that objects in the same group (called a ) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval, bioinformatics, data compression, and computer graphics. Cluster analysis clustering cluster Clustering divides objects based on unknown features. Machine chooses the best way Going further, we will mainly look into clustering techniques and associated algorithm with it, then we will look K-Means in detail and go hands-on with one problem statement using python & Jupyter notebook Some of the typical Use Cases To Make Things More Clearer: Nowadays used: For market segmentation (types of customers, loyalty)To merge close points on a mapFor image compressionTo analyze and label new dataTo detect abnormal behavior Going further, we will mainly look into clustering techniques and associated algorithms with it, then we will look at K-Means in detail and go hands-on with one problem statement using python & Jupyter notebook. you must have used Apple Photos and Google Photos in your phone, here a more complex level of clustering is involved. Your pics are clubbed based on all the faces in photos to create albums of your friends. The concerned app doesn’t know how many friends you have and how they look, but it’s trying to find the common facial features, it is a typical case of unsupervised learning analysis, which involves clustering. For example, Similarly e-Commerce giants like amazon used customer segmentation to offer product recommendations based on group behavior and similarly of the attributes of the consumers in that particular group. Another visual example below to help you understand the clustering mechanism: src Clustering Techniques: 3 major type of clustering techniques are: Hierarchical Clustering - -Partitional Clustering -Bayesian Clustering The below diagram gives the details of the type of clustering mechanism employed in unsupervised learning. src We will mainly look into two of them in today’s journey. Hierarchical Clustering K-Means Clustering A. Hierarchical Clustering : Hierarchical clustering is an algorithm that groups similar objects into a cluster where each successive cluster is formed based on the previously established cluster. The endpoint is a set of clusters where each cluster is different & distinct from each other, and also the attributes within each cluster are broadly similar to each other. , Strategies for hierarchical clustering generally fall into two types : This is a bottom-up approach: where each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy. : This is a top-down approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy. Agglomerative Divisive B. Partitional Clustering: These clustering method partitions the objects into k clusters and each partition form one cluster. This method is used to optimize an objective criterion similarity function such as when the distance is a major parameter example etc. This mechanism determines all the cluster at once. K-means, CLARANS (Clustering Large Applications based upon randomized Search), Partitional clustering decomposes a data set into a set of disjoint clusters. Given a data set of N points, a partitioning method constructs K (N ≥ K) partitions of the data, with each partition representing a cluster. That is, it classifies the data into K groups by satisfying the following requirements: (1) each group contains at least one point, and (2) each point belongs to exactly one group. Notice that for fuzzy partitioning, a point can belong to more than one group. One of the popular partitional clustering methods is K-Means Clustering: -means clustering is a method to partition observations into clusters where each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. k n number of k The outcome of the K-means clustering algorithm is: The centroids of the clusters, which can be used to label new dataLabels for the training data (each data point is assigned to a single cluster) K How K-Means Algo Works: It aims to partition a set of observations into several clusters (k), resulting in the partitioning of the data into Voronoi cells. It can be considered a method of finding out which group a certain object belongs to. This algorithm works in the following manner: K points are placed into the object data space representing the initial group of centroids.Each object or data point is assigned to the closest k.After all, objects are assigned, the positions of the k centroids are recalculated.Steps 2 and 3 are repeated until the positions of the centroids no longer move. Understanding Hierarchical Clustering In Detail With Example: A. Agglomerative Clustering : It is a Bottom-up hierarchical clustering algorithm, where we treat each data point as a single cluster to start with and then successively merge (or ) pairs of clusters until all clusters have been merged into a single cluster that contains all data points. While we do this process of merging the cluster based on similarity we end up with a tree called a agglomerate dendrogram The root of the tree is the unique cluster that gathers all the samples, the leaves being the clusters with only one sample src: Agglomerative Hierarchical Clustering To understand how the above merging happens we need to understand the following key concepts: Measures of distance (similarity) The distance between two given clusters is computed using the distance metrics like Euclidean distance. There are other distance metrics too, like Euclidean squared, Manhattan, etc., but the choice of a distance metric purely depends on the domain and the business problem statement which one is trying to solve. Linkage Criteria: Once we decide to select the right distance metric, one needs to decide from which point distance is being calculated. We achieve this by using various types of linkage options. H the pairs merge in Agglomerative clustering, involves calculating a dissimilarity between each merged pair and the other sample, which is achieved by the linkage criteria. ow Single Linkage: Here the distance between two clusters is the distance between the nearest neighbors in those clusters. The distance between clusters “r” and “s” to the left is equal to the length of the arrow between their two closest points. Example: In single-link clustering, we consider the distance between one cluster and another cluster to be equal to the shortest distance from any member of one cluster to any member of the other cluster. Complete Linkage: the distance between clusters “r” and “s” to the left is equal to the length of the arrow between their two furthest points Example: In this type of clustering (it is also called the or method), we consider the distance between one cluster and another cluster to be equal to the longest distance from any member of one cluster to any member of the other cluster. diameter maximum Here the distance between two clusters is the distance between the furthest points in those clusters. Average Linkage: In clustering, we consider the distance between one cluster and another cluster to be equal to the average distance from any member of one cluster to any member of the other cluster. average-link : the distance between clusters “r” and “s” to the left is equal to the average length each arrow between connecting the points of one cluster to the other. Example Here the distance between two clusters is the average of the distances between all the points in those clusters. : finds the centroid of cluster 1 and centroid of cluster 2, and then calculates the distance between the two before merging. Centroid-linkage Divisive Clustering : It is a top-down clustering method and is less commonly used. It works similarly to agglomerative clustering but, in the opposite direction. Here, all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy. To Summarize: Here Data starts as one single combined cluster.The cluster splits into two distinct parts, based on some degree of similarity.Clusters split into two again and again until the clusters only contain a single data point. Divisive Clustering method is rigid i.e., once a merging or splitting is done, it can never be undone Hands-on With Hierarchical Clustering Using Python: We will make use of Wholesale customer Data sets. This file has been sourced from UCI: https://archive.ics.uci.edu/ml/datasets/wholesale+customers The data set refers to the clients of a wholesale distributor. It includes the annual spending in monetary units (m.u.) on diverse product categories Note! We expect that you have installed anaconda from here , to use Jupyter as your code editor. we will be making use of Python version 3 which will come pre-loaded in anaconda installation. Attribute Information: FRESH: annual spending (m.u.) on fresh products (Continuous) MILK: annual spending (m.u.) on milk products (Continuous) GROCERY: annual spending (m.u.)on grocery products (Continuous) FROZEN: annual spending (m.u.)on frozen products (Continuous) DETERGENTS_PAPER: annual spending (m.u.) on detergents and paper products (Continuous) DELICATESSEN: annual spending (m.u.)on and delicatessen products (Continuous); CHANNEL: customers Channel — Horeca (1: Hotel/Restaurant/Cafe) or Retail channel (Nominal). Objective: We will try to cluster this given dataset into a customer segment, which is achieved based on the purchase made by various consumers. The intent is to cluster similar customers together using a hierarchical clustering technique Code Time : I have created a simple Kaggle notebook to help you get hands-on experience in solving the problem statement and achieve the desired objective. Here is the link: https://www.kaggle.com/pramodchandrayan/understanding-clustering-in-unsupervised-learning What's Next? We will look deeper into another important data clustering technique called . K-Means Clustering #ChandrayanLogy: Data science works well when the two exponent of  machine learning mechanism named supervised and unsupervised works closely as a team. When you want your data to make more sense and to be structured well employ unsupervised learning methodology of data clustering & dimensionality reduction using PCA.Now this data will be more useful to be used in supervised learning algo to help you build effective models.

Understanding The Concept of Clustering In Unsupervised Learning

More data we have, better performance we can achieve. However, it is very too luxury to annotate large amount of training data. Therefore, proper data augmentation is useful to boost up your model performance. Authors of (Xie et al., 2019) proposed Unsupervised Data Augmentation (UDA) assistants us to build a better model by leveraging several data augmentation methods. Unsupervised Data Augmentation In natural language processing (NLP) field, it is hard to augmenting text due to high complexity of language. Not every word we can replace it by others such as a, an, the. Also, not every word has synonym. Even changing a word, the context will be totally difference. On the other hand, generating augmented image in computer vision area is relative easier. Even introducing noise or cropping out portion of image, model can still classify the image. Xie et al. conducted several data augmentation experiments on image classification (AutoAugment) and text classification (Back translation and TF-IDF based word replacing). After generating large enough data set of model training, authors noticed that model can easily over-fit. Therefore, they introduce Training Signal Annealing (TSA) to overcome it. Augmentation Strategies This section will introduce three data augmentation in computer vision (CV) and natural language processing (NLP) field. AutoAugment for Image Classification AutoAugment is found by google in 2018. It is a way to augment image automatically. Unlike traditional image augmentation library, AutoAugment is designed to find the best policy to manipulate data automatically. You may visit for model and implementation. here Back translation for Text Classification Back translation is a method to leverage translation system to generate data. Given that we have a model for translating English to Cantonese and visa versa. Augmented data can be retrieved by translating the original data from English to Cantonese and then translating back to English used back-translation method to generate more training data to improve translation model performance. Sennrich et al. (2015) TF-IDF based word replacing for Text Classification Although back translation helps to generate lot of data, there is no guarantee that keywords will be kept after translation. Some keywords carries more information than others and it may be missed after translation. Therefore, Xie et al. uses to tackle this limitation. The concept of TF-IDF is that high frequency may not able to provide much information gain. In another word, rare words contribute more weights to the model. Word importance will be increased if the number of occurrence within same document (i.e. training record). On the other hand, it will be decreased if it occurs in corpus (i.e. other training records). TF-IDF IDF score is calculated by DBPedia corpus. TF-IDF score will be computed for each token and replacing it according to TF-IDF score. Low TF-IDF score will have a high probability to be replaced. If you are interested to use TF-IDF based word replacing for data augmentation, you may visit for python implementation. nlpaug Training Signal Annealing (TSA) After generated large amount of data by using aforementioned skill, Xie et al. noticed  that  model will be over-fitting easily. Therefore, they introduce TSA. During model training, examples with high confidence will be removed from loss function to prevent over-training. The following figure shows the value range of ηt while K is number of categories. If the probability is higher then ηt, it will be removed from loss function. 3 calculation of ηt are considered for different scenarios: Linear-schedule: Growing constantly Log-schedule: Growing faster in the early stage of training Exp-schedule: Growing faster at the end of training. Recommendation The above approach is designed to solve problems which authors are facing in their problem. If you understand your data, you should tailor made augmentation approach it. Remember that golden rule in data science is garbage in garbage out. Like to learn? I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. Feel free to connect with on or . me LinkedIn Github Extension Reading Data Augmentation for Text Unofficial AutoAugment implementation Reference R. Sennrich, B. Haddow and A Birch. . 2015 Improving Neural Machine Translation Models with Monolingual Data E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan and Q. V. Le. . 2018 AutoAugment: Learning Augmentation Strategies from Data Q. Xie, Z. Dai, E Hovy, M. T. Luong and Q. V. Le. . 2019 Unsupervised Data Augmentation

Unsupervised Data Augmentation

How can you apply unsupervised learning on audio data? Unsupervised Learning Photo by on Jan Střecha Unsplash Unsupervised learning approach is demonstrated by state-of-the-art NLP model (e.g. BERT, GPT-2) to be a good way to learn feature for downstream task. Researchers demonstrated data-driven learned features provide a better audio feature than traditional acoustic feature such as Mel-frequency cepstrum (MFCC). This story will discuss about how can you use unsupervised learning to learn audio feature and applying it to downstream tasks. Unsupervised feature learning for audio classification Lee et al. propose to use convolutional deep belief network (CDBN, aksdeep learning representation nowadays) to replace traditional audio features (e.g. spectrogram and Mel-frequency cepstrum (MFCC)). The original input is spectrogram from each utterance and window size is 20ms with 10ms overlaps. Small window and overlapping setting is common when handling audio input. Believe that computer resource is limited in that time (it was 2009), they leverage principal component analysis (PCA) to reduce dimension before feeding into neural network. The setting for the neural network includes 2 convolutional neural network (CNN) layer with 300 dimensions, filter length of 6 and max-pooling ratio of 3. Visualization of pre-trained layer (Lee et al., 2009) Lee et al. evaluated data-driven feature (CDBN) with traditional features which are spectrogram (i.e. RAW in the following figure) and MFCC for speaker identification, speaker gender classification, phone classification, music genre classification and music artist classification. Test classification accuracy for speak identification by using summary statistics (Lee et al., 2009) Test classification accuracy for speak identification by using all frame (Lee et al., 2009) Test classification accuracy for gender classification (Lee et al., 2009) Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging Xu et al. use spectrogram as raw input to learn vector representation. Asymmetric de-noising auto-encoder (aDAE) is presented in the research paper. The network architecture include encoder (first three layers) and decoder (last three layers) parts. Spectrogram is extracted and feeding into the encoder while the training objective is predicting the middle of frame by using previous and next frames. It is similar to Continuous Bag-of-Words (CBOW) in NLP. CBOW use surroundings to predict the target words. Network architecture of aDAE (Xu et al., 2016) The following model comparisons demonstrate aDAE achieve a better result in general. Model comparison results among labels (Xu et al., 2016) Unsupervised feature learning for audio analysis Meyer et al. also use spectrogram as raw input to learn vector representation. The training object is using previous frame to predict next frame which is similar to language model in NLP. Audio frame predictor (AFP) is presented in this paper. The network architecture include encoder and decoder parts. Spectrogram is extracted with 2.56s sliding window size and 0.64s overlaps and feeding into encoder which includes multiple ConvLSTM layer. ConvLSTM setting use 3x3 filter kernal with ReLu activation and batch normalization function. Model of the audio frame predictor (Meyer et al., 2017) Meyer et al. use two steps procedure to train the data-driven representation. The network is trained by minimizing the mean squared error (MSE) (i.e. encoder to decoder) in the first 6 epochs. In the sixth to ninth epochs, pairwise loss training objected is added to adjust the representation simultaneously. Vector distribution from pre-trained feature (Meyer et al., 2017) Tess accuracy for classification and clustering tasks (Meyer et al., 2017) Like to learn? I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. Feel free to connect with on or following me on or . me LinkedIn Medium Github Reference H. Lee, Y. Largman, P. Pham and Andrew Y. Ng. . 2009 Unsupervised feature learning for audio classification using convolutional deep belief networks Y. Xu, Q.Huang, W. Wang, P.Foster, S. Sigtia, P. J. B. Jackson, and M. D. Plumbley. . 2016 Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging M. Meyer, J. Beutel and L. Thiele. . 2017 Unsupervised Feature Learning for Audio Analysis

How can you apply unsupervised learning on audio data?

Learn how to train and evaluate an unsupervised machine learning model — principal component analysis in this article by Jillur Quddus, a lead technical architect, polyglot software engineer and data scientist. There are numerous real-world use cases, where the number of features available, which may potentially be used to train a model, is very large. A common example is economic data and using its constituents, stock price data, employment data, banking data, industrial data, and housing data together to predict the ( ). Such types of data are said to have high dimensionality. Though they offer numerous features that can be used to model a given use case, high-dimensional datasets increase the computational complexity of machine learning algorithms and more importantly may also result in over fitting. gross domestic product GDP Over fitting is one of the results of the , which formally describes the problem of analyzing data in high-dimensional spaces (which means that the data may contain many attributes, typically hundreds or even thousands of dimensions/features) but where that analysis no longer holds true in a lower-dimensional space. curse of dimensionality Informally, it describes the value of additional dimensions at the cost of model performance. ( ) is an technique used to preprocess and reduce the dimensionality of high-dimensional datasets while preserving the original structure and relationships inherent to the original dataset so that machine learning models can still learn from them and be used to make accurate predictions. Principal component analysis PCA unsupervised Movie recommendation system To better understand PCA, let’s study a movie recommendation use case. Our aim is to build a system that can make personalized movie recommendations to users based on historic user-community movie ratings. The historic user-community movie ratings data that we will use for our case study has been downloaded from GroupLens, a research laboratory based at the University of Minnesota that collects movie ratings and makes them available for public download at . For the purposes of this case study, we have transformed the individual and datasets into a single pivot table where the 300 rows represent 300 different users, and the 3,000 columns represent 3,000 different movies. https://grouplens.org/datasets/movielens/ movies ratings This transformed, pipe-delimited dataset can be found in the GitHub repository at . You can also find the code file for this article at . https://github.com/PacktPublishing/Machine-Learning-with-Apache-Spark-Quick-Start-Guide/tree/master/Chapter05/data/movie-ratings-data and is called movie-ratings-data/user-movie-ratings.csv https://github.com/PacktPublishing/Machine-Learning-with-Apache-Spark-Quick-Start-Guide/blob/master/Chapter05/chp05-02-principal-component-analysis.ipynb A sample of the historic user-community movie ratings dataset that we will study looks as follows: In this case, each movie is a different feature (or dimension) and each different user is a different instance (or observation). This sample table, therefore, represents a dataset containing 5 features. However, our actual dataset contains 3,000 different movies and therefore 3,000 features/dimensions. Furthermore, in a real-life representation, not all users would have rated all the movies, so there will be a significant number of missing values. Such a dataset, and the matrix used to represent it, is described as . These issues would pose a problem for machine learning algorithms, both in terms of computational complexity and the likelihood of over fitting. sparse To solve this problem, take a closer look at the previous sample table. It seems that users that rated Movie #1 highly (Toy Story) generally also rated Movie #2 highly (Monsters Inc.) as well. We could say, for example, that User #1 is of all fans of computer-animated children’s films, and so we could recommend to User #2 the other movies that User #1 has historically rated highly (this type of recommendation system where we use data from other users is called ). representative collaborative filtering At a high level, this is what PCA does — it identifies , called , within a high-dimensional dataset so that the dimensions of the original dataset can be reduced while preserving its underlying structure and still be representative in dimensions! These reduced datasets can then be fed into machine learning models to make predictions as normal, without the fear of any adverse effects from reducing the raw size of the original dataset. Our formal definition of PCA can therefore now be extended so that we can define PCA as the identification of a linear subspace of lower dimensionality where the largest variance in the original dataset is maintained. typical representations principal components lower Returning to our historic user-community movie ratings dataset, instead of eliminating Movie #2 entirely, we could seek to create a new feature that combines Movie #1 and Movie #2 in some manner. Extending this concept, we can create new features where each new feature is based on all the old features, and thereafter order these new features by how well they help us in predicting user movie ratings. Once ordered, we can drop the least important ones, thereby resulting in a reduction in dimensionality. So how does PCA achieve this? It does so by performing the following steps: 1. First, we standardize the original high-dimensional dataset. 2. Next, we take the standardized data and compute a covariance matrix that provides a means to measure how all our features relate to each other. 3. After computing the covariance matrix, we then find its and corresponding . Eigenvectors represent the principal components and provide a means to understand the direction of the data. Corresponding eigenvalues represent how much variance there is in the data in that direction. eigenvectors eigenvalues 4. The eigenvectors are then sorted in descending order based on their corresponding eigenvalues, after which the top eigenvectors are selected representing the most important representations found in the data. k 5. A new matrix is then constructed with these eigenvectors, thereby reducing the original -dimensional dataset into reduced dimensions. k n k Covariance matrix In mathematics, refers to a measure of how spread out a dataset is and is calculated by the sum of the squared distances of each data point, , from the mean , divided by the total number of data points, . This is represented by the following formula: variance xi x-bar N refers to a measure of how strong the correlation between two or more random variables is (in our case, our independent variables) and is calculated for variables and over dimensions: Covariance x y i If the covariance is positive, this implies that the independent variables are positively correlated. If the covariance is negative, this implies that the independent variables are negatively correlated. Finally, a covariance of zero implies that there is no correlation between the independent variables. Here, we are computing the covariance between all variables. A is a symmetric square matrix where the general element ( , ) is the covariance, , between independent variables and (which is the same as the symmetric covariance between and ). Note that the diagonal in a covariance matrix actually represents just the between those elements, by definition. covariance matrix i j cov(i, j) i j j i variance The covariance matrix is shown in the following table: Identity matrix An identity matrix is a square matrix in which all the elements along the main diagonal are 1 and the remaining elements are 0. Identity matrices are important for when we need to find all of the eigenvectors for a matrix. For example, a 3 x 3 identity matrix looks as follows: Eigenvectors and eigenvalues In linear algebra, eigenvectors are a special set of vectors whose remains unchanged when a linear transformation is applied to it, and only changes by a factor. In the context of dimensionality reduction, eigenvectors represent the principal components and provide a means to understand the direction of the data. direction scalar Consider a matrix, , of dimensions ( x ). We can multiply by a vector, (of dimensions x 1 by definition), which results in a new vector, (of dimensions x 1), as follows: A m n A x n b m In other words, However, in some cases, the resulting vector, , is actually a scaled version of the original vector, . We call this scalar factor , in which case the formula above can be rewritten as follows: b x λ We say that is an of matrix , and is an associated with . In the context of dimensionality reduction, eigenvalues represent how much variance there is in the data in that direction. λ eigenvalue A x eigenvector λ In order to find all the eigenvectors for a matrix, we need to solve the following equation for each eigenvalue, where is an identity matrix with the same dimensions as matrix : I A Once all of the eigenvectors for the covariance matrix are found, these are then sorted in descending order by their corresponding eigenvalues. Since eigenvalues represent the amount of variance in the data for that direction, the first eigenvector in the ordered list represents the principal component that captures the most variance in the original variables from the original dataset, and so on. For example, as illustrated in the figure below, if we were to plot a dataset with two dimensions or features, the first eigenvector (which will be the first principal component in order of importance) would represent the direction of most variation between the two features. The second eigenvector (the second principal component in order of importance) would represent the direction of second-most variation between the two features: To help choose the number of principal components, , to select from the top of the ordered list of eigenvectors, we can plot the number of principal components on the axis against the cumulative explained variance on the axis, as illustrated in the next figure, where the explained variance is the ratio between the variance of that principal component and the total variance (that is, the sum of all eigenvalues): k x y Using this as an example, we would select around the first 300 principal components, as these describe the most variation within the data out of the 3,000 in total. Finally, we construct a new matrix by projecting the original dataset into -dimensional space represented by the eigenvectors selected, thereby reducing the dimensionality of the original dataset from 3,000 dimensions to 300 dimensions. This preprocessed and reduced dataset can then be used to train machine learning models as normal. k PCA in Apache Spark Let’s now return to our transformed pipe-delimited user-community movie ratings dataset, , which contains ratings by 300 users covering 3,000 movies. We will develop an application in Apache Spark that seeks to reduce the dimensionality of this dataset while preserving its structure using PCA. To do this, we will go through the following steps: movie-ratings-data/user-movie-ratings.csv 1. First, let’s load the transformed, pipe-delimited user-community movie ratings dataset into a Spark dataframe using the following code. The resulting Spark dataframe will have 300 rows (representing the 300 different users) and 3,001 columns (representing the 3,000 different movies plus the user ID column): user_movie_ratings_df = sqlContext.read

.format('com.databricks.spark.csv').options(header = 'true',

inferschema = 'true', delimiter = '|')

.load('

print((user_movie_ratings_df.count(),

len(user_movie_ratings_df.columns))) 2. We can now generate feature vectors containing 3,000 elements (representing the 3,000 features) using 's , as we have seen before. We can achieve this using the following code: MLlib MLlib VectorAssembler feature_columns = user_movie_ratings_df.columns

feature_columns.remove('userId')

vector_assembler = VectorAssembler(inputCols = feature_columns,

outputCol = 'features')

user_movie_ratings_features_df = vector_assembler

.transform(user_movie_ratings_df)

.select(['userId', 'features']) 3. Before we can reduce the dimensionality of the dataset using PCA, we first need to standardize the features that we described previously. This can be achieved using 's estimator and fitting it to the Spark dataframe containing our feature vectors, as follows: MLlib StandardScaler standardizer = StandardScaler(withMean=True, withStd=True,

inputCol='features', outputCol='std_features')

standardizer_model = standardizer

.fit(user_movie_ratings_features_df)

user_movie_ratings_standardized_features_df =

standardizer_model.transform(user_movie_ratings_features_df) 4. Next, we convert our scaled features into an instance. A is a distributed matrix with no index, where each row is a vector. We achieve this by converting our scaled features data frame into an RDD and mapping each row of the RDD to the corresponding scaled feature vector. We then pass this RDD to 's (as shown in the following code), resulting in a matrix of standardized feature vectors of dimensions 300 x 3,000: MLlibRowMatrix RowMatrix MLlib RowMatrix() scaled_features_rows_rdd =

user_movie_ratings_standardized_features_df

.select("std_features").rdd

scaled_features_matrix = RowMatrix(scaled_features_rows_rdd

.map(lambda x: x[0].tolist())) 5. Now that we have our standardized data in matrix form, we can easily compute the top principal components by invoking the method exposed by 's . We can compute the top 300 principal components as follows: k computePrincipalComponents() MLlib RowMatrix number_principal_components = 300

principal_components = scaled_features_matrix

.computePrincipalComponents(number_principal_components) 6. Now that we have identified the top 300 principal components, we can project the standardized user-community movie ratings data from 3,000 dimensions to a linear subspace of only 300 dimensions while preserving the largest variances from the original dataset. This is achieved by using matrix multiplication and multiplying the matrix containing the standardized data by the matrix containing the top 300 principal components, as follows: projected_matrix = scaled_features_matrix

.multiply(principal_components)

print((projected_matrix.numRows(), projected_matrix.numCols())) The resulting matrix now has dimensions of 300 x 300, confirming the reduction in dimensionality from the original 3,000 to only 300! We can now use this projected matrix and its PCA feature vectors as the input into subsequent machine learning models as normal. 7. Alternatively, we can use 's estimator directly on the dataframe containing our standardized feature vectors to generate a new dataframe with a new column containing the PCA feature vectors, as follows: MLlib PCA() pca = PCA(k=number_principal_components, inputCol="std_features",

outputCol="pca_features")

pca_model = pca.fit(user_movie_ratings_standardized_features_df)

user_movie_ratings_pca_df = pca_model

.transform(user_movie_ratings_standardized_features_df) Again, this new dataframe and its PCA feature vectors can then be used to train subsequent machine learning models as normal. 8. Finally, we can extract the explained variance for each principal component from our PCA model by accessing its attribute as follows: explainedVariance pca_model.explainedVariance The resulting vector (of 300 elements) shows that, in our example, the first eigenvector (and therefore the first principal component) in the ordered list of principal components explains 8.2% of the variance; the second explains 4%, and so on. In this example, we have demonstrated how we can reduce the dimensionality of the user-community movie ratings dataset from 3,000 dimensions to only 300 dimensions while preserving its structure using PCA. The resulting reduced dataset can then be used to train machine learning models as normal, such as a hierarchical clustering model for collaborative filtering. If you found this article interesting, you can explore Machine Learning with Apache Spark Quick Start Guide to combine advanced analytics including machine learning, deep learning neural networks and natural language processing with modern scalable technologies including apache spark to derive actionable insights from big data in real-time. Machine Learning with Apache Spark Quick Start Guide can help you become familiar with advanced techniques for processing a large volume of data by applying machine learning algorithms.

UNSUPERVISED

unsupervised.com

-- employees

Since n.d.

COMPANY RANKING

UNSUPERVISED

EVERGREEN INDEX #11489

Representing this company? Click here to claim and customize this page!

Unsupervised WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

UNSUPERVISED

unsupervised.com

-- employees

Since n.d.

COMPANY RANKING

UNSUPERVISED

EVERGREEN INDEX #11489

RELATED COMPANIESRANK

STORIES ABOUT UNSUPERVISED

Representing this company? Click here to claim and customize this page!

Unsupervised WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!