Valiance offers AI-based customized solutions for Data Enrichment, Process Automation, Predictive Analytics \u0026 Customer Engagement that cut across businesses and industries

Valiance Solutions

I recently started PhD studies in machine learning at Ruhr University Bochum. One of the main research topics of the group I joined is called Slow Feature Analysis (SFA). To learn about a new topic, I like seeing examples and intuitive explanations if possible before submerging myself in mathematical rigor. I wrote this blog post for others who like approaching subjects in a similar manner, as I believe that SFA is quite powerful and interesting. In this post I’ll lead with a code example where SFA is applied, to help motivate the method. Then I’ll go into more detail about the math behind the method and finally provide links to other good resources on the material. 1. Determining a smooth latent variable SFA is an unsupervised learning method to extract the smoothest (slowest) underlying functions or from a time series. This can be used for dimensionality reduction, regression and classification. For example, we can have a highly erratic series that is determined by a nicer behaving latent variable. features Let’s start by generating time series and D S: <a href="https://medium.com/media/eb21f9e38f260e44cc189d0519724165/href">https://medium.com/media/eb21f9e38f260e44cc189d0519724165/href</a> This is known as the logistic map. By plotting the series , we can inspect its chaotic nature. The underlying time series that’s driving the behavior of the curve above is much simpler: S D <a href="https://medium.com/media/c1b2f8e7cef9961881db6fec59c437d1/href">https://medium.com/media/c1b2f8e7cef9961881db6fec59c437d1/href</a> How can we determine the simple, underlying driving force from the erratic time series? We can use SFA to determine the most slowly varying features of a function. In our case, we would start off with data like and end up with , without necessarily knowing beforehand how is generated. S D S Implementations of SFA aim at finding features of the input that are . But as we can see from our example, the driving force is highly non-linear! This can be remedied by doing a non-linear expansion of the time series first, then finding linear features of the expanded data. By doing this we find non-linear features of the original data. linear D S Let’s create a new multivariate time series by stacking time delayed copies of on it: S <a href="https://medium.com/media/74b5d3081145588c5f4a721a2a57ff93/href">https://medium.com/media/74b5d3081145588c5f4a721a2a57ff93/href</a> Next we do a cubic expansion of the data and extract the SFA features. A cubic expansion turns a 4 dimensional vector [ , , , ]ᵀ into the 34 element vector with elements , t²v, tvu, t_²_, tv, t_ for distinct ∈{a, b, c, d}. a b c d t_³ t, u, v <a href="https://medium.com/media/69ae30d6cb09be6d313ae5a161f42fe4/href">https://medium.com/media/69ae30d6cb09be6d313ae5a161f42fe4/href</a> Keep in mind that the best number of time delayed copies to be added varies from problem to problem. Alternatively, if the original data is too high-dimensional then dimensionality reduction needs to be done, for example with . Principal Component Analysis Consider thus the following to be the hyperparameters of the method: the method of dimensionality expansion (reduction), the output dimension after expansion (reduction) and the number of slow features to be found. Now, after adding the time delayed copies, the length of the time series changed from 300 to 297. The corresponding length of the slow feature time series is thus 297 as well. For nicer visualization here, we turn it to length 300 by prepending the first value to it and appending the last value two times. The features found by SFA have zero mean and unit variance, so we normalize as well before visualizing the results. D <a href="https://medium.com/media/ebfa6a0a4a087c1a9943cfc9875099cc/href">https://medium.com/media/ebfa6a0a4a087c1a9943cfc9875099cc/href</a> Even considering only 300 data points, the SFA features manage to almost completely recover the underlying source - which is quite impressive! <a href="https://medium.com/media/538bc4a47e52faeb3abf5f8c1eb6399d/href">https://medium.com/media/538bc4a47e52faeb3abf5f8c1eb6399d/href</a> 2. So what’s going on under the hood? Theoretically, the SFA algorithm accepts as input a (multivariate) time series and an integer indicating the number of features to extract from the series, where is less than the dimension of the time series. The algorithm determines functions X m m m such that the average of the squared time derivative of two successive time points of each is minimized. Intuitively, we want to maximize the slowness of the features: yᵢ where the dot indicates the time derivatives, in the discrete case: The objective function (1) measures the slowness of the feature. The zero-mean constraint (2) makes the second moment and variance of the features equivalent and simplifies the notation. The unit variance constraint (3) discards constant solutions. The final constraint (4) decorrelates our features and induces an ordering on their slowness. This means that we first find the slowest feature, then we find the next slowest feature that is orthogonal to the one before it and so on. Decorrelating the features ensures that we capture the most information. In the following I glance over important details and skip steps, but I want to include it for completeness. I suggest looking at the links below as well for more thorough explanations. Let’s consider only linear features: The time series X can be “raw data” or its non-linear expansion, see example above. Remember that even though those are linear features of the expanded data, they can still be non-linear features of the original data. Assuming zero-mean , the linear features are found by solving the = . We determine eigenvalue-eigenvector tuples ( , ) such that = where we have X generalized eigenvalue problem AW BWΛ m λᵢ Wᵢ A Wᵢ λᵢ B Wᵢ, The scalars signify the slowness of the features, i.e. the smaller the , the more slowly varying the corresponding . If you are familiar with the generalized eigenvalue problem, note that the eigenvalues here are increasing — not decreasing. Lastly, the eigenvectors are the transformation vectors that define our learned features. λᵢ λᵢ yᵢ Wᵢ 3. Further reading The original paper: https://www.ini.rub.de/PEOPLE/wiskott/Reprints/WiskottSejnowski-2002-NeurComp-LearningInvariances.pdf Application of SFA to classification: http://cogprints.org/4104/1/Berkes2005a-preprint.pdf Example above is adapted from: http://mdp-toolkit.sourceforge.net/examples/logmap/logmap.html

A brief introduction to Slow Feature Analysis

Introduction We now have an arsenal of algorithms that can handle any problem we throw at them, thanks to breakthroughs in machine learning and deep learning. But the majority of these sophisticated and complex algorithms have a problem; they need to be simpler to understand. Nothing compares to the simplicity and interpretability of linear regression when it comes to the interpretability of machine learning models. However, there may be some problems with the interpretability of the linear regression, particularly if the multicollinearity assumptions of the linear regression are broken. In this article, we will be looking at the concept of multicollinearity, its detection & treatment through a vital statistical metric - “Variance Inflation Factor”. The concept behind Multicollinearity The variance inflation factor is a statistical metric used to detect multicollinearity in supervised predictive models. First of all, let us understand the concept of multicollinearity before jumping inside the intuition of VIF. Consider that we need to build a regression model to predict the salary of a person. We have the following independent variables in our dataset- Age Years of experience Job Gender City Cost of living index in the place they are residing Our data looks like this- So, this is a traditional multiple linear regression problem and any multiple linear regression can be represented using the mathematical formula- Y = m1x1 + m2x2 + m3x3+.....+mnxn + c Where, Y is the target variable x1, x2, x3…xn are all independent variables m1, m2, m3, mn are all slopes of x1, x2, x3, and xn respectively c is the intercept According to our use case, we define our problem as- Salary = (0.5 * Age) + (0.6 * Years of experience) + (0.1 * Job) + (0.2 * Gender) + (0.6*City) + 0.7(Cost of living index) (Just assume some random values for the coefficients or slopes to understand the concept in a simple way). The above formula intrinsically means that- Whenever Age is increased by one unit, Salary will increase by 0.5 units keeping all other variables constant. Whenever the Cost of living index is increased by one unit, the Salary will increase by 0.7 units keeping all other variables constant. Now, we can logically think about a few things according to the real-life scenario- All of the above-mentioned independent variables seem to be having some relationship with the target variable (Salary). However, we know that Age and Years of experience are having a straightforward linear relationship with each other. Assuming that a person starts to work around the age of 25, His/Her working experience is going to be increased by one year whenever his/her age increases by one year. The primary objective of any regression model is to learn about the variance in the dataset. Thereby, finding how each of the independent variables affects the target variable (by identifying a proper coefficient value)  and predicting the target for unseen data points. Whenever we use a specific independent variable, it should be capable of providing some information to the model regarding the target variable that other variables are not able to capture. If 2 variables are explaining the same variance (affecting the target variable in an approximately unique way) then there is no point in keeping both the independent variables since it will be an additional burden to the model that can increase the time and space complexity without any usage. For example, Consider that you have 4 friends - John, James, Joseph, and Jack. All of them went to watch a movie. Unfortunately, you were not able to get the tickets. So, your friends said that they will watch the movie and explain the story scene by scene. The movie’s duration was 3 hours. Assume that- John explained the story of the first 60 minutes James explained the story from the 61st minute to the 120th minute Joseph  explained the story from the 121st  minute to the 180th minute Now, it's Jack’s turn. He doesn't know that the other 3 people already covered all parts of the story. He again explained to you the story from the 121st  minute to the 180th minute. Now, You have heard the last 60 minutes’ story twice. Either Joseph or Jack was enough to explain the last 60 minutes. Since they both explained the same thing, there is no additional information that they can provide to you. Technically, you heard the story for 240 minutes, although the entire information can be covered in 180 minutes. The time and energy spent by Jack went in vain and you also lost time by hearing unwanted information. So, Jack and Joseph are multicollinear !!! Multicollinearity refers to a situation in which more than two explanatory variables in a multiple regression model are highly linearly related. You will have an obvious doubt about what the issue is if 2 variables are highly collinear. We discussed the coefficients in the multiple linear regression formula, According to our problem statement, it is - Salary = (0.5 * Age) + (0.6 * Years of experience) + (0.1 * Job) + (0.2 * Gender) + (0.6*City) + 0.7(Cost of living index) Here, we know that Age and Years of experience are highly correlated. If we remove one of these collinear variables from the model then we can see that the model adapts itself into a new set of coefficients for all of the independent variables. Sometimes, if there is approximate collinearity, it might not lead to a drastic decrease in the predictive ability of the model but it will certainly increase the time and space complexity of the model. However, if multiple variables are perfectly correlated then it may lead to the collapse of the model leading to poor performance. For maintaining the assumptions of linear models, we should never take a risk by incorporating multiple collinear variables. A linear model always expects that all of the predictor variables are perfectly independent ie a specific predictor variable should not be able to get explained by establishing some relationships among certain other predictor variables in the data. In such a scenario, all of the coefficients will be very efficient leading to a generalized hyperplane that can maintain the consistency of prediction in any unseen set of data points The basic intuition behind VIF The next question is - How can we identify whether  multicollinearity is present in a dataset or not ? The answer is “VIF”. Now, let's understand the intuition behind VIF. We discussed earlier that a specific predictor variable should not be able to get explained by establishing some relationships among certain other predictor variables in the data. Before moving to the example, let's understand the meaning of R Squared. R Squared aka Coefficient of determination is one of the most widely used evaluation metrics for linear regression models. R squared is considered a goodness of fit metric which most of the time ranges around 0 to 1. The higher value of R Squared is examined as higher the coherence and predictive ability of the model. The VIF is represented with a mathematical formula- VIF - 1/(1-R Squared). In order to identify the multicollinearity, we need to derive the VIF of all the predictor variables present in the dataset. Let’s say, if we need to calculate the VIF for Age then we should consider “Age” as the target variable, and all other predictor variables as independent variables and should design a multiple linear regression model. For example, For finding the VIF of Age, Let us build a model like this- Age  =  (0.6 * Years of experience) + (0.1 * Job) + (0.2 * Gender) + (0.6*City) + 0.7(Cost of living index) We will be getting an R Squared value for this model. Let's assume that the R Squared value we got is 0.85 Now, VIF = 1 /(1- 0.85) VIF = 1/ 0.15 VIF = 6.67 One thing we should note here is that if the R Squared is a large number (near 1). Then denominator of the VIF formula will become a small number (Because the denominator is 1- R Squared) If the denominator is a small number then the value of VIF will be a large number (Because we are dividing 1 by the denominator ie  1 - R Squared). So, R squared is proportional to VIF. For example, We got a high VIF value of 6.67 earlier because we had a high R Squared value (0.85). If we had a low R Squared value like 0.2 then our VIF would have been low like- VIF = 1 /(1- 0.2) VIF = 1/ 0.8 VIF = 1.25 Indirectly, this conveys that- If VIF is a low number then that means other predictor variables together are not properly able to explain the variance and establish a stable relationship with our specified response variable. In other words, “The response variable is statistically independent or isolated”. If VIF is a high number then that means another predictor variable together is somewhat properly able to explain the variance and establish a stable relationship with our specified response variable. In other words, “The response variable is statistically not perfectly independent”. We need to use a similar procedure as used for “Age” to find the VIF of all other independent variables. Usually, a VIF value greater than 5 is not considered as good. A rule of thumb for interpreting the variance inflation factor: 1 = not correlated. Between 1 and 5 = moderately correlated. Greater than 5 = highly correlated. Consider that, we got the VIF for each variable as mentioned in the table- It is very clear that “Age” and “Years of experience” are highly inflated and correlated since their value is above 5. Hence, we can remove one of these variables from this model (Preferably, we can remove “Age” as it has the highest VIF for now). So, after removing the “Age”, the number of independent variables has come down to 5 from 6. We can repeat the above-mentioned process one more time- Now, we can see that the VIF of Years of experience has come down. Also, the City and Cost of living seem to be inflated. Hence, we can drop one of those variables from the model. In the next iteration of modelling, we got the VIF values as shown below- Now, all of the variables look non-correlated and independent. Hence, we can proceed with the actual modeling considering “Salary” as the target variable. We started with 6 independent variables for the regression model. With the help of VIF, we were able to identify multicollinearity in the data and identified 2 variables for dropping from the model. Now, our regression model will be more generalized, accurate and less complex. Conclusion As a learning from the whole article, we can summarize the entire content via the following points- In a multivariate regression model, the variance inflation factor (VIF) measures the multicollinearity among the independent variables. While multicollinearity does not lower the model's explanatory power, it does reduce the statistical significance of the independent variables, therefore detecting it is crucial. A large VIF on an independent variable denotes a strongly collinear link to the other variables, which should be taken into account or accounted for in the model's structure and independent variable selection. In this article, we discussed one of the most important but foundational fundamental concepts in applied statistics. This VIF metric is available as an inbuilt library in most data science-oriented programming languages such as Python or R. Hence, it is easy to implement this once you understand the theoretical intuition. I have added the links to some of the advanced materials in the references section, where you can dive deep into the complex calculations if you are interested. References Statistics Solutions. (2013). . [online] Available at: What is Logistic Regression? — Statistics Solutions https://www.statisticssolutions.com/what-is-logistic-regression/. Jason Brownlee (2016). . [online] Machine Learning Mastery. Available at: Logistic Regression for Machine Learning https://machinelearningmastery.com/logistic-regression-for-machine-learning/. ]Rogel-Salazar, J. (2017). . Boca Raton, Fl: CRC Press, Taylor & Francis Group. Data science and analytics with Python ‌Miller, T.W. (2015). . Upper Saddle River: Financial Times/Prentice Hall. Marketing data science: modelling techniques in predictive analytics with R and Python

Variance Inflation Factor - A Pertinent Statistical Metric for the Discernment of Multicollinearity 

From mathematical formalism to a description of available solutions (+ Kaggle benchmark) by Julien Jouganous , Romain Savidan and Axel Bellec (Data Scientists at Cdiscount ) — Photo by on the hyperparameters jungle chuttersnap Unsplash What is AutoML and why do we need it? Auto ML is the fact of simplifying data science projects by automating the tasks. The below presents a classical machine learning pipeline for supervised learning tasks. machine learning figure 1 Figure 1: A machine learning process. In the process presented above, several blocks must be tuned to extract most of the predictive power of the data: First of all, you have to select relevant data that potentially explain the target you want to predict. Once these raw data are extracted, you generally need to process them. A non-exhausive list of preprocessing steps is given below: text vectorization categorical data encoding ( , one hot) e.g. missing values and outliers processing rescaling ( , normalization, standardization, min-max scaling) e.g. variables discretization dimensionality reduction Last but not least, you have to choose a machine learning algorithm depending on the kind of task you are facing: supervised or not, classification or regression, online or batch learning… Most machine learning algorithms need parameterization and even if some empirical strategies can help ( provides guidelines for ), this optimization is complex and there is generally no deterministic way to find the optimal solution. But that is only the tip of the iceberg as the whole process involves choices and manual interventions that will impact the efficiency of the machine learning pipeline. this article XGBoost Welcome to the hyperparameters jungle! To illustrate this, consider the classical : using the content of emails, you want to predict whether an email is a spam or not. To do that, we have a database with emails tagged as SPAM or NOT_SPAM so this is a supervised binary classification problem. spam detection problem Figure 2: Spam detection As machine learning algorithms generally deal with numerical vectors, the first step is to vectorize our emails. We can use a approach. But before, we have to tokenize the emails (turn sentences into lists of words). We may also want to apply more advanced text preprocessing such as lowercasing, , or spelling correction. tf-idf stemming, lemmatization Now the emails are tokenized and preprocessed, we can vectorize them computing the tf-idf weights but here again the algorithm needs to be parameterized. Indeed, we generally want to specify stopwords, truncate the document frequency range (_i.e._remove from the dictionnary rare and/or frequent words), or even include (sequences of consecutive words) into our dictionnary. n-grams n We have a vectorized representation of our emails. We may or may not want to apply a dimensionality reduction technique such as PCA. We may also want to add other features such as the number of words in the mail, the average word length for example. It is now time to train our machine learning model but which one will we use? For this kind of batch supervised binary classification task, we are spoilt for choice: logistic regression, naive Bayes classifier, random forest, gradient boosting, neural nets… We probably want to try and compare some of these algorithms but here let say that we have chosen the . How many decision trees do we need? What is the optimal tree depth? How do we control the leaves granularity? How many samples and features do we want to include in each tree? random forest Just have a look at the to have an overview of the amount of hyperparameters you can tune. scikit learn documentation Finally, each component of the pipeline involves choices and parameters that can impact the performances of the algorithm. Auto ML simply consists in automating all these steps and optimizing data preprocessing, algorithm choice and hyperparameters tuning. Hyperparameters optimization strategies Mathematical formalism Consider 𝚨 the possible strategies space. It may sound a bit abstract but let’s just say that each element ∈ 𝚨 is a parameterization of the machine learning pipeline from the data processing to the model hyperparameters tuning. The auto ML process aims at finding * such that: A A where and are train and validation sets obtained via cross validation and K-fold is the loss for pipeline trained on training dataset and evaluated on validation dataset . A i i Random search, grid search The first and simplest tuning approach that may come to mind is just picking randomly several parameterizations of the pipeline and keep the best one. A variant called grid search consists in testing the nodes of a discretization grid of 𝚨. For more details, these algorithms are implemented in scikit learn, is the documentation. here These Monte-Carlo-like algorithms can be a good starting point for low dimensional problems but, as we saw before, we generally have a consequent amount of parameters to tweak so they will converge very slowly. Moreover, in the grid search case, if optimal strategies are far from the nodes you defined, you have no chance to discover them. Metaheuristics To overcome the poor convergence properties of random search, one may want to implement more “clever” algorithms. Metaheuristics are a class of generic optimization techniques based on the exploitation / exploration trade-off and often mimicking biological or physical phenomena. They can be efficient to solve convex and non-convex problems. Among the tons of metaheuristics, we can mention for instance: is a method inspired from annealing in metallurgy that consists in alternating cooling and heating phases to minimize the material enregy modifying its micro structure and physical properties. In analogy, we explore the parameters space to try to minimize the system energy (our objective function). We also introduce the temperature of the material . At each step, we test a configuration in the neighborhood of the previous one. The higher the temperature is, the more the atoms are free to move and the larger the neighborhood is. Consequently, at the beginning of the process, when is high, we explore pretty much all the search space and as this temperature decreases, we focus more and more on the areas deemed promising. See for a more detailled description. Simulated annealing T T here uses a population of candidate solutions (or particles) to explore the space 𝚨. Each particle moves in 𝚨 depending on its current location, its best known past position and the whole population’s best position. The trade-off between exploitation and exploration can be tuned by playing whith the relative importance of these different factors. gives more details about the algorithm and some variants. Particle swarms This paper is a family of algorithms inspired by biological evolution. We consider a population of candidate solutions (or individuals) and we apply simplified evolution laws to have them optimize the objective function called . At each step or generation, we select the best (meaning here the fittest) individuals. The next generation is built via the reproduction of these selected individuals. During this step we apply genetic operations: (exploitation) is the process by which the parents features are combined to form children solutions, whereas , introduce random (exploration) perturbations. This way, the average population’s fitness is supposed to improve from one generation to the next and the fittest individual to converge to a good solution of your optimization problem. Evolutionary algorithms fitness recombination mutations Bayesian approaches Each evaluation of the objective function we try to minimize can be very expensive as we have to train and evaluate a machine learning pipeline. Bayesian optimization provides an efficient method to explore the parameter set and converge quickly to a convenient solution. In short: it helps focus the exploration of the parameter space 𝚨 on promising areas. The below ( ) gives a simple example in one dimension (in this case, we want to maximize the function): figure 3 source Figure 3: Bayesian optimization. In this example, the objective function is approximated through a model. This modelling technique provides a probability density function for the values of based on priors (points where the value of the function is known). This density is represented by the blue area and the mean is plotted in solid line. The distribution is very peaked (small variance) close to observations whereas we are not confident (high variance) with our estimations of in unsampled areas. f Gaussian Process regression f f At each step, the new point to explore is selected as the maximum of a function called (the green one). This function synthesizes the knowledge and the uncertainty about the function to optimize. It is high where the predicted objective function is high (or low if we want to minimize ) and where the uncertainty is also high. In other words, it is high where there is a good chance we find better solutions for our optimization problem. activation function a priori f The opposing forces In this section we provide a non exhaustive list of auto ML solutions currently available. We arbitrarily chose a few summary features to compare them: Is it a open source package? Which backend models implementation and hyperparameters tuning algorithms are used? How many contributors? These features, when they where available, were gathered in the table below. Note that it is a snapshot of these libraries at the time this article was written. Figure 4: AutoML solutions snapshot (05/2018) We could play with a few of these auto ML solutions. Our tests are summarized in the following section. Benchmark Protocol We chose four frameworks to test. The main selection criteria where the diversity and community involved (number of contributors, recency of the last commit). AWS Machine Learning (version available on 2017/12/05) Autosklearn 0.2.0 H2O 3.16 TPOT 0.9.1 These packages were benchmarked on three classical datasets available on : Kaggle A problem: binary classification Titanic survival prediction A challenge: based on the MNIST dataset multiclass classification Digit recognizer A problem: regression House prices This way, we can compare the results to the general leaderboards (and an army of talented data scientists) on different kinds of supervized learning problems. We trained each solution on the training set during on server and then submitted the predictions on the test set on Kaggle. The results are provided in the figures below. 1 hour 32 cores — 256Go RAM Results For each challenge, we plotted the metric value depending on the ranking of the solution for the whole leaderboard and located the libraries we tested on these plots. Basic baseline solutions are also provided. Titanic challenge (binary classification): metric: (percentage of correctly predicted labels) accuracy number of challengers: 11164 teams (11700 competitors) number of features: 9 number of samples: 891 in the training set, 418 in the test set baseline: predictions based on gender (men die and women survive) Figure 5: Titanic challenge benchmark This curve shows for instance that the top 20% solutions reached more than 80% accuracy. MNIST (multiclass classification): metric: accuracy number of challengers: 2288 teams (2345 contributors) number of features: 784 number of samples: 42000 in the training set, 28000 in the test set baseline: k nearest neighbors with k=1 Figure 6: MNIST challenge benchmark House prices (regression): metric: root mean squared error (RMSE) number of challengers: 5123 teams (5397 competitors) number of features: 268 number of samples: 1460 in the training set, 1459 in the test set baseline: linear regression (area, number of bedrooms, year and month of sale) Figure 7: House prices challenge benchmark Conclusion In practice, the frameworks needed some and (binarization, null values handling…). Only commercial solutions offer the complete pipeline, from features engineering to models deployment. open source data preparation cleaning Moreover, hyperparameters tuning algorithms sometimes need to be . It is for instance the case for the genetic algorithm used by TPOT. parameterized Open source Auto ML solutions data scientists (top 20–80% on the leaderboard depending on the challenge) but can help get convenient levels of accuracy with a minimum effort and amount of time. does not perform better than human Keep in mind that feature engineering is the key: “One of the holy grails of machine learning is to automate more and more of the feature engineering process” — s, Pedro Domingo A Few Useful Things to Know about Machine Learning This is where most of the effort in a machine learning project goes because it is a very time-consuming task. Further readings about particle swarms optimization about simulated annealing about bayesian optimization This post was originally published on . Cdiscount Techblog

A brief overview of Automatic Machine Learning solutions (AutoML)

Ingestion and Processing of Data For Big Data and IoT Solutions

Portfolio Management: All The Ways AI Is Transforming Modern Asset Strategies

পোর্টফোলিও ম্যানেজমেন্ট: সমস্ত উপায় এআই আধুনিক সম্পদ কৌশলগুলিকে রূপান্তরিত করছে

Portfoliomanagement: Wie KI moderne Anlagestrategien verändert

Gestión de carteras: todas las formas en que la IA está transformando las estrategias de activos modernas

Gestion de portefeuille : toutes les façons dont l'IA transforme les stratégies d'actifs modernes

पोर्टफोलियो प्रबंधन: वे सभी तरीके जिनसे AI आधुनिक परिसंपत्ति रणनीतियों को बदल रहा है

ポートフォリオ管理: AI が現代の資産戦略をどのように変革しているか

포트폴리오 관리: AI가 현대 자산 전략을 변화시키는 모든 방법

Gerenciamento de portfólio: todas as maneiras pelas quais a IA está transformando as estratégias modernas de ativos

Управление портфелем: все способы, которыми ИИ трансформирует современные стратегии управления активами

Portföy Yönetimi: Yapay Zekanın Modern Varlık Stratejilerini Dönüştürdüğü Tüm Yollar

Quản lý danh mục đầu tư: Tất cả các cách AI đang chuyển đổi các chiến lược tài sản hiện đại

投资组合管理：人工智能正在改变现代资产战略

A Sparse Bayesian Learning for Diagnosis of Nonstationary and Spatially Correlated Faults:References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jihoon Chung;
(2)  Zhenyu (James) Kong. Table of Links Abstract & Introduction
Review of Related Work
Proposed Research Methodology
Numerical Case Studies
Real-World Simulation Case Studies
Conclusion
References II. REVIEW OF RELATED WORK The related existing studies of fault diagnosis in manufacturing systems are reviewed in Section II A. Then, the literature related to sparse Bayesian learning is provided in Section II-B. Afterward, the research gaps in the current work are identified in Section II-C. A. Fault Diagnosis Methodologies in Manufacturing Systems Numerous studies have focused on fault diagnosis methodologies for manufacturing systems, utilizing the fault-quality model outlined in Eq. (1). [14] developed a PCA-based orthogonal diagonalization strategy to transform the measurement data. It enabled the estimation of the variance of KCCs in a multistation assembly system. [13] presented a fault diagnosis method in the multistation assembly systems integrating the state space model of the process and matrix perturbation theory. [25], [26] proposed a fault diagnosis method in the machining process considering the process physics regarding how fixtures generate the patterns. Using this method, root cause identification was conducted sequentially. The approaches mentioned above assume that the number of measurements is greater than the number of KCCs (i.e., M > N in Eq. (1)). However, this assumption may not always be consistent with industrial practice. These approaches become ineffective when this assumption is violated because the fault-quality linear model leads to an underdetermined system, resulting in a nonunique solution. To overcome an underdetermined system in the fault-quality linear model, sparse learning can be utilized, which has gained considerable attention in fault diagnosis and detection within manufacturing systems. For fault diagnosis in the manufacturing system, [27] developed a fault diagnosis method using dictionary learning and sparse representationbased classification. [28] proposed a novel root cause diagnostic framework satisfying the assumption that sparse inputs affect the process output. [29] proposed a group-sparsity learning approach for bearing fault diagnosis. In addition to fault diagnosis and detection in the general manufacturing system, sparse learning has been widely utilized to address the issue of an underdetermined system in the multistation assembly system. In particular, sparse Bayesian learning has been widely utilized to incorporate the sparsity of process faults as the prior distribution. [5] proposed a fault diagnosis approach by enhancing the relevant vector machine to detect process faults using the sparse estimate of the variance change of KCCs. [8] developed a spatially correlated sparse Bayesian learning to consider the spatial correlation of KCCs in sparse estimation. The work identifies the KCCs with mean shifts. [10] proposed a Bayesian model to identify the KCCs that have variance increases in a multistation manufacturing system using the sparse variance component prior. [9] developed a novel sparse Bayesian learning to figure out the KCCs with mean shifts by considering the temporal correlation of KCCs and the prior knowledge of process faults. B. Sparse Bayesian Learning After the introduction of sparse Bayesian learning (SBL) by [30], many researchers have extended this approach significantly. For instance, [31] was the first to apply SBL to sparse estimation for the single measurement vector model given in Eq. (1). Subsequently, [32] further extended it to the MMV model (Eq. (2)) by developing the MSBL algorithm under the common support assumption. The notable advantage of SBL and MSBL is that their global minimum is always the sparsest solution, while that of the minimization-based sparse algorithms [17] [33] is usually not the sparsest solution [16], [34]. Based on the MMV model, many previous studies exploit the spatial correlation in solution vectors (i.e., rows in X in Eq. (2)). [35] proposed a block structure to exploit the intra-block correlation for sparse estimation. [36] developed a Bayesian method for recovery of block-sparse solution whose block-sparse structures are entirely unknown. [37] modeled the spatial structure of the solution as Markov dependency by the Beta process. Besides considering the spatial correlation of sparse solutions, work that considers nonstationary sparse solutions has been studied recently under the MMV model in the SBL framework. [38] developed a method using the Dirichlet process to cluster the measurements into groups with common sparsity patterns. Compared to [38], [2] proposed a more general method having two sparsity components: a commonly shared sparsity and an individual sparsity to deal with outliers that deviated from the uniform sparsity pattern in each group. C. Research Gap Analysis The research presented in Section II-A focuses on utilizing the sparsity of process faults for accurate fault diagnosis when low dimensional measurements exist in actual manufacturing systems. However, there is a lack of efforts to identify process faults by considering the spatial correlation and the nonstationary process faults, which is common in industrial practice. Section II-B introduces methods in SBL. It introduces the work concerning the spatial correlation in the solution vector and dealing with the nonstationary sparse solution individually. However, it still lacks the work that uses both properties simultaneously. Therefore, this paper proposes a novel SBL method considering both properties for accurate fault diagnosis. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jihoon Chung; (2)  Zhenyu (James) Kong. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jihoon Chung; (2)  Zhenyu (James) Kong. Table of Links Abstract & Introduction Review of Related Work Proposed Research Methodology Numerical Case Studies Real-World Simulation Case Studies Conclusion References Abstract & Introduction Abstract & Introduction Review of Related Work Review of Related Work Proposed Research Methodology Proposed Research Methodology Numerical Case Studies Numerical Case Studies Real-World Simulation Case Studies Real-World Simulation Case Studies Conclusion Conclusion References References II. REVIEW OF RELATED WORK The related existing studies of fault diagnosis in manufacturing systems are reviewed in Section II A. Then, the literature related to sparse Bayesian learning is provided in Section II-B. Afterward, the research gaps in the current work are identified in Section II-C. A. Fault Diagnosis Methodologies in Manufacturing Systems Numerous studies have focused on fault diagnosis methodologies for manufacturing systems, utilizing the fault-quality model outlined in Eq. (1). [14] developed a PCA-based orthogonal diagonalization strategy to transform the measurement data. It enabled the estimation of the variance of KCCs in a multistation assembly system. [13] presented a fault diagnosis method in the multistation assembly systems integrating the state space model of the process and matrix perturbation theory. [25], [26] proposed a fault diagnosis method in the machining process considering the process physics regarding how fixtures generate the patterns. Using this method, root cause identification was conducted sequentially. The approaches mentioned above assume that the number of measurements is greater than the number of KCCs (i.e., M > N in Eq. (1)). However, this assumption may not always be consistent with industrial practice. These approaches become ineffective when this assumption is violated because the fault-quality linear model leads to an underdetermined system, resulting in a nonunique solution. A. To overcome an underdetermined system in the fault-quality linear model, sparse learning can be utilized, which has gained considerable attention in fault diagnosis and detection within manufacturing systems. For fault diagnosis in the manufacturing system, [27] developed a fault diagnosis method using dictionary learning and sparse representationbased classification. [28] proposed a novel root cause diagnostic framework satisfying the assumption that sparse inputs affect the process output. [29] proposed a group-sparsity learning approach for bearing fault diagnosis. In addition to fault diagnosis and detection in the general manufacturing system, sparse learning has been widely utilized to address the issue of an underdetermined system in the multistation assembly system. In particular, sparse Bayesian learning has been widely utilized to incorporate the sparsity of process faults as the prior distribution. [5] proposed a fault diagnosis approach by enhancing the relevant vector machine to detect process faults using the sparse estimate of the variance change of KCCs. [8] developed a spatially correlated sparse Bayesian learning to consider the spatial correlation of KCCs in sparse estimation. The work identifies the KCCs with mean shifts. [10] proposed a Bayesian model to identify the KCCs that have variance increases in a multistation manufacturing system using the sparse variance component prior. [9] developed a novel sparse Bayesian learning to figure out the KCCs with mean shifts by considering the temporal correlation of KCCs and the prior knowledge of process faults. B. Sparse Bayesian Learning B. Sparse Bayesian Learning After the introduction of sparse Bayesian learning (SBL) by [30], many researchers have extended this approach significantly. For instance, [31] was the first to apply SBL to sparse estimation for the single measurement vector model given in Eq. (1). Subsequently, [32] further extended it to the MMV model (Eq. (2)) by developing the MSBL algorithm under the common support assumption. The notable advantage of SBL and MSBL is that their global minimum is always the sparsest solution, while that of the minimization-based sparse algorithms [17] [33] is usually not the sparsest solution [16], [34]. Based on the MMV model, many previous studies exploit the spatial correlation in solution vectors (i.e., rows in X in Eq. (2)). [35] proposed a block structure to exploit the intra-block correlation for sparse estimation. [36] developed a Bayesian method for recovery of block-sparse solution whose block-sparse structures are entirely unknown. [37] modeled the spatial structure of the solution as Markov dependency by the Beta process. Besides considering the spatial correlation of sparse solutions, work that considers nonstationary sparse solutions has been studied recently under the MMV model in the SBL framework. [38] developed a method using the Dirichlet process to cluster the measurements into groups with common sparsity patterns. Compared to [38], [2] proposed a more general method having two sparsity components: a commonly shared sparsity and an individual sparsity to deal with outliers that deviated from the uniform sparsity pattern in each group. C. Research Gap Analysis C. Research Gap Analysis The research presented in Section II-A focuses on utilizing the sparsity of process faults for accurate fault diagnosis when low dimensional measurements exist in actual manufacturing systems. However, there is a lack of efforts to identify process faults by considering the spatial correlation and the nonstationary process faults, which is common in industrial practice. Section II-B introduces methods in SBL. It introduces the work concerning the spatial correlation in the solution vector and dealing with the nonstationary sparse solution individually. However, it still lacks the work that uses both properties simultaneously. Therefore, this paper proposes a novel SBL method considering both properties for accurate fault diagnosis.

Sparse Bayesian Learning: Review of Related Work

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
Authors:
(1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr;
(2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction
Organization and notation
Problem Setting and Preliminaries
Generalized Wiener algebra and Fréchet derivative
Characterization of coefficients
Coefficients from an ergodic process
Conclusion & References Conclusion In this mathematical study, we delve into the realm of statistical inference and introduce a novel approach to variational non-Bayesian inference. Most significantly, we propose a new method for uniquely determining the hidden PDF solely from a random sample while leveraging theory. Beyond merely approximating the shape of the unrevealed PDF, we yield results that provide practical assistance in inferring important moments, such as the mean and variance, from the estimated PDF. These days, methods utilizing artificial intelligence are widely employed, but they generally exhibit a drawback of depending on initial conditions, i.e., a prior distribution, and iterative computations, i.e., a backpropagation. The limitations of such approaches lie in their reliance on these conditions to achieve global optimization. We emphasize the significant contribution of the proposed method to predictive and classification models, even without any information on populations, highlighting its potential applications in various domains such as finance, economics, weather forecasting, and machine learning, all of which present unique challenges. The most well-known method for approximating hidden PDFs is the variational approach in a Bayesian context. In contrast, our study extends this problem by seeking to precisely determine unknown PDFs through a system of equations. Our approach includes proving the Fréchet differentiability of entropy to establish the uniqueness of the energy function space in the Wiener algebra. We then derive the unique determination of the energy function through the minimization of KL-divergence. Leveraging the Ergodic theorem, we elucidate that solutions to equations comprising polynomial function series are the coefficients of the energy function and numerically substantiate the convergence of partial sums of the energy function obtained from a finite number of equations. In summary, our mathematical exploration has unveiled the potential of variational Non-Bayesian inference in Wiener space. We anticipate that these mathematical ideas can offer an innovative framework for probability density estimation and predictive modeling. Ultimately, our research emphasizes its potential contribution to reshaping mathematical horizons and expanding the boundaries of knowledge in the field of statistical methodology. REFERENCES [1] K. Bharath A. Saha and S. Kurtek, A geometric variational approach to bayesian inference, Journal of the American Statistical Association 115 (2020), no. 530, 822–835. [2] P. Alquier and J. Ridgway, Concentration of tempered posteriors and of their variational approximations, The Annals of Statistics 48 (2020), no. 3, 1475–1497. [3] A. R. Barron and C. Sheu, Approximation of density functions by sequences of exponential families, The Annals of Statistics 19 (1991), no. 3, 1347–1369. [4] C. M. Bishop and M. E. Tipping, Probabilistic principal component analysis, J. R. Statist. Soc. B 61 (1999), no. 3, 611–622. [5] F. W. Olver, D. Lozier, R. F. Boisvert and C. W. Clark, NIST Handbook of Mathematical Functions, Cambridge University Press, 2010. [6] C Brouder and F Patras, One-particle irreducibility with initial correlations, (2011). [7] I. Csiszar, I-divergence geometry of probability distributions and minimization problems, Ann. Probab. 3 (1975), no. 1, 146–158. [8] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2 (1989), 303–314. [9] L. Grafakos, Classical and Modern Fourier Analysis, Prentice Hall, 2003. [10] D. E. Rumelhart, G. E. Hinton and R. Williams, Learning representations by back-propagating errors, Nature 323 (1986), 533–536. [11] G. E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, University of Toronto, 2010. [12] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), no. 2, 251–257. [13] Y. Katznelson, An introduction to harmonic analysis (third ed.), New York: Cambridge Mathematical Library, 2004. [14] D. M. Blei, A. Kucukelbir and J. D. McAuliffe, Variational inference: A review for statisticians, Jour. of the American Statistical Association 112 (2017), no. 518, 859–877. [15] S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statistics 22 (1951), no. 1, 79–86. [16] H. M. Möller and R. Tenberg, Multivariate polynomial system solving using intersections of eigenspaces, Journal of Symbolic Computation 32 (2001), no. 5, 513–531. [17] B. Mourrain, V. Y. Pan and O. Ruatta, Accelerated solution of multivariate polynomial systems of equations, SIAM Journal on Computing 32 (2003), no. 2, 435–454. [18] W. Rudin, Functional Analysis, McGraw-Hill, 1991. [19] D. W. Stroock, Probability theory, an analytic view, Cambridge University Press, 1993. [20] B. Sturmfels, Solving systems of polynomial equations (cbms regional conference series in mathematics), Amer. Math. Soc., 2002. [21] N. Wiener, Tauberian theorems, Annals of Mathematics 33 (1932), no. 1, 1–100. [22] F. Zhang and C. Gao, Convergence rates of variational posterior distributions, The Annals of Statistics 48 (2020), no. 4, 2180–2207. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction Organization and notation Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Coefficients from an ergodic process Conclusion & References Introduction Introduction Organization and notation Organization and notation Problem Setting and Preliminaries Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Characterization of coefficients Coefficients from an ergodic process Coefficients from an ergodic process Conclusion & References Conclusion & References Conclusion In this mathematical study, we delve into the realm of statistical inference and introduce a novel approach to variational non-Bayesian inference. Most significantly, we propose a new method for uniquely determining the hidden PDF solely from a random sample while leveraging theory. Beyond merely approximating the shape of the unrevealed PDF, we yield results that provide practical assistance in inferring important moments, such as the mean and variance, from the estimated PDF. These days, methods utilizing artificial intelligence are widely employed, but they generally exhibit a drawback of depending on initial conditions, i.e., a prior distribution, and iterative computations, i.e., a backpropagation. The limitations of such approaches lie in their reliance on these conditions to achieve global optimization. We emphasize the significant contribution of the proposed method to predictive and classification models, even without any information on populations, highlighting its potential applications in various domains such as finance, economics, weather forecasting, and machine learning, all of which present unique challenges. The most well-known method for approximating hidden PDFs is the variational approach in a Bayesian context. In contrast, our study extends this problem by seeking to precisely determine unknown PDFs through a system of equations. Our approach includes proving the Fréchet differentiability of entropy to establish the uniqueness of the energy function space in the Wiener algebra. We then derive the unique determination of the energy function through the minimization of KL-divergence. Leveraging the Ergodic theorem, we elucidate that solutions to equations comprising polynomial function series are the coefficients of the energy function and numerically substantiate the convergence of partial sums of the energy function obtained from a finite number of equations. In summary, our mathematical exploration has unveiled the potential of variational Non-Bayesian inference in Wiener space. We anticipate that these mathematical ideas can offer an innovative framework for probability density estimation and predictive modeling. Ultimately, our research emphasizes its potential contribution to reshaping mathematical horizons and expanding the boundaries of knowledge in the field of statistical methodology. REFERENCES [1] K. Bharath A. Saha and S. Kurtek, A geometric variational approach to bayesian inference, Journal of the American Statistical Association 115 (2020), no. 530, 822–835. [2] P. Alquier and J. Ridgway, Concentration of tempered posteriors and of their variational approximations, The Annals of Statistics 48 (2020), no. 3, 1475–1497. [3] A. R. Barron and C. Sheu, Approximation of density functions by sequences of exponential families, The Annals of Statistics 19 (1991), no. 3, 1347–1369. [4] C. M. Bishop and M. E. Tipping, Probabilistic principal component analysis, J. R. Statist. Soc. B 61 (1999), no. 3, 611–622. [5] F. W. Olver, D. Lozier, R. F. Boisvert and C. W. Clark, NIST Handbook of Mathematical Functions, Cambridge University Press, 2010. [6] C Brouder and F Patras, One-particle irreducibility with initial correlations, (2011). [7] I. Csiszar, I-divergence geometry of probability distributions and minimization problems, Ann. Probab. 3 (1975), no. 1, 146–158. [8] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2 (1989), 303–314. [9] L. Grafakos, Classical and Modern Fourier Analysis, Prentice Hall, 2003. [10] D. E. Rumelhart, G. E. Hinton and R. Williams, Learning representations by back-propagating errors, Nature 323 (1986), 533–536. [11] G. E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, University of Toronto, 2010. [12] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), no. 2, 251–257. [13] Y. Katznelson, An introduction to harmonic analysis (third ed.), New York: Cambridge Mathematical Library, 2004. [14] D. M. Blei, A. Kucukelbir and J. D. McAuliffe, Variational inference: A review for statisticians, Jour. of the American Statistical Association 112 (2017), no. 518, 859–877. [15] S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statistics 22 (1951), no. 1, 79–86. [16] H. M. Möller and R. Tenberg, Multivariate polynomial system solving using intersections of eigenspaces, Journal of Symbolic Computation 32 (2001), no. 5, 513–531. [17] B. Mourrain, V. Y. Pan and O. Ruatta, Accelerated solution of multivariate polynomial systems of equations, SIAM Journal on Computing 32 (2003), no. 2, 435–454. [18] W. Rudin, Functional Analysis, McGraw-Hill, 1991. [19] D. W. Stroock, Probability theory, an analytic view, Cambridge University Press, 1993. [20] B. Sturmfels, Solving systems of polynomial equations (cbms regional conference series in mathematics), Amer. Math. Soc., 2002. [21] N. Wiener, Tauberian theorems, Annals of Mathematics 33 (1932), no. 1, 1–100. [22] F. Zhang and C. Gao, Convergence rates of variational posterior distributions, The Annals of Statistics 48 (2020), no. 4, 2180–2207.

VALIANCE SOLUTIONS

valiancesolutions.com

-- employees

Since n.d.

COMPANY RANKING

VALIANCE SOLUTIONS

EVERGREEN INDEX #11556

Representing this company? Click here to claim and customize this page!

Valiance Solutions WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

VALIANCE SOLUTIONS

valiancesolutions.com

-- employees

Since n.d.

COMPANY RANKING

VALIANCE SOLUTIONS

EVERGREEN INDEX #11556

RELATED COMPANIESRANK

STORIES ABOUT VALIANCE SOLUTIONS

Representing this company? Click here to claim and customize this page!

Valiance Solutions WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!