Semsee

semsee

Let's Build an MLOps Pipeline With Databricks and Spark - Part 1

Random forest is one of the most popular algorithms for multiple machine learning tasks. This story looks into random forest regression in R, focusing on understanding the output and variable importance. If you prefer Python code, . here you go Decision Trees and Random Forest When decision trees came to the scene in , they were better than classic multiple regression. One of the reasons is that decision trees are easy on the eyes. People without a degree in statistics could easily interpret the results in the form of branches. 1984 Additionally, decision trees help you avoid the synergy effects of interdependent predictors in multiple regression. Synergy (interaction/moderation) effect is when one predictor depends on another predictor. is a nice example from a business context. Here On the other hand, regression trees are not very stable - a slight change in the training set could mean a great change in the structure of the whole tree. Simply put, they are not very accurate. But what if you combine multiple trees? Randomly created decision trees make up a , a type of based on , .i.e. . First, you create various decision trees on bootstrapped versions of your dataset, i.e. random sampling with replacement (see the image below). Next, you aggregate (e.g. average) the individual predictions over the decision trees into the final random forest prediction. random forest ensemble modeling bootstrap aggregating bagging Notice that we skipped some observations, namely Istanbul, Paris and Barcelona. These observations, i.e. rows, are called and used for prediction error estimation. out-of-bag Random Forest Regression in R Based on CRAN’s , 63 R libraries mention random forest. I recommend you go over the options as they range from bayesian-based random forest to clinical and omics specific libraries. You could potentially find random forest regression that fits your use-case better than the original version. Still, I wouldn’t use it if you can’t find the details of how exactly it improves on Breiman’s and Cutler’s implementation. If you have no idea, it’s safer to go with the original - . list of packages randomForest Code-wise, it’s pretty simple, so I will stick to the example from the documentation using . 1974 Motor Trend data ### Import libraries
library(randomForest)
library(ggplot2)

set.seed(4543)
data(mtcars)
rf.fit <- randomForest(mpg ~ ., data=mtcars, ntree=1000,
                       keep.forest=FALSE, importance=TRUE) I will specifically focus on understanding the performance and . So after we run the piece of code above, we can check out the results by simply running . variable importance rf.fit > rf.fit

Call:
 randomForest(formula = mpg ~ ., data = mtcars, ntree = 1000, 
              keep.forest = FALSE, importance = TRUE) 

               Type of random forest: regression
                     Number of trees: 1000
No. of variables tried at each split: 3

          Mean of squared residuals: 5.587022
                    % Var explained: 84.12 Notice that the function ran random forest regression, and we didn’t need to specify that. It will perform nonlinear multiple regression as long as the target variable is numeric (in this example, it is Miles per Gallon - ). But, if it makes you feel better, you can add mpg type= “regression”. The mean of squared residuals and % variance explained indicate how well the model fits the data. Residuals are a difference between prediction and the actual value. In our example, 5.6 means that we were wrong by 5.6 miles/gallon on average. If you want to have a deep understanding of how this is calculated per decision tree, watch . this video You can experiment with, i.e. increase or decrease, the number of trees ( ) or the number of variables tried at each split ( ) and see whether the residuals or % variance change. ntree mtry If you also want to understand what the model has learnt, make sure that you do as in the code above. importance = TRUE Random forest regression in R provides two outputs: decrease in mean square error (MSE) and node purity. Prediction error described as MSE is based on permuting out-of-bag sections of the data per individual tree and predictor, and the errors are then averaged. In the regression context, Node purity is the total decrease in residual sum of squares when splitting on a variable averaged over all trees (i.e. how well a predictor decreases variance). MSE is a more reliable measure of variable importance. If the two importance metrics show different results, listen to MSE. If all of your predictors are numerical, then it shouldn’t be too much of an issue - read more . here The built-in will visualize the results, but we can do better. Here, we combine both importance measures into one plot emphasizing MSE results. varImpPlot() ### Visualize variable importance ----------------------------------------------

# Get variable importance from the model fit
ImpData <- as.data.frame(importance(rf.fit))
ImpData$Var.Names <- row.names(ImpData)

ggplot(ImpData, aes(x=Var.Names, y=`%IncMSE`)) +
  geom_segment( aes(x=Var.Names, xend=Var.Names, y=0, yend=`%IncMSE`), color="skyblue") +
  geom_point(aes(size = IncNodePurity), color="blue", alpha=0.6) +
  theme_light() +
  coord_flip() +
  theme(
    legend.position="bottom",
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank()
  ) Conclusion In terms of assessment, it always comes down to some theory or logic behind the data. Do the top predictors make sense? If not, investigate why. “Rome was not built in one day, nor was any reliable model.” ( ) Good & Hardin, 2012 Modeling is an iterative process. You can get a better idea about the predictive error of your random forest regression when you save some data for performance testing only. You might also want to try out . other methods

Random Forest Regression in R: Code and Interpretation

Regresión de bosque aleatorio en R: código e interpretación

A classification approach with natural language processing in R Audience This short blog post is intended for someone who has basic understanding of the natural language processing algorithm in addition to familiarity with machine basics, particularly in classification techniques, and is keen to learn the actual integration of the two in R. We will go over detailed data preparation as well as model optimization. It is also recommended to review for a brief introduction on word2vec, learning this post word2vec. Objective The objective is to learn how the Dow Jones Industrial Average would be affected by daily popular news on . Reddit Dataset The datasets can be found . The simplified dataset is consisted of daily Dow Jones data and top daily headlines from Reddit WorldNews Channel for the past 8 years. The target variable is Label (0 is down and 1 is up). here Screenshot of the stock news data set Approach We will use the algorithm to first create word vectors of 100 dimensions on all the headlines. We will then transform the headlines into sentence vectors by averaging the word vectors that every sentence is consisted of. Finally we will use a variety of classification algorithms to predict whether Dow Jones Industrial Average would go up or down. word2vec Data Preparation 1…Remove undesired characters , including punctuations and upper cases 2…Remove English stop words such as “i,” “my,” “myself” Screenshot of the data set with unwanted features removed 3…Input cleaned headlines into model to obtain word vectors word2vec Screenshot of the word vectors generated from cleaned headlines 4…Apply and average word vectors to all headlines 5…Return 2-dimension vector representations on headlines using t-SNE 6…Create dataset with target variable and 2-dimension vectors Screenshot of the finished data set Model Evaluation Here we will use the package in R to evaluate the prediction accuracy on a variety of algorithms, including CART (rpart), logistic regression (glm), linear discriminate analysis (LDA), k-Nearest Neighbors (kNN), Support Vector Machine.with Radial Basis Function (svmRadial), and eXtreme Gradient Boosting (xgbLinear). caret It is observed that all algorithms above have unsatisfying performance, with close to 50% accuracy and poor Kappa values. However, SVM with the Radial Basis Function kernel tends to perform better than other algorithms, with a 95% confidence interval on accuracy between 51% and 55%. Performance on various classification algorithms As a result, we will explore a few optimization options. Performance Optimization 1…Stacking Multiple Models We will first try to stack all the models to see if performance will improve. Unfortunately, not much improvement is observed. 2…Averaging All Headlines Instead of treating top headlines as individual variables, we will combine all headlines to create one set of 2-dimension vector. Performance on the algorithms with averaged headlines A slight increase of 0.1% in mean accuracy is observed in SVM algorithm. 3…Shifting Target Variable We will try shifting the target variable back 1–7 days, as daily news may have lagged effects on . Alternatively, we can also shift the target variable forward a couple of days, however, this approach will contradict with the objective of stock marketing performance using daily top news. stock performance predicting Performance on the algorithms with 3 day lag on target variable We can observe that CART (rpart) now has the best performance in terms of mean accuracy at 53.66%. There is also another slight increase of 0.1% in mean accuracy is observed in the SVM algorithm. 4…Tuning Model Parameters Here we will demonstrate how to tune parameters of the CART and SVM algorithms to see if we can increase the accuracy on our predictions. CART In rpart, the tuning parameter is , or . We will examine the accuracy on different values of . The optimal cp value is found to be 0.15, as shown below. cp complexity parameter cp SVM In SVM with Radial Basis Function, the tuning parameters are and ( ). We will examine the accuracy on different values of and . The optimal values are shown below, with and . sigma C cost sigma C sigma = 0.05 C = 3 R Code library(devtools)library(tm)library(ggplot2)library(text2vec)library(rword2vec)library(wordVectors)library(tidyverse)library(Rtsne)library(caret)library(mlbench)library(caretEnsemble) setwd("~/Desktop/stock")#Simple Visualization on DJIA datastock <- read.csv("DJIA_table.csv")stock$Date <- as.Date(stock$Date) ggplot(data=stock,aes(x=Date,y=Open))+geom_line()+labs(title="Dow Jones Industrial Average") #Read the simplified data setdata <- read.csv("stock.csv") #Remove unwanted columns for natural language processingtext <- data text$Date <- NULLtext$Label <- NULL #Remove unwanted characters/punctuations/upper-casesfor (i in 1:25){text[,i] <- gsub('b"',"",text[,i])text[,i] <- gsub("b'","",text[,i])text[,i] <- gsub("[[:punct:]]", "", text[,i])text[,i] <- tolower(text[,i])} write.table(text,"text.txt") #Rename column namesnew_text <- subset(text,select=paste("Top",1,sep=""))new_text <- setNames(new_text,"x") for (i in 2:25){new_text2 <- subset(text,select=paste("Top",i,sep=""))new_text2 <- setNames(new_text2,"x")new_text <- rbind(new_text,new_text2)} write.table(new_text,"text.txt") final_text <- new_text #Remove stopwordsstopWords <- stopwords("en")'%nin%' <- Negate('%in%') train <- lapply(final_text$x, function(x) {t <- unlist(strsplit(x, " "))t[t %nin% stopWords]}) train.df = as.data.frame(do.call(rbind, train))train.df$x <- do.call(paste, c(train.df[,1:45], sep=" ")) finaltext <- subset(train.df,select="x") write.table(finaltext,"text_data.txt") #Input words to word2vecmodel = train_word2vec("text_data.txt", output="vec.bin",threads = 4, vectors = 100, window = 12, min_count = 5, iter=10, force=TRUE) #Plot the word vectorsplot(model)#Convert binary to text formatbin_to_txt("vec.bin","model1text.txt") #Remove first rows manually in the .txt file #Convert .txt to .cvsm1 <- read.table("model1text.txt",header = F,quote = "", row.names = NULL, stringsAsFactors = FALSE) colnames(m1)[1] <- "word" #Apply word vectors to headlines and return 2-dimension representations with t-SNElabel <- subset(data,select="Label")labeltext <- cbind(label,text)labeltext$id <- 1:nrow(labeltext)labeltext$id <- as.factor(labeltext$id) for (i in 2:24){s <- strsplit(labeltext[,i], split= " ")new <- data.frame(V1 = rep(labeltext$id, sapply(s, length)), V2 = unlist(s))colnames(new) <- c("id","word")new <- merge(new,m1,by="word")new2 <- aggregate(V2 ~ id, new, mean)for (j in 4:102){new3 <- aggregate(new[,j] ~ id, new, mean)new2 <- merge (new2,new3,by="id")}colnames(new2)[2:101] <- paste("V", 1:100, sep="")tsne <- Rtsne(new2, dims = 2, perplexity=100, verbose=TRUE, max_iter = 500)t = as.data.frame(tsne$Y)colnames(t) <- c(paste("top",i,"x",sep=""),paste("top",i,"y",sep=""))labeltext <- cbind(labeltext,t)} write.csv(labeltext,"labeltext.csv") #Add back Label informationmydata <- labeltextmydata[,2:27] <- NULL mydata$Label[mydata$Label 0]<- "No" 1]<- "Yes"mydata$Label[mydata$Label #Classfication Modelscontrol <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE)algorithmList <- c('rpart','glm','lda','knn','svmRadial','xgbLinear')set.seed(1)models <- caretList(Label~., data=mydata, trControl=control, methodList=algorithmList)results <- resamples(models)summary(results)dotplot(results) #-----Performance Optimization #1: Stacking Multiple Models-----#Model CorrelationsmodelCor(results)splom(results)#Model Stacking with GLMstackControl <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE)set.seed(2)stack.glm <- caretStack(models, method="glm", metric="Accuracy", trControl=stackControl)print(stack.glm) #-----Performance Optimization #2: Averaging All Headlines-----mydata2 <- mydatamydata2$headline <- rowMeans(mydata2[,2:45], na.rm = TRUE)mydata2[,2:45] <- NULL control <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE)algorithmList <- c('rpart','glm','lda','knn','svmRadial','xgbLinear')set.seed(1)models <- caretList(Label~., data=mydata2, trControl=control, methodList=algorithmList)results <- resamples(models)summary(results)dotplot(results) #-----Performance Optimization #3: Shifting Target Variable-----shift <-subset(mydata,select="Label")shift1 <- as.data.frame(shift[-1,])shift2 <- as.data.frame(shift[-c(1,2),])shift3 <- as.data.frame(shift[-c(1,2,3),])shift4 <- as.data.frame(shift[-c(1,2,3,4),])shift5 <- as.data.frame(shift[-c(1,2,3,4,5),])shift6 <- as.data.frame(shift[-c(1,2,3,4,5,6),])shift7 <- as.data.frame(shift[-c(1,2,3,4,5,6,7),]) mydata1 <- mydata[-1989,]mydata2 <- mydata[-c(1988,1989),]mydata3 <- mydata[-c(1987,1988,1989),]mydata4 <- mydata[-c(1986,1987,1988,1989),]mydata5 <- mydata[-c(1985,1986,1987,1988,1989),]mydata6 <- mydata[-c(1984,1985,1986,1987,1988,1989),]mydata7 <- mydata[-c(1983,1984,1985,1986,1987,1988,1989),] mydata1$Label <- NULLmydata2$Label <- NULLmydata3$Label <- NULLmydata4$Label <- NULLmydata5$Label <- NULLmydata6$Label <- NULLmydata7$Label <- NULL mydata1 <- cbind(shift1,mydata1)colnames(mydata1)[1] <- "label1day"mydata2 <- cbind(shift2,mydata2)colnames(mydata2)[1] <- "label2day"mydata3 <- cbind(shift3,mydata3)colnames(mydata3)[1] <- "label3day"mydata4 <- cbind(shift4,mydata4)colnames(mydata4)[1] <- "label4day"mydata5 <- cbind(shift5,mydata5)colnames(mydata5)[1] <- "label5day"mydata6 <- cbind(shift6,mydata6)colnames(mydata6)[1] <- "label6day"mydata7 <- cbind(shift7,mydata7)colnames(mydata7)[1] <- "label7day" control <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE)algorithmList <- c('rpart','glm','lda','knn','svmRadial','xgbLinear')set.seed(1)models <- caretList(label3day~., data=mydata3, trControl=control, methodList=algorithmList)results <- resamples(models)summary(results)dotplot(results) #-----Performance Optimization #4: Tuning Model Parameters-----set.seed(3)control <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE) #Tuning cp in CARTrpartGrid <-  expand.grid(cp = c(0.01,0.05,0.1,0.15)) rpartFit <- train(label3day ~ .,data = mydata3,method = "rpart",tuneLength = 10,tuneGrid=rpartGrid) rpartFitggplot(rpartFit) #Tuning sigma and C in SVMsvmGrid <-  expand.grid(sigma = c(0.01,0.05,0.1,0.15),C = c(1,3,5,7,9)) svmFit <- train(label3day ~ ., data = mydata3,method = "svmRadial",trControl = control,verbose = FALSE,tuneGrid = svmGrid) svmFitggplot(svmFit) Questions, comments, or concerns?jchen6912@gmail.com

Stories, News & Mentions about SEMSEE

Categories

Trending Topics

Stories, News & Mentions about SEMSEE

Semsee’s Stories on HackerNoon

Categories

Trending Topics