Ensemble modeling can reduce variance, minimize modeling method bias and decrease chances of overfitting. Read on to learn the secrets of ensemble modeling and why it works better than single models. Overfitting I remember the excitement when I first achieved 97% accuracy. However, the moment I ran the model on new (unseen) data, the accuracy went down to 57% and the excitement was gone 😢. The task was to predict whether a patient would develop Alzheimer's or not. This was my first experience with overfitting. Overfitting happens when your algorithm can't generalize data patterns because it learns the data too well. When you present new data to an overfitted model, the predictive performance goes down, as in my case. Occam's Razor A good model fit, one that can generalize data well, should be simple. This is the idea behind the law of parsimony, often called Occam's razor. "Don't elaborate the nature of something beyond necessity." The frequent interpretation of this statement is that simple solutions are better than complex solutions. In other words , . We can apply this thinking to the process of choosing the best model fit. complex == bad simple == good Remember my initial 97% accuracy that went to 57% on the new data? A model that learns all the relationships in the training data becomes too complex. Then, when it comes to predicting outcomes for the new observations, it performs poorly because the model is inflexible and tailored only to the training data. On the other hand, a simple model can have a looser fit on training data, but it can also achieve a good fit on unseen data. Ensemble modeling, among other methods, can prevent overfitting. Ensemble Modeling Ensemble models are models consisting of multiple models or algorithms. Individual models can be combined through various methods such as . bagging, boosting, and stacking In general, we can differentiate between two types of ensembles. They are either based on 1) many that are stronger together (e.g. boosting) or 2) fewer well-thought-out models combined via stacking into a . weaker models metamodel You can imagine the weaker models as ants. One ant can't build an anthill, but the whole colony of ants, well, that's a different story. The second group of ensemble methods is more like Avengers. (Yes, it wasn't just a hook in the title.) Every Avenger has a unique skill, and they are superheroes already, but they are stronger together. This is because their individual disadvantages are covered by their teammates. Why Ensemble Modeling Works? We established at the beginning that according to Occam's razor, simple models are better. But ensemble modeling is about combining multiple models, so how is that simpler? There is something I didn't mention earlier when talking about Occam's razor. As summarized by in 1999, it has two interpretations: Domingos "Simplicity is a goal in itself." "Simplicity leads to greater accuracy." Turns out the second bit is not always valid. I talked about overfitting and how it occurs when a model or an algorithm overlearns the training data. However, as empirical have shown, this doesn't hold true all the time. Additionally, judging complexity by the number of models is not ideal. studies A solution combining multiple models is more complex than one model for sure. However, looking at how a model or algorithm functions, i.e. predicts, can help us understand this mystery. Simplicity doesn't always lead to greater accuracy. Let's go back to the analogy world. When ants are building their anthills, they need every ant they can get. It's similar for the weak learners, although you can still overfit this type of ensemble, so you need to be careful. When Avengers meet for the first time, some of them are overconfident and don't appreciate teamwork. But after they go through challenges together, they change and improve for the better. Ensemble modeling averages out the biases of the individual models or methods. This causes the predictions to be more stable with lower variance. In other words, the predictions based on ensemble modeling are simpler in terms of their statistical characteristics. Ensemble Modeling in Practice To consolidate how great ensemble modeling is, I have an example of their success in making an impact. In 2013, the Centre for Disease Control and Prevention in the U.S. started in a "Predict the Influenza Season Challenge" from research teams across the country. crowdsourcing forecasting models These seasonal challenges became popular, and the idea developed into a where you can click through individual and combined predictions. The main takeaway was that from all of the individual models, the forecasts from models were the most . I recommend you have a look. website ensemble accurate Read more about or how to avoid : ensemble modeling overfitting Boosting and Bagging: How To Develop A Robust Machine Learning Algorithm Automatic Feature Selection in Python: An Essential Guide Other resources: John Elder (2018) The Apparent Paradox of Complexity in Ensemble Modeling Robert Nisbet, Gary Miner & Ken Yale (2018) Model Evaluation and Enhancement Abdulhamit Subasi (2020) Machine Learning Techniques

Different

Bayesian Brain: Is Your Brain a Data Scientist?

Foundation Series: Data Science, Psychohistory, and the Future of Humanity

Instagram

Avengers Ensemble: How Ensemble Modeling Helps You Avoid Overfitting

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3 Data Distributions for Counts in Layman’s Terms

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

10 Cutest Pink Pokémon of All Time

3 Data Distributions for Counts in Layman’s Terms

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

10 Cutest Pink Pokémon of All Time

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps