Ensemble Learning Techniques Ensemble Learning in Machine Learning Ensemble learning is a supervised learning technique used in machine learning to improve overall performance by combining the predictions from multiple models. Ensemble Learning - Types of Ensemble Methods Voting (Averaging) Bootstrap aggregation (bagging) Random Forests Boosting Stacked Generalization (Blending) Ensemble Learning – Voting (Averaging) Voting is an ensemble machine learning algorithm that involves making a prediction that is the average (regression) or the sum (classification) of multiple machine learning models. Ensemble Learning – Bootstrap aggregation (bagging) Bootstrap Aggregating, also known as bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms like classification and regression. It decreases the variance and helps to avoid overfitting. It is usually applied to decision tree methods. Bagging is a special case of the model averaging approach. Input data Train 1 | Train 2 | ... | Train N (Representing subsets of input data) Model | Model | ... | Model Pred 1 | Pred 2 | ... | Pred N Bagging classifier Prediction Ensemble Learning – Random Forest Random forest is a commonly-used machine learning algorithm. A Random forest is an ensemble learning method where multiple decision trees are constructed and then they are merged to get a more accurate prediction. Random Forest Input data Average of all predictions Random forest prediction Ensemble Learning – Boosting Boosting is an ensemble modeling technique that attempts to build a strong classifier from the number of weak classifiers. It is done by building a model by using weak models in series. Firstly, a model is built from the training data. Then the second model is built which tries to correct the errors present in the first model. This procedure is continued and models are added until either the complete training data set is predicted correctly or the maximum number of models is added. Ensemble Learning – Boosting Input data | Weighted data | Weighted data Ensemble Learning – Stacked Generalization (Blending) Stacking, Blending and Stacked Generalization are all the same thing with different names. It is a kind of ensemble learning. In traditional ensemble learning, we have multiple classifiers trying to fit to a training set to approximate the target function. Since each classifier will have its own output, we will need to find a combining mechanism to combine the results. This can be through voting (majority wins), weighted voting (some classifier has more authority than the others), averaging the results, etc. This is the traditional way of ensemble learning. Ensemble Learning – Stacked Generalization (Blending) In stacking, the combining mechanism is that the output of the classifiers (Level 0 classifiers) will be used as training data for another classifier (Level 1 classifier) to approximate the same target function. Basically, you let the Level 1 classifier to figure out the combining mechanism. Input data Train 1 | Train 2 | ... | Train N Pred 1 | Pred 2 | ... | Pred N Stacked dataset Final Model Prediction