ensemble learning One of the challenging problems is to select a model out of several models, so there are a lot of models out there. For example, let me just focus on classification. Uh, first, let's list some of the classifiers. Let's say KNN, that's K-Nearest Neighbor classifier, that's one classifier. Another is, let's say, Support Vector Machines. Um, maybe another classifier is Logistic Regression. And there are several other classifiers. There are several. Let's just focus on these two, these three. It turns out that there is no best classifier at the end of the day for every dataset. Um, which means for a given dataset, you may have to choose one of the models that best suits your data or the new data. And this model selection is really difficult and, I mean, time-consuming. And sometimes, finding out an ideal model by cross-validation and stuff like that is really, really expensive. An alternative way is that, given a pool of models — just think about classification — given a pool of models, that's again an SVM and Support Vector Machine, why don’t we actually train or use all these models on our training data? So, that's the training data. And let's use KNN for it. KNN uses the same training data and trains, let’s say, SVM, use the same training data, and let’s build a Logistic Regression model on that. And then, somehow, combine these different models to make one model, sometimes called the meta-model. At the end, this seems really intuitive and maybe the right thing to do when we do not know which model is the best. And it turns out, depending upon the choices of different models, depending upon whether we are using the whole dataset or subsets of data, and depending upon the way we are combining the different models together to make one meta-model, it all goes to ensemble learning. And different ensemble learning techniques actually vary based on making the way they use the sample data, whether they use subsets or the whole training data for each model. The way they use different kinds of models, whether all the models are homogeneous or heterogeneous. Heterogeneous means different kinds of models like KNN, SVM, Logistic Regression — they are different models. Uh, maybe we use KNN for a particular subset, KNN for another subset, KNN for another subset. That's kind of a heterogeneous homogeneous model. And then, uh, whether we are treating all the classifiers independently or in parallel, or whether we are training the classifiers in a particular order or sequential way, and finally, how to combine different classifiers. All the techniques of ensemble learning actually vary based on these kinds of criteria. One important thing is the choice of the number of models to use, and it turns out this is a hyperparameter. I mean, it has been shown recently that there is an ideal number of models to be used in ensemble learning. Um, and going below or above that number may deteriorate the performance. But in practice, people use some rules of thumb or make cross-validation to find out the number of components or the number of models to be used in ensemble learning. So, the take-home for this slide is: ensemble learning uses several models and combines them, and that whole thing becomes one model. That whole thing eventually becomes a model, and that is called ensemble learning.