bagging So, let's see that in bagging, sometimes known as bootstrap aggregating. What happens is, um, let me list certain things that are special to bagging. Number one: you use subsets of data for different models. So, for example, that's a model — let's say classifier one. I'm just using the example of classification; the whole thing works for regression as well. So, subset of data, different subsets. This is one subset, this is another subset of data. Maybe we decide on a particular percentage, let's say 50 or 60 percent of data for subset creation. We generate subsets of data, and the way we generate subsets of data is with replacement. So, with replacement, let me tell you what that means. Replacement means that if, for example, this point — let me highlight this point — if that point is, let's say, point number one, and let's say it's at a certain index, this is point number one. Then, this point number one can appear in the first subset for the first model, and it can appear in the second subset for the second model. And it can appear — I mean, it is allowed to resample this for different subsets. On top of that, the same point can be resampled for the same subset as well. And that's particularly the case why it's called a bag data structure rather than a subset, because if we treat this as a subset, a set does not contain duplicate entries. And that's why people call it a "bag," because bags normally are multisets. Now, okay, the real question is, um, let's say we do this sampling, making different bags. Each data point is allowed to appear in different bags, and each data point is allowed to appear in the same bag multiple times. That's fine. That's okay with bagging. That's one kind of distinction of bagging in sampling, compared to or with respect to the rest of the ensemble learning techniques. Second: on each of these bags, a particular model is trained. So, let's say, that's kind of, let's say, KNN with some K value. On a different subset, you get another KNN. On another subset, you may use another KNN. That's the most basic and original form of bagging: that all the models belong to the same category of models. For example, all are KNNs. And if you're doing regression, then all are the same regression models. However, in general settings, all these models can be different. Classifier 1 can be any classifier, like SVM, and classifier 2 can be a different classifier, maybe like logistic regression, and so on. So, in general settings, all these models can be different, but in the most basic setting, all the models are the same, looking at the different subsets of data. And then finally, this KNN will make a prediction on a particular data point — let's say y₁ or yᵢ (on the ith point). Um, this also — so let me call it yᵢ for KNN. Um, then for yᵢ₁ for model 1, this makes, let's say, the prediction yᵢ₂. And this makes the prediction, let's say, yᵢ₃. And then finally, what happens is, um, we need to combine these predictions together to make a final prediction. In the very basic form of bagging, the combination is just voting. And in the case of regression, the combination is simply averaging. So, just average out yᵢ₁ + yᵢ₂ + yᵢ₃, and divide by 3. That's the average. One particular property of this bagging is that all the models are weighted equally. One model is not preferred over the other when combining these kinds of models. But wait a second! We are discussing that bagging works on subsets. Why subsets, and not the whole dataset? There is a reason for that, and that reason relates bagging to overfitting. So, consider for example several data points, and consider right now the task is regression. So, we have several data points — let’s say all these red data points. The feature is only one feature, let’s say this is feature *x*, and this is the target *y*, and these are all the training data that are available in this bluish kind of color. Let’s say the first model subsamples only these points that are in the same bluish color, and you, for example, fit a regression model that looks like it overfits on the subset here. I’m again using the term subset. Let me use the term *bag* because maybe one point is picked many times in the same bag. But even if the point is picked so many times, the model will stay roughly the same. So next, what we really do is, let’s say the second model is again the same regression model, but it looks at the different subsets, like these green points, and it fits its regression to these green points. And it looks like the model is overfitting on that subset, which is fine; that makes sense, it actually is doing that. Now, consider a third model which actually sees all these black points and fits a regression model on that, which looks like overfitting on that. But what happens if we average all these models together? And this kind of funny curve is actually the average model I’ve just drawn. You can see the average model is much smoother and less overfitted to every one of the data points. So what happens is, that’s your feature space — different models look at different portions of your feature space, different portions, and may learn the local structure of that subset or that portion or region of the feature space. That may lead to overfitting for the classifier in that particular local region, thereby learning a lot about the local structure of that region. But if we have several classifiers that are randomly looking at different portions and learning the local structures, all the variation or overfitting that is caused by the local structure will actually be averaged out significantly when we average out all the predictions together, thereby making the model less overfitted or a low-variance model in the end. And that’s the primary reason why bagging focuses on subsets of data. Several other ensemble learning techniques actually focus on subsets of data, although some actually take into account all the training data. But several of those, including bagging, focus on the subset, which makes much more sense as it avoids overfitting at the time of combination because the variances or overfitting factors are averaged out. Yeah, so that’s about it — about the overfitting for bagging. You can also see that the average model, or the combined model, performs well in general, or the generalization performance on the test data looks like it is much better than any of these individual models. So combining makes sense. But, it really turns out that in practice, bagging avoids overfitting, but it really depends on how you subsample the data, how many models you’re using, and what type of models these are. It really depends on that. In theory, bagging will avoid overfitting or not, but in theory, it is hard to explain that in general settings. No matter what models you choose, no matter how many models you choose, or no matter how the subsampling for different bags is done, in general, then bagging cannot be proven to avoid overfitting. But in practice, in so many fruitful or good settings, bagging works really well.