boosting in machine learning Boosting machine learning is one such technique that can be used to solve complex data-driven real-world problems. So, what is boosting in machine learning? Boosting refers to the process of creating a strong learner from a collection of weak learners. A weak learner is a model that performs only slightly better than random guessing on the training data. By iteratively adjusting the weights of the training instances, the boosting algorithm assigns higher importance to misclassified instances, forcing subsequent weak learners to focus on these challenging samples. The final prediction is determined by aggregating the predictions of all weak learners, with higher emphasis placed on those that demonstrate superior performance. Boosting is an ensemble learning technique that uses a set of machine learning algorithms in order to convert or combine weak learners into strong learners to increase the accuracy of the model. So, guys, boosting is actually a very effective method to increase the efficiency of your model. In most of these competitions that you see on Kaggle or any machine learning competition, the majority of the competitors, or the winners, usually implement boosting and bagging, or any other ensemble learning technique. So, as you can see from the figure, by combining the outputs or the predictions that we get from all our weak learners or our rules, we are able to get a strong learner. So now, let's move forward and see the types of boosting in machine learning. The first one is AdaBoost (Adaptive Boosting). AdaBoost is the most popular boosting algorithm. It assigns weights to training instances and adjusts these weights based on the performance of weak learners. It focuses on misclassified instances, allowing subsequent weak learners to concentrate on these samples. The final prediction is determined by aggregating the predictions of all weak learners through a weighted majority vote. The second one is Gradient Boosting. Gradient Boosting is a widely used boosting algorithm that builds an ensemble of decision trees. The third one is the XGBoost algorithm, which is also known as Extreme Gradient Boosting. XGBoost is an advanced boosting algorithm that combines gradient boosting with regularization techniques. The fourth one is LightGBM (Light Gradient Boosting Machine). LightGBM is a high-performance boosting algorithm that uses a leaf-wise approach to construct decision trees. It is implemented by combining several weak learners into a single strong learner. So, the couple of steps that this adaptive boosting algorithm follows: Adaptive boosting starts by assigning equal weight to all of your data points, and you draw out a decision stump for a single input feature. The next step is that the results you get from the first decision stump are analyzed, and if any observations are misclassified, they are assigned higher weights. This is exactly explained in the previous slide as well. After that, a new decision stump is drawn by considering the observations with the higher weights as more significant. So, whichever data point was misclassified, they are given a higher weight. In the next step, you'll draw another decision stump that tries to classify the data points by giving more importance to the data points with higher weights. Again, if there are any observations that are misclassified, they are given higher weight, and this process will keep continuing. It will keep looping until all the observations fall into the right class. So, the end goal here is to make sure that all your data points are classified into the correct classes. Adaptive boosting, or AdaBoost, can also be used for regression problems. It's not restricted to classification only; it can be used for both classification and regression, but it's more commonly seen in classification problems. So, that was a brief overview of adaptive boosting. Now, let's understand gradient boosting. Gradient boosting is also based on the sequential ensemble learning model. Here, what happens is the base learners are generated sequentially in such a way that the present base learner is always more effective than the previous one. Basically, the overall model improves sequentially with each iteration. Now, the difference in this type of boosting is that the weights for misclassified outcomes are not incremented. You're not going to increment or add weights to the misclassified outcomes. Instead, in gradient boosting, what you do is try to optimize the loss function of the previous learner by adding a new adaptive model that adds weak learners in order to reduce the loss function. Now, the main idea here is to overcome the errors in the previous learner's prediction. This type of boosting has three main components. It has something known as the loss function. The loss function is the one that needs to be optimized, meaning that you need to reduce the error. The other component is that the weak learners are needed for computing predictions and forming strong learners. Then, you need an additional model that will regularize the loss function, meaning that it will try to fix the loss or the error from the previous weak learner. So, you keep adding a model that will regularize the loss function from the previous learner. Just like adaptive boosting, gradient boosting can also be used for both classification and regression problems. Now let's discuss the last type of boosting, which is XGBoost. XGBoost is basically an advanced version of gradient boosting. It literally means "extreme gradient boosting." So, XGBoost actually falls under the category of the distributed machine learning community. It's a more advanced version of the gradient boosting method. The main aim of this algorithm is to increase speed and to improve the efficiency of your computations and overall model performance. The reason why this model was introduced was because the gradient boosting algorithm was computing the output at a very slow rate. Right? Because there's sequential analysis of the dataset, it takes a longer time. That's why XGBoost was introduced. It basically boosts or "extremely boosts" the performance of the model. So, XGBoost is mainly going to focus on your speed and your model efficiency. In order to do this, it has a couple of features. It supports parallelization by creating decision trees in parallel—there's no sequential modeling in this. It also implements something known as distributed computing methods for evaluating any large and complex models. It uses out-of-core computing in order to analyze huge and varied datasets. Additionally, it implements cache optimization in order to make the best use of your hardware and resources overall. So, guys, these were the basics of the different types of boosting algorithms.