CS229: Gradient Descent in Machine Learning

University:
Stanford University
Course:
CS229 | Machine Learning
Academic year:

2023
Views:

439

Pages:

1
Author:

anon

Gradient Descent in Machine Learning: An Overview A fundamental machine learning approach called gradient descent is used to reduce a model's cost function. This approach can be used to train a broad variety of models, including deep learning and linear regression models, and it can be applied to different cost functions. In this article, we'll go into great detail on gradient descent and explain how it works to optimize a model's parameters. Gradient Descent: What Is It? Finding the minimum value of a cost function is done using the optimization technique of gradient descent. The parameters are initially estimated by the algorithm, which then iteratively changes them to lower the cost until it is as low as possible. By altering the parameters in a way that lowers the cost, gradient descent seeks to minimize the cost function. The operation of Gradient Descent Calculating the gradient of the cost function with respect to the parameters is how gradient descent operates. The gradient shows where the cost function's greatest growth is occurring. To cut costs, the algorithm adjusts the parameters in the gradient's opposite direction. The learning rate, a hyperparameter that must be specified prior to training the model, determines the size of the update. The parameters are adjusted in the manner described below at each cycle of gradient descent: 𝑤 = 𝑤 − 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 · 𝑏 = 𝑏 − 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 · 𝑑𝑤 𝑑𝑤 𝑑𝑏 𝑑𝑏 Where dw/dw and db/db are the gradients of the cost function with respect to w and b, respectively, and w and b are the parameters. The cost function is what? A gauge of how well the model matches the data is the cost function. The discrepancy between expected and actual output is how it is defined. The mean squared difference between the output predicted and the actual output is generally used to calculate the cost function in linear regression. Although the cost function in deep learning models may be more sophisticated, lowering costs is still the key objective.