Gradient Descent for Linear Regression

University:
Stanford University
Course:
CS229 | Machine Learning
Academic year:

2023
Views:

54

Pages:

2
Author:

anon

Understanding Linear Regression and the Cost Function Linear Regression is a commonly used statistical technique that is used to model the linear relationship between the independent and dependent variables. It can be used for prediction and forecasting of future data points, and it is widely used in the field of machine learning. In this article, we will take a deep dive into the linear regression model, the cost function, and the gradient descent algorithm. By the end of this article, you will have a comprehensive understanding of these concepts and how they work together to produce accurate predictions. The Linear Regression Model Linear Regression is a simple and straightforward model that assumes a linear relationship between the independent and dependent variables. The model is represented by the equation: 𝑦 = 𝑤𝑥 + 𝑏 where w is the weight or coefficient, b is the bias, x is the independent variable, and y is the dependent variable. The line's slope is represented by the weight w, and its intercept is represented by the bias b. Price Function The cost function is used to assess how well the linear regression model is working. It is determined using the mean squared error (MSE) formula and measures the discrepancy between the expected and actual values: 𝑚 1 2 𝑖 2 𝑖 𝐽(𝑤,𝑏) = ( 𝑚) · ∑ (𝑦 − (𝑤𝑥 + 𝑏)) 𝑖=1 𝑖 Where 𝑤𝑥 + 𝑏 is the actual value, yi is the predicted value, and m is the total number of data points. The aim of the linear regression model is to give the most precise forecasts by minimizing the cost function. Algorithm for Gradient Descent The linear regression model is improved by minimizing the cost function using the gradient descent approach. The algorithm iteratively adjusts the bias b and the weight w until convergence. The following formulas are used to create the updates: 𝑤=𝑤−𝑎·( 1 𝑚 𝑚 𝑖 𝑖 𝑖 ) · ∑ (𝑦 − (𝑤𝑥 + 𝑏)) · 𝑥 1 𝑖=1 𝑚 𝑖 𝑖 𝑏 = 𝑏 − 𝑎 · ( 𝑚 ) · ∑ (𝑦 − (𝑤𝑥 + 𝑏)) 𝑖=1 Where is the learning rate, which controls how much of w and b are updated. The gradient descent algorithm is an iterative procedure that keeps going until either a halting requirement is satisfied or the cost function achieves a low value.