Feature Scaling: The Key to Faster Gradient Descent in ML

University:
Stanford University
Course:
CS229 | Machine Learning
Academic year:

2023
Views:

411

Pages:

2
Author:

anon

Understanding Feature Scaling for Improved Gradient Descent in Machine Learning Machine learning is a rapidly evolving field with new techniques being developed all the time to improve model performance. One such technique is feature scaling, which plays a crucial role in enabling gradient descent to run much faster. This article will explore the relationship between the size of a feature and its associated parameter, and how feature scaling can improve the performance of gradient descent. The relationship between feature size and parameter size In order to understand the relationship between feature size and parameter size, let's take a concrete example of predicting the price of a house. We'll use two features, the size of the house (𝑥1) and the number of bedrooms (𝑥2), to make our prediction. Let's say that 𝑥1 typically ranges from 300 to 2000 square feet, while 𝑥2 in the data set ranges from 0 to 5 bedrooms. When we consider a house that has a size of 2000 square feet, five bedrooms, and a price of $500,000, we can see that different choices of parameters can result in vastly different predicted prices. For example, if 𝑤1 is 50 and 𝑤2 is 0.1, and b is 50, the estimated price would be 100,000 + 0.5 + 50, which is over $100 million. Clearly, this is not a reasonable estimate. On the other hand, if 𝑤1 is 0.1 and 𝑤2 is 50, and b is still 50, the predicted price would be 0. 1 × 2000 + 50 × 5 + 50, which is equal to $500,000, a much more reasonable estimate. The impact of feature scaling on gradient descent Now let's consider the relationship between feature scaling and gradient descent. If we plot the training data on a scatter plot, with the size of the house as the horizontal axis and the number of bedrooms as the vertical axis, we can see that the horizontal axis has a much larger range of values compared to the vertical axis. This can have a significant impact on the cost function in a contour plot, where the horizontal axis may have a much narrower range, while the vertical axis takes on much larger values. As a result, the contours formed by the cost function may be elongated and narrow in one direction, with a very small change in w1 having a large impact on the estimated price. This can result in slow convergence and a suboptimal solution. On the other hand, if we apply feature scaling to both features, so that their ranges of values are more similar, the contours will be more circular and the algorithm will converge faster to a better solution.