Perceptron Algorithm and Binary Classification 2

Gradient Descent for Linear Regression Gradient descent minimizes the Mean Squared Error (MSE) which serves as the loss function to find the best-fit line. Gradient Descent is used to iteratively update the weights (coefficients) and bias by computing the gradient of the MSE with respect to these parameters. • The algorithm computes the gradient of the MSE with respect to the weights and biases. • It updates the weights (w) and bias (b) using the formula: • Calculating the gradient of the loss with respect to the weights. • Updating weights and biases iteratively to maximize the likelihood of the correct classification Gradient of J(w,b) with respect to b • Here we have considered the linear regression. So that here the parameters are weight and bias only. • But in a fully connected neural network model there can be multiple layers and multiple parameters but the concept will be the same everywhere. And the belowmentioned formula will work everywhere. 𝑃𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 =𝑃𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 −𝛾 Gradient Descent Learning Rate • The learning rate is a critical hyperparameter in the context of gradient descent, influencing the size of steps taken during the optimization process to update the model parameters. • When the learning rate is too small, the optimization process progresses very slowly. The model makes tiny updates to its parameters in each iteration, leading to sluggish convergence and potentially getting stuck in local minima. • an excessively large learning rate can cause the optimization algorithm to overshoot the optimal parameter values, leading to divergence or oscillations that hinder convergence. • Achieving the right balance is essential. A small learning rate might result in vanishing gradients and slow convergence, while a large learning rate may lead to overshooting and instability. Backpropagation • Backpropagation is a technique used in deep learning to train artificial neural networks particularly feedforward networks. It works iteratively to adjust weights and bias to minimize the cost function. • Backpropagation often uses optimization algorithms like gradient descent or stochastic gradient descent. • The algorithm computes the gradient using the chain rule from calculus allowing it to effectively navigate complex layers in the neural network to minimize the cost function. Working of Backpropagation Algorithm • The Backpropagation algorithm involves two main steps: the Forward Pass and the Backward Pass. • Forward pass: • In forward pass the input data is fed into the input layer. These inputs combined with their respective weights are passed to hidden layers. For example in a network with two hidden layers (h1 and h2 as shown in Fig.) the output from h1 serves as the input to h2. Before applying an activation function, a bias is added to the weighted inputs. • Each hidden layer applies an activation function like ReLU (Rectified Linear Unit) which returns the input if it’s positive and zero otherwise. This adds nonlinearity allowing the model to learn complex relationships in the data. • Finally the outputs from the last hidden layer are passed to the output layer where an activation function such as softmax converts the weighted outputs into probabilities for classification. Backward Pass • In the backward pass the error (the difference between the predicted and actual output) is propagated back through the network to adjust the weights and biases. One common method for error calculation is the Mean Squared Error (MSE) • Once the error is calculated the network adjusts weights using gradients which are computed with the chain rule. These gradients indicate how much each weight and bias should be adjusted to minimize the error in the next iteration. The backward pass continues layer by layer ensuring that the network learns and improves its performance.