Gradient Descent - Algorithm

This is how the gradient descent algorithm looks like, but the gradient descent algorithm expects two parameters. The first parameter is the training example, and the second one is the learning rate. The training example is a pair (X, T), where X is the vector of input values and T is the target associated with that particular example. The learning rate, denoted by "n," is usually set to a very small number, which will determine the factor by which the weight should be updated. It should be a small number, but not too small, because if it is very small, it may take a long time to converge to the proper weights. On the other hand, if it is too large, the algorithm may not converge at all. Next, what we need to do is initialize each of these particular weights. For example, if we have a certain number of input variables, we need to have an equal number of weights. All these weights should be initialized with a small random value. Once you initialize all these inputs with a small random number Next, what we need to do is, unless and until a termination condition is satisfied, that is nothing but, after every iteration, we calculate something called error. If that error is acceptable, then we stop executing or updating the weights. Otherwise, we will go on doing these particular steps again and again. That's one termination condition. The second termination condition may be the number of epochs. For example, if I set the value of epochs equal to 5, these steps will be executed five times. After the fifth iteration, whatever value you will get for the weights, that will be the final weights. So, that is how actually this termination condition will be determined in this particular case. Now, what we do in these particular steps is update the weights. So, what is the weight updation equation? This is how it looks like: Wi = Wi + ΔWi, where Wi is the old weight. To this particular old weight, we add, let's say, the change in weight. That change in weight is always equivalent to n, which is the learning rate, multiplied by the summation of (TD - OD) over all training examples, and multiplied by XID, where TD is the target output, OD is the calculated output, and XID is the input in this particular case. So, once you get this ΔWi, we will add it to Wi to get the new weight in this case. Now, how to do this particular thing is written in these particular steps over here. First, we will initialize this ΔWi to 0 for every iteration before we continue.We will initialize this to 0 here first. Next, what we do is, we will take one example at a time. For that particular example, what we do is, let us say that the first example I have considered. What I do is, learning rate multiplied by Target minus calculated output multiplied by x₁ in this case. So, that is what I will do. Similarly, I will go back and then take the second one, that is nothing but, once again, n multiplied by T₂ minus O₂ multiplied by x₂. So, that is the second input. The same thing will be repeated again and again, and that partial error will be added to this particular Δwi. So, initially, it is 0. After seeing the first example, it will become Δwi = Δwi + that error. When it sees the second input, it will become Δwi = previous error + second example error. The same thing will be done for all the inputs. In this case, once all the inputs are over, we will get the final change in weight. That particular change in weight will be added to this particular wi. That is written here: for each linear input weight, Wi = Wi + Δwi, what we have calculated in the previous step. So, this Δwi is calculated over all training examples. Once it is calculated, we will add the Δwi to the previous weight, and we will get the new weight in this case. So, this is how the gradient descent algorithm works in this case. Now, coming back to the second one, that is the stochastic version of the gradient descent algorithm. Before we go to that particular thing, first, we will see what the disadvantage is. The disadvantage in this case is that the error or the change in weight is calculated by considering all examples. For example, if you have a thousand number of examples, you have to do this particular thing for all the thousand examples. You will get the change in weight that you have to update here, and then you need to calculate whether the termination condition is satisfied or not. If not, again, we have to calculate the change in weight over all training examples, and then we have to add it here. So, that will take a lot of time, and sometimes, what happens is it may not converge to the final weights over here.