Multi-Layer Perceptron Learning - Solved Example

In this case, we have been given a multi-layer perceptron network with the input layer, one hidden layer, and the output layer. In this case, the input layer has four neurons, the hidden layer has two neurons. You can notice here, X1, X2, X3, X4 are the input layer neurons, X5 and X6 are the hidden layer neurons, and X7 is the output layer neuron. In this case, X0 is a bias with respect to the hidden layer neurons as well as the output layer neuron. In this case, we need to train the multi-layer perceptron by updating the weights as well as the biases in the network. The learning rate given to us is 8 in this case, and the inputs are X1, X2, X3, X4, which are 1, 1, 0, 1, respectively, and the desired output, that is, the target output in this case, is one. That is what is expected from this particular multi-layer perceptron network. Along with this, the weights are also given to us. I have represented that particular weight in this particular table. X1, X2, X3, X4 are the inputs. In this case, \(W_{15}\), that is nothing but the weight from X1 to X5, which is equivalent to 3, is given in the network. The same thing I have represented in this particular table. \(W_{16}\) is the weight from X1 to X6, here that is nothing but 0.1. Here, the same thing I have written in this case with the same note. I have written all these particular initial weights. \( \Theta_5 \) is the weight from this particular bias, that is, \(X_0\) to this particular \(X_5\), and \( \Theta_6 \) is the weight from \(X_0\) to \(X_6\). Here, they are 2 and 0.1, respectively. The same thing I have written here, and \( \Theta_7 \) is the weight from this particular bias to this \(X_7\), that is equal to -0.3 in this case. Now, given this particular thing, what we need to do is, we need to train the multi-layer perceptron network, and then we need to update these particular weights. In the first step of multi-layer perceptron learning, we need to calculate the input and output of these particular input layer neurons. So, there are four input layer neurons, and we need to calculate the input as well as the output here. The input to these particular X1, X2, X3, X4 are already given to us. The output of these particular neurons is exactly the same because we don't do any modification in this particular input layer. So, the output at these particular input layer neurons is exactly similar to this input. Input is 1, 1, 0, 1, and the output is 1, 1, 0, 1, respectively, over here. So, once you calculate the input and output at these input layer neurons, now we need to calculate the net input and the output at the hidden layer neurons as well as the output layer neurons here. So, that is what I'm doing in this particular table. The hidden layer neurons are X5 and X6 here, and the output layer neuron is X7 in this case. So, first, we need to calculate the net input for X5, and then we need to calculate the output here. Now, the question is, how to calculate the net input? Net input is nothing but, in this case, you can see here, I want to calculate the net input at X5. That is nothing but, we need to multiply the input and the weights over here. So, input is \(X1 \times W_{15}\) here, plus \(X2 \times W_{25}\), plus \(X3 \times W_{35}\), and \(X4 \times W_{45}\), here. Along with this, we need to add the bias and its associated weight also, that is \(X0 \times \Theta_5\). Here, the same thing I have written in this particular equation, that is nothing but \(X1 \times W_{15} + X2 \times W_{25}\) and so on. We know the values of all these particular variables. We need to put them over here, and you will be getting the net input in this case. The net input at this X5 is equivalent to 0.4 here. Once you get the net input, we need to calculate the output at this particular neuron. That is, \(O_5 = \frac{1}{1 + e^{-\text{net input}}}\). Here, what is the net input we have calculated? That is 0.4, so \( \frac{1}{1 + e^{-0.4}} \). Once you solve it, you will get 0.599 in this case. Similarly, we need to calculate the net input at X6, as well as we need to calculate the output here. The net input at this particular X6 is \(X1 \times W_{16} + X2 \times W_{26} + X3 \times W_{36} + X4 \times W_{46} + X0 \times \Theta_6\). Here, that is what I have written in this particular equation, and then we need to put the values from this particular table. You will get the net input at X6 is equal to 1.2 here, and the associated output, that is \(O_6 = \frac{1}{1 + e^{-\text{net input}}}\). The \(I_6\) value is known to us, that is 1.2. Put that particular value in this equation, and you will get the output as 0.769 in this case. Similarly, we need to calculate the net input at X7. X7 is the output layer neuron, and then we need to calculate the output over here. The output layer net input is calculated here, and then the output at the output layer neuron is 0.419 in this case. So, this is the output of this particular multi-layer perceptron network in this case. So, once you calculate this output at this output layer neuron, we need to calculate the error. In this particular case, the error is always equivalent to \( O_{\text{desired}} \), that is nothing but the desired output minus the estimated output. Here, what is the desired output? The desired output is equivalent to one; that is what is given in the problem definition, and the estimated output, that is nothing but the calculated output, is 0.419. That is what we have calculated in the previous slide over here. Now, once you put these particular values in this equation, you will get the error at the output layer is 0.581 in this case. So, this is the error we got after the first epoch, or you can say that the first iteration in this case. Now, the question is, if you are happy with this particular error, there is no need to do anything. Here, you can stop the training of this particular multi-layer perceptron network, and then you can deploy it in the real world. But if you're not happy with this particular error term, what we need to do is we need to propagate this particular error backwards, and then we need to update the weights so that we can reduce this particular error. So, we have been given the weights; we need to update these particular weights so that you can reduce this particular error in this case. So, what we do is we will try to propagate this particular error backwards, and then we will try to update the weights, and then we will try to see whether the error is reduced or not in the next iteration. The question comes in point of view as: how can we propagate this particular error backwards? Now, if you want to propagate this particular error backwards, you need to remember these two equations. The first one is the error at output layer neurons, and the error at this particular hidden layer neurons here. The error at output layer neuron is always calculated with this equation: Here, \(O_k\) is the output at the output layer neuron, \(1 - O_k\) is the factor for the sigmoid activation function, and \(O_{\text{desired}} - O_k\) is the difference between the desired output and the calculated output. The error at hidden layer neurons can be calculated with this equation: Where \(O_J\) is the output at the hidden layer neuron, \(1 - O_J\) is the factor for the sigmoid activation function, and the summation term is over all the output layer neurons (\(k\)). In this case, we have only one output layer neuron. If you have multiple output layer neurons, we need to take the summation term over here. The error at the output layer neuron (\(\text{Error}_k\)) is multiplied by the weight \(W_{JK}\), where \(J\) is the hidden layer neuron and \(K\) is the output layer neuron. With this particular equation, we will be able to calculate the error at these particular hidden layer neurons. Once you calculate this particular error, we need to update the weights over here. So, first we will try to calculate the error at each neuron. In this case, the output layer neuron is \( X_7 \). For this particular \( X_7 \), the error is: Here, \( O_7 \) is the calculated output, which we have already calculated in the previous slide and is equal to 0.419. The desired output \( Y_{\text{desired}} \) is 1. Now, we plug these values into the equation: Once you solve this, you will get the error term at the output layer neuron as 1.141 in this case. Similarly, we will try to calculate the error at the hidden layer neurons, that is \( X_6 \) and \( X_5 \). For \( X_6 \), the error is: Here, we have only one output layer neuron, so the summation term will involve only \( k = 7 \). So, for \( X_6 \), the equation becomes: Once you put all the values, you will get the error term at \( X_6 \) in this case. Similarly, for \( X_5 \), the error term is calculated using the same method: Once all these calculations are done, we have the error terms for the output layer neuron and the hidden layer neurons. Now, once you do this particular thing, we need to update the weights. To update the weights, we need to know the learning rate. Learning rate is already given to us, that is, equivalent to 0.8, as well as we need to know the equation to update the weights. That is, \( W_J = W + \alpha \), that is, the old weight plus alpha (that is nothing but the learning rate) multiplied by the error at that particular neuron multiplied by the output at that particular neuron here. So, once you solve this particular equation, you are going to get the updated weights over here. I will show you a few calculations, the remaining you can do it by yourself. \( W_{15} \), that is, we are trying to modify or update the weights at \( W_{15} \) here. So, \( W_{15} = W_{15} \), that is, the old weight. We know the old weight; what is the old weight over here? That is nothing but 3 here. Plus, 0.8 is nothing but what? The learning rate, multiplied by error 5, that is error 5 we have already calculated in the previous slide, multiplied by \( O_i \). \( i \) is nothing but what? That's, in this case, you can see here \( W_{15} \), that is \( W_{i,j} \), so \( i \) is equal to 1. So, that is nothing but \( O_1 \) here. So, \( O_1 \) is nothing but what? The output at neuron 1. In this case, the output at neuron 1, the neuron 1 is this. This is \( X_1 \); the output at this particular neuron is nothing but the input only. That is nothing but 1 over here. So, that is what I have added in this particular case. So, once you solve this particular equation, you will get the new weight in this case. Similarly, we need to calculate or update the weights at \( W_{16}, W_{25}, W_{26}, W_{35}, W_{36}, W_{45}, W_{46} \), and so on. The updated weights are written in this particular table here. So, once you update these particular weights, the next step is to update the bias. To update the bias, we use a similar approach as we did for the weights. We use this particular equation, that is, Theta J is equal to Theta J plus the learning rate multiplied by the error. Here, we don't consider the output because the output is always equal to one in this particular case, so we don't consider the output. We only consider the old weight plus the learning rate multiplied by the error at that particular neuron in this case. So, Theta 5 is equal to Theta 5 plus Theta 5 (the old weight) plus Alpha (that is, the learning rate, which is equal to 8) multiplied by error 5. Error 5 is already known to us. We put all these particular values, and we will get the new bias here. Similarly, we need to calculate Theta 6 and Theta 7 in this case. So, once you calculate the updated weights as well as the updated biases here.Now, what we need to do is, we need to propagate the input again from the input layer neuron to the output layer neuron, and then we need to calculate the error over here. So, in the first step, again, we need to calculate the output at the input layer neuron. The output at the input layer neuron is exactly similar to the input here. So, input is one, output is one here; input is one, output is one here; and so on. So, there is no need to do any calculation at the input layer neuron. Now, we need to calculate the net input as well as the output at the hidden layer neuron, as well as the output layer neuron here. So, that is nothing but X5. We need to calculate the net input as well as the output over here. This equation I have written already in the previous slide; the only thing is we need to put the updated values here, that is the updated weights. So, once you put this particular updated weight, we are getting I5 = 3.68, and O5 = 0.591 here. Similarly, we need to calculate the net input for this X6, as well as the output here, and the net input for X7, as well as the output here. This output is nothing but the output of this particular multi-layer perceptron network after updating the weights here. This is the final output of this particular multi-layer perceptron learning network. So, we will try to calculate the error again here. So, error is equivalent to 1 - 0.474. What is the value here? 0.526 in this particular case. So, this is the error we are getting after updating the weights in this case. Now, in the previous case, what is the error we got? In the previous case, we got the error as 0.581. Now, we are getting the error as 0.526. If you calculate the difference, you will get the change in error in this case. The change in error is 0.055. The meaning is the previous error was this much, the current error is this much, and the error is getting reduced by this particular value here. The meaning is after updating the weights, the error of this particular perceptron network has been reduced in this case. Now, if you're happy with this particular error, you can stop here. If you're not happy, again, we need to modify the weights by updating the errors at each and every neuron, and then we need to do this particular calculation again and again. So, in this case, I have shown only two epochs. If you're happy with this particular error, you can stop here. If you're not happy, we need to repeat this particular process again and again over here so that you will get the minimum error at the end of the day.