Working of a Neural Network

So, let's take a little bit of a detailed look into the working of a neural network. So, let's say we want to classify some images between dogs and cats. How do we do this? This is known as a classification process, and we are trying to use neural networks and deep learning to implement this classification. So, how do we do that? This is how it works. You have a four-layer neural network: there is an input layer, there is an output layer, and then there are two hidden layers. What we do is we provide labeled training data, which means these images are fed to the network with the label saying, "Okay, this is a cat." The neural network is allowed to process it and come up with a prediction, saying whether it is a cat or a dog. Obviously, in the beginning, there may be mistakes. A cat may be classified as a dog, so we then say, "Okay, this is wrong. This output is wrong." But every time it predicts correctly, we say, "Yes, this output is correct." So, that learning process, it will go back and make some changes to its weights and biases. We again feed these inputs, and it will give us the output. We will check whether it is correct or not, and so on. This is an iterative process, which is known as the training process. So, we are training the neural network. And what happens in the training process? These weights and biases—you remember there were weights like w1, w2, and so on—these weights and biases keep changing every time you feed these, which is known as an epoch. So, there are multiple iterations. Every iteration is known as an epoch, and each time the weights are updated to make sure that the maximum number of images are classified correctly.So, once again, what is the input? This input could be like 1000 images of cats and dogs, and they are labeled because we know which is a cat and which is a dog. We feed those thousand images, and the neural network will initially assign some weights and biases for each neuron. It will try to process and extract the features from the images, and it will try to come up with a prediction for each image. That prediction, which is calculated by the network, is compared with the actual value, whether it is a cat or a dog. And that's how the error is calculated. So, let's say there are a thousand images, and in the first run, only 500 of them have been correctly classified. That means we are getting only 50% accuracy. So, we feed that information back to the network, further update these weights and biases for each of the neurons, and we run these inputs once again. It will try to calculate, extract the features, and it will try to predict which of these are cats and which are dogs. This time, let's say out of a thousand, 700 of them have been predicted correctly. So, that means in the second iteration, the accuracy has increased from 50% to 70%.All right, then we go back again. We feed this maybe for a third iteration, fourth iteration, and so on. Slowly and steadily, the accuracy of this network will keep increasing, and it may reach, probably, you never know, 90-95%. There are several parameters that are known as hyperparameters that need to be changed and tweaked, and that is the overall training process. Ultimately, at some point, we say, "Okay, you will probably never reach 100% accuracy," but then we set a limit, saying, "Okay, if we reach 95% accuracy, that is good enough for our application," and then we say, "Okay, our training process is done." So, that is the way training happens. And once the training is done, now, with the training data set, the system has, let's say, seen all these thousand images. Therefore, what we do is the next step, like in any normal machine learning process: we do the testing. We take a fresh set of images and feed them to the network, a fresh set which it has not seen before as part of the training process. And this is, again, nothing new in deep learning; this was there in machine learning as well. So, you feed the test images and then find out whether we are getting a similar accuracy or not. So maybe that accuracy may reduce a little bit. While training, you may get 98%, and then for the test, you may get 95%. But there shouldn't be a drastic drop. Like, for example, if you get 98% in training and then you get 50% or 40% with the test, that means your network has not learned, and you may have to retrain your network. So, that is the way neural network training works. And remember, the whole process is about changing these weights and biases and coming up with the optimal values of these weights and biases so that the accuracy is the maximum possible. All right, so a little bit more detail about how this whole thing works. This is known as **forward propagation**, where the data or the information is going in the forward direction. The inputs are taken, a weighted summation is done, a bias is added here, and then that is fed to the activation function. After that, the result comes out as an output. So, that is forward propagation. The output is then compared with the actual value, and that will give us the **error**—the difference between them is the error. In technical terms, that is also known as our **cost function**, and this is what we would like to minimize. There are different ways of defining the cost function, but one of the simplest ways is mean square error. So, it is nothing but the square of the difference of the errors, or the sum of the squares of the difference of the errors. And this is also nothing new. We have, probably, if you're familiar with machine learning, you must have come across this mean square. Now, there are different ways of defining cost function. It need not always be the mean square error, but the most common one is this. So, you define this cost function and you ask the system to minimize this error. So, we use what is known as an optimization function to minimize this error, and the error itself is sent back to the system as feedback, and that is known as backpropagation.