So, the cost function in machine learning is the error representation. It shows how our model is predicting compared to the actual values or actual output labels. The lesser the cost function, the more accurate our model is, and the higher the cost function, the lesser the accuracy of our model in a neural network. We use different types of cost functions based on the application we are trying to make or the problem we are trying to solve. In this video, we will look at the cost functions for three different types of applications, which are for: 1. Regression or linear regression problems 2. Binary classification, where the output label can only take either of the two values 3. Multi-class classification, where the output can be in the form of any number of categories. Let's first start by looking at the cost function for the regression type of problem. In regression or linear regression, the output can take any number. An example for the regression type of problem can be house price prediction. Now, as the cost function is just the error representation, we can represent the error for one observation by taking the difference between the predicted value and the actual value, where here \( y_i \) is the actual output label and \( a_i \) is the predicted value. Now, we can just take the modulus of it to get the absolute value of this error, but this is the error for only one observation. Let's say if we have \( m \) number of observations, then we can take the average of all those errors, and we will get the cost function. So, the cost function can be given by taking the average of all those errors. Now, we can also square this error and take the \( \frac{1}{2m} \), and this will again be represented as a cost function and can be used as a different type of cost function that does the same job for linear regression. So, the other form of the cost function can be given by taking \( \frac{1}{2m} \) and summing over all the errors, just taking the square here, and both of these can be represented as the cost function for regression-type problems. This cost function is called mean absolute error, while this is called mean squared error. Now, also note one more thing here: you might have heard of the term *loss* or *loss function*. So, what is that? The loss is the error for one observation. So here, our error was \( a_i - y_i \), so the loss will be given by \( e_i - y_i \) for the \( i^{th} \) observation, and the cost function will be given by the average of all those errors. Now, let's look at the cost function for the binary classification. Now, we cannot use the previously seen cost function in binary classification because the output takes only 0 or 1 as its value, so we use a different type of cost function. An error for one observation in this can be given by minus of \(y_i\) multiplied by log of \(a_i\) plus \(1 - y_i\) log of \(1 a_i\), and the cost function will again be given by taking the average of all these errors. Now, let's look at how this term can represent the error for one observation. The graph of the negative of the logarithm looks something like this. So, when, let's say, the output value or output prediction \(y_i\) is equal to 0, then this term will be canceled. So, we will be only left with the other term, which was minus of \(1 - y_i\) log of \(1 - a_i\). And as \(y_i\) is 0, so it will be minus of log of \(1 - a_i\). Now, if you see, let's say \(a_i\) value approaches to 0, which is, let's say, close to 0, then \(1 - y_i\) will be closer to 1, and thus the log of \(1 - a_i\) will be around this region. So, you can see the value is pretty close to 0. Thus, we can say that the error is less when \(a_i\) is approaching 0, and let's say when \(a_i\) is approaching 1, then \(1 - a_i\) will be approaching 0, and the value for the minus of log will be very high. So, when the \(a_i\) value is approaching 1, then we have high error. Thus, you can see that as \(y_i\) is close to the \(a_i\), we get less error, and if the \(a_i\) is far from \(y_i\), we get high error. Similarly, let's say if \(y_i\) is equal to 1, then this term will be canceled, and we will only be left with this term. So, our error will be equal to minus of log of \(a_i\), and if the \(a_i\) is approaching 1, then we will be having a smaller error, and if the \(a_i\) is approaching 0, then we will be having a higher error. So, here it also makes it clear that as \(a_i\) approaches the value closer to \(y_i\), we get less error, and if the \(a_i\) is far from \(y_i\), we get more error. Thus, this term can be represented as the error for one observation, and this will be the formula for the cost function for binary classification type of problem. If you know about logistic regression, then you will know that this is the formula we use for logistic regression, and the same formula will be used for binary classification in the neural network. So, this can also be called the logistic regression cost function, and the other name for this cost function is also binary cross-entropy. You might come across this term a lot while dealing with neural networks because, in neural networks, we can call this function binary cross-entropy. Now, for binary classification, the output can take many categories, so again, we cannot use the previously seen binary classification cost function in this. For multi-class classification, the output label for the \(i\)-th observation is going to be given by a one-hot representation. One-hot representation means there is a 1 at one position, which can be any of these positions, and 0 at the other positions. So, this 0 is for this neuron, this 1 is for this neuron, and similarly, these 2, and the predictions for the \(i\)-th observation can be given by this vector, where it takes these values. So, it will be 0.1, 0.5, 0.3. Thus, the error for this observation can be given by taking the minus of \(y_{i1}\), where \(y_{i1}\) is this value, multiplied by log of \(a_{i1}\), where \(a_{i1}\) is this value, plus \(y_{i2}\) log of \(a_{i2}\), plus \(y_{i3}\) log of \(a_{i3}\), plus \(y_{i4}\) log of \(a_{i4}\), so \(a_{i2}\), \(a_{i3}\), and \(a_{i4}\) are these values. Again, if we have \(m\) number of observations, then we can just take the average of these and it will be our cost function. Also, note one more thing here: for every observation, we are going to have one at one position and zero at the rest of the positions. So, for this observation, all these terms are going to be zero, and our error is just going to be this value, which is going to be minus of log of 0.5. Thus, the cost function for multi-class classification can be given by this formula, where \(n\) represents the total number of neurons in the output layer, while \(m\) is the total number of observations. This cost function is also called the categorical cross-entropy.