K-Fold Cross-Validation

So, in k-fold cross-validation, we basically split the dataset into k number of folds, or you can think about these folds as subsets. One chunk of data is used as test data for evaluation, and the remaining part of the data is used for training the model. Each time, a different chunk will be used as the test data. So, this is the formal explanation for this k-fold cross-validation. So, let's say that we have a dataset. This can be any dataset, so it can be the diabetes prediction dataset, where you have data about people with diabetes and people without diabetes. There can be, , you know, many factors such as their age, BMI, blood sugar level, etc. Okay, and let's say that there are about a thousand data points that are present in this dataset. When we do the train-test split, within these thousand data points, we will take 80% of the data as training data and 20% of the data as test data, where this 80% of the training data will be used to train our model, and the 20% remaining data will be used to test our model. Right? So, everyone knows up to this point. Instead of doing this train-test split, what we will do is, while using the k-fold cross-validation, we give some value for this k. Okay, so generally, we give the value as 5 or 10. So, it's the general thumb rule. So, let's say that we are giving the k value as 5. So, now what happens is this dataset will be split into 5 chunks, or you can call this as folds. This is the technical term that is used in. So, this is called as k-fold cross-validation. Again, when we talk about cross-validation, there are different types, and one of the mainly used types is k-fold cross-validation. In the end of this video, I’ll tell you what are all the different types. So, we have k number of folds, and now, when the k value is 5, we will be splitting the dataset into k number of folds. Okay, and, you can see here, so we have,, five folds here, right? So, here we have five iterations. In the first iteration, so, let's say that the fifth chunk of data or the fifth fold of data will be used as our test dataset. Okay, so as I’ve told you, let’s assume that there are thousand data points. So, if we split it into five folds, each fold will, in that case, contain 200 data points, right? Because 5 into 200 would be thousand, the entire data points number. So, each fold will have 200 data points, and this fifth chunk will be used as the test data for the first iteration. Let's say that we are using the logistic regression model for all the iterations. So, for the first iteration, our logistic regression model will be trained with all these four chunks of training data, and once the training is completed, it will be tested on the test data. This is the first iteration. For the second iteration, the fourth fold of the data, or you can think about this as the fourth chunk of data, so this fourth subset of the data will be used as the test data, and the remaining data will be used as the training data. So, this is the procedure. For the third iteration, this third chunk will be used as test data, and for the fourth iteration, the fourth, , the second, , you know, fold is used as test data, and the remaining data is used as training data, and so on. So, similarly, we have, , you know, five iterations. So, the number of folds of your data is, , you know, probably equal to the number of iterations that you have. Okay, and what we will do is, in each step, we will try to find the accuracy of the model on the test data. Okay, so as we know that once we train our model, we will evaluate our model using the test data, right? And if it is a classification problem, we may get accuracy like 80%, 90%, or 95%. So, 90% accuracy means, out of 100 data points, your model can predict correctly for 90 data points out of 100. So, that is called as 90% accuracy rate. So, for each iteration, we will try to get this accuracy score, and once we have all these five accuracy scores, we will, , you know, take the average of that accuracy score, and this will be the final accuracy score. Whereas, if you think about train-test split, what we will do is we will just take, , the entire data, as I have told you before. So, 80% of the data will be training data, and 20% of the data will be your test data, right? So, if we think about this number, the training data will be 800, and the test data will be 200, right? So, the advantage of k-fold cross-validation is that you are testing your model with different chunks of data, you know, different data points. So, I'll, , explain to you what is the advantage with an analogy or an example. Let's say that a student, you know, is a secondary student preparing for his JEE exam or something, and we know that there is, you know, physics, chemistry, and so on. Let's say that there is a chapter called semiconductors, which we all would have studied, right? And that particular student has studied all the concepts and formulas, and now, you know, we are going to test it based on the previous year questions. Okay, and let's say that in the question bank, there are about, , you know, 200—let’s say that there are about 100 total previous questions that are based on the chapter semiconductors. Now, let's say that we are taking the entire, , you know, questions.So, out of those 100 questions, we are going to, , test that particular student on 20% of data. So, out of the 100 questions, we will be asking only 20 questions to him, and we will check how he is performing on those 20 questions, right? And let's say that he has, , answered correctly for 15 questions out of that particular 20 questions. Now, we will try to, , you know, evaluate him, and what is this percentage, and so on. Now, the problem with this approach is that the particular student has seen only that 20% of data, and this is probably not a good evaluation. That requires, let’s say, that we are, you know, subjecting that particular student to a different set of questions, so it would give a more reliable idea on how much he has understood that particular subject, right? So, that is the same idea here. So, when you have a small dataset, when you just, , use the train-test split function, your model will be evaluated on a very small portion of your data, and we cannot use that as the proper accuracy score. Whereas, if we use k-fold cross-validation, we kind of use different test data for each, , you know, training session. Okay, and when you have that five accuracy scores, we will get a more reliable score. So, that is the idea behind it. And, , another thing that you need to note here is, so once you, let's say, that you are training your model for this first iteration where this fifth chunk of your data is the test data, so after that iteration—so that is like, , and, you know, once we find this accuracy score, that is all that we are doing. So, we will create a new instance of the logistic regression model, and we will train that with the test data.So, here you need to understand that we are not training the same model which has been already trained for the second iteration. So, we have to use a new instance of that particular model. So, this is very, very important because, like, let's say that you are using the, you know, same model, same trained model. So, in that case, there is a possibility that that model has already seen the data that is present in this test, right? So, the model already knows what is present in this test data because this is the training data in the previous step. So, I hope you understand this. I'll just, you know, quickly repeat this particular thing because it is very, very important. Let’s say that we are training our logistic regression model with this first iteration, and in this case, the first four chunks of data are your training data, and the final fifth chunk is your test data. If you use the same model for training on top of it, now the model knows what this test data is because it has already trained on that particular chunk. So, in each iteration, we will, , you know, create a new instance of that logistic regression model or whatever model we are using and test it for different accuracy scores with different test data. So, this is very important. Okay, now you will get a reliable, , score. So, the basic idea is you need to, , check whether your model is performing on the overall dataset. So, that’s the idea behind this. And one of the main applications where we use this k-fold cross-validation is when it comes to model selection. Let’s say that we have this support vector machine and logistic regression, and for a particular, , case or a particular dataset, we want to determine which model is the most efficient and which model is the accurate one. Okay, so in that case, instead of going with a single train-test split and a single evaluation metric, which is accuracy score, we can use k-fold cross-validation, and it will give a more reliable accuracy score or evaluation metric. So, that’s the idea. So, similarly, let’s say that we have a dataset, and let’s say that this is our, , you know, the same diabetes dataset that we have been working on lately. So, you know, it is a famous dataset, so we all know that, right? So, we have the diabetes dataset, and in here, and let’s say that the k-fold or the k value is five, and we are going to implement this k-fold cross-validation. First, we are using a support vector machine. So, first, we will load an instance of this support vector machine and train it with the first four folds, or four chunks. Here, I mean folds and chunks, which, , basically mean the same thing here, okay? So, we are taking the first four folds or the chunks of data as training data, and the fifth chunk of data is the test data, and then we will calculate the accuracy for this particular case. So, the accuracy on test data, which is very important, and not on training data. In the second iteration, we will load a new instance of, , the support vector machine model, and now, , the fourth chunk of data will be your test data, and the remaining data will be used as the training data. Now, let’s say that the accuracy score is 83%. So, your accuracy will definitely change because here, the test data is different, okay? And the training data is again different. So, there will be a slight change in your accuracy. In most cases, the accuracy change won’t be that much, but in some cases, where the dataset is, you know, very complex, and, , when some of the data are missed in this split and all that, there will be a significant change in this accuracy.So, let’s say the accuracy in the second iteration is 83%, and in the third iteration, where this third chunk is used as the test data, let’s say the accuracy is 86%. And for the fourth iteration, let’s say it’s 81%, and the final iteration, it’s 84%. So, as you can see here, there is a slight change in accuracy. So, instead of, , you know, taking one accuracy, which we will do while we are using train-test split, in k-fold cross-validation, we will take all these accuracies into consideration, and we will divide it by five because there are, you know, totally five accuracies. So, we are just taking the mean accuracy of all the accuracy values. The average of this particular accuracy will be 84.4%. So, you can say that the reliable accuracy value for the support vector machine for this particular dataset is 84.4%. So, once we do this with the support vector machine, we will do the same procedure with a logistic regression model. Okay, so every procedure is the same. So, in this case, now we have 90% accuracy for the first iteration, and the second iteration, it’s 88%. The third is 86%, the fourth is 91%, and the fifth is 85%. So, these are all the five accuracies we are getting for logistic regression, and in this case, let’s say that the mean accuracy is 88%. Now, we can compare this accuracy with the mean accuracy of the support vector machine model. we can basically compare them and say in this case the logistic regression model works.Well, so this is one of the things that we do while we are, you know, selecting the model. For now, understand that this can be used as one of the steps in the model selection procedure, where we need to select which is the more optimum model for a particular dataset. So, here we can say that the average accuracy for the support vector machine model for a particular dataset is 84.4%, whereas the mean accuracy or the average accuracy for the logistic regression model is 88%. So, we can say that this is the best model for this particular case. Okay, so this is how we use cross-validation in order to select our model. Now, let's try to understand what are all the advantages of or, you know, what are all the pros of using this k-fold cross-validation. The first step is, you know, it's a better alternative for train-test split when the dataset is small. So, this is very, very important. So, if you, let’s say, that you have a very large dataset—so let’s say that that particular dataset has about 10,000 data points or 100,000 data points or something like that. So, in that case, you can actually go with the train-test split because the test dataset will contain a huge proportion of the data, even for 20% of the data. Okay, so if you think about 20% of data for the 10,000 data points or 20% data for 100,000 data points, that is pretty large. Okay, so in large datasets, you can actually go with train-test split. Whereas, if your dataset is small, then k-fold cross-validation is a better idea because, in that case, you can check how your model is performing for the overall dataset and so on. And the other reason, , k-fold cross-validation is not advisable for large datasets is because, so, when your dataset is large, the training is going to take a long time, right? So, in that case, k-fold cross-validation is not a proper, , step because, like, each iteration may take several hours for training that particular model, and we cannot train the model for, you know, five iterations if we don't have that much training time and so on. So, it is a trade-off between the amount of time that you can spend and the dataset and other things and so on. So, it is a trade-off between time and, , the accuracy of the model. So, that is what I would say. So, if it is a small dataset, then k-fold cross-validation is actually a better idea. , it's up to you. So, if you have, you know, a good amount of time, you can actually use k-fold cross-validation in a large dataset itself, but it is, you know, as I told you, it's completely up to you whichever you can, , you know, use. But k-fold cross-validation is always a better idea when we compare it to train-test split.