Training, Development, Testing Set

Course : Machine Learning Lecture : Joseph Credit Hours : 3 Credits Semester Sessions : 3 Semester, 2021/2022 Session Prepared By : Mindie LECTURE NOTES Training, Development, Testing Set There are two important terms in building machine learning models namely: training and testing. Training is the process of building models and testing is the process of testing the performance of a learning model. Datasets are data set (sample in statistics). This sample is our data used to create models and evaluate machine learning modelsing. Generally, datasets are divided into three nonintersecting types (one samples in certain sets do not appear in other sets):  Training set is a set of data used to train or building models. In this book, the term training data(set) refers on the training set.  Development set or validation set is a data set that is use it to optimize when training the model. The model is trained to use training sets and in general performance during training is tested using bro development set. This is useful for generalization (so that the model able to recognize generic patterns). In this book, the term development/validation data(set) refers to the same thing.  Testing set is a set of data used to test the model.after the training process is complete. In this book, the term testing data(set) or test set refers to the testing set. We need to emphasize, testing set is unseen data. This means that models and humans cannot see each other. This mop during the training process. Many people are tempted to look at testing set during the training process even though it is bad behavior.because it causes bias. We call one sample in a data set a data point or instans (instances) which represent a statistical event (event). Please remember, training, development, and testing data are taken (sampled) from.the same distribution and have the same characteristics (independently.and identically distributed). Distribution on each of these datasets as well.it should be balanced and cover all cases. For example, a.binary classification dataset should contain 50% positive cases and 50%.negative case. In general, the dataset sharing ratio is (80% : 10% : 10%) or (90% : 5% : 5%) (training:development:testing). Development set on Generally it can not be used if the dataset is small (only divided be a training and testing set only). In this case, dataset sharing.into training and testing sets generally have a ratio of (90% : 10%), (80% : 20%), (70% : 30%), or (50% : 50%). In this case, the current performance training is tested using a training set (known as closed testing). When not using the development set (only training and testing set), we also have the option to evaluate the model with the K-method cross-validation11. That is, we divide the training dataset into K parts. We use K − 1 parts for training, then test the performance model during training (validation) using one part. This is repeated.K times where a piece of data is used as a testing set.a lot (take turns).