Optimization Of Hyper Parameters Of Convolutional Neural Networks

Article Study Case OPTIMIZATION OF HYPER PARAMETERS OF CONVOLUTIONAL NEURAL NETWORKS USING ANT COLONY OPTIMIZATION Amelia Rose Smith Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA ameliaroses2019@gmail.com Abstract Fields such as agriculture and healthcare face challenges in image classification that can be addressed through various methods. One approach used is combining convolutional neural networks (CNN) with deep learning. The performance of these methods is greatly affected by hyperparameters such as loss functions, activation functions, and optimizers, which require optimization. Existing methods for optimization, such as genetic algorithms and ant colony optimization, can be applied for this purpose. Ant colony optimization has been proven effective in optimizing deep learning. This research contributes to the automatic tuning of various hyperparameters using ant colony for image classification. In this study, the MNIST dataset is used to identify digits in images. The dataset is divided into two parts: a training dataset with 33,600 images and a validation dataset with 8,400 images. The results show that ant colony optimization achieves 97.46% accuracy in the validation data and 99.69% in the training data, which is superior to the genetic algorithm that achieves 94.60% and 97.59% accuracy in the validation and training data, respectively. In addition, the ant colony optimization took 27.94 seconds for training, while the genetic algorithm took 22.25 seconds. Keywords: optimization, hyper parameters, convolutional neural, networks, ant colony, optimization INTRODUCTION Efforts have been made to address image classification problems through various approaches, such as machine learning techniques including support vector machines (SVM) and random forests, with the aim of achieving better results. Many studies use deep learning approaches, in conjunction with machine learning, to solve image classification problems because deep learning has proven to be very effective in this task. Deep learning has proven to be effective in solving many image classification problems in various domains, including the healthcare field. For example, deep learning can be used to classify COVID-19 based on lung CT scan images. In addition, deep learning can help in solving classification problems in remote sensing. In agriculture, it can be used to detect plant diseases and pests. AMELIA ROSE SMITH 1 Amelia Rose Smith Deep learning is able to classify different types of problems because it can learn important characteristics of image data. Important features of an image can be extracted using a convolutional neural network (CNN). The effectiveness of deep learning is not only influenced by the significant features obtained from the extracted image, but many factors can affect the performance of deep learning. One of these factors is the loss function used. Ali Akbari's research has shown that choosing the right loss function is very important in improving deep learning performance. In addition to the loss function aspect, the quantity and diversity of data used in deep learning can affect deep learning performance. According to the study conducted by S. Vani, optimizer selection can also affect deep learning performance. Furthermore, the key factor to get good performance results is for experts to make the appropriate modeling or architecture, as the model or architecture used can have a significant impact. loss function, activation function, optimizer, and the previously mentioned factors are hyperparameters. This study aims to configure the hyperparameters of deep learning models automatically, eliminating the need to try all the potential hyperparameters available. STUDY OF LITERATURE Optimization techniques can be used to auto-tune the hyperparameters in deep learning models. There are various methods that can be used in optimization techniques. One of them is ant colony optimization. This optimization is inspired by the behavior of ants that track the pheromone trails of other ants. Ant colony optimization is proven to work together with deep learning in solving various problems, such as Human gait recognition (HGR) which aims to be an alternative choice of biometry techniques. In addition to biometry, the health field also uses deep learning and ant colony optimization to classify various stomach diseases. Another example in agriculture uses ant colony optimization and deep learning to create a crop prediction recommendation system. All the above-mentioned experiments have proven that using ant colony optimization techniques and combining them with deep learning can yield great results. In the experiment by Khan Awais in making alternative biometry using Human gait recognition (HGR) by utilizing ant colony optimization to improve feature extraction from videos combined with deep learning. Whereas in the experiment conducted by MA Khan in classifying gastric diseases using deep learning and ant colony optimization for preprocessing AMELIA ROSE SMITH 2 Article Study Case and image filtering. Then another study was conducted by Micheal Olaolu Arowolo in detecting malaria using analysis of variance (ANOVA) and ant colony optimization (ACO) as a hybrid feature selection in selecting relevant genes. After knowing that ant colony can improve deep learning performance in various cases, it is important to know that there are many things that affect deep learning performance. One example is model configuration. Other things that can affect performance besides model configuration include the optimization used in training deep learning models. There are various types of optimizers available, such as Adam, RMSprop, SGD, Adadelta, Adagrad, and Adamax, each with its own strengths and weaknesses. For example, Stochastic Gradient Descent (SGD) is an optimizer used in many areas, ranging from machine learning and big data to deep learning. SGD is a popular choice due to its simple iterative structure, which is suitable for solving optimization problems. A study by Avinash Kumar in malaria detection found that using SGD optimization resulted in a high performance accuracy of 95%. This study also showed that the choice of optimizer can have a significant impact on the performance of a particular model and data set. The data set used to train the model can have a significant impact on the effectiveness of deep learning, especially when there is limited data available. In addition to model and optimizer configurations, the variety and size of datasets have an impact on deep learning effectiveness. This study uses the MNIST dataset, which is commonly used for image recognition tasks, to identify digits in images. The dataset consists of 42,000 images with 10 classes ranging from 0 to 9. The dataset is divided into two parts: training dataset and validation dataset. The training dataset contains 33,600 images, which is 80% of the total data, while the validation dataset contains 8,400 images, which is the remaining 20% of the total data. After knowing the various uses of ant colony optimization combined with deep learning and knowing the various hyperparameters that can affect the performance of deep learning, the next chapter will provide an implementation and explanation of applying ant colony optimization for automatic hyperparameter tuning. AMELIA ROSE SMITH 3 Amelia Rose Smith PROPOSED MODEL Before the training process, the hyperparameters are set automatically. This research combines Convolutional Neural Networks (CNN) with ant colony. To improve the performance of CNN, Dropout layer is used to avoid overfitting, and Batch Normalization is adopted to speed up the training process. Figure 1. Basic Model The architecture used in this experiment can be seen in Figure 1. The model begins with the capture of an input image with dimensions of 28 pixels long, 28 pixels wide, and 3 pixels deep. The image is then processed through two convolutional layers, followed by a dropout layer whose value is determined by the ant colony, and then a max pooling layer. The image is then flattened into vector form and passed through a batch normalization layer, two dense layers, the number of neurons determined by the ant colony, and another dropout layer whose value is determined by the ant colony. After this process, the model is able to classify the image into the ten available classes, with the dense layer containing ten neurons located at the end of the model. More details on the automatic hyperparameter tuning process using the ant colony method can be found below. AMELIA ROSE SMITH 4 Article Study Case Representation of the solution in the proposed Ant Colony 1. number of neurons on the first dense screen 2. Number of neurons on the second dense screen 3. value on the first dropout layer 4. value on the second dropout layer 5. number of batches to be used 6. Activation function to be used 7. The optimizer that will be used 8. loss function to be used Figure 2. Proposed Coding With Ant Colony A depiction of the coding model created by the ant colony algorithm to enable automatic hyperparameter tuning is shown in Figure 2. The model includes eight tunable hyperparameters, with further information and possible choices available in Table 1. Table 1. Eight Tunable Hyperparameters and Available Options for Each Hyperparameter Hyperparameter Option Number of neurons in the first dense layer Number of neurons in the second dense layer The value in the first dropout layer Values on the second dropout layer Number of batches to be used The activation function to use AMELIA ROSE SMITH 5 Amelia Rose Smith Optimizer to be used The loss function will be used 2. each ant chooses a path 1. initialize 3. update pheromone 4. Evaporation of pheromones 5. Evaluation 4C. Testing 4B. Training 4A. Coding Figure 3. Complete Hyperparameter Autotuning Architecture Figure 3 is a visualization of the entire ant colony performed in this experiment in order to automatically tune the hyperparameters. There are five main steps with ant colony, starting from the initialization stage. In the initialization stage, 1 route consisting of 8 hyperparameters is determined. It is determined that 20 ants will go through the route that has been created, then the evaporation value is set at 0.25, then we start by creating a set of solutions called routes. The ants will traverse these routes. This study consists of 8 hyperparameters for each solution or route created. Then each ant will go through the route or solution that has been created. At this stage, 20 ants will pass through the path that has been created. In the next stage, after the ants have passed the selected path, pheromones will be added to the path that has been traveled. Then the pheromone evaporation stage is carried out for the evaporation rate according to the value set at the initialization stage. The process of giving and evaporating AMELIA ROSE SMITH 6 Article Study Case pheromones is very important because it will be used to create a route that the ants should follow next, then the evaluation stage is carried out. The evaluation stage consists of 3 main stages, namely coding, training, and testing. In the coding stage, the solution created in ant colony is translated into the hyperparameters previously described. After successfully translating, training is carried out with the hyperparameters created with ant colony, and testing is carried out. EXPERIMENT RESULTS Table 2 displays the environment used in conducting this research. Table 2. Used Environment A comprehensive overview of the research environment can be found in Table 2. Replication of this study with different environments may yield different results, underscoring the importance of this information. This study compares ant colony with genetic algorithm to perform automatic hyperparameter adjustment. Both methods' base models, data sets, and hyperparameter options are the same for a fair comparison. For the genetic algorithm setup, there are 10 solutions in 1 population. For the best parent selection method, a new solution was created using the Roulette Wheel Selection (RWS) method. To perform crossover using the single point method. While to do mutation using swap method. AMELIA ROSE SMITH 7 Amelia Rose Smith Figure 4: Comparison of Accuracy with Training Data Figure 4 illustrates the comparison of accuracy obtained by utilizing the training dataset on the model with optimal hyperparameters. The ant colony technique achieved a remarkable accuracy rate of 99.69%, while the genetic algorithm only achieved an accuracy of 97.59%. Figure 5: Comparison of accuracy with validation data Figure 5 presents a comparison of the accuracy results obtained from the validation dataset on the model that has the most optimal hyperparameters. The ant colony algorithm achieved an accuracy score of 97.46%, while the genetic algorithm obtained a lower accuracy score of 94.60%. Figure 6: Loss Comparison with Training Data AMELIA ROSE SMITH 8 Article Study Case Figure 6 shows the loss comparison with the training dataset between the models with the best hyperparameters. The ant colony approach produces a loss value of 0.0094, while the genetic algorithm method produces a loss value of 0.0777. Figure 7. Loss Comparison with Validation Data Figure 7 shows a comparison of the loss value achieved by the model with the optimal hyperparameter using the validation dataset. The ant colony algorithm gets a loss value of 0.2130, while the genetic algorithm gets a loss value of 0.1874. Figure 8. Comparison of Training Time A comparison of the training time required in seconds for the model with the best hyperparameters is illustrated in Figure 8. The ant colony method takes 27.94 seconds, while the genetic algorithm takes 22.25 seconds. AMELIA ROSE SMITH 9 Amelia Rose Smith Figure 9. ROC Comparison The best ROC value on the model with the best hyperparameters was obtained by ant colony as shown in Figure 9, exceeding the genetic algorithm. Conclusion Ant colony is used to perform automatic hyperparameter setting for image classification in this study. Compared to the genetic algorithm, ant colony gives the best results in terms of accuracy, but the training time is slightly longer than the genetic algorithm AMELIA ROSE SMITH 10 Article Study Case References AMELIA ROSE SMITH 11 Amelia Rose Smith AMELIA ROSE SMITH 12 Article Study Case AMELIA ROSE SMITH 13