This paper investigates the impact of hyperparameter optimization on the performance of fine-tuned Convolutional Neural Network (CNN) models. The authors explore four hyperparameter optimization methods—grid search, random search, Bayesian optimization, and the Asynchronous Successive Halving Algorithm (ASHA)—and compare their effectiveness in improving CNN classification accuracy. They also examine the feasibility of using a subset of the training data for hyperparameter optimization, considering both balanced and imbalanced datasets. The study uses three publicly available datasets: CIFAR-100, Stanford Dogs, and MIO-TCD. Key findings include: 1. **Hyperparameter Importance**: The functional analysis of variance (FANOVA) method is used to assess the importance of hyperparameters, identifying the learning rate and input image size as the most influential parameters. 2. **Optimization Method Comparison**: ASHA and Bayesian Optimization outperform grid search and random search in terms of classification accuracy and computational efficiency. 3. **Subset of Training Data**: Using a subset of the training data can effectively optimize hyperparameters, with balanced subsets showing a 4% increase in accuracy and imbalanced subsets with additional modifications (class weighting and augmentation) achieving a 4-5% improvement. 4. **Conclusion**: The choice of hyperparameter optimization method depends on the specific dataset and task. ASHA and Bayesian Optimization are particularly effective, while prior knowledge can help narrow down the hyperparameter space. Balancing the training data is crucial for successful hyperparameter optimization. The research highlights the importance of hyperparameter optimization in fine-tuned CNN models and provides insights into the most effective methods and strategies for achieving optimal performance.This paper investigates the impact of hyperparameter optimization on the performance of fine-tuned Convolutional Neural Network (CNN) models. The authors explore four hyperparameter optimization methods—grid search, random search, Bayesian optimization, and the Asynchronous Successive Halving Algorithm (ASHA)—and compare their effectiveness in improving CNN classification accuracy. They also examine the feasibility of using a subset of the training data for hyperparameter optimization, considering both balanced and imbalanced datasets. The study uses three publicly available datasets: CIFAR-100, Stanford Dogs, and MIO-TCD. Key findings include: 1. **Hyperparameter Importance**: The functional analysis of variance (FANOVA) method is used to assess the importance of hyperparameters, identifying the learning rate and input image size as the most influential parameters. 2. **Optimization Method Comparison**: ASHA and Bayesian Optimization outperform grid search and random search in terms of classification accuracy and computational efficiency. 3. **Subset of Training Data**: Using a subset of the training data can effectively optimize hyperparameters, with balanced subsets showing a 4% increase in accuracy and imbalanced subsets with additional modifications (class weighting and augmentation) achieving a 4-5% improvement. 4. **Conclusion**: The choice of hyperparameter optimization method depends on the specific dataset and task. ASHA and Bayesian Optimization are particularly effective, while prior knowledge can help narrow down the hyperparameter space. Balancing the training data is crucial for successful hyperparameter optimization. The research highlights the importance of hyperparameter optimization in fine-tuned CNN models and provides insights into the most effective methods and strategies for achieving optimal performance.