Hyperparameters significantly affect the performance of machine learning models on unseen data. This research note highlights the lack of transparency in reporting hyperparameters and their tuning in political science publications. Of 64 machine learning-related papers analyzed in three leading political science journals between 2016 and 2021, only 13 (20.31%) fully reported hyperparameters and their tuning. This lack of transparency hinders the replication and evaluation of results, as readers and reviewers cannot assess the quality of the manuscript without access to the replication code. Hyperparameters are crucial because they determine how well a model generalizes from training data to new data. Properly tuning and documenting hyperparameters is essential for robustness checks in machine learning. The study emphasizes the importance of transparency in hyperparameter selection and tuning to ensure the reliability and reproducibility of results. The paper explains what hyperparameters are, why they need to be tuned, and offers best practices for selecting and tuning them. It also illustrates the dangers of not being transparent about hyperparameters by comparing the performance of several machine learning models in predicting electoral violence from tweets. The results show that hyperparameter tuning significantly improves model performance, and without proper tuning, conclusions about model performance can be misleading. The research underscores the need for political scientists to follow best practices in hyperparameter tuning and documentation to ensure the reliability and reproducibility of their findings. The study calls for greater attention to hyperparameter transparency in political science research to build trust in the use of machine learning.Hyperparameters significantly affect the performance of machine learning models on unseen data. This research note highlights the lack of transparency in reporting hyperparameters and their tuning in political science publications. Of 64 machine learning-related papers analyzed in three leading political science journals between 2016 and 2021, only 13 (20.31%) fully reported hyperparameters and their tuning. This lack of transparency hinders the replication and evaluation of results, as readers and reviewers cannot assess the quality of the manuscript without access to the replication code. Hyperparameters are crucial because they determine how well a model generalizes from training data to new data. Properly tuning and documenting hyperparameters is essential for robustness checks in machine learning. The study emphasizes the importance of transparency in hyperparameter selection and tuning to ensure the reliability and reproducibility of results. The paper explains what hyperparameters are, why they need to be tuned, and offers best practices for selecting and tuning them. It also illustrates the dangers of not being transparent about hyperparameters by comparing the performance of several machine learning models in predicting electoral violence from tweets. The results show that hyperparameter tuning significantly improves model performance, and without proper tuning, conclusions about model performance can be misleading. The research underscores the need for political scientists to follow best practices in hyperparameter tuning and documentation to ensure the reliability and reproducibility of their findings. The study calls for greater attention to hyperparameter transparency in political science research to build trust in the use of machine learning.