This paper discusses the validation of classification models in metabolomics using cross model validation and permutation testing. The main issue is that PLSDA (Partial Least Squares Discriminant Analysis) can overfit data, leading to overly optimistic results if not validated properly. The authors propose using cross model validation and permutation testing to assess the validity of classification models. They argue against using PLSDA score plots for inference of class differences, as they can be misleading due to the high dimensionality of metabolomics data. The paper highlights that when data is randomly assigned to classes, PLSDA can produce seemingly good separation between classes, but this does not reflect real class differences. The Q² value, commonly used to assess model performance, can be misleading as it does not account for the true class separation. The authors suggest using permutation testing to generate a distribution of values under the null hypothesis of no class difference, allowing for a more accurate assessment of model performance. Cross validation is often used to validate classification models, but it is frequently performed improperly. The paper describes three cross validation strategies: FIT, 1CV, and 2CV. FIT uses a single cross validation step to select the optimal number of PLS components, but this is not sufficient for proper validation. 1CV leads to an average number of misclassifications that is still too optimistic. Only 2CV provides a validation that is independent of the test set, leading to a more accurate assessment of model performance. The authors also discuss the use of permutation testing to assess the significance of classification results. By permuting class labels, they can determine if the observed classification results are statistically significant. This approach helps to avoid over-optimism in model validation and provides a more reliable assessment of model performance. The paper concludes that proper validation of classification models in metabolomics is essential to avoid over-optimistic results. Cross model validation and permutation testing are recommended as effective methods for validating classification models. The use of PLSDA score plots for inference of class differences is discouraged, as they can be misleading in high-dimensional data. Instead, the authors advocate for using predictions rather than fitted values to assess class separation.This paper discusses the validation of classification models in metabolomics using cross model validation and permutation testing. The main issue is that PLSDA (Partial Least Squares Discriminant Analysis) can overfit data, leading to overly optimistic results if not validated properly. The authors propose using cross model validation and permutation testing to assess the validity of classification models. They argue against using PLSDA score plots for inference of class differences, as they can be misleading due to the high dimensionality of metabolomics data. The paper highlights that when data is randomly assigned to classes, PLSDA can produce seemingly good separation between classes, but this does not reflect real class differences. The Q² value, commonly used to assess model performance, can be misleading as it does not account for the true class separation. The authors suggest using permutation testing to generate a distribution of values under the null hypothesis of no class difference, allowing for a more accurate assessment of model performance. Cross validation is often used to validate classification models, but it is frequently performed improperly. The paper describes three cross validation strategies: FIT, 1CV, and 2CV. FIT uses a single cross validation step to select the optimal number of PLS components, but this is not sufficient for proper validation. 1CV leads to an average number of misclassifications that is still too optimistic. Only 2CV provides a validation that is independent of the test set, leading to a more accurate assessment of model performance. The authors also discuss the use of permutation testing to assess the significance of classification results. By permuting class labels, they can determine if the observed classification results are statistically significant. This approach helps to avoid over-optimism in model validation and provides a more reliable assessment of model performance. The paper concludes that proper validation of classification models in metabolomics is essential to avoid over-optimistic results. Cross model validation and permutation testing are recommended as effective methods for validating classification models. The use of PLSDA score plots for inference of class differences is discouraged, as they can be misleading in high-dimensional data. Instead, the authors advocate for using predictions rather than fitted values to assess class separation.