The study evaluates the psychometric properties of the predictive-inference task, a widely used paradigm to assess cognitive flexibility. Using a large-scale test-retest design with 330 participants at T1 and 219 at T2, the researchers assessed internal consistency and test-retest reliability of key measures. The main measures capturing flexible belief and behavioral adaptation showed good internal consistency and satisfactory test-retest reliability. However, more complex markers of flexible behavior exhibited lower psychometric quality. The findings suggest that while some measures are reliable, others should be used cautiously. The study highlights the importance of psychometric soundness in the task for accurate inferences about individual differences. The task's reliability is crucial for studies in psychiatric and clinical contexts, where cognitive flexibility is linked to various disorders. The study also shows that measures like learning rate and confidence are stable across time, while others, such as learning rate at change points, show lower reliability. The results emphasize the need for careful selection of measures in future studies to ensure valid and reliable data. The study provides guidance on which measures are suitable for investigating cognitive flexibility and its implications for mental health research.The study evaluates the psychometric properties of the predictive-inference task, a widely used paradigm to assess cognitive flexibility. Using a large-scale test-retest design with 330 participants at T1 and 219 at T2, the researchers assessed internal consistency and test-retest reliability of key measures. The main measures capturing flexible belief and behavioral adaptation showed good internal consistency and satisfactory test-retest reliability. However, more complex markers of flexible behavior exhibited lower psychometric quality. The findings suggest that while some measures are reliable, others should be used cautiously. The study highlights the importance of psychometric soundness in the task for accurate inferences about individual differences. The task's reliability is crucial for studies in psychiatric and clinical contexts, where cognitive flexibility is linked to various disorders. The study also shows that measures like learning rate and confidence are stable across time, while others, such as learning rate at change points, show lower reliability. The results emphasize the need for careful selection of measures in future studies to ensure valid and reliable data. The study provides guidance on which measures are suitable for investigating cognitive flexibility and its implications for mental health research.