This study evaluates the psychometric properties of the predictive-inference task, a widely used paradigm for assessing flexible behavior in humans. The task involves participants predicting the landing position of a particle, which shifts suddenly at change-points (CPs). Using a large-scale, test-retest design with 330 participants at the first time point (T1) and 219 participants at the second time point (T2), the study assesses internal consistency and test-retest reliability of key measures. The results show that measures capturing flexible belief and behavioral adaptation, such as learning rate and confidence, exhibit good internal consistency and overall satisfying test-retest reliability. However, more complex markers of flexible behavior, like learning rate at CPs or associations with Bayesian model predictions, show lower psychometric quality. These findings have implications for the reliability of previous studies using this task and guide the selection of appropriate measures for future research. The study highlights the importance of psychometric soundness in cognitive flexibility research, particularly in psychiatric contexts, where cognitive flexibility is linked to various disorders.This study evaluates the psychometric properties of the predictive-inference task, a widely used paradigm for assessing flexible behavior in humans. The task involves participants predicting the landing position of a particle, which shifts suddenly at change-points (CPs). Using a large-scale, test-retest design with 330 participants at the first time point (T1) and 219 participants at the second time point (T2), the study assesses internal consistency and test-retest reliability of key measures. The results show that measures capturing flexible belief and behavioral adaptation, such as learning rate and confidence, exhibit good internal consistency and overall satisfying test-retest reliability. However, more complex markers of flexible behavior, like learning rate at CPs or associations with Bayesian model predictions, show lower psychometric quality. These findings have implications for the reliability of previous studies using this task and guide the selection of appropriate measures for future research. The study highlights the importance of psychometric soundness in cognitive flexibility research, particularly in psychiatric contexts, where cognitive flexibility is linked to various disorders.