Psychometric assessment of three newly developed implementation outcome measures

Psychometric assessment of three newly developed implementation outcome measures

2017 | Bryan J. Weiner, Cara C. Lewis, Cameo Stanick, Byron J. Powell, Caitlin N. Dorsey, Alecia S. Clary, Marcella H. Boynton, Heather Halko
This study aimed to develop and psychometrically assess three new measures—Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM)—to evaluate implementation outcomes. The measures were designed to address the lack of conceptual clarity and psychometric properties in existing measures. The study involved three phases: 1. **Substantive and Discriminant Content Validity**: A web-based survey was administered to 63 implementation scientists and mental health professionals to assess the content validity of the measures. Most items demonstrated acceptable content validity, with five items removed due to insufficient validity. 2. **Structural Validity, Reliability, and Known-Groups Validity**: An experimental vignette study with 326 mental health counselors was conducted to refine the scales and assess structural validity, reliability, and known-groups validity. The refined 4-item scales showed acceptable factor loadings and model fit, with high internal consistency (Cronbach's alphas ranging from 0.85 to 0.91). ANOVA results indicated significant main effects, supporting known-groups validity. 3. **Test-Retest Reliability and Sensitivity to Change**: A subsample of 192 participants from the vignette study was re-administered the same or opposite vignettes to assess test-retest reliability and sensitivity to change. The measures demonstrated acceptable test-retest reliability (Cronbach's alphas ranging from 0.83 to 0.88) and sensitivity to change, with regression analysis showing significant associations between vignette assignment order and measure scores. The study concluded that the newly developed measures are valid and reliable, with potential for widespread use in implementation research and practice. Future research will focus on predictive validity and pragmatic properties of the measures.This study aimed to develop and psychometrically assess three new measures—Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM)—to evaluate implementation outcomes. The measures were designed to address the lack of conceptual clarity and psychometric properties in existing measures. The study involved three phases: 1. **Substantive and Discriminant Content Validity**: A web-based survey was administered to 63 implementation scientists and mental health professionals to assess the content validity of the measures. Most items demonstrated acceptable content validity, with five items removed due to insufficient validity. 2. **Structural Validity, Reliability, and Known-Groups Validity**: An experimental vignette study with 326 mental health counselors was conducted to refine the scales and assess structural validity, reliability, and known-groups validity. The refined 4-item scales showed acceptable factor loadings and model fit, with high internal consistency (Cronbach's alphas ranging from 0.85 to 0.91). ANOVA results indicated significant main effects, supporting known-groups validity. 3. **Test-Retest Reliability and Sensitivity to Change**: A subsample of 192 participants from the vignette study was re-administered the same or opposite vignettes to assess test-retest reliability and sensitivity to change. The measures demonstrated acceptable test-retest reliability (Cronbach's alphas ranging from 0.83 to 0.88) and sensitivity to change, with regression analysis showing significant associations between vignette assignment order and measure scores. The study concluded that the newly developed measures are valid and reliable, with potential for widespread use in implementation research and practice. Future research will focus on predictive validity and pragmatic properties of the measures.
Reach us at info@study.space
[slides] Psychometric assessment of three newly developed implementation outcome measures | StudySpace