Psychometric assessment of three newly developed implementation outcome measures

Psychometric assessment of three newly developed implementation outcome measures

2017 | Bryan J. Weiner, Cara C. Lewis, Cameo Stanick, Byron J. Powell, Caitlin N. Dorsey, Alecia S. Clary, Marcella H. Boynton, Heather Halko
This study presents the psychometric assessment of three newly developed implementation outcome measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). These measures were developed to assess key implementation outcomes: acceptability, appropriateness, and feasibility. The study involved three phases: content validity assessment, structural validity and reliability testing, and test-retest reliability and sensitivity to change evaluation. In the first phase, 36 implementation scientists and 27 mental health professionals evaluated the content validity of 31 items assigned to the three constructs. The results showed that all but five items had substantive and discriminant content validity. In the second phase, 326 mental health counselors participated in an experimental vignette study to assess structural validity, reliability, and known-groups validity. The results indicated that the three-factor confirmatory factor analysis model had acceptable fit and high factor loadings. The measures also demonstrated good reliability and known-groups validity. In the third phase, 192 participants were asked to re-rate the implementation outcomes after receiving either the same or opposite vignettes, demonstrating test-retest reliability and sensitivity to change. The results showed that the three measures demonstrated promising psychometric properties, with high reliability and validity. The measures were found to be sensitive to change in both directions, indicating their usefulness in assessing the impact of implementation strategies on practitioners' perceptions of acceptability, appropriateness, and feasibility. The study also highlights the importance of developing valid and reliable measures for implementation research, as well as the need for pragmatic measures that are useful, compatible with settings, easy to use, and acceptable to stakeholders. The measures developed in this study are considered to be important implementation outcomes and leading indicators of other implementation outcomes. The study contributes to the field of implementation science by developing valid and reliable measures of important implementation outcomes. The measures are also considered to be generalizable and have the potential for widespread use across implementation studies. The study also outlines a systematic process for measure development and testing that is replicable and feasible. The study has limitations, including the high correlations among the three factors and the need for further research on discriminant validity. Future directions include prospectively administering the measures to a large sample of providers and evaluating whether their perceptions of the acceptability, appropriateness, and feasibility of an EBP predict their adoption of the EBP. The study also emphasizes the importance of stakeholder input in defining pragmatic features of measures and developing rating criteria for assessing their pragmatic properties. The study concludes that the three measures are valid, reliable, and have the potential for widespread use in implementation research and practice.This study presents the psychometric assessment of three newly developed implementation outcome measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). These measures were developed to assess key implementation outcomes: acceptability, appropriateness, and feasibility. The study involved three phases: content validity assessment, structural validity and reliability testing, and test-retest reliability and sensitivity to change evaluation. In the first phase, 36 implementation scientists and 27 mental health professionals evaluated the content validity of 31 items assigned to the three constructs. The results showed that all but five items had substantive and discriminant content validity. In the second phase, 326 mental health counselors participated in an experimental vignette study to assess structural validity, reliability, and known-groups validity. The results indicated that the three-factor confirmatory factor analysis model had acceptable fit and high factor loadings. The measures also demonstrated good reliability and known-groups validity. In the third phase, 192 participants were asked to re-rate the implementation outcomes after receiving either the same or opposite vignettes, demonstrating test-retest reliability and sensitivity to change. The results showed that the three measures demonstrated promising psychometric properties, with high reliability and validity. The measures were found to be sensitive to change in both directions, indicating their usefulness in assessing the impact of implementation strategies on practitioners' perceptions of acceptability, appropriateness, and feasibility. The study also highlights the importance of developing valid and reliable measures for implementation research, as well as the need for pragmatic measures that are useful, compatible with settings, easy to use, and acceptable to stakeholders. The measures developed in this study are considered to be important implementation outcomes and leading indicators of other implementation outcomes. The study contributes to the field of implementation science by developing valid and reliable measures of important implementation outcomes. The measures are also considered to be generalizable and have the potential for widespread use across implementation studies. The study also outlines a systematic process for measure development and testing that is replicable and feasible. The study has limitations, including the high correlations among the three factors and the need for further research on discriminant validity. Future directions include prospectively administering the measures to a large sample of providers and evaluating whether their perceptions of the acceptability, appropriateness, and feasibility of an EBP predict their adoption of the EBP. The study also emphasizes the importance of stakeholder input in defining pragmatic features of measures and developing rating criteria for assessing their pragmatic properties. The study concludes that the three measures are valid, reliable, and have the potential for widespread use in implementation research and practice.
Reach us at info@futurestudyspace.com
[slides and audio] Psychometric assessment of three newly developed implementation outcome measures