Test Case Prioritization: A Family of Empirical Studies

Test Case Prioritization: A Family of Empirical Studies

February 2002 | Sebastian Elbaum, Alexey G. Malishevsky, Gregg Rothermel
This paper presents a family of empirical studies on test case prioritization, focusing on how prioritization techniques affect the rate of fault detection in regression testing. The study addresses three key research questions: 1) Can version-specific prioritization improve fault detection? 2) How do fine-granularity (statement-level) techniques compare to coarse-granularity (function-level) techniques in terms of fault detection? 3) Can fault proneness predictors improve the effectiveness of prioritization techniques? The study compares 18 prioritization techniques, including comparator techniques, statement-level techniques, function-level techniques, and techniques incorporating fault proneness information. The results show that all prioritization techniques improve the rate of fault detection compared to random or untreated test orders. Fine-granularity techniques generally outperformed coarse-granularity techniques, but the differences were relatively small. Incorporating fault proneness measures did not significantly improve fault detection rates, contrary to expectations. The study also shows that the effectiveness of prioritization techniques varies across different programs. The practical significance of these results depends on the cost factors associated with the testing process. The study includes both controlled experiments and case studies on various programs, including two open-source Unix utilities and an embedded real-time subsystem of a RAID storage system. The study introduces a metric called APFD (Average Percentage of Faults Detected) to quantify the effectiveness of prioritization techniques. The results show that version-specific prioritization can significantly improve fault detection rates compared to general prioritization. Function-level techniques were found to be less effective than statement-level techniques, but the differences were relatively small. The study also shows that the effectiveness of prioritization techniques can vary significantly across different programs and testing scenarios. The findings suggest that further research is needed to understand the sources of this variance and to incorporate this understanding into prioritization techniques and the selection of techniques.This paper presents a family of empirical studies on test case prioritization, focusing on how prioritization techniques affect the rate of fault detection in regression testing. The study addresses three key research questions: 1) Can version-specific prioritization improve fault detection? 2) How do fine-granularity (statement-level) techniques compare to coarse-granularity (function-level) techniques in terms of fault detection? 3) Can fault proneness predictors improve the effectiveness of prioritization techniques? The study compares 18 prioritization techniques, including comparator techniques, statement-level techniques, function-level techniques, and techniques incorporating fault proneness information. The results show that all prioritization techniques improve the rate of fault detection compared to random or untreated test orders. Fine-granularity techniques generally outperformed coarse-granularity techniques, but the differences were relatively small. Incorporating fault proneness measures did not significantly improve fault detection rates, contrary to expectations. The study also shows that the effectiveness of prioritization techniques varies across different programs. The practical significance of these results depends on the cost factors associated with the testing process. The study includes both controlled experiments and case studies on various programs, including two open-source Unix utilities and an embedded real-time subsystem of a RAID storage system. The study introduces a metric called APFD (Average Percentage of Faults Detected) to quantify the effectiveness of prioritization techniques. The results show that version-specific prioritization can significantly improve fault detection rates compared to general prioritization. Function-level techniques were found to be less effective than statement-level techniques, but the differences were relatively small. The study also shows that the effectiveness of prioritization techniques can vary significantly across different programs and testing scenarios. The findings suggest that further research is needed to understand the sources of this variance and to incorporate this understanding into prioritization techniques and the selection of techniques.
Reach us at info@study.space