JULY/AUGUST 1999 | Victor R. Basili, Fellow, IEEE, Forrest Shull, and Filippo Lanubile, Member, IEEE Computer Society
The paper discusses the challenges and importance of experimentation in software engineering, emphasizing the need for a cohesive framework to organize and integrate results from multiple studies. The authors, based on their own experiences, propose a framework for organizing sets of related studies, which can help build knowledge incrementally through replication and integration. The framework is applied to a set of experiments on software reading techniques, focusing on the effectiveness of different reading methods in detecting defects or anomalies in software documents. The paper highlights the difficulties in software engineering experiments, such as the lack of statistical power due to the small number of data points and the variability in human performance. It also discusses the importance of valid measurement, process conformance, and context modeling to ensure the reliability and generalizability of experimental results. The authors emphasize the need for a systematic approach to define and interpret goals, variables, and models, using the goal/question/metric (GQM) template as a tool to structure and evaluate experiments. The paper concludes by suggesting that a well-structured framework can help advance the field of software engineering by building a body of evidence and knowledge.The paper discusses the challenges and importance of experimentation in software engineering, emphasizing the need for a cohesive framework to organize and integrate results from multiple studies. The authors, based on their own experiences, propose a framework for organizing sets of related studies, which can help build knowledge incrementally through replication and integration. The framework is applied to a set of experiments on software reading techniques, focusing on the effectiveness of different reading methods in detecting defects or anomalies in software documents. The paper highlights the difficulties in software engineering experiments, such as the lack of statistical power due to the small number of data points and the variability in human performance. It also discusses the importance of valid measurement, process conformance, and context modeling to ensure the reliability and generalizability of experimental results. The authors emphasize the need for a systematic approach to define and interpret goals, variables, and models, using the goal/question/metric (GQM) template as a tool to structure and evaluate experiments. The paper concludes by suggesting that a well-structured framework can help advance the field of software engineering by building a body of evidence and knowledge.