Building Knowledge through Families of Experiments

Building Knowledge through Families of Experiments

JULY/AUGUST 1999 | Victor R. Basili, Fellow, IEEE, Forrest Shull, and Filippo Lanubile, Member, IEEE Computer Society
This paper discusses the challenges of experimentation in software engineering and proposes a framework for organizing related studies into families of experiments. Experimentation is necessary but difficult due to the large number of context variables and the need for a mechanism to motivate studies and integrate results. A community of researchers is needed to replicate studies, vary context variables, and build models that represent common observations. The authors present their experiences with a collection of experiments, emphasizing the importance of a framework for organizing studies and building knowledge incrementally through replication. The framework supports the development of a unifying theory by creating a list of specific hypotheses investigated in an area. The paper also discusses the difficulties of drawing general conclusions from empirical studies in software engineering, as results depend on a potentially large number of context variables. The authors illustrate their framework with examples of experiments on software reading techniques, including Defect-Based Reading (DBR), Perspective-Based Reading (PBR), Use-Based Reading (UBR), Second Version of PBR (PBR2), and Scope-Based Reading (SBR). The paper also discusses the GQM goal template as a tool for experimentation, emphasizing the variables across which studies are unified. The framework helps in defining experiments, combining them to overcome validity problems, and generating laboratory manuals to support the framework. The paper concludes that building a body of software engineering knowledge requires families of experiments and a set of unifying principles that allow results to be combined and generalized. The framework helps in choosing a focus for future studies, determining important attributes of the models used in an experiment, and understanding the true effects of both the process being studied and the environmental variables.This paper discusses the challenges of experimentation in software engineering and proposes a framework for organizing related studies into families of experiments. Experimentation is necessary but difficult due to the large number of context variables and the need for a mechanism to motivate studies and integrate results. A community of researchers is needed to replicate studies, vary context variables, and build models that represent common observations. The authors present their experiences with a collection of experiments, emphasizing the importance of a framework for organizing studies and building knowledge incrementally through replication. The framework supports the development of a unifying theory by creating a list of specific hypotheses investigated in an area. The paper also discusses the difficulties of drawing general conclusions from empirical studies in software engineering, as results depend on a potentially large number of context variables. The authors illustrate their framework with examples of experiments on software reading techniques, including Defect-Based Reading (DBR), Perspective-Based Reading (PBR), Use-Based Reading (UBR), Second Version of PBR (PBR2), and Scope-Based Reading (SBR). The paper also discusses the GQM goal template as a tool for experimentation, emphasizing the variables across which studies are unified. The framework helps in defining experiments, combining them to overcome validity problems, and generating laboratory manuals to support the framework. The paper concludes that building a body of software engineering knowledge requires families of experiments and a set of unifying principles that allow results to be combined and generalized. The framework helps in choosing a focus for future studies, determining important attributes of the models used in an experiment, and understanding the true effects of both the process being studied and the environmental variables.
Reach us at info@study.space
[slides and audio] Building Knowledge through Families of Experiments