The paper presents a methodology for collecting valid software engineering data to evaluate software development methodologies and study the software development process. The method uses goal-directed data collection, where the goals are defined based on the claims made for the methodologies. Data are collected during software development and validated concurrently through interviews with those who supply the data. The methodology has been applied to five different projects in two environments, demonstrating its feasibility and usefulness. Key aspects include establishing clear goals, developing a list of questions, defining data categories, designing and testing data collection forms, collecting and validating data, and analyzing the data. The paper also discusses the importance of validation, the consequences of omitting it, and the need for careful form design and programmer training. The authors conclude that the methodology can be used to evaluate methodologies and improve the software development process, but limitations include the inability to isolate single factor effects and the inability to recapture lost data.The paper presents a methodology for collecting valid software engineering data to evaluate software development methodologies and study the software development process. The method uses goal-directed data collection, where the goals are defined based on the claims made for the methodologies. Data are collected during software development and validated concurrently through interviews with those who supply the data. The methodology has been applied to five different projects in two environments, demonstrating its feasibility and usefulness. Key aspects include establishing clear goals, developing a list of questions, defining data categories, designing and testing data collection forms, collecting and validating data, and analyzing the data. The paper also discusses the importance of validation, the consequences of omitting it, and the need for careful form design and programmer training. The authors conclude that the methodology can be used to evaluate methodologies and improve the software development process, but limitations include the inability to isolate single factor effects and the inability to recapture lost data.