This technical report presents a methodology for collecting valid software engineering data. The method uses goal-directed data collection to evaluate software development methodologies based on claims made for them. The methodology involves defining data collection goals, establishing questions of interest, defining data categorization schemes, and designing a data collection form. Data collected are based on changes made to the software during development and are obtained when the changes are made. Validation is performed concurrently with software development and data collection to ensure accuracy. Results show that data validation is essential for accurate change data collection, as up to 50% of data may be erroneous without it. The methodology was applied to five different projects in two environments, demonstrating its feasibility and usefulness. The methodology includes six criteria: data must contain information to identify errors and changes, include the cost of changes and error correction, be defined based on clear study goals, include studies of projects from production environments, be analyzed historically but collected and validated concurrently with development, and use carefully specified data classification schemes for repeatability. The data collection process involves designing and testing a data collection form, collecting and validating data, and analyzing the data. The form is used to collect data on changes made to software, including the reason for the change, the components affected, and the effort required. Data validation is crucial to ensure accuracy, as errors can occur without it. The validation process involves checking the forms for correctness, consistency, and completeness, and interviewing those who filled out the forms to resolve any issues. The methodology was applied to three SEL projects, resulting in significant improvements in data quality. The validation process revealed that up to 50% of data could be inaccurate without validation. The data collection process was integrated with configuration control procedures, ensuring data completeness and accuracy. The results showed that careful validation, including programmer interviews, is essential for accurate data collection. The methodology also highlights the importance of defining clear goals and questions of interest, as well as designing forms that are easy to use and understand. The study also identified several pitfalls in data collection, such as misclassification of errors and changes, and emphasized the need for careful planning and training to avoid these issues. The methodology provides a framework for collecting valid software engineering data that can be used to evaluate software development methodologies and gain insights into the software development process.This technical report presents a methodology for collecting valid software engineering data. The method uses goal-directed data collection to evaluate software development methodologies based on claims made for them. The methodology involves defining data collection goals, establishing questions of interest, defining data categorization schemes, and designing a data collection form. Data collected are based on changes made to the software during development and are obtained when the changes are made. Validation is performed concurrently with software development and data collection to ensure accuracy. Results show that data validation is essential for accurate change data collection, as up to 50% of data may be erroneous without it. The methodology was applied to five different projects in two environments, demonstrating its feasibility and usefulness. The methodology includes six criteria: data must contain information to identify errors and changes, include the cost of changes and error correction, be defined based on clear study goals, include studies of projects from production environments, be analyzed historically but collected and validated concurrently with development, and use carefully specified data classification schemes for repeatability. The data collection process involves designing and testing a data collection form, collecting and validating data, and analyzing the data. The form is used to collect data on changes made to software, including the reason for the change, the components affected, and the effort required. Data validation is crucial to ensure accuracy, as errors can occur without it. The validation process involves checking the forms for correctness, consistency, and completeness, and interviewing those who filled out the forms to resolve any issues. The methodology was applied to three SEL projects, resulting in significant improvements in data quality. The validation process revealed that up to 50% of data could be inaccurate without validation. The data collection process was integrated with configuration control procedures, ensuring data completeness and accuracy. The results showed that careful validation, including programmer interviews, is essential for accurate data collection. The methodology also highlights the importance of defining clear goals and questions of interest, as well as designing forms that are easy to use and understand. The study also identified several pitfalls in data collection, such as misclassification of errors and changes, and emphasized the need for careful planning and training to avoid these issues. The methodology provides a framework for collecting valid software engineering data that can be used to evaluate software development methodologies and gain insights into the software development process.