Jun. 2011 | Xuejun Yang, Yang Chen, Eric Eide, John Regehr
The paper "Finding and Understanding Bugs in C Compilers" by Xuejun Yang, Yang Chen, Eric Eide, and John Regehr from the University of Utah's School of Computing, presents a comprehensive study on improving the quality of C compilers through automated testing. The authors developed Csmith, a randomized test-case generation tool, which has been used to identify over 325 previously unknown bugs in mainstream C compilers, including GCC and LLVM. These bugs often manifest as crashes or incorrect code generation when valid inputs are provided.
Csmith generates programs that cover a wide range of C language features while avoiding undefined and unspecified behaviors, ensuring that each generated program has a single interpretation. The tool's design goals include producing well-formed programs and maximizing expressiveness while adhering to these constraints. Csmith uses static analysis and run-time checks to avoid undefined behaviors, such as integer overflow and pointer dereferences, and ensures that generated programs do not depend on unspecified behaviors.
The paper reports on five experiments using Csmith:
1. Uncontrolled bug finding over three years.
2. Quantitative comparison of GCC and LLVM versions over a million random programs.
3. Bug-finding performance as a function of test-case size.
4. Comparison of Csmith's bug-finding power with other random program generators.
5. Analysis of the impact of adding Csmith-generated programs to existing test suites on code coverage.
The results show that Csmith significantly improves compiler quality, with the greatest number of bugs found in programs around 81 KB in size. The paper also provides detailed examples of wrong-code bugs found in GCC and LLVM, highlighting issues with safety checks, transformations, and analyses. The authors conclude that Csmith is an effective tool for finding serious, previously unknown bugs in C compilers, and that it is particularly useful for testing the middle end of compilers.The paper "Finding and Understanding Bugs in C Compilers" by Xuejun Yang, Yang Chen, Eric Eide, and John Regehr from the University of Utah's School of Computing, presents a comprehensive study on improving the quality of C compilers through automated testing. The authors developed Csmith, a randomized test-case generation tool, which has been used to identify over 325 previously unknown bugs in mainstream C compilers, including GCC and LLVM. These bugs often manifest as crashes or incorrect code generation when valid inputs are provided.
Csmith generates programs that cover a wide range of C language features while avoiding undefined and unspecified behaviors, ensuring that each generated program has a single interpretation. The tool's design goals include producing well-formed programs and maximizing expressiveness while adhering to these constraints. Csmith uses static analysis and run-time checks to avoid undefined behaviors, such as integer overflow and pointer dereferences, and ensures that generated programs do not depend on unspecified behaviors.
The paper reports on five experiments using Csmith:
1. Uncontrolled bug finding over three years.
2. Quantitative comparison of GCC and LLVM versions over a million random programs.
3. Bug-finding performance as a function of test-case size.
4. Comparison of Csmith's bug-finding power with other random program generators.
5. Analysis of the impact of adding Csmith-generated programs to existing test suites on code coverage.
The results show that Csmith significantly improves compiler quality, with the greatest number of bugs found in programs around 81 KB in size. The paper also provides detailed examples of wrong-code bugs found in GCC and LLVM, highlighting issues with safety checks, transformations, and analyses. The authors conclude that Csmith is an effective tool for finding serious, previously unknown bugs in C compilers, and that it is particularly useful for testing the middle end of compilers.