Modern programs can have many different configurations. There are a large number of possible errors that need handling and usually imply complex, almost infinitely many possible scenarios. The more complex your program, the more complex it is to test. This means that manually crafted test cases are painful and time-consuming, and only find 15% of the bugs.
QuickCheck is an innovative software testing tool. Instead of writing test cases, users write a property that should hold for the software. QuickCheck automatically generates thousands of different test cases to verify whether that property indeed holds. Whenever a test generated by QuickCheck fails, QuickCheck will automatically start analysing the failing case by testing a large number of “almost the same” tests, until it finds the smallest set of test cases that precisely indicates what the problem is. QuickCheck takes you quickly from specifications to identified bugs.
Quick Check means you can:
QuickCheck lets the developer specify what the software should do and then generates hundreds of meaningful test cases that explore that property. It almost always finds a test case that the programmer has not considered. Properties are much easier to maintain than individual test cases and cover the subject under test better. QuickCheck takes tests fast from specification to identified bugs in three steps:
Write a QuickCheck specification instead of test cases – general properties your system should always satisfy.
QuickCheck uses controlled random generation to test your code against the spec.
Failing cases are automatically simplified before they are presented to you.
Your QuickCheck specifications consist of properties and generators which describe system behaviour in a specified set of cases. Each property can generate many different test cases – so specifications can be much more concise and maintainable than test suites. At the same time, many more cases can be generated, so testing is more thorough. Like the functional programming paradigm it is based on, QuickCheck keeps specifications concise and readable.
Conventional random testing generates failing cases in which – like failures from the field – the signal causing the failure is obscured by a great deal of random noise, making fault diagnosis difficult and costly. QuickCheck automatically simplifies failing cases to minimal examples that provoke the failure, making fault diagnosis easy. Once again, you are in control: while useful simplification strategies are built-in, you can easily extend or replace them with your own.
The specification language for QuickCheck is Erlang, which is developed and maintained as open-source. Only the QuickCheck libraries themselves are proprietary. Linking QuickCheck specifications to code under test in other languages (such as C), or via TCP/IP, is easy to do thanks to Erlang's built in support for external languages.
QuickCheck tests your properties in randomly generated cases, either interactively or invoked from your test server. Pure random generation makes for poor test data, but QuickCheck’s simple and powerful interface puts you in control, making it easy to generate complex data with specific properties, with tight control over the distribution of test cases.
It is often convenient to model the system under test via a state machine. QuickCheck’s state machine libraries let you model the state declaratively, write pre– and postconditions and state transition functions for each operation, then generate minimal call sequences that provoke violations. QuickCheck can exercise a wider variety of call sequences than you can possibly write manual test cases for. Testing a pre-release version of Ericsson’s Media Proxy, with commands to Add, Modify, and Subtract callers, provoked numerous failures.
For open source developers, there is a free version called QuickCheck Mini. The commercial version of QuickCheck offers testing of stateful systems with state machine specifications, testing for concurrency errors such as race conditions, the possibility to use grammars as specifications, and much more.
Ask questions >
Successes in using QuickCheck include: