JAUT (Java Automated Unit Testing) is a tool to generate automatically unit tests for Java Bytecode Programs. Test suites obtained with other automatic test cases generators (e.g. Pex or JCute) do not always cover all the statements. JAUT can then improve the code coverage by using a technique based on symbolic execution made in a backward way and on constraint programming to solve complex path constraints including reference aliasing on parameters.
We have used several examples (numerical or with complex conditions on references) to show the capacity of JAUT to complete the test suite obtained with Pex and JCute when one is trying to generate some unit test cases to cover the code of programs. Programs are in C# for Pex and in Java for JCute and JAUT, the source codes for our examples are the same in both languages.
The source code of the programs under test can be downloaded here.
The three tools have not exactly the same goal and the same functionalities. JCute and Pex aim at covering paths of the method under test whereas JAUT aims at reaching specific statements given by the user as test targets, for example the statements not covered by other tools. Pex and JCute generate the scripts to execute the unit tests, so the test data they find are realistic data that can be obtained by sequences of calls to methods and constructors. JAUT has not yet this functionality, so to permit the comparison all the attributes of the classes are public in our examples: as a consequence putting the objects in the right state is not a difficulty even if there is still a time overhead to generate scripts for JCute and Pex, what must be taken into account to compare the generation time.
The test objectives for a method under test are determined by using the bytecode program. We compare the ability of each tool to generate test suite to reach the statements beginning each of the statement blocks at the bytecode level. We have made coverage and time measures for these test objectives.
Experiments have been computed with an Intel Pentium, 2.16GHZ machine running Windows XP with 2.0GB of RAM.
We have used the version 0.15.40714.1 of Pex. The test drivers used with Pex for each example can be found here.
The test drivers used with JCute can be found here.
Here is the version 1.0 of JAUT, with the explanations to reproduce the experiments with the tool (in the README file). SICStus prolog is needed to use JAUT.
Table 1 : Time and coverage with JCute, Pex and JAUT.
**: timeout of 3600 sec
Explanations to read the table :
#Bc is the number of bytecode instructions in the method, #To is the number of test objectives. In all three cases, we report the percentage of coverage obtained, which we have computed ourselves on a common basis. Indeed, PEX reports dynamic bytecode coverage which does not necessarily correspond to bytecode branch coverage. Note also that coverage percentage is influenced by the number of static elements considered: the number of source code instructions is usually distinct from the number of bytecode instructions. So, we computed the coverage by looking at the covered portions of code, rather than taking into account the numbers provided by the tools. We selected two search modes for jCUTE: the former corresponds to depth-first search on the execution tree (both columns noted jCUTE) while the second corresponds to random path exploration (both columns noted jCUTEr). We also measured the CPU time required by JAUT to complete the test set generated by PEX (column compl. JAUT) when it was incomplete. When this measure was unnecessary, n/a was reported.