SOTA User's Guide - Version 1.0 |
The following three tutorials explain the behaviour of the three ways to use SOTA in manual program testing, program testing with an external test program and the test as a library in an automatic testing system.
The basis for the tutorial of the manual program test is a simple Java program which tries to read a positive rational number from a string. The program consists of the class Digit with a main function and the function evaluateDigitSequence which evaluates the string.
The string can either be passed as a parameter of the program or provided in the source code. The 'hardwired' string in the source code will be evaluated once the program is called without parameters. The output of the program is either the number in case the program was able to evaluate the string, or '-1' in case of an error, i.e. the string did not contain such a number.
public class Digit {
public static void main(String[] args) {
if(args.length==0)
System.out.println(evaluateDigitSequence("."));
else
System.out.println(evaluateDigitSequence(args[0]));
}
private static double evaluateDigitSequence(String inDigitString) {
double value = 0.0;
double accuracy = 1.0;
String where = "prePoint";
boolean errorfree = true;
int position = 1;
while(position <= inDigitString.length() &errorfree) {
String chr = inDigitString.substring(position-1, position);
if(chr.matches("[0-9]")) {
if(where.equals("postPoint"))
accuracy = accuracy / 10.0;
value = 10.0*value + Double.parseDouble(chr);
}
else if(chr.equals(".") & where.equals("prePoint"))
where = "postPoint";
else
errorfree = false;
position ++;
}
if(!errorfree | inDigitString.length()==0 | ((where.equals("postPoint")&inDigitString.length()==1)))
return -1.0;
else
return value*accuracy;
}
}
The program 'Digit' is supposed to be written in Eclipse and then tested using a structure oriented program test. This procedure is split into 4 phases:
The following data flow diagram reflects all important actionas as well as inputs and outputs. The four phases result from switching between Eclipse and SOTA and are depicted in differing colors. The interface between Eclipse and SOTA is realised only with the specified files.
The test program should be written in Eclipse. In case Eclipse is not installed yet, the user can do this following these instructions: http://wiki.eclipse.org/Eclipse/Installation.
Firstly, the user has to create a new project for the test program. This is done via the menu File -> New ->
Java Project. A dialog opens where the name for the project 'Digit'
has to be entered, for all other options the default values can be used. Therefore the user can close the wizard on the first page
clicking Finish.
To create a class Digit the user has to select File -> New ->
Class which calls the appropriate wizard. Here the name of the class - Digit - has to be entered before closing the dialog.
Then a Java file is opened into which the source code above has to be copied.
Next it is necessary to include the logging component as a library in the project. In order to do this, the user has to copy
the file ASCLogger.jar from the SOTA directory into a folder lib in the root directory
of the porject (..\workspace\Digit\). On refreshing the project overview in Eclipse (F5) this file as well as the folder lib
appear (see figure). Now the file has to be added to the build path of the project by right-clicking ASCLogger.jar in the context menu
and selecting Build Path -> Add to Build Path.
The library is then included in the Referenced Libraries. With this the first phase - writing the program and preparing it
for instrumentation by SOTA- is completed.
After starting SOTA via SOTA.exe the user has to create a project in SOTA for the test program. Selecting the menu
item New Project opens a two-page wizard guiding the creation process.
On the first page the name of the project - Digit - as well as the root directory (project directory) of the proejct created in point 1 have to be specified. The execution directory of the project is automatically linked to the same directory by SOTA needing no user input since both directories are identical in this case. The first page of the wizard is now completed and clicking the Next button calls the second page.
On the second page all sources of the project need to be imported. The project 'Digit' only consists of one file, so marking the root directory is sufficient. The Finish button closes the wizard and then SOTA reads and parses the source file.
Saving the project succesfully completes the creation process. When SOTA is started again, the project can be loaded via the
menu item Open Project. Immediately after creating a project or loading
an exisiting project respectively, the source code can be viewed in the view Source and the control flow graph of each function
is visible in the view CFG after selecting one in the view Project. The static metrics that were computed while parsing
the sources are now listed in the view Metrics.
The next step is to instrument the project for the next test run, i.e. adding instructions that effect in writing data into the
log file during program execution, allowing its complete reconstruction. For this reason, the user has to select the menu item
Start Test. A dialog opens and a test name as well as an instrumentation
scheme have to be selected. The test name also determines the name of the log file where all the log data is saved. The
instrumentation scheme specifies the manner of instrumentation for all structures of the project. Usually, the user should select
the IScheme Level 2 instrumentation which leads to a minimal instrumentation but computes all coverage metrics for all files.
After confirming the dialog the file Digit.java is saved as Digit.java.backup and then the instrumentation directions are added to the original file. With this the preparation of the instrumentation in SOTA is completed and the testing phase may begin.
At first the modified source code of the entire project has to be loaded in Eclipse. To do this, the project Digit has to be selected in the project overview and then refreshed by clicking 'F5' or via the context menu -> Refresh. Subsequently, Eclipse automatically compiles the new source files.
The instrumented sources require the library ASCLogger.jar which was included in step 2. Without the correct integration of the library the compilation process will evoke error messages.
After succesfully compiling the program it is ready for the test. In Eclipse it is started via the button
Run As... . On the first start Eclipse prompts whether
the program is supposed to be started as an application or applet. Here, the user should choose Application.
The following dialog asks for the application where the correct choise is Digit. Then the program starts.
For the following program starts Eclipse should always choose the selected start configuration, so it is sufficient to click
the button
now labeled with Run Digit.
The parameters for the program start in Eclipse could be written into the start configurations of the project, but since this is quite intricate for the simple test, it is recommended to change the string in the source code line
System.out.println(evaluateDigitSequence("."));
and then start the program without parameters. The console in Eclipse should now show the succesful initialisation output of the ASCLogger as well as the result of the evaluation of the string.
Additionally, the corresponding log file with the name of the test should appear in the project overview of Eclipse after the first program test. Repeating the test adds the new log data to this file.
Remark
It is also possible, and often more convenient, to execute the test cases in an automated way, so that you don't have to enter the
test data manually over and over again. Eclipse supports the creation, execution and evaluation of JUnit test-cases. The procedure
is the following:
DigitTest
(If required, allow Eclipse to add build path for JUnit)After that, the final project content should look like the following:
As can be noticed, in this test file, all of the test-cases are grouped according to the cover measure they should cover, which is stressed by the title of a method. Additionally, some formatting was added, so that test-cases are more readable. For each test-case, the form of a source code line is:
System.out.println("z: " + Digit.evaluateDigitSequence("."));
so that the resulting line in a log file would be "test-case" : "test-result".
After this, user should perform the actual test. This is performed by starting the test run with Run As → JUnit Test. The
test will run as long as necessary and the results of that run can be observed in a Console window of Eclipse. With the above
mentioned set of test-cases, the results are:
When the tests in Eclipse are completed, the user has to inform SOTA of this by selecting the menu item
Stop Test. The sources are then reconstructed into their original state. Alternatively, the sources can also be rebuilt via
the menu item
Restore Sources.
In order to read the log files, a dialog opens immediately after selecting the menu item
Stop Test. Here, all files in the execution directory ending on log are listed for the import. The log files selected there are read,
analysed for further evaluation and finally appear in the view TestLogs. The TestLogs marked there will now be used for computing the
soverage metrics and determine the presentation of the coverage in the views Source, CFG and Coverage.
If the user wishes to import the log files when SOTA is not in testing mode, the menu item
Read Logs evokes the same behaviour.
The program test is completed by creating a report. This is done via the menu item
Create Report which, depending on the settings in the preferences, either opens a dialog for entering a file name, or
automatically generates a name and creates a report file in the root directory of the test program.
SOTA offers the possibility to execute the manual program test in SOTA by using two scripts. The procedure matches the procedure described in point 6.1.2 in principle, but the third phase, the testing phase, is not done in Eclipse which is only needed for creating the project. The following data flow diagram represents an overview of the steps taken in SOTA and the points at wich the scipts complement the procedure.
The detailed procedure matches the procedure described in point 6.1.2 and only differing steps are ecplained.
Additionally to creating the project as in 6.1.2, an xml build file is exported from Eclipse. This can be done by
selecting Export ... in the context menu. A dialog opens
where the user should select General -> Ant Buildfiles.
In the second dialog window only the corresponding project (here: Digit) has to be chosen and after finishing an xml file named 'build.xml' is created in the root directory of the project which enables compiling the complete project.
Finally, a batch file 'Digit.bat' has to be created in the root direcotry of the project 'Digit'. This is necessary to start the program. The file has to contain the following Java command including the class path:
java -cp bin;lib/ASCLogger.jar; Ziffer
Creating the project in SOTA works as in point 6.1.2 with the only difference that the files 'build.xml' and
'Digit.bat' created in the first step, have to be imported on the first wizard page. However, it is also possible
to add these files at a later point via the menu item Configure Project.
In order to succesfully compile the project using the Ant build files, an Ant file 'ant.bat' that SOTA can execute also has to be included in the preferences under Preferences -> General -> Location of Ant. Since the settings in the preferences are effective for all projects, it is sufficient to make this entry once.
By including the two files the manual program test is available in SOTA. The dialog Start Test now offers the option Build Project, and as soon as this has been marked also the option Run Project. If the first option is active, the first Ant build file will be executed thus compiling the instrumented sources. The second option also runs the start script so that the program will be started. However, the start option of the current version of SOTA does not offer parameter passing to the test program which strongly limits testing options for our program 'Digit'. In order to test several strings, it is necessary to readjust the start script.
The remaining steps of the manual program test are consistent with the description in 6.1.2.
To use ATOSj as an external testing system and of HUSemOrg as a test program they need to be installed.
Instructions can be found here:
he testing procedure is similar to the manual program test with the exception that the actual testing is done using the external testing system. This leads to the following phases:
As in 6.1.3 Eclipse is only used to create the program and plays no further role in the program test as an Ant buildfile is used. The following data flow diagram indicates the phases transferred to the external testing system.
The detailed procedure matches the procedure of the manual test with only a few exception which are described below.
The program HUSemOrg should be unpacked into the Eclipse folder workspace and is already adapted to usage with SOTA. As in point 6.1.2 a project husemorg has to be created in Eclipse and the library ASCLogger.jar added to its build path.
The next three steps only refer to the preparation in SOTA and are identical to the manual test.
After the instrumented source files have been compiled ATOSj is started for the program test itself. The regression tests are performed on the instrumented classes. SOTA may be closed for this. The usage of ATOSj is in no way affected by the preperation phase with SOTA, there is no interaction between the two programs.
The remaining steps match the description in 6.1.2.
SOTA-ATM (Automatic Test Module) is a library containing the test functionality of SOTA and needing no GUI at all. This permits controlling the instrumentation of projects as well as the evaluation of log files with other programs.
There are two control options offered for the module. On the one hand, it is possiblet to start SOTA-ATM as an executable Jar which can be controlled and started via command line parameters enabling its usage in simple scripts. On the other hand, SOTA-ATM is a library which can be integrated making it an interface between functions for testing the project and other programs.
SOTA-ATM is an executable Jar and can be called via command line with several paramters. The necessary values about the project can either be passed to SOTA-ATM by importing a project file (-p) or by passing them on calling SOTA-ATM (-n). A project file can be generated via the graphical user interface of SOTA, by using the option -n of SOTA-ATM or manually.
The other options call the different components of the module. The main aspects are the instrumentation of source files (-i), the reconstruction of the original sources (-z) and the evaluation of tests by reading the appropriate log files (-t) as well as creating a report thereafter (-r).
Additional functions can be combined with the ones above. Those are the compilation (-c) and start of the test program (-s), provided the necessary files are available. The order of options is irrelevant.
Usage: java -jar SOTA-ATM.jar [-options]
options include:
-c [ <ant-buildfile> ]
compile sourefiles; only if ant-buildfile is provided
-i ( Level1 | Level2 | Level3 | <ischeme-name> )
instrument sourcefiles according to chosen level or IScheme
-n <name> <lang> <project-dir> [ <exec-dir> <src dir> ]
create new project file
-p <name>.project
open the project file
-r [ <report-file> ]
create report-file; if no name is provided it will be stored in "report.html"
-s [ <runscript> ]
start project; only if runscript is provided
-t <testname>.log [ <testname>.log ... ]
name of testlog to create or to import
-z restore original sources
The parameter c evokes the compilation of the project after the file operations have been completed. This option requires the project file to contain a reference to Apache Ant and to an Ant build file. The build file may also be passed via command line right after -c.
With the parameter i all sources of the project are saved as a backup and then instrumented. It has to be followed by either a name of an IScheme specified in the project file or one of the values 'Level1', 'Level2', 'Level3' which stand for a complete instrumentation of the entire project according to the respective levels. The project will only be instrumented, if none of the source files is instrumented already.
The parameter i requires the declaration of a test name via the parameter t and subsequently the usage of the parameters r to create a report and z to rebuild the sources.
The parameter n creates a new project on the basis of the passed values. A project name, the employed programming language and the reference to the project directory have to be indicated at the least. The execution directory as well as the source directory are optional values and are set to the project directory if not specified otherwise.
The project file will be saved as <name>.project. Using the paramter n prohibits the usage of parameter p.
With this parameter the user can pass a project file which contains the data to characterize the project. Such a project file can be generated by using the graphical user interface, by calling SOTA-ATM with the parameter n (see above) or manually. Using the paramter p prohibits the usage of parameter n.
This parameter evokes the creation of a report containing the computed coverage metrics, on the basis of the test's log files that have to be included by calling the parameter t. If a reference to an html file follows the parameter, the report will be written into it. Otherwise a file 'report.html' will be created or overwritten.
The parameter r requires at least one log file passed via t and finally the paramter i evoking the instrumentation of the source files. If the source files are already instrumented, they will be restored to their original state before the test is read.
The parameter s starts the test program. It is necessary to reference a run script either as a parameter or in the project file.
This parameter is used in two contexts. When an instrumentation via i is called, it defines the name of the test which is also the name of the log file that will be created. For creating a report with the parameter r all log files to be imported are listed after the parameter t.
The parameter z restores all source files of the project from the backup files. It cannot be used together with the parameter i for the instrumentation.
A project 'Digit' of the programming language Java is created. Its project directory is '/workspace/Digit'. From there all java files are imported into the project. After successful completion all project information is saved in the file 'Digit.project'. Various settings, e.g. build files that should be used, can be changed in that file.
This call of the module has the effect that the project 'Digit' is parsed, fully instrumented according to level 2 and then compiled and started. Additionally, the value 'test1' is passed as a test name. The order of the options is irrelevant.
With this call the original state of the sources is restored.
This call of the module creates a report for the project 'Digit'. The test 'test1' is read and the coverage metrics computed there are written into the standard report file 'report.html'. If the source files are instrumented, they will be restored to their original state first.
SOTA-ATM can be imported as a library into a program thus enabling the usage of the non-gui-functionality of SOTA for the static analysis and the coverage test. The javadoc documentation is available here.
The central class for the program test is SotaATM. Each of its objects represent a test instance of SOTA. A test instance can be configured by either loading a project file or passing an instance of ProjectConfiguration which contains all relevant data for the program test similar to a project file. For this instance the full functionality of SOTA, from instrumentation to report creation, is available (see javadoc).
Right after opening the project a Metrics object containing all metrics of the static analysis can be returned from the test instance. The coverage metrics are available in the very same object after testing the program and evaluating the generated log files.
To configure the instrumentation for a test, it is possible to define variable ISchemes which can assign a different instrumentation level to each structure (file, class, function) of the test project . The overall instrumentation of the project according to one instrumentation level is available via a GlobalIScheme.
The following java program code reads the given project file, starts a test with instrumentation level 2, compiles and starts the test program. After completing the test all source files are restored to their original state, the project is recompiled and an html report is generated from the imported test.
String projectFile = "D:/Development/eclipse/husemorg.project";
String reportFile = "report_test_1.html";
String testName = "husemorg_test_1";
String testDesc = "Test 08/15";
TreeSet<String> testSet = new TreeSet<String>();
testSet.add(testName);
SotaATM atm = new SotaATM(fileName);
atm.startTest(testName, testDesc, new GlobalIScheme("Level2", 2) , true);
atm.stopTest(testSet, true);
atm.createReport(reportFile);