1 Introduction
2 Overview
3 Installation and Start of Program
4 User Interface and Functionality
4.1 Menus und Toolbar
4.1.1 Menu Project
4.1.2 Menu Tasks
4.1.3 Menu Configuration
4.1.4 Menu Help
4.2 Views
4.2.1 View Project
4.2.2 View Testlogs
4.2.3 View IScheme
4.2.4 View Source
4.2.5 View CFG
4.2.6 View Coverage
4.2.7 View Metrics
4.3 Preferences
4.3.1 Preferences View CFG
4.3.2 Preferences View Coverage
4.3.1 Preferences General
4.3.1 Preferences Report
4.3.1 Preferences View Source
4.4 Delete Project
5 Files
6 Tutorials
7 Appendix

4 User Interface and Functionality

Next to the compulsory menu bar and the toolbar offering quick access to the most common actions the user interface consists of different views available in three areas by clicking the corresponding tabs. The upper left area offers the view with the project outline. Arranged below the project outline are two views listing the test cases and the instrumentation schemes of the project respectively. The right area takes up the largest part of the window, consisting of different views supplying detailed information about the source code, control flow graphs as well as coverage and metrics reports.

The views are linked so that e.g. selecting a file in the ProjectView causes the corresponding source code to appear in the SourceView. Selecting a test in the TestView updates the coverage metrics for the entire project and the representation of the coverage for the selected file. Wizards and dialogs are at the user's disposal in order to guide larger actions such as creating a new project.

Lastly, a status bar can be found at the bottom of the window, providing information about the status of the original source files as well as the parsed source code SOTA is displaying. The original state is designated as CLEAN whereas the instrumented state is marked as DIRTY.

Fig.: main window of SOTA

4.1 Menus und Toolbar

4.1.1 Menu Project

Fig.: menu Project

New Project

Clicking the menu item New Project opens the two-page wizard for creating a project. On the first page the user has to indicate a name for the project, as well as the project directory and run directory. From the project name SOTA generates the file <project name>.project placing it in the SOTA directory.

The project directory is the root directory of the test program (corresponds to ".. / workspace / project-name" in Eclipse projects). This directory contains the imported source code as well as the reports created by SOTA at the end of a test. In the run directory the test program is started. In most cases, the run directory corresponds to the home directory of the test program, except for Eclipse-RCP-projects where it is the Eclipse base directory ".. \ eclipse \". In the run directory the ASCLogger.ini is created which is read by the ASCLogger during the test. After the test the testlogs are read from the run directory as well

Choosing a programming language in the wizard is necessary, even though the current version of SOTA only supports JAVA. Finally, the user can include an Ant build file for compiling and a batch file for creating the test project.

On the second page the user has to include the source files to be considered for the test. They can either select a directory thus including all source files in all sub directories, or simply individual files.

After finishing the wizard, SOTA creates the project and loads all the selected source files.

Fig.: first page of wizard
Fig.: second page of wizard


Open Project

Clicking the Open Project menu item opens a project created previously. A standard dialog for loading a file from the root directory of SOTA appears, where the desired project can be selected.
After confirming the selection by clicking OK, SOTA opens the project and loads all the associated source files.

Fig.: dialog Open Project

Save Project

Saving the project by clicking Save Project creates, if done for the first time, a project file in the root directory of SOTA. The name of the file is identical with the project and ends on 'project'. This file contains the project data and all instrumentation schemes. If the file already exists, it will be overwritten with the current data.

Close Project

Clicking the menu item Close Project closes the current project. Consequently, SOTA is in its starting state again.

Recover Project

The menu item Recover Project's task is to restore corrupted projects. Analogous to the menu item Open Project, a dialog for selecting a project appears. As a result, all source files of the chosen project are restored and then opened.

Exit

Clicking the menu item Exit closes SOTA.

4.1.2 Menu Tasks

Fig.: menu Tasks

New InstrumentationScheme

Upon clicking the menu item New Instrumentation Scheme a dialog allowing the creation of a new instrumentation scheme (short: IScheme) for the current project appears. An IScheme represents an instrumentation pattern for all source files of the project. It can adopt different degrees. Depending on the test case, the memory requirements of the log files can be very high. The purpose of this configuration option is to allow the user to set a reasonable limit for the size of those files.

Fig.: dialog New IScheme

For creating an IScheme a name for management of the scheme inside the project, must be specified in the dialog. Entering a description is optional. The field below the description field represents the hierarchical structure of the current project in form of a tree. It shows all files, their classes and the individual functions. A level of instrumentation from 0 to 3 can be assigned to each of these structures by clicking the matching button for the level after choosing a certain structure. Consequently, each structure is colored according to their respective levels. Furthermore, the substructures associated with them receive the same level of instrumentation. The individual levels have the following effects:

For the program test, instrumentation level 2 is recommended as it allows the identification of all coverage metrics. If necessary, a lower level can be used in order to save capacity. Using level 3 is suggested when programs or functions are not terminating properly and the user wishes to identify the exact point of abortion or where the exception is called.

After confirming the dialog, the instrumentation scheme as well as all selected settings are stored in the project-specific .ischeme-file which is from then on available for the program test.


Start Test / Restart Test

Fig.: dialog Start Test

By clicking the menu item Start Test the program test is initiated in SOTA. A dialog box for configuring the test appears. The user has to enter a name for the test which also serves as a name for the test log SOTA creates at the end of a test, i.e. limitations for file names set by the respective operating system must be considered. Adding a description of the test is also possible but not mandatory.

The instrumentation of the project has to be configured next. In addition to the user-generated ISchemes, the 3 basic ISchemes which instrument to whole project according to Level 1, Level 2 and Level 3 are also available. The hierarchical project list shows the specific instrumentation of the individual project structures for a selected IScheme. The list can be fully expanded or collapsed by clicking the buttons Expand All or Collapse All respectively.

Once a name for the test and an IScheme have been selected, the program test can be started by confirming the dialog. SOTA stores the information relevant for the logging component in the file 'ASCLogger.ini' by adjusting the execution path of the project. Subsequently, a backup of all original source files is produced by changing the file extensions to '.backup' and the source code is instrumented. Then the project is parsed again. Finally, the user can view the instrumented source code in the SourceView. Now the instrumented version of the source code can be compiled and e.g. tested systematically with an external testing system.

Three more options can be selected by ticking the check boxes at the bottom of the dialog. Rerun configuration merely leads to a change of the name of the test and its description in the ASCLogger.ini leaving the source code unchanged. This enables re-testing the code with the same configurations but without having to instrument and compile it again. Build Project causes SOTA to compile the source code after the instrumentation. For that purpose it is necessary to embed an Ant version by providing the path to the file 'ant.bat' under the point General of the preferences. Furthermore, the user has to include an appropriate xml-file enabling SOTA to compile using Ant. This option is only available if a script exists and the option Rerun configuration has not been selected. Finally, the option Run project actually causes the project to start after compilation as long as a startup script has been specified. This will be possible, only if one of the other two options are selected.

If a test is running already or SOTA finds the source code instrumented during parsing, the option Start Test will be exchanged with Restart Test. It is not possible to re-instumentate the sources without having stopped the current test and restored the original files. However, The instrumented test program may be started again with a different name for the test. This only updates the initialization file of the logging component with the new name for the test, but no source files or binaries are changed. Restarting the test is equivalent to the rerun option of the normal Start-Test-dialog.


Stop Test

Fig.: dialog Read Logs

The Stop Test menu item terminates the current test run. Then the original source files are restored and re-read. If a corresponding Ant buildfile is available, the original sources will also be compiled again.

Finally, a dialog appears allowing the user to select log files from the run directory which SOTA will then read, analyze and list in the view TestLogs. Subsequently, the coverage metrics for individual test logs or combinations thereof can be computed and displayed.


Create Report

Fig.: dialog Create Report

By clicking the menu item Create Report it is possible to create a coverage report based on the selected test logs for the current project. The user can adjust the preferences so that for each report a prompt appears for setting the file name and path. Otherwise, the report is stored in the root directory of the project under the name 'report.html'. In the preferences there is also a setting for preventing SOTA from overwriting report files. In that case a new file is generated for each report according to the format 'report_<date>_<index>.html'. These and other settings for creating reports can be made in the preferences menu under the item Report

Build Project

The menu item Build Project enables compiling the project in SOTA on the basis of the current source files regardless of the test status. This menu item will only be activated, if an xml buildscript is specified for the current project and an Ant version is included in the preferences.

Run Project

The menu item Run Project will be available, if a startup script is specified for the current project. After clicking this item, the project will be executed.

Restore Sources

In order to translate the project into its original state, it is also possible to select the menu item Restore Sources. Unlike the Stop Test menu item the source files are only restored and read. They are not be re-compiled (assuming a buildfile exists), nor does the dialog for reading the test logs appear.

Read Logs

With the menu item Read Logs the log files can be read manually regardless of the test status of the project. For that purpose, the same dialog as for the Stop Test menu item appears allowing the import of log files. After selecting the log files from the run directory of the project, SOTA reads the files and processes the test results.

Show Coverage

Activating the menu item Show Coverage leads to a colored display of the coverage information in the source code and coverage views.

4.1.3 Menu Configuration

Fig.: menu Configuration

Configure Project

Fig.: dialog Configure Project

Project-specific settings can be made via the menu item Configure Project. In detail, these are the two paths of the test project relevant to SOTA, namely the root directory and the execution directory, and, underneath them, the optional scripts imported for the purpose of compilation and start of the project. Confirming these values by selecting Ok overwrites the settings made when the project was created.

Select Sources

In order to modify the source files included in the project, the user has to select the menu item Update Sources. Subsequently, a dialog identical to the second page of the wizard for creating a project appears. The source files can be selected analogically. After confirming the selection, the current source files are replaced by the new ones.

Preferences

Under the menu item Preferences the user can make changes to the general settings of SOTA that apply to all projects and will also be saved when exiting the program. A comprehensive explanation of the variuos configuration options can be found under 4.3 Preferences.

4.1.4 Menu Help

Fig.: menu Help

Manual

Selecting the menu item Manual opens the user documentation on hand in the system's standard browser

About

The menu item About opens a dialog containing information about the current SOTA version as well as the plug-in status of the application.

4.2 Views

4.2.1 View Project

Fig.: view Project
(hierarchical presentation)

In the view Project all source code files of the test program and their subordinate structures, as well as classes and methods are listed in a tree structure. The top nodes represent the source code files imported on creating the project. The top-level classes and methods and their respective internal classes and methods are represented as children in hierarchical order up to an arbitrary nesting level. Using the two buttons and the user can fully expand or collapse the tree. The button alternately causes the tree to be displayed in ascending and descending order. On selecting an element of the tree the views Source and CFG are updated automatically.

The individual structures listed in the tree are:

If instructions for instrumentation are detected on parsing the file, e.g. during a test run on instrumented sources, the Icon will be displayed with a red exclamation mark (e.g.:).

Fig.: view menu
Fig.: view Project
(flat presentation without files)

Via the menu selectable on the top right corner of the view, the presentation of the project structures can be configured. The two presentation options Flat Presentation and Hierarchical Presentation are available. The hierarchical presentation matches the listing in a tree structure as described above which is the default setting. In the flat presentation all structure units are at root level, i.e. file, classes and functions are listed coequally.

The additional options allow the following adjustments of the presentation:


4.2.2 View TestLogs

Abb.: View TestLogs (unlocked)

If log files are imported into the project, they will be listed in the view TestLogs. In case a log file already exists, the new log data are appended to its end, so that a test log can contain a variety of test case data. As a default, the individual test cases are hidden from the user and only the test logs are listed. In order to examine the content of the test logs in detail, the user has to unlock the button Lock TestLogs on the toolbar of the view, which lists all test cases as well as ISchemes of every test log right beneath each one.

Since not every log file may have been created by SOTA and not all the test data belongs to the current project, the view TESTLOG uses the following icons to represent compatibility:

If valid test logs and test cases are selected, the coverage information in the views Source, CFG and Coverage will be updated automatically. Clicking the buttons and changes the entire selection. To select several test logs at once, the keys <Shift> or <Ctrl> respectively have to be held down while selecting the desired test logs.

Selected test logs can be deleted with the Delete TestLogs button. That not only removes them from the project but from the system. This option is not available for test case data, i.e. individual parts of a test log.

Fig.: dialog TestLogs

On double-clicking onto one of the listed test logs or test case data a dialog appears containing detailed information on the test data. Listed first are the name and description of the test as well as the name of the instrumentation scheme. Next, the number of paths (i.e. function cycles) contained in total in the test case and a list of all functions the test hit are indicated. For each function the number of associated paths is also listed. By clicking the column headers the presentation of the list can be sorted in ascending or descending alphabetical order by function name and number of paths.


4.2.3 View IScheme

Fig.: view ISchemes

The view ISchemes contains only the instrumentation schemes (short: ISchemes) related to the project. For every project three ISchemes are created automatically, instrumenting the entire project into the levels 1, 2 and 3 respectively. The user can create additional ISchemes which will then appear in this view via the menu item New IScheme.

Double-clicking on an IScheme opens a dialog listing, analogous to the menu item New IScheme, stored instrumentation settings of the IScheme and enabling the user to change all information. The button Delete IScheme removes an IScheme from the project.

The standard ISchemes across levels 1,2 and 3 are exempt from these changes and cannot be deleted.


4.2.4 View Source

In the view Source the source code of the file selected in the view Project is shown. If individual classes or methods are selected, only the corresponding source code lines will appear.

In case the coverage indicator Show Coverage in the toolbar is enabled, the coverage of the source code by the test logs selected in the view TestLogs is indicated with colors. Green lines in the source code were covered by the tests, whereas red ones were not traversed. A line containing several statements is marked green as soon as one of the instructions was covered. The corresponding colors for the line coverage and the syntax highlighting can be set in the menu item Preferences.

Fig.: view Source

4.2.5 View CFG

While the view Source only shows line coverage and is primarily intended to give an overview of the source code coverage, the view CFG (Control Flow Graph) provides detailed information on the coverage of source code structures. On selecting a function in the view Project, the upper part of the view CFG shows the corresponding control flow graph. For every function appears at least one node for entering and one for exiting the function with the latter being the joint of all edges leaving the function. Each branching structure is represented by a node labeled as follows:

For assigning nodes to their corresponding parts of the source code, it is sufficient to click on a node. Then the bottom of the view CFG focuses on the appropriate section of the source code and the line is highlighted in yellow. If the option Pin SourceView is selected in the toolbar of the view, only on the first selection of a node the corresponding source code line will be displayed, afterwards the source code remains 'pinned' and does not scroll automatically anymore. Selecting the option Show Number of Paths displays the number of edge traversals during the current test on the left of every edge of the control flow graph.

In order to be able to also view large graphs clearly, the user can scale the illustration of the control flow graph via the buttons Zoom Out and Zoom In. SOTA offers seven zoom levels. The first three zoom levels still display the condition coverage and the fourth level still shows the caption of the nodes. The three smallest zoom levels reduce the graph to blank squares. More detailed information is available via the tooltip or the node information dialog (see below).

Fig.: view CFG

If test logs or test case data are selected in the view TestLogs, the color of the nodes and edges will change according to the coverage of the selected test case data. Also, the coverage data is updated automatically. Covered nodes and edges are colored green, whereas not covered ones are colored red. To detect covered nodes whith multiple outputs of which not all are covered- which might be the cause for not covered sections of the source code- these are colored yellow. The user can configure this just like all other colors and also line style and thickness in the preferences.

Fig.: tooltip
(iteration-node)

The user can obtain additional information by navigating the cursor over a node. The tooltip that opens indicates the type and number of hits ('nrHits') by test case data of each node, as well as its internal project ID and the line number where it can be found. Branching nodes also contain information about the number of times each branch was taken. Therefore, the tooltip for if-nodes lists a value for each the true and the false branch, whereas switch-nodes contain an overview of the chosen selections and a list of the number of times each case was called. The case-node also contains this value as 'nrSelects' which can differ from the number of hits. The tooltip for iteration-nodes lists, next to the number of hits, details about how often the loop body was skipped ('nrSkips'), the number of times it was executed exactly one time ('nrSingleLoops') and two or more times('nrMultipleLoops') as well as the overall number of executions ('nrLoops'). For the try-nodes which initiate the exception handling, it is mentioned how often the try block could be completed without an exception.

Fig.: node information with MCDC pair

Information on condition coverage is viewable in two ways. On the one hand, located on the right side of each node containing a non-trivial condition are four small boxes representing the different degrees of condition coverage. From top to bottom, these are the single, minimal multiple, MC/DC and multiple condition coverage. The color indicates the degree of coverage from green meaning 100% to dark red meaning 0%. The threshold values and colors can be set in the preferences, where their presentation can also be turned off, and the user can select two additional sets of coverage related data to be displayed.

For more details about the coverage of individual nodes with or without a condition a dialog box which can be opened via double clicking the node is available. Firstly, all information from the tooltip and all relevant coverage data of this node are listed both by percentage and numerically. If the node contains a non-trivial condition, the logical structure of the condition as well as all assignments to their elements are also shown in a table. The first column represents the truth vector for the entire condition as read from the test log file. The second column contains the evaluation of the entire condition and the subsequent columns show the evaluation for each element. For each elementary condition the appropriate MCDC pair, if present, will be highlighted by clicking the corresponding table head.


4.2.6 View Coverage

The view Coverage offers a means for essentially analyzing the coverage data of the project. Analogous to the view Project all structures of the project are listed hierachically in a tree, which can be configured through the menu of the view to adopt a flat representation. For each project structure the percentage of the degree of coverage by various coverage metrics is specified in the corresponding columns. If the option is activated in the toolbar, the values will be highlighted in favour of clarity. The individual colors and thresholds can be defined in the preferences.

The listed coverage metrics are: Function Entry Exit Coverage (FEEC), instruction coverage (C0), branch coverage (C1), single condition coverage (C2), minimal multiple condition coverage (MMDC), modified condition/decision coverage (MCDC), multiple condition coverage (C3), modified boundary-interior path coverage (MBI) and boundary-interior path coverage (BI). The definitions of the individual coverage metrics are listed in the Annex.

If a line instead of a value is displayed in the view, the corresponding metric is not appicable to the structure, because the class does not include e.g. conditions or instructions. The user can change the presentation of the value itself, via the option Change Info, from percentage to ratio representation and back again. With the two buttons Expand All and Collapse All the hierarchical presentation of the project structure can be fully collapsed or expanded.

The user can sort the entire table by each column, i.e. by name and coverage ratio. In order to do that, it is necessary to click the corresponding column head. The sorting is based solely on the root elements of the tree, i.e. in hierarchical representation on the values of the files and classes respectively. However, in flat representation the structures can be sorted by functions.

Fig.: view Coverage

4.2.7 View Metrics

SOTA calculates a number of static metrics during the syntactic analysis of the source code while parsing. These can be evaluated immediately after the project was read in the view Metrics. Similar to the view Coverage this view consists of a project tree and a mapping of the following metrics to each project unit: cyclomatic complexity, essential complexity, lines of code, number of instructions, number of branches, number of modified boundary-interior path segments, number of boundary-interior paths, number of instructions evaluating conditions, number of elements in all conditions and number of conditions. Explanations as well as definitions of individual metrics are listed in the Annex.

As in the view Coverage, the user can change the presentation of the project structure in the menu of the view and expand and collapse the presentation by clicking the buttons Expand All and Collapse All. Sorting the table also works through clicking the column heads.

Abb.: View Metrics

4.3 Preferences

4.3.1 Preferences View CFG

Fig.: Preferencees View CFG

In the first options block of the preferences page of the view CFG, the user can determine which coverage metrics will be displayed as small square labels next to each node of the control flow graph. The default setting shows only four labels for the condition coverage metrics (C2, MMDC, MCDC, C3) next to all nodes containing non-trivial conditions, and at the function node. If the display of path coverage metrics are enabled (second option), three more labels will appear next to the function node indicating the overall modified boundary-interior path coverage, the coverage of modified boundary-interior paths by the function body and the boundary-interior path coverage. Similar to the second label each iteration node also receives a label for the coverage of modified boundary-interior paths corresponding to the iteration. The last option causes the remaining three coverage metrics (FEEC, C0, C1) to appear at the function node.

In the second block, the user can configure the presentation of the nodes of the control flow graph. Colors for normal nodes without coverage information, covered and not covered nodes can be selected. For a more differentiated presentation branching nodes which are covered, but whose outputs are not covered completely, are highlighted. The color of the highlights can be set under Branching CFG-nodes. Removing the tick on the option above switches the differentiated presentation off.

Additionally, in the third block the color of the edges of the control flow graph for the presentation of edges without coverage, as well as for covered and not covered edges can be configured. The line thickness of the edges can also be set to a value from one to three.

Finally, the user can also change the color for highlighting lines in the source code corresponding to the selected node of the control flow graph.


4.3.2 Preferences View Coverage

Fig.: Preferences view Coverage

Here, the colors for highlighting the percentage of the coverage in the view Coverage can be defined as well as the percentage bounds.

Two elemental bounds are complete (100%) and no (0%) coverage. The user can add percentage values for five additional bounds, with the result that each cell is assigned a color according to the respective values. The default limits are: 25%, 50%, 75%, 90% and 99%. If the user specifies a bound with a value larger than one of the bounds above it, this bound will be ignored.

Subsequently, the user can assign a color to each bound. In the table of the view Coverage, all cells where the percentage of the coverage reaches the specified bounds will be colored as defined.


4.3.3 Preferences General

Fig.: Preferences General

The user can change the general settings of SOTA under the preferences item General. In order to use an Ant buildfile to compile a project automatically, the corresponding Ant file (\bin\ant.bat) of an installed version of Ant has to be embedded here. Then, it is possible to select the option Build Project in the dialog Start Test, and the menu item Build Project, as well as its equivalent in the toolbar are enabled. In Eclipse an Ant buildfile for compiling the project can be generated via File -> Export -> Ant Buildfile.

If Create log file is activated, system messages of SOTA will be saved as log files of the format sota_<YY-MM-DD>_<index>.log on each program boot. Also enabling the option Overwrite existing log file causes SOTA to create only one log file sota.log and overwrite it every time the program is booted.

The last two options define more general aspects of the behavior of SOTA. If Parse instrumented source code is activated, the project will be parsed again after launching the test, the view of coverage metrics will be disabled and the presentation of the project in all views will be based on the instrumented sources. In this case, the source code displayed in the view Source is always identical with the current sources, i.e. the two status indicators on the status bar always show the same value. Should the option be disabled, the backup will be parsed and displayed instead of the instrumented file. This allows an evaluation of the test logs independent of the status of the source files. The last option determines whether the test project will be compiled automatically, provided that a corresponding Ant script was included, in addition to restoring the original source files on stopping the test. Otherwise, the binaries of the test program would remain instrumented and continue creating logs.


4.3.4 Preferences Report

Fig.: Preferences Report

Under the preferences item Report the output of the report in a html-file can be configured.

Activating the option Prompt for file name in the first block, causes a file selection dialog to appear on selecting Create Report in the menu. The dialog asks for the name of the report file which will be created. Otherwise, SOTA creates the file report.html in the root directory of the test program overwriting it every time a new report is generated, or SOTA saves each report according to the name scheme report_<date>_<index>.html. This behavior is determined by the option Overwrite existing report.

In the second block the user can define the content and the presentation of the report. It is possible to choose between using all test logs for the report or only the currently selected ones. Also, the user can select elements from the following list to be included in the report: the applied tests with ISchemes and descriptions, an overview of the coverage of all classes (including or excluding inner classes) and/or an overview of all functions, sorted according to their classes. Finally, the font size for the report file can be specified.

If Use Colors is selected, the coverage metrics in the report file will be highlighted in color according to the degree of coverage similar to the presentation in the view Coverage. The values for the bounds are adopted from the preferences item view Coverage, but here the colors of each bound can be defined separately for the report file.


4.3.5 Preferences View Source

Fig.: Preferences view Source

The highlighting of syntax in the view Source can be adjusted in the correspondent preferences item. The colors for the keywords of the language, comments, strings, and those comments added by instrumentation through SOTA can be chosen arbitrarily, and subsequently the font size for the presentation of the source code can be defined.

To view the coverage of the source code in the view source, it is possible to select the background color for covered and not covered source code lines here. The colored highlighting will be applied as soon as test logs are read and if the option Show Coverage is activated.


4.4 Delete Project

A function 'Delete Project' was not implemented. This means that SOTA files corresponding to a project with the exception of the log files (see View Testlogs) have to be deleted by hand.

All SOTA files are described in 5.1. The files created by SOTA that have to be deleted in order to clear the system are in detail:

It is absolutely necessary to restore any instrumented source files to their original state (by choosing the menu item Restore Sources), before deleting the backup files as this is hardly possible without them!