Gathering Regression List for Structural Coverage Analysis

By Shricharan Gaddam, eInfochips - An Arrow company

Introduction:

Structural coverage analysis is a method to ensure RBTs parse the code structure to verify every line of code for its correctness.

RBTCA is used to determine the efficiency our tests verifying implementation of the software requirements. The intention is to check whether code and requirements are in sync. This analysis shows that:

1. Test cases exist for each software requirement.
2. Test cases assure the benchmarks/guidelines of normal and robustness testing.
3. Redundant test cases can be identified.

RBT entails verifying that software meets application(s) requirements. Every requirement embraces set of tests signifying eminent realization of that requirement. In order to make our software bug-free, a measure of high-test coverage lowers chances of containing undetected software bug. If we keep the measure as 100% test coverage, then more percentage of source code is realized/executed while testing. The coverage criteria for measurement in category wise are as follows:

1. Function Coverage
2. Statement Coverage
3. Branch Coverage
4. Condition Coverage
5. Decision Coverage (optional)
6. Modified Condition/Decision Coverage

Let us understand briefly on each coverage mentioned above with the help of the source code below:

int demoFunc (int x, int y)
{
int z = 0;
if ((x > 0) && (y > 0))
{
z = x;
}
return z;
}

Function coverage analysis – This checks if each function (or subroutine) in the program is been called. Throughout the course of test suite execution, if subroutine 'demoFunc' is called no less than once, then subroutine coverage for this subroutine is fulfilled.

Statement coverage analysis – This checks if every instruction in the source code has been executed no less than once. Statement coverage is achieved for demoFunc() when tests calls this function with x=1 and y=1 as inputs, covers every line in the function including z = x.

Branch coverage analysis – This checks if every branch of every control structure (code statements as in ‘if’ and ‘switch case’ control statements) has to be executed. Let’s consider an instance, for an ‘if’ condition, true part and false part has to be realized. Tests with demoFunc(1,1), demoFunc(0,1) achieves branch coverage, in the initial test case, required conditions (x > 0), (y > 0) are achieved and z = x instruction is executed, where as in the later test case, condition (x > 0) is never achieved, and hence instruction z = x is never executed.

Condition coverage analysis – This checks if every boolean sub-expression is verified for true and false conditions. Tests which call demoFunc(1,0) and demoFunc(0,1) will satisfy condition coverage. In the initial test case, (x > 0) evaluates to true, while (y > 0) evaluates to false; where as in the later test case, (x > 0) evaluates to false, while (y > 0) evaluates to true.

Condition coverage might be inferred as branch coverage. To evaluate the difference let’s observe the source code instruction:

"if p and q then"

Condition coverage can be achieved by following test cases:

p=true, q=false

p=false, q=true

In any of the above test cases branch coverage is not achieved since, none of the test cases achieves ‘if’ condition. However, it is necessary to inject faults to make sure that all conditions and branches that handles code exceptions are covered while testing.

Decision coverage – All the entry and exit points of the source code are been verified and each decision in the source code is verified for all possible outcomes at least once.

Modified condition/Decision coverage – The Structural Coverage Analysis verifies if all the entry and exit points of the source code are been verified, each decision in the source code is verified for all possible outcomes at least once, and it is ensured that each condition independently impacts the output of a decision. To elaborate the above-mentioned independence criteria an example is sown below,

For instance, let’s observe the below source code instruction:

if (p or q) and r then

With below mentioned tests, the condition and decision coverage is achieved

p=true, q=true, r=true

p=false, q=false, r=false

In any of the above-mentioned test cases modified condition/decision coverage is not achieved, it is vivid that the output is not influenced by either value of 'q' in the initial test case or by value of 'r' in the later test case. Hence the below mentioned test cases are required to achieve MC/DC:

p=false, q=true, r=false

p=false, q=true, r=true

p=false, q=false, r=true

p=true, q=false, r=true

Objective:

The objective of this paper is to discuss an approach or an idea for gathering the regression list to perform SCA for the modified or updated source code, but in the case of structural coverage analysis, the objective is to ensure that the source code is covered through the mentioned verification methods (test cases, analysis, inspection).

Problem Statement:

Many a times, SCA teams end up executing all the test cases/procedures for a regression activity, which in turn takes lot of time for execution and dependency of the target time (if any) increases. Required time to complete SCA activity increases which in turn increases cost of the project.

For example, if we are monitoring different parts of a car, like the tire pressure monitoring system, fuel monitoring system, navigation systems, ambient lightning, ambient temperature, etc.

Let’s assume that after delivering all the above mentioned systems monitoring applications. Later if one wants some updates in fuel monitoring system, the structural coverage analysis ensures that the system is well verified and all the test cases/procedures for that particular update are verifying the desired purpose of requirements.

In order to perform the structural coverage analysis, there is no need to consider test cases for other systems. For e.g. the test cases of fuel monitoring system are more than enough and to be more specific test cases related to updated requirements are enough to ensure that requirements-based test cases exercised the code structure.

Clarification(s):

1. This paper discusses about the regression list for SCA, which may vary with the regression list of verification or rfs/dry run. The reason is that, when a source code is modified or added, it is mandatory for the verification team; to verify the modified code and ensure that it is tested as per the Requirement based test (RBT) approach; also, no other functionality in the system is hampered by it. In order to ensure this, the verification team might go with all test cases or with selective test cases (where justification will be given for selecting the specific test case).
2. It is suggested that verification activity is performed using Requirements Based Testing (RBT) method.
3. Here the tools which are used for (not limited to) this activity are mentioned below:
   1. Requirements maintaining and tracing to high level or low level or to the verification test cases/procedures can be done using DOORS or Jama.
   2. SCA activities are done using “Rapita and Vectorcast”. These tools can be customized to our needs; in consultations with the vendors of the tool while or after purchasing them; for generating the analysis report.
4. For any SCA activity, normal build is not used. The source code is ‘instrumented’ for structural coverage analysis and this ‘instrumented’ code is called as ‘instrumented’ build. Test cases/procedures are executed on this ‘instrumented’ build for SCA activity.
5. Output of the instrumented build; when test cases are executed; is either ".dat" or ".txt" file for Vectorcast or Rapita respectively.
6. It is suggested to have the source code designed and implemented components wise, or functionality wise. All these components can be integrated and there can be one build for all the components combined. So that only one executable is needed. However, it is the user’s choice to have component wise build and executables also.

Solution(s):

Generally, for any modification or addition of the source code, a change request must be raised. This change request drives further the process of requirement gathering, validation, design, development, integration and verification. There are few methods to perform the SCA activity for this source code.

Solution 1:

Consider there are multiple change requests; based on each change request; using the requirements maintenance tool, we need to trace and identify the verification test cases and procedures; if the list of test cases/procedures is more some automation script can be written.

Advantages of Solution 1:

1. This final filtered test cases/procedures list will be very less compared to the list used to verify the whole project/build.
2. Target required time/target occupied time required to execute these test cases is less.
3. Test procedures execution time on instrumented build is less.

All the above factors lead to cost reduction also.

Solution 2:

When a change request is driving a change only in a particular component or functionality, test cases related to that component are taken for execution on the instrumented build. Since the verification is performed based on the Requirement Based Test methodology, these component related test cases will be sufficient for performing structural coverage analysis. Here it is suggested or recommended that test cases and procedures are placed in the respective components folders.

For example, let’s consider same example of development of monitoring different parts of a car; like the tire pressure monitoring system, fuel monitoring system, navigation systems, ambient lightning, ambient temperature, etc. and the change request is for updating fuel monitoring system.

Then we must place the test cases and test procedures of fuel monitoring system in the respective folder only not in a different component folder. This helps in identifying the test cases and procedures quickly and easily for performing the structural coverage analysis.

Advantages of Solution 2:

1. Test cases and procedures are organized based on their components and functionality.
2. The final test case/procedure list will be smaller as compared to the list used to verify the whole project/build.
3. ‘Target required time’/’target occupied time’ required to execute these test cases is less.
4. Test procedures execution time on instrumented build is less.

All the above factors lead to cost reduction also.

Further Improvements:

We can further filter/optimize these test cases based on identifying the changes driving in that component and choosing them to use the requirements maintenance tool.

Solution 3:

While purchasing an SCA tool (like Rapita or Vectorcast, etc...) we can ask for customizations in tool to include features for identifying list of test cases for any specific part of code or IDs (In SCA, while instrumenting the source code, IDs are assigned to file, function and statements). This may be possible based on mapping IDs to test cases.

Advantages of Solution 3:

1. The only job of an engineer will be, to give the list of files or functions or code statements or IDs (to be more specific about the identification of test cases) to the tool to get the test cases list.
2. The final test case/procedure list will be smaller as compared to the list used to verify the whole project/build.
3. Target required time/target occupied time required to execute these test cases is less.
4. Test procedures execution time on instrumented build is less.
5. Ramp-up time for engineers is also less.

All the above factors lead to ‘Cost Reduction’ as well.

Conclusion:

It is preferred to use optimized or optimal test list for regression activity for structural coverage analysis instead of using whole test lists. This method leads to less ramp-up time for engineers, target required time is less, test procedures execution time on instrumented build is less. All these factors lead to cost reduction and less delivery time. eInfochips, an Arrow company, is a leading global provider of product engineering and semiconductor design services. With over 500+ products developed and 40M deployments in 140 countries, eInfochips continues to fuel technological innovations in multiple verticals.

For more information connect with eInfochips today.

Abbreviations:

RBTCA: Requirement Based Test Coverage Analysis

RBT: Requirement Based Test

References:

1. https://en.wikipedia.org/wiki/Code_coverage

2. https://doc178b.blogspot.com/2013/08/do178b.html

3. https://www.rapitasystems.com/products/rapicover

4. https://www.vector.com/int/en/products/products-a-z/software/vectorcast/#c176623

About Author

Shricharan Gaddam, works as Sr. Engineer at eInfochips -An Arrow company, holds a Master’s Degree in Digital Electronics and Communication Systems from JNTU Hyd. He has work experience on multiple embedded domains like Multimedia, Medical/Health Care devices, Avionics, Networking. His responsibilities incudes Requirement gathering, validating, development and verification in the software development life cycle.

If you wish to download a copy of this white paper, click here