Platform Software Verification Framework Solution for Safety Critical Systems

Nirav S Patel and Rahul K Patel, eInfochips

Abstract

Aerospace and Automotive industries are growing rapidly, from mechanical controls to electronic controls. This has increased the complexity of software running on powerful airborne hardware/systems. Thus, the safety of these software relies heavily on the verification. Advancing aerospace and automotive technologies require complex designs of hardware and RTOS. So, verification of platform software per safety-critical standards (i.e. DO 178B/C[2]) is significant, involving more resources in terms of cost and time. Verification of platform software components needs a mixture of different approaches [1]. Here, in this paper, we shall discuss the challenges involved in various verification platform software framework solutions for ARINC-653-based operating system and how a single framework, PSW eIHTestCon can resolve these challenges.

Introduction

Over the last few decades, airborne safety-critical systems are heavily relying on electronics and software to overcome weight, cost, size, performance, feature enhancement of the products and time-to-market factor. This has also increased the complexity of platform software such as VxWorks 653 [5], Lynx OS-178 [6], Deos etc. and platform software components such as Core Board Support Package (BSP), Chip Support Package (CSP), Program-specific FPGAs (Network Engine, Graphics Engine, Hardware Monitor), Avionics Network Driver, USB etc. In few programs, we have come across architecture that has multiple operating systems running on a hypervisor. It is a proven technology in the other industries, however, as we often deal with exteremly safety-critical systems in aero / auto, it is more important to verify it robustly using different techniques. The verification of platform software and its component per DO-178B/C is costly and time-consuming as it involves following challenges:

• Diversity across platform software components
• Dependency on platform software component to develop test procedures
• Complexity in terms of technology and design
• Complexity of hardware
• Analysis of the Interface Performance

To deal with these challenges, it is very important to have a common verification framework for platform software that provides an easy interface with following features:

• Portable and reusable test procedure across different platforms
• Easy-to-develop platform software verification test procedures
• Automated testing without user intervention running from Host
• Easy to debug failures from host PC
• Consolidated result logs on host without storage constraints
• Easy-to-customize Host-based test report generation as per need and integration with other tools
• Consistent result format across different module
• Simple architecture from tool certification perspective

A good solution of framework reduces time and cost of verification with significant increases in the quality of verification. eInfochips PSW eIHTestCon solution provides a capability to verify all level of platform software components with ease. In this paper, we will provide common platform verification framework solutions, their merits, demerits and understand details of PSW eIHTestCon design to overcome shortcomings of other solutions.

Platform Software Verification Framework Solutions

Platform Software components are much more diversified in implementation i.e. Boot loader, CSP, BSP, and Device Driver. So, in most of the case, platform software verification engineers make their own verification framework based on the need of platform software component.

Mostly platform software is written in the C Programing Language. Here, we are presenting basics of different types of Platform Software Verification Framework Solutions to support C Language-based Platform Software and Platform Software Component Verification.

[A] Config File-based PSW Verification Framework:

Description:

This test approach is mainly used for verification of library where the requirements are API-based and user interfaces are clearly defined. The test config file defines the API, input parameter and expected result. The test framework uses this config file as an input to call interface level APIs of Module under test with given input parameter values. The actual outcome of APIs is compared against the expected value defined in the config file to generate the test result. Configuration files can be in CSV, Text or XML format.

Pros:

• Easy to define multiple combinations of inputs
• Easy to understand test procedure
• Less time consumption while drafting test procedure

Cons:

• It is challenging to verify intermittent functionalities which are not visible at the interface level.
• It does not provide a capability of simulating the behavior of Internal APIs / Hardware. Example malloc returns ERROR or failure of device interrupt generation.

[B] Host Script-based PSW Verification Framework:

Description:

This test approach is preferred for API based verification. The test framework resides on both target and host systems. The host test framework called as a server, interprets test procedure scripts, translates it into target commands, controls the execution of tests, compares actual v/s expected results and generates the result data. The target test framework called as a client interprets commands received from the host, translates the received commands to test APIs, executes test APIs and sends back the response to the host.

Pros:

• Test control from host system
• Easy to translate test case in test procedure
• Less time consumption while drafting test procedure for the same API

Cons:

• Test logic needs to be defined on both target and host systems
• Challenging to verify complex data structure
• Knowledge of test framework internals are required to draft the test procedures

[C] Debugger-based PSW Verification Framework:

Description:

This test approach is used for boot up platform software components i.e. Bootloader, CSP, BSP. Verification is challenging as test framework cannot get the control over the execution of such platform software components.

The test procedures load the debug symbol file, halt the execution at the function under test and alter the input parameters using debugger interface APIs. This approach requires the hardware debugger i.e. ICE, Lauterbach to be connected between the target and host system. The test procedure uses the test-harness APIs to compare actual vs expected value and capture the result data. Most of the debuggers are supporting GNU Debugger (GDB) and Python/TCL interfaces for automation.

Pros:

• Easy to simulate any failures
• Easy to verify Boot up level components
• Simple to verify functionality of assembly language

Cons:

• Time-related functionality cannot be verified as debugger introduces addition overhead
• Source code is visible to verification engineers (white box approach)
• Framework relies on stability of debugger hardware
• Less preferred due to white box test approach

[D] C Language-based PSW Verification Framework:

Description:

This test approach is usually used to verify the PSW Components developed in C. So, it is easy to develop and integrate test procedure in C. The test procedure is built with the test framework and operational software. The test procedure uses the test harness, Stub driver and Test driver to test the platform software components. The result logger is used to transfer the result to Host. The test procedures running in user space uses the Stub driver and Test driver to simulate the input condition and capture the kernel space data.

Pros:

• Easy to understand as developed in C language
• Easy to integrate with platform software environment
• Capable to test the kernel level and user level components
• Easy to simulate the failure condition using stub driver

Cons:

• Test framework and test procedures are not independent, whenever Build Test procedure, needs to build test framework and data load on target
• Assembly components of the PSW cannot be verified
• No Control from host to run selective test procedures
• Longer time to test procedure debugging and fixing it

PSW eIHTestCon Framework

PSW eIHTestCon stands for eInfochips Host Based Test Controller for Platform Software Verification. It is designed and developed to provide complete verification framework solution for all PSW components’ verification. The test procedures are developed in two parts, test procedure logic in C language and test configuration in an XML file. The test configuration file carries test setup information. The Host Test Controller cross compiles [3] the test procedure files. It generates an executable image with dynamically linked [4] to target image and transfers to Target. The Target Test Agent loads the received executable in predefined reserved memory and starts test execution. The Target Test Agent transfers the test result data back to Host Test Controller. And Host Test Controller generates the result file in the required format. The test procedures are developed using the test harness, stub driver, and test driver.

Pros:

• Uniform test procedure development across all PSW components i.e. Bootloader, BSP, CSP, OS, Drivers, Libraries, and Applications.
• Easy to control test execution from Host.
• Multiple test result formats support.
• Easy to integrate with Structural coverage tools.
• ~90% independent of Platform Architecture.
• Simple APIs for developing test procedure with Test Harness.
• Easy to integrate, dry run and debug Test procedure.
• GUI, web-based and command line interface support.

Conclusion:

This paper has explained various platform software verification framework designs. The eIHTestCon design overcomes challenges of other frameworks. It also supports verification for all platform software components under one solution. The paper has also covered pros and cons of individual frameworks. The following table provides additional data points from safety critical standard perspective [2].

Sr. No	Objectives	[A]	[B]	[C]	[D]	eIHTestCon
1	Test equivalence and boundary inputs	x	x	x	x	x
2	Test Invalid inputs	x	x	x	x	x
3	Time-related functions				x	x
4	State transitions during normal operations	x	x	x	x	x
5	State transitions that software requirements do not allow			x	x	x
6	Incorrect Interrupt/exception handling			x	x	x
7	Errors in hardware/software interfaces			x	x	x
8	Incorrect behaviors of feedback loops			x	x	x
9	Violation of software partitioning				x	x
10	Software integration failures			x	x	x
11	Compute out-of-range loop counts			x

[A] Config file based PSW Verification framework
[B] Host Script based PSW Verification framework
[C] Debugger based PSW Verification framework
[D] C Language based PSW Verification framework

Visit www.einfochips.com for more information

References:

[1] https://www.design-reuse.com/articles/40117/platform-software-verification-approaches-for-safety-critical-systems.html

[2] RTCA/DO178C, “Software Considerations in Airborne Systems and Equipment Certification”

[3] https://linux.die.net/man/1/gcc

[4] https://linux.die.net/man/1/ar

[5] https://www.windriver.com/products/product-overviews/vxworks-653-product-overview/

[6] http://www.lynx.com/products/real-time-operating-systems/lynxos-178-rtos-for-do-178b-software-certification/

If you wish to download a copy of this white paper, click here