When Traceability Catches What Verification Does Not

By Paul Graykowski, Arteris IP

Through a complex design process, some mistakes will inevitably slip by even the most expert system-on-chip (SoC) design teams. A disciplined V-model process (diagrammed below) will guard against many of these errors in the concept-to-design process (left arm) and in the verification-to-validation process (right arm). Within the industry, teams have developed significant tools and methodology automation for the right arm. The existing process is very good at testing what designers have been tasked to build from the design specification. There are also excellent point tools for the left arm. But methodology automation has not been as strong because system design tools and user domains span a wider range. Moving down the left arm still depends on manual translation from step to step, with the potential for misunderstanding and oversight. Overall, the V process is ineffective at testing the system requirements until the last step of the right arm - validation, though that is rather late to find basic errors.

Click to enlarge

The path traceability must track, from system requirements to validation.

The Design Specification as System Oracle

Testing depends on a reference specification that claims to offer a precise definition of one or more requirements. In testing jargon, this is known as an oracle, an ultimate arbiter of truth. Oracles may be design or use-case specifications or some other concrete definition of requirements. How precisely an oracle builder adheres to the original system objectives depends on the accuracy, clarity and definition of the input the designer receives. Understandably, a glitch anywhere in this path will propagate. Worse still, it will disseminate with all the authority of an oracle. Every step beyond that point will read that flaw as a non-negotiable requirement until a wider scope test hopefully reveals a disconnect.

It is routine in articles on verification to talk about the cost of finding a bug increasing exponentially the later it is found. The solution is always to shift left by doing more testing early – static verification, formal verification, emulation, etc. Accelerating all these tests is important. Verifiers will catch disconnects between low-level oracles and the design but cannot catch oracle problems. Issues commonly spring from two sources: ambiguity in natural language specifications and schedule pressure forcing incomplete or dropped requirements. Whichever cause applies, no amount of early verification will catch such problems.

Requirements Verification Aids Correct-by-Construction Design

W. Edwards Deming made this observation decades ago: “Inspection to improve quality is too late, ineffective, costly. Quality comes not from inspection, but from the improvement of the production process.” Or in design terms, it is better to architect the first time accurately than to depend on verification to correct the implementation. Verification is still essential to confirm the appropriate behavior. The same statement could be made for updates and bug fixes.

High-level design languages attract some interest but are still quite low-level compared to the ultimate system design and do not have a better connection with system-level requirements. A superior approach is to compare directly with the requirements, minimizing error-prone intermediate levels of interpretation. Designers can compare against requirements from the next higher level in the left arm, leveraging the low-level details of implementation and intuition about higher-level system objectives. This comparison encourages correct-by-construction design. Verifiers equally and independently benefit when building tests. Both can tap into an oracle much closer to the original requirements when implementations are created and modified.

How can the SoC team know that the oracles are reliable? Requirements elaboration down the left arm of the V is ultimately what provides input against which designers will implement the SoC and against which verifiers will build their test suites. Traceability provides links from system specifications down through low-level requirements to implementation, unit tests, and system-level and full-system validation tests. These links ensure that architects, designers and verifiers can follow each requirement to ensure it is correctly interpreted. This process provides more eyes on each oracle, design and verification artifact, and more opportunities to catch misinterpretations.

Traceability’s Untapped Value in Design, Verification and Validation

Traceability is most often associated with compliance testing for safety in automotive and other markets. But it can offer much more in the accuracy of implementation to complex requirements in any domain as a method to encourage correct-by-construction design. It also complements verification left-shift through up-shifting attention to system specifications. Requirements traceability keeps oracles front and center for designers and verifiers. Verification ensures the correctness of the design to those oracles.

Want to learn more? Check out the Harmony Trace product.