How to choose a verification methodology

The tools and techniques to be used in a project have to be decided upon early in the design cycle to get the best value for these new verification methods. Companies often end up making costly mistakes by underestimating or sometimes overestimating the complexity of the design and skill set required to run these new tools and techniques.

The higher the abstraction level, the easier it is to design; by the same token, the higher the abstraction level, the easier it is to make a bigger mistake. Making an architectural flaw can end up hurting the entire chip, as opposed to a misconnection of wires at the gate level that can be fixed by a re-spin.

Verilog, for example, makes it possible to design at a fairly abstract level, but it is very easy to make mistakes if one does not know the nuances of the language. The same argument holds true with the many verification techniques and languages available today.

This article gives the reader an overview of the prevalent verification techniques (formal verification, random, directed, constrained random, assertions, property checking) and languages (SystemC, C/C++, SystemVerilog, OpenVera, e). It also examines the place for the various verification techniques and the time one should use them during a traditional digital ASIC design flow.

1.1 Bottlenecks

1.1.1 Design bottleneck
Time is very essential to a company's success. Historically, "product time" — the time it takes for a concept to become a production part — has been mainly a function of design time. Whenever a new technology or process is introduced, design time was and continues to be the primary bottleneck.

Design time is a function of silicon complexity. This gives rise to system complexity, which affects time to market, as shown in Figure 1.

Figure 1 — Technology cycle

Following an exponential increase in the number of transistors in designs, a linear increase in compute time or number of engineers was not adequate to reduce design time. To solve this problem, the EDA industry stepped in to introduce the concept of design abstraction through automation. Language-based solutions such as Verilog and VHDL were introduced.

This design catch-up game is still in place. The latest in the languages to be introduced and supported by EDA world are SystemC and SystemVerilog. For the current technology processes, design complexity is well understood and design bottleneck has been overcome to some extent, thanks to the productivity gains through the use of EDA tools.

Having solved the first round of problems, the focus now is on solving the effects of the first order problems such as the verification bottleneck.

1.1.2 Verification bottleneck
Design productivity growth continues to remain lower than complexity growth — but this time around, it is verification time, not design time, that poses the challenge. A recent statistic showed that 60-70% of the entire product cycle for a complex logic chip is dedicated to verification tasks. Verification of complex functions that we can build using new design tools poses a challenge to reducing the total product time.

Figure 2 — Design and verification gaps

The verification bottleneck is an effect of raising the design abstraction level for the following reasons:

1) Designing at a higher abstraction level allows us to build highly complex functions with ease. This increase in design complexity then results in almost doubling the verification effort. We have doubled the functional complexity and hence its verification scope.


2) Using a higher level of abstraction for design, transformation, and eventual mapping to the end product is not without information loss and misinterpretation. For instance, synthesis takes an HDL-level design and transforms it to the gate level. Verification is needed at this level to ensure that the transformation was indeed correct, and that design intent was not lost. Raising the level of abstraction also brings about the question of interpretation of the code that is used to describe the design during simulation.

1) Increase in functional complexity because of the heterogeneous nature of designs today; for example, co-existence of hardware and software, analog and digital.


2) The requirement for higher system reliability forces verification tasks to ensure that a chip level function will perform satisfactorily in a system environment, especially when a chip level defect has a multiplicative effect.

• Chip flaws because of design errors. 82% of designs with re-spins resulting from logic and functional flows had design errors. This means that corner cases were not covered during the verification process and that bugs remained hidden in the design all the way through tapeout.
• Chip flaws because of specification errors. 47% of designs with re-spins resulting from logic and functional flaws had incorrect or incomplete specification. 32% of designs with re-spins resulting from logic and functional flaws had changes in specifications.
• Problems with reused IP and imported IP. 14% of all chips that failed had bugs in reused components or imported IP.
• Effects of a re-spin. Re-spins can cost a company up to $100,000. In addition, it delays product introduction and adds to expenses due to failing systems that use these defective chips.

• Reduce chip complexity. Practically, this is not possible because of customer demand for more functionality.
• Reduce number of designs. This solution affects a company's long-term goal of being profitable.
• Increase resources. Another alternative is to increase the number of designers or verification engineers. This alternative works well to some extent, but does not meet today's demands for verifying exponentially complex chips with a limited amount of time and money.
• Increase productivity of designers. This area is open for optimization. Productivity gains have been achieved by improving compute power, using personal tools such as Microsoft Excel. While they have been of great help in capturing test and verification plans, the majority of the time is spent in coding the test cases, running them, and debugging.
• Increase verification productivity. This has obvious potential for gains in productivity. The rest of this article concentrates on this last point.

As complexity continued to grow, new verification languages were created and introduced that could verify complex designs at various levels of abstraction. Along with new verification languages came technologies and tools that supported them.

So what does all this mean for the chip vendors? They have to evaluate new tools. Engineers have to be trained on these new tools and technologies. New tools and resources have to be included in the cost structure of R&D expenses. The company as a whole has to overcome a learning curve in a short time. Risk evaluation for these new tools needs to be performed. And integration and interoperability of new tools with existing technologies needs to be considered.

1.2 Verification versus validation

In addition to the verification problem, chip companies are grappling with validation time. This section describes how "validation" differs from "verification" and sets the stage for the subsequent section on various verification technologies.

Kropf [3] defines "validation" as the "process of gaining confidence in the specification by examining the behavior of the implementation." Recently there was discussion on the subject of "verification vs. validation" in the on-line Verification Guild. Many views were presented regarding the difference. One view was that "validation ensures it is the right design, while verification ensures that the design is right." Another view was "verification means pre-silicon testing (Verilog/VHDL simulations) while validation is post-silicon testing (testing silicon on boards in the lab)."

Whether it is validation or verification, two things need to happen to ensure that the silicon meets the specification:

• The chip specification is interpreted correctly (typically through documentation and sometimes modeling).
• The interpretation is captured and implemented correctly (typically through HDL) and synthesized into silicon and packaged as a chip.

Figure 3 — Typical design flow

Depending on the complexity of the function being implemented, some of these steps may be skipped or more steps added. For example, if we know that a certain design is purely hardware-oriented and does not involve drivers or software, one can directly jump to abstraction level 1 from abstraction level 3 (no need for hardware/software trade off). An example of this would be a PLL (Phase Locked Loop) design.

It is important to note that equivalence must always be maintained as we step down the levels of abstraction to ensure that the lowest level of abstraction meets the requirements of the system specifications. For example:

1) Equivalence between chip specification (typically a text document) and its C-model will be made when the C-model is put in a system environment and meets all the system requirements as described in the specification. Equivalence of this type is often said to be functional in nature.


2) Equivalence may be established between a C-model of a specification and its HDL implementation by comparing the outputs of the C-model (now reference or golden) and the HDL implementation for a given application. In the absence of a C-model, an "expected data model" (behavioral model that has passed the functional equivalence test as described in [1]) is used. Equivalence of this type is also considered functional.


3) The HDL (now reference or golden) implementation and the gate level description (after synthesis) are established to be equivalent by using a logic equivalence check. At this point the equivalence will be logical in nature because the design is in the form of bare logic gates, and functions can now be expressed as logical expressions.

Figure 4 — Verification methodologies

1.3.1 Dynamic functional verification
The most widespread method of functional verification is dynamic in nature. The reason it is called "dynamic" is because input patterns/stimulus are generated and applied over a number of clock cycles to the design, and the corresponding result is collected and compared against a reference/golden model for conformance with the specification.

A simulator is used to compute all the values of all signals and compare the specified expected values with the calculated ones. Currently, industry has provided the choice of two types of simulators:

• Cycle based simulator. The simulator has no notion of what happens within a clock, and it evaluates signals in a single shot once per cycle. This type of simulator is typically faster because of the low execution time.
• Event based simulator. These simulators take events (within a clock or at the clock boundary) and propagate them through the design until a steady state is achieved.

When the logic gets more complex, the verification space increases. This brings about random dynamic simulation, which provides random stimulus to the design in an effort to maximize the functional space that can be covered. The problem with random testing is that for very large and complex designs, it can be an unbounded problem.

To solve this problem, the EDA industry introduced higher-level verification languages such as Open Vera, e, and SVL (SystemC Verification Library). These introduced concepts such as constrained-random stimulus, random stimulus distribution and reactive test benches.

In addition to the introduction of randomization features, new verification languages and tools increased productivity by decreasing the amount of time companies spent on building various test case scenarios for stimulus generation. For example, the test scenarios can be written at the highest level of abstraction and can be "extended" to any lower level of abstraction by using powerful object-oriented constructs.

When using dynamic verification, companies typically want an estimate of the functional space covered and captured in quantifiable terms. These include:

• Number lines of code that were verified (line coverage)
• How many of the logical expressions were tested (expression coverage)
• How many states in a FSM design were reached (FSM coverage)
• The number of ports and registers that were toggled both ways during a simulation run (toggle coverage)
• Number of logical paths in the design code that were covered (path coverage)

Assertions
Designers use assertions as placeholders to describe assumptions and behavior (including temporal) associated with a design. Assertions get triggered during a dynamic simulation if the design meets or fails the specification or assumption. Assertions can also be used in a formal/static functional verification environment.

1.3.2 Hybrid functional verification
Typically in this method, dynamic simulation is performed and the results are captured and used as inputs for static verification. During static verification, logic equations/symbols are propagated through the design as opposed to values (as in dynamic simulation). This technique is less exhaustive than formal verification, but may prove more effective than pure dynamic simulation, as it starts off where dynamic simulation left off.

1.3.3 Static functional verification
In static functional verification there is no input stimulus applied to the design. Instead, the design is mapped on a graph structure that describes its function using BDDs (Binary Decision Diagrams) or other mathematical representations that specify the design function over all time periods. Proving or disproving the properties with it then verifies these mathematical representations. This is done by determining contradictions in the mathematical structure by passing values to and against the signal flow.

Current tools target the static verification market in two ways:

• Using assertions. These are design constraints that are specified and formulated in the model/design itself (using design/verification languages such SystemVerilog or Open Vera/Verilog/VHDL).
• Using Properties.These allow specification of the properties using a property language (such as PSL, Sugar).

Some practical reasons for the design representations to be different are as follows:

• Synthesis algorithms/heuristics. Depending on the constraints on the synthesis tools (area, time, power), the synthesis tools will optimize the logic to derive appropriate gate level representation. In order to do so, the synthesis tools will use heuristics and logic minimization algorithms.
• Abstraction level. Sometimes the design may be implemented using HDL, which could be different from the designer's intent due to language limitations, or because of lack or inability to predict how synthesis tools interpret and transform a particular language construct into a gate level representation.

This section describes the trends and forces that are shaping the world of verification.

Technology perspective
Design errors could be located at the interfaces of modules or inside the modules that eventually get integrated to form chips. Using dynamic simulation, bugs not found at the module level, for instance, can show up during the subsystem level or system level. Fixing a module level bug during integration could possibly introduce new bugs at that level, causing the turn-around time to increase further.

Using static functional verification, each step of module development is exhaustively verified, ensuring radically better subsystem/system quality and reliability. It has been claimed that using static/formal functional verification, we can find more bugs faster [3].

On the flip side, the drawbacks of using static functional verification are:

• Current tools can be run at module level only.
• It typically cannot handle large designs (typically handles 100K-150K gates).
• It cannot handle highly complex designs.
• The current formal tools sometimes have a problem with being time unbound (this is mainly because they tend to be based on a set of heuristics and not on any single algorithm). This means that formal functional verification could, in some cases, show longer times for verification than dynamic simulation.
• The design has to be written in synthesizable RTL.

Static verification has done the same thing to functional simulation that static timing analysis (STA) did to dynamic timing analysis (gate level simulation). But, gate level simulation is not dead (see article, Similarly, dynamic simulation will continue to dominate the functional verification space until formal verification tools provide a method to converge on results and, in general, mature more.

Language perspective
A new trend points towards combining design and verification languages. This will bring about a tremendous boost in productivity, improve system reliability, and enhance design quality because the ambiguity and misunderstanding involved with the use of multiple tools and languages is now eliminated.

Until the time this trend is proven on real world designs, companies continue to rely on current high-level design languages (Verilog/VHDL) and use either proprietary verification languages (Open Vera, e) or good old-fashioned Verilog/VHDL.

SystemC and other high-level design description languages play an important role in design flow that involve hardware/software tradeoffs and designs that have software running on hardware, as in SoCs (Systems-on-Chip). SystemC and other high-level design languages continue to play an important role in architectural modeling and validation. SystemC is also used where the architectural model components can be used for verification using transactional models.

1.4 Criterion for choosing the right verification methodology

Engineers are grappling with extreme design complexities in an environment of decreasing time to market and tighter cost constraints. In these types of environments, it seems that filling in the holes in existing methodologies will be sufficient, and that spending time on new technologies can be postponed if the value proposition is high. It has been shown that companies that spend a higher percentage of R&D on new technologies make more efficient use of their R&D spending, enjoy faster time-to-market, grow faster and are more profitable.

Having said the above, it needs to be reiterated that companies have to evaluate the methodologies and technologies based on their individual needs and their core values. Before introducing new technologies into their tool flow, they should ask themselves the following questions and make appropriate tradeoffs.

• Are we a market leader or follower in a certain product line?
  
• Are the CAD infrastructure, tools, and methodology centralized or distributed?
  
• Is the evaluation of new verification technology for a first generation product?
  
• What is the direct and indirect effect of new verification technologies on cost and time-to-market?
  
• Can the new tool handle varying design capacities?
  
• How easy is the tool to use?
  
• What is the level of documentation and support available for the tool?
  
• Will the new tools and methodology interoperate with existing in-house tools and methodologies?
  
• What will be the ROI (Return on Investment) for new tools (including service, training, consulting, compute, and manpower resources)?
  
• Can it support multiple geographical design locations?
  
• Do the designers and verification engineers we employ have a mindset for new technology?
  

[1] SNUG (Synopsys Users Group) paper: "The Next Level of Abstraction: Evolution in the Life of an ASIC Design Engineer"
[2] Roloff, M./Chan, A./Cote, C./Srinivasan R.: Growth through Product Development New Research Insights. PRTM Insight, 2000.
[3] Kropf, T.: Formal hardware verification: Methods and systems in comparison. Springer, 1997

Rangarajan (Sri) Purisai is a senior logic design engineer at Cypress Semiconductor's Network Processing Solutions Group. Currently, he is working on the architecture and design of high performance network search engines and co-processors.