IP reuse requires a verification strategy

One of the first things a design team must address is evaluating if the design IP meets requirements as provided. In most cases the IP will require some level of reconfiguration or redesign.

Another challenge that must be dealt with early on is determining if the IP meets the physical requirements of the process in which it will be implemented. You must answer questions like, "Does ASIC/FPGA Vendor X have a DLL/PLL that meets the IP's functional requirement?" or "Does ASIC/FPGA Vendor X have the correct I/O Cell, Hard Macro, or PHY that this IP may require?" Another challenge is determining if the IP can be implemented within the speed, area, and power limitations of the targeted silicon technology.

After conquering the most obvious obstacles, design teams may start to feel confident that the hard part is over. Unfortunately, there are several other large and complex tasks that must be addressed to ensure successful IP integration and reuse.

The most critical of these is the functional verification of the design IP, which is usually the most underestimated task. "Why do I need to test it?" and "The IP vendor already tested it, so I don't need to test it!" are phrases often heard when this subject is first broached. Unfortunately for people striving to reuse design IP, it's not as simple as that. In reality, just as with the previous obstacles, solid engineering work must be done to understand this aspect of IP reuse and to ensure success.

Choosing a verification strategy

The process of determining the verification strategy for a piece of design IP involves a number of important steps. One of the first is to understand the IP provider's verification process and review their verification plan and efforts to date. The particular verification methodology employed by IP vendor can greatly affect the quality and reusability of the IP.

A common mistake made by design teams is to read too much into the fact that Vendor X's IP has been built in silicon by customer Y. This is certainly valuable information that can allow customers to gain confidence that IP implementation in a specific technology is viable and achievable, but it does little to address the question of whether the IP meets your current needs from a functional perspective.

In order to effectively evaluate the IP provider's verification strategy, the IP consumer must start by creating their own verification plan specific to the target application. Users need to sit down and come up with a list of important features and functions that are critical for their implementation.

It's also useful at this point to define performance metrics and goals for these metrics so that the end user can validate not only that the function of the core in their system is correct, but also that system can meet its required performance goals.

In creating any verification plan, it's very important to prioritize test plan items. Once that effort has been initially completed it becomes much more straightforward to review the provider's verification plan, and see how well it is aligned with the user goals for this usage of the IP.

Areas that are of critical importance to the end user that were not adequately covered from a functional and performance perspective constitute holes in the verification plan. These holes in the verification plan will need to be plugged by the end user to control risk and deliver successful silicon.

In my experience, most applications of complex IP only utilize a subset of the full features and functionality of the IP. It's quite unlikely, especially with complex IP, that the subset of functions used by different customers will overlap completely. Therefore, it is essential to evaluate how (or if) the specific features that are important to you were verified.

It is equally important to ensure that the IP provider has a viable process and infrastructure for changing the functional configurations of the IP, and verifying those configuration changes. Understanding the methodology used, how different configurations were tested, and what types of coverage were measured will greatly help the verification team to understand what, and how much, they need to test.

But the vendor said it was working!

Let me provide an example to illustrate this point. During one project, our team decided to use a DRAM controller from one of the top tier ASIC vendor IP libraries. Inquiries with the vendor indicated that they had over five working pieces of the silicon using this IP core in the technology we were targeting.

Good news, right? We plugged the IP into our system and I ran a simple directed random test algorithm against the core and it immediately started failing. The very first test case I wrote found five bugs in the controller.

How was this possible? The vendor said it was working in silicon! The answer lies in the configuration and functionality of the core. Upon further investigation with the vendor, we determined that in all previous uses of this DRAM controller it was connected to a CPU cache controller.

This cache controller only utilized 8 byte long reads or writes. In addition, these transactions were always 8 byte aligned. Our application and my random transaction generator was generating reads and writes with various lengths and alignments that completely broke this piece of IP.

Again, this is a true story from a real world design project. The vendor was a little dumbfounded when the issue was presented to them. I will not use any real names to protect the guilty, but unfortunately this is not an isolated incident.

This example shows why IP integrators need to form their own verification plan based on the criteria being described here. If you take some time to evaluate the vendor test plan and methodology against your needs and expectations, identifying issues or holes similar to this can be done up front and planned for. This will help the IP evaluators assess the IP verification quality, associated risk, and help quantify the effort it will take to reach verification complete.

In my experience, the quality and robustness of commercial IP varies widely; the wide variety of techniques used for verification result in vastly different quality levels. With only one exception in the last 10 years, myself and the teams I worked closely with found extensive and critical bugs in every single commercial IP that was being reused.

Even if the IP is functionally perfect, teams still need to address how they will verify that the IP performs correctly in their system-level environment, and how they will verify their own logic that interfaces to the IP core. Don't assume that just because you believe the core to be functionally correct that the IP will achieve the desired function and performance within the context of the target system.

The importance of verification IP

Good quality Verification Intellectual Property (VIP) can offer tremendous value to teams trying to integrate a commercial IP core into their device, regardless of whether teams are testing for compliance or just testing the integration of the core into the target system. In the recent past, commercial VIP often only consisted of a bus-functional model (BFM), and may have included a monitor to do some protocol checking. For VIP to be effective in enabling reuse of complex design IP, it needs to provide many more features and functions than just a BFM.

More modern commercial VIP offerings tend to be highly reconfigurable, and offer more robust verification functionality such as directed random stimulus generation, protocol and temporal checking, functional coverage metrics, and reusable stimulus libraries (scenarios). The concept of highly reconfigurable VIP stems from the nature of the highly configurable IP that people are trying to test. If the VIP cannot be reconfigured to match the ever-changing design IP, then it clearly will not be of much value in verifying configurable IP.

Protocol and temporal checking is also very important so that the verification engineers will be aware of any illegal behavior that may occur during testing. Modern VIP typically provides means to measure the protocol coverage as well. This is very important in making sure you exercised critical functions for your system.

Due to the complexity of today's protocols, directed stimulus is not an efficient or effective way of verifying designs any more. The time and effort required to generate stimulus by hand has become overwhelming. In its place, random constraint solvers that allow the user to generate interesting, complex, and often unforeseen stimulus — with minimal effort — are becoming widely accepted.

But having a constraint solver alone is not enough. For verification reuse to occur, VIP must provide easily or automatically configured stimulus libraries (scenarios) that are able to adapt to many different configurations and verification situations. These scenarios are more than simple test cases. They are reusable building blocks that enable teams to quickly and efficiently create the necessarily complex test cases.

In the case of interface IP blocks, the IP vendor can use the VIP to encapsulate similar libraries of scenarios or streams of interface traffic for the customer to 'replay' in the system environment. The end user can benefit from these scenarios in several ways. They will reduce the time it takes to get the first meaningful test running, but probably more importantly, allow you reach your testing complete goals with less effort and time.

Finally, functional coverage metrics give the VIP user an objective way to measure how much of the IP's functions and features have been exercised. In this case, functional coverage can be thought of as an electronic and executable version of a written test plan.

Functional coverage is an essential compliment to random stimulus generation. With directed random testing, users are not specifically targeting features, functions, or even corner cases, but rather are using constrained randomness to exercise the design. Functional coverage measures if all interesting cases and features of the design have been hit. This is essential in answering the question, "Are we done testing yet?"

Despite all these challenges, there are clearly benefits to design IP reuse. Today's IP cores have the potential to not only address complex functionality, configurability, and performance, but also provide a solution that allows for rapid and constant change of the design requirements.

Experts often cite functional verification effort at as much as 70% of the overall design effort. With such a large percentage of a project's resources devoted to verification, it only makes sense that verification reuse is as critical — if not more critical — than design reuse.

One of the conclusions we can draw is that feasible reuse of IP cores requires that vendors must invest in a solid methodology for dealing with the verification challenges associated with rapidly changing and widely varying IP configurations. Verification IP plays a key role in overcoming the verification and quality issues, which loom as one of the key barriers to IP reuse today.

Commercial IP vendors are increasingly leveraging configurable VIP solutions to increase the quality of IP core verification. Likewise, designers are rapidly investing in reusable verification environments that leverage the same VIP for IP integration and local quality assurance, and to facilitate more efficient system-level verification.

Embracing verification reuse is a primary key in bringing customers and vendors one step closer to realizing the substantial benefits of IP reuse. If verification is up to 70 percent of problem and users are primarily focusing on design IP, then they are only addressing 30 percent of the problem.

For real and substantial reuse benefits to be achieved, design teams must look at the problem of IP reuse as a whole, and include all its aspects when looking for a solution. Design, verification, timing closure, and test all play a significant part in the adoption and integration of a piece of IP. Don't make the mistake of focusing solely on the IP core when there are larger issues like verification and system performance that must also be addressed to really achieve the benefits of reuse.

Sean Smith has an extensive background in system design verification and is Chief Verification Architect at Denali Software. Formerly, he was a verification engineer at Cisco Systems Inc. in Research Triangle Park, North Carolina. Previously, he worked for IBM as a verification engineer.