Verification Planner in QuestaSim

Kishan Kalavadiya, Bhavinkumar Rajubhai Patel (eInfochips)

Abstract

Verification of complex SoCs (System on Chip) require tracking of all low level data (i.e. Regression results, Functional and Code coverage). Usually, verification engineers do this type of tracking manually or using some automation through scripting. Manual efforts in order to get above information while verifying complex SoC may lead us towards delays in project execution. A proper verification planner can help reduce such manual efforts and make the tracking process more efficient. Mentor Graphics has introduced such Verification Planners for QuestaSim known as “Questa Testplan”. This article contains detailed steps on using this verification planner along with key features which can reduce the time in verification cycle to track the verification progress.

Keywords

Verification Planner, SoC, Coverage, Questa Testplan, Regression, Automation

Introduction

Mentors’ Verification Planner facilitates the verification process management across all relevant aspects. Without such automation, tracking of verification progress requires a lot of manual efforts in order to get information like coverage and regression status at a common place. Functional and Code Coverage numbers are key parameters in verification and without it, verification can not be closed. Regression analysis (Individual Test status) is also one important aspect to track the verification progress. Verification Planner automates the tracking of verification progress by offering the analysis and optimization facilities followed by additional features like trend analysis, back annotation of results, filtration of specific data and such other features. It provides the top level summary of all verification aspects which provides real time visibility into the project to all parties i.e. designers, verification engineers, project managers etc. Due to this visibility, it becomes easy to manage risk factors and efficiencies in project execution can be improved.

Motivation for Questa Testplan Development

Functional Verification has been described as a major challenge in SoC design with proper visibility into the verification process as a major factor contributing to this challenge. This lack of visibility impacts design quality, schedule predictability and cost.

A testplan is a document which captures the important features of a design regarding how they will be verified. Questa testplan is one such plan which can be linked directly to the coverage database and results can be annotated in that plan itself. By putting such a plan in place that captures the list of verification intents, one can organize the tracking of verification progress better. While verifying the complex SoC (System on Chip), it may be possible that all the features we have listed in Questa Testplan do not have equal priority. In that case, one can add separate user defined attribute in testplan to provide priority for individual feature. Verification engineers can focus their coverage closure efforts on the critical features that have been identified as a hole followed by important features and then by the “nice to have” features.

Basic flow for QuestaSim Verification Planner

Fig 1. Basic Flow Diagram

Questa Testplan includes feature-wise test scenarios and functionalities covered by that feature (functional coverage), assertions, code coverage and such more in XML format. After preparing such Questa Testplan, conversion of such plan in UCDB (Unified Coverage Database) format is required using xml2ucdb command. After running the regression, separate coverage databases will be available for individual test scenarios. At the time of merging of these coverage databases, Questa Testplan in UCDB format needs to be merged so that the results can be linked with the data available in Questa Testplan. One can back annotate the results in XM fformat of Questa testplan and the same results will be available in HTML report or in GUI as well. The creation of the Questa Testplan required certain steps and syntaxes, which is explained in upcoming section.

Detailed Steps for Questa Testplan Development

In many cases, the features will be verified in simulation and recorded as verified using coverage analysis. For Questa Testplan which facilitates back annotation of information, a linking between the testplan and simulations is required. For that, certain document format must be followed. The format for Questa Testplan is described below with detailed steps.

Step: 1. Install Questa add-in in Microsoft Excel

By default, Microsoft Excel does not contain Questa option in toolbar. So, the user needs to install Questa add-in which is open source and freely available. After installing this add-in, it will be available in toolbar as shown in fig 2.

Fig 2. Questa add-in in Microsoft Office Toolbar

Step: 2. Prepare the Questa Testplan

For creating the Questa Testplan, select create testplan option from Questa VM option as shown in fig 3.

Fig 3. Questa VM options and Testplan Details

Questa's Verification Planner requires four distinct pieces of information for each requirement captured. They are Section, Title, Link and Type. Link is one of the attributes used while developing the Questa Testplan, which is required to link plan information to the actual results. The additional information for each requirement includes a Description, a Weight and a Goal. While these additional fields are not required, they are used the majority of the time. Questa's Verification Planner also has the flexibility to allow user defined fields. If we look at the typically used fields in a spreadsheet format, each field is represented by a column in the spreadsheet as shown in fig 4.

Fig 4. Sample Questa Testplan Format

After preparing the Questa Testplan, it can be saved as XML or can be directly exported to XML from Questa VM options as shown in fig 3.

A User can track different information by specifying the different options in Type field. Description for options available for Type field is as shown in table 1,

Coverage Construct — in “Type” field	Description
Assertion	Assertion statement
Bin	Coverage item bin
Branch	Branch coverage scope
Condition	Condition coverage scope
CoverGroup	SystemVerilog covergroup statement
CoverPoint	SystemVerilog coverpoint statement
CoverItem	Generic name for any coverage or design object in a UCDB. This can be used to specify any objects not fitting into another category of construct.
Cross	SystemVerilog cross-coverage statement
Directive	PSL cover directives and SystemVerilog "cover" statements/properties
DU	All coverage on a given design unit
Expression	Expression coverage scope
Formal_Proff Formal_Assumption	Formal property of assertion object
FSM	State Machine coverage scope
Instance	All coverage on a given instance
Rule	Forms a link using an automatically created virtual covergroup “User Rules” — either from a set of pre-defined Rules, or one you create.
Tag	Forms a link using any coverage tag command arguments which are specified in the Link column.
Test	Link to test attribute record. This is the test name.
Toggle	Toggle coverage scope
XML	Triggers hierarchical (nested) testplan import.

Table 1. Coverage Construct Type Field options

According to the requirement, user can set several rules as per the option available in Add Rule option. Here in below example, rule is applied to get the recursive toggle and statement coverage for some particular module,

Fig 5. Add Rule option in Questa Test plan

Step: 3. XML to UCDB Conversion of Questa Testplan

Convert this XML Questa Testplan in UCDB using below command,

xml2ucdb -format Excel verification_plan.xml -ucdbfilename verification_plan.ucdb

Step: 4. Regression and Coverage Database generation

After preparing the Testplan, next step is to develop test cases to verify the features listed in plan. After running the regression of those test cases, command for merge all separate coverage databases is as shown below,

vcover merge merged_cov.ucdb <test1.ucdb> <test2.ucdb>…<testn.ucdb> verification_plan.ucdb

We can back annotate the results in existing XML format Questa Testplan using below option,

Questa VM > coverage data > annotate

Step: 5. HTMP Report generation

Command for the HTML report generation from merged coverage database is as shown below,

vcover report -html -htmldir covreport merged_cov.ucdb -details -testhitdata

Verification Results Analysis

Questa’s Verification Planner facilitates with multiple formats for analysis of the results hierarchically so that it will become easy to address the failures identified in regression and track the coverage holes to complete the verification process. Find below sample reports for analysis in different format,

Click to enlarge

Fig 6. Verification Management Tracker in GUI

Click to enlarge

Fig 7. Questa Testplan in HTML

Click to enlarge

Fig 8. Annotated Questa Testplan in XML

Fig.6 and Fig.7 contains the Top level summary of Questa Testplan in GUI and HTML format respectively. Annotated Questa Testplan in XML format in fig.8 contains the same information which is shown in Fig.6 and Fig.7. Fig.8 shows the syntax for different Link options as per the requirements.

Asterisk (*) is permissible in case of the same naming convention. Here in above annotated Questa Testplan, section 2.5 containing 4 random test cases, out of which, 3 are passing and 1 is failing. To link these test cases with merged database, ‘ * ’ is being used in Link attribute. Here, pass/fail status of these test cases is in terms of coverage itself (i.e. Getting 75% coverage for 4 objects means 3 test cases are passing out of 4 random cases). Similarly, we can get the results in cases of multiple iteration of same test case.

Filtration is also one of the key features of Verification planner using which user can filter out user specific results from the whole report. User can use Attribute Name, Coverage Numbers, Weight, Link Type, Instance Type and such more for filter the required data. The user needs to create individual filter according to the requirements using below options in GUI,

Select Tools > Filter > Setup > Create

User can apply the filter using below option,

Select Tools > Filter > Apply > Name_of_Filter

Example for the filtration feature is shown in fig 9,

Click to enlarge

Fig 9. Filtration in Verification Planner

Trend Analysis

Trend analysis offers the ability to track the progress of the coverage over time of all the objects. For trend analysis, user needs to manage one trend UCDB after each regression. Trend UCDB is nothing but a special purpose UCDB file which contains the coverage data over the period for trend analysis.

Trend report offers 2-dimensional representation of coverage number over the period for individual feature or instance of DUT. Trend report can be viewed in multiple formats (i.e. HTML Report, XML Report etc.)

Trend report appears in GUI using trend UCDB at Verification Management > Browser > Trend Analysis

Command for trend UCDB generation from command line is shown as below,

vcover merge -trend [-output] <trend ucdb> <ucdb inputs>

Sample trend report and graphical representation for the same is shown in fig 10 and fig 11 respectively,

Click to enlarge

Fig 10. Date wise trend report in VM Trender (GUI)

Fig 11. 2-Dimensional Graph in Trend Analysis

Conclusions

Verification of complex SoC projects is a complex process to manage without Verification Management. Automation of Verification Planner provides deeper visibility into the regression process to allow for throughput optimization. Questa’s Verification Planner Tool enables us to analyze the regression failures and coverage holes more efficiently by providing the reports in multiple formats. Use of verification planner can reduce the manual effort which is required to update the verification documents at the time of closure and quicker the documentation in more appropriate format.

References

Questa SIM Verification Management User Manual (Software Version 10.4c)

If you wish to download a copy of this white paper, click here