Using Transactions to Effectively Debug Large SoC Designs

Rich Edelman, Alain Gonier (Mentor Graphics)

Abstract :

SoC designs are large designs made by combining other large designs. A typical SoC has many communication pathways and a large amount of parallel activity. In order to debug these kinds of designs effectively new approaches must be taken. This paper discusses the requirements for effective debug in the face of today's large SoCs, outlining real world example and making some recommendations for easier solutions. This includes transaction based debug, with special attribute recording. These special attributes help identify "connected" transactions.

Figure 1 - Large SoC - ARM CCN-504

PROBLEM

When a large SoC is being tested, the communication on the SoC can be modeled using transactions. Communication between two points can be modeled a transaction. The collection of related transactions that make up a "complete transaction" or data transfer across the entire SoC is a transaction. In a simulation there may be hundreds of thousands or millions of transactions. This is a large dataset to use for debug. It is hard to find the transactions that are deemed "interesting". Simply putting the transactions in the waveform window is not practical.

Figure 2 - Simplified view of an SoC

SOLUTION

The Stripeviewer is a concept that has been around for many years [1]. Stripe Charts go all the way back to the mid 1850's when seismographs (stripe charts) recorded ground shaking over time – ground shaking transactions.

Using a stripechart or stripeviewer today allows the user to think abstractly about transactions "of interest". The "uninteresting" transactions can be ignored for the current analysis.

What is needed is a "database" of these transactions, and a way to query the database for just the transactions that are interesting. For example, in a database of a million transactions, you may only be interested in the transactions that originate in MASTER1, using the ADDRESS=0x01234567, and arrive at SLAVE2 during some time period. Something like a database search or a regular expression matching algorithm immediately comes to mind. This search or find needs to be flexible and powerful, and able to collect transactions together into a set or list. This set or list can then be processed or analyzed in many ways. This is the stripeviewer.

The stripeviewer is used to compose these queries, and then to display the results. Furthermore, these results can be "sent" to other viewing and debug windows. For example, given the query above, 10 transactions may be displayed in the stripeviewer. Those 10 transactions are the "result" of the query. Now we can send this result to the wave window, and color those transactions RED. We could issue a new query, and send the new results to the wave window colored YELLOW. Now we can go to the wave window and see the RED and YELLOW transactions in context with the other transactions. We've identified the most interesting transactions, but we can see them in context with the remaining transactions.

Before transactions can be used for debug they must be "recorded". This recording could be a monitor monitoring bus transactions. It could be any code that has knowledge of when a transaction occurs. The recorded transaction can be recorded into a proprietary database, an SQL database or a simple ASCII log file.

Figure 3 - Monitoring and Recording Communication

EFFECTIVE DEBUG

Effective debug consists of many phases, but the basic goal is to provide visibility where there is no visibility. Adding logic probes to a wired board adds visibility. Logging RTL signals to a waveform database adds visibility. Monitoring RTL signals, and converting them to transactions adds visibility.

Using higher levels of abstractions in debug allows for higher level thought processes. Using transactions is one way to do that. Transactions can be simple like a READ_ADDR_CHANNEL; more complex like READ, or even abstract like SEND_PACKET.

Transactions are a useful medium for sending information from one place to another. [7][8]

DATABASE FOR TRANSACTION DEBUG

The transaction recording can be performed in any manner feasible, including proprietary database, ASCII file, transcript or SQL database. [8]

The most important parts of the transaction database are recording useful information, and having fast access to that information.

Each transaction has to be identified with an ID or other symbol that makes it unique. After that, each transaction needs information that is specific to the particular protocol or application. A transaction has a beginning and an end. A transaction has attributes (which are [name, value] pairs). A transaction has relationships to other transactions. Using this simple data model a transaction database can be created which will provide visibility into cause and effect, and also allow higher level reasoning about behavior and performance.

The transaction model is quite simple. [10][5]. It consists of streams, transactions attributes and relations. A stream has transactions on it. A transaction has a begin time and end time, and has attributes. An attribute is a [name, value] pair. A relation is a named link from one transaction to another.

Figure 4 - Transaction Model

Attribute names are strings. Attribute values may be any legal datatype. In SystemVerilog this might be strings, bits, bytes, integers, reals, enumerations, class handles. Furthermore, an attribute value can be an array of any of these. An attribute value can be a struct or a union or a user-defined type. Getting attribute values correct is important to debug, since attributes are the pieces of information moving from point to point.

Relations are the key to grouping transactions. A relation could be an explicit relation (tr1 is a parent of tr34) or an implicit relation (address == 0x034 in any transaction).

Recording transactions can happen in many ways. Whether recording happens as part of the UVM, or as part of a VIP or as part of a customized monitor the recorded transaction should will contain all the useful information about the transaction. These bits of useful information will vary with different protocols (transaction types). For example in an ACE transaction a READ channel, a WRITE channel and a RESPONSE channel might be recorded. For a USB transaction, a completely different set of channels are recorded.

When transactions are monitored and recorded an opportunity exists to link "related" transactions to each other. This linking is quite difficult in general, but can be managed with smart monitors. This paper suggests that fabric and protocol designers incorporate the idea of a transaction id or tag that is associated with a transaction, and follows it and any descendants for the lifetime of the "complete transaction".

In an AMBA® ACE™ [4] transaction, a recorded transaction might include the following attributes and values:

Figure 4 - Subset of ACE Attributes

There are many ways to record a transaction. For example a simple $display statement in SystemVerilog:

$display(“trans=%p”, transaction_handle);

In the SystemVerilog [2] UVM [3] there is a transaction recording API which can either generate an ASCII logfile, or can generate a vendor specific database. In either case the recording API works the same way. This UVM transaction recording API is built-in to the sequence start and end. It can also be used with custom drivers and monitors. The UVM transaction recording API will not faithfully record the proper attribute types for all attributes. This is problem is easily solved using the `uvm_record_field macro.

TYPICAL WAVEFORM DEBUG

In a typical debug environment, a waveform with RTL signals is used to figure out what is going on, and what the cause for misbehavior was. Transactions can easily be added to this typical environment, but transactions suffer some problems. They consume vertical real estate in the wave window. They also consume horizontal real estate. And there are many of them. [Figure 6] Although transactions raise the abstraction level, in a large simulation there are going to be many transactions, so we still have the same problem as debugging using RTL signals – there is too much information.

Figure 5 - A Sea of Transactions

DEBUG ANALYSIS

In order to effectively debug using transactions, the number of transactions that are being considered needs to be made smaller. In order to reduce the number of transactions under consideration searches, sorting and filtering can be applied.

For example a search could be created to find the transaction with a specific ID.

stripeviewer –with ID==12

Or a search could be created for all transactions within a time window, and then the results could be sorted by size of payload.

stripeviewer 10 1000 \
–sortby PAYLOAD

Or a refinement of the previous search to filter out any traffic from MASTER2 could be created.

stripeviewer 10 1000 \
-sortby PAYLOAD \
–exclude MASTER==2

These are simple examples, but the idea is to search for transactions that are interesting, then sort them or filter them until the result makes sense and becomes usable.

Each search/sort/filter is a useful expression which can be named and saved for later reuse.

PROBLEMS

Using transactions for debug presents its own problems including the previously mentioned vertical and horizontal real estate and quantity.

A more significant problem is recording transactions in a way that can identify them. For example, all transactions should be recorded with an ID.

Relations are another potential problem area. Recording transaction relationships sometimes requires monitoring which is quite sophisticated. The monitors have corner cases which are hard to exercise and hard to debug.

Real estate problems can be solved easily enough by limited what gets recorded or filtering what gets displayed.

The ID problem is hard to solve. Some protocols specify an ID solution, but as blocks of IP are composed into larger blocks sometimes IDs are not kept consistent as transactions traverse the hardware. Single transactions can be split into sub-transactions or multiple transactions can be coalesced into a single transaction. How IDs are handled in these cases is not always well defined.

The relation problem is equally hard to solve directly. It can be solved indirectly quite easily. One solution is to “relate” transactions together when they operate together. For example, all transactions that read or write to the same address are related. Sorting by address, and then by time may identify transactions which are related to each other. A second solution is to add symbolic names or tags to transactions. These tags specify that the tagged transaction is a member of a set. The tag is an attribute of the transaction and may be easier to record than a true relation.

Despite these limitations transaction based debug can be quite powerful and is sometimes the only way to debug complex problems in a large SOC with long running simulations.

PRACTICAL DEBUG

A stripeviewer is really a way to organize the transactions of interest during a debug session. Transactions get searched, sorted and filtered according to regular expressions, membership in a list, non-membership in a list, comparison operators, range specifications, bit selects, logical operations, bit masks and shifts. Any expression comparing transaction attributes should be supported.

Transaction attributes include the protocol or application specific attributes (i.e. READ_OR_WRITE, ADDR, DATA, etc), but also “internal attributes” like START_TIME, END_TIME and STREAM_NAME.

For the ACE transaction above, an expression to identify all the transactions between 1000 and 2000 with address 42 could look like:

stripeviewer 1000 2000 \
–with “ADDR==42” /streams*

In the SoC pictured below, to identify the transactions with address 42 between 1000 and 2000 that appeared on MASTER1 and SLAVE2:

stripeviewer 1000 2000 \
–with “ADDR==42” \
/MASTER1 /SLAVE2

These queries result in a list of transactions. This results list can be used as-is or can be applied to other window. For example the results of a query can be clicked on to either display the transaction in the wave window, or to produce the source code line.

Applying the list to other windows is as simple as having the other windows display or highlight just those transactions. For example, a list of transactions could be displayed in a “watch window” for future monitoring.

Another usage of the stripeviewer search and filter is to create lists of transactions satisfying the search and filter criteria, and then apply math or other algorithms to them. The amount of data transferred can be calculated from the transaction, summed and reported as bandwidth (bytes/second).

With the proper filter, transactions originating on one master or another can be identified. Then calculations or debug display can be performed with the “list of transactions originating on MASTER 1, meeting the remaining search expressions and filters”.

Expressions can be written to find “quiet times” on a channel. If the time between the previous transaction end and the next transaction start is greater than 100 clocks then include the two transactions in the search list.

APPLICATION SPECIFIC DEBUG

So far the transactions discussed have been protocol transactions or simple transactions made up of multiple protocol transactions. In reality any communication or action can be modeled as a transaction. For example, a cache state change is a transaction.

Writing application specific debug searches for cache state changes is easy:

stripeviewer 1000 2000 \
–with state==UD \
-with master==M1 \
/cache_transactions

This expression finds all the transactions between 1000 and 2000 on the “cache_transactions” stream that have the attribute master set to “M1” and the state set to “UD”.

Another application specific search might be to find all the SNOOP transactions on MASTER2 with ID in the range 100 to 200:

stripeviewer 1000 2000 \
-with master==M2 \
-with {ID inside [100:200]} \
/top/snoop*

SUMMARY

Using transactions monitors and a database, transactions can be saved for later analysis.

Figure 6 - Instrumented Fabric

Once the instrumentation is available a data is created, and a search/filter/sort is created to move from Figure 4 - A Sea of Transactions to Figure 8 - Transaction Debug Details.

Figure 7 - Transaction Debug Details

And finally focus in to the exact time when two transactions interfere with each other:

Figure 8 - Potential Transaction Conflict

CONCLUSION

Using a stripeviewer with searching, filtering and sorting provides effective debug for large SoC designs. Using transactions is a key – higher level abstractions. But without additional help transaction based debug of SoC designs will be limited.

There are two places which need to be addressed. The first is a transaction ID. Some identifier needs to be attached to transactions that follows them through as they traverse fabrics and interconnects. The second, related, thing that needs to be addressed is an efficient way to specify relations. Currently relations are an afterthought, and are quite hard to recreate. They should become a natural part of transaction generation and usage.

REFERENCES

[1] Stripe Charts. http://en.wikipedia.org/wiki/Chart_recorder

[2] SystemVerilog LRM. IEEE Std 1800-2012.

[3] SystemVerilog UVM, www.accellera.org

[4] AMBA ® AXI™ and ACE™ Protocol Specification, Version E. February 22, 2013.

[5] Edelman, Rich and Gonier, Alain, "Improving SystemVerilog UVM Transaction Recording and Modeling”, IPSOC 2011.

[6] Edelman, Rich, “Transaction Analysis and Debug Across Language Boundaries and Between Abstraction Levels”, IPSOC 2009.

[7] Edelman, Rich, “Transactions in an OVM SystemVerilog Verification Environment”, IPSOC 2008.

[8] Edelman, Rich, “Transaction Recording, Modeling and Extensions for SystemVerilog”, IPSOC 2006.

[9] Edelman, Rich, “A SystemVerilog DPI Framework for Reusable Transaction Level Testing, Debug and Analysis of SOC Designs”, IPSOC 2005.

[10] Edelman, Rich, “Debugging SOC Designs with Transactions”, IPSOC 2004.