Challenges in verifying PCI Express in complex SoCs

By Siddharth Garg, Umesh Pratap Singh (Freescale)

PCI-Express (PCIe) is the backbone of today’s complex systems requiring high speed data communication with high throughput. It is being used extensively in different applications like computer cards, graphic cards, automotive, networking, industrial and consumer applications. In automotive applications, PCIe is useful for communication of data coming at a very high speed from real-time graphics and video processing. It is very useful in the Advanced driver assistance systems (ADAS).

With a focus shift towards high speed serial interface in auto electronics contents, in this paper, we will be discussing how to verify PCIe in the SoCs. Functional verification is just a part of the complete verification methodology required for verifying high speed interfaces like PCIe. In this paper, we will be covering the areas which can be covered using functional verification. Here, we will be focusing on the various scenarios which can be potential barriers in the designing of a high quality system.

Introduction to PCIe:

PCI Express is a third generation high performance I/O bus used to interconnect peripheral devices in applications such as computing and communication platforms. PCI Express is an all encompassing I/O device interconnect bus that has applications in the mobile, desktop, workstation, server, embedded computing and communication platforms.

PCIe is used to provide the connections between motherboard peripherals like graphics card, ethernet card to the CPU and main memory. It also allows add-on peripheral devices such as external graphics card, external hard disk to be seamlessly connected to the motherboard devices in a plug-and-play manner.

PCIe is a packet-based serial bus, provides a high-speed, high-performance, point-to-point, dual simplex, differential signaling Link for interconnecting devices. Data is transmitted from a device on one set of signals, and received on another set of signals.

PCIe link and lane: A PCI Express Link is the physical connection between two devices. A Lane consists of signal pairs in each direction. A x1 Link consists of 1 Lane or 1 differential signal pair in each direction for a total of 4 signals. A x32 Link consists of 32 Lanes or 32 signal pairs for each direction for a total of 128 signals.

PCIe clocking and speed: No clock signal exists on the Link. Each packet to be transmitted over the Link consists of bytes of information. Each byte is encoded into a 10-bit symbol. All symbols are guaranteed to have one-zero transitions. The receiver uses a PLL to recover a clock from the 0-to-1 and 1-to-0 transitions of the incoming bit stream. PCIe Gen1 supports 2.5 GTs, PCIe Gen2 supports 5GTs and PCIe Gen3 supports 8GTs. PCIe Gen1 and Gen2 use 8/10 bit encoding while PCIe Gen3 uses 128/130 bit encoding scheme.

The maximum supported speed and link width depends on the application and use case of the SoC in which it is being used.

PCIE TOPOLOGY:

A Root Complex (RC) denotes the root of an I/O hierarchy that connects the CPU/memory subsystem to the I/O.

Endpoint (EP) refers to a type of Function that can be the Requester or Completer of a PCI Express transaction either on its own behalf or on behalf of a distinct non-PCI Express device (other than a PCI device or Host CPU), e.g., a PCI Express attached graphics controller or a PCI Express-USB host controller.

A Switch is defined as a logical assembly of multiple virtual PCI-to-PCI Bridge devices. A PCI Express to PCI/PCI-X Bridge provides a connection between a PCI Express fabric and a PCI/PCI-X hierarchy.

The following figure illustrates a single fabric instance referred to as a hierarchy – composed of a Root Complex (RC), multiple Endpoints (I/O devices), a Switch, and a PCI Express to PCI/PCI-X Bridge, all interconnected via PCI Express Links.

PCIE has three layered architecture for communication between two devices. The layers are given below:

Transaction Layer Services: The Transaction Layer tracks flow control credits for TLPs across the Link, generates TLPs from device core requests, converts received Request TLPs into Requests for the device core.

Data Link Layer Services: The Data Link Layer is responsible for reliably exchanging information with its counterpart on the opposite side of the Link.

Physical Layer Services: The Physical Layer is responsible for interface initialization, maintenance control, and status (Reset/Hot-Plug) tracking.

PCIE TRANSACTIONS AND ADDRESSING FORMAT:

PCIe supports posted (P), non-posted (NP) and completion transactions (Cpl). Posted transactions are those for which completion is not required such as MWr and MSG. Non posted transactions are those for which completion is returned such as all reads (MRd/IORd/IOWr).

Transaction Types	Description	Address format supported	Completion returned
Memory – Read, Write and atomic requests	Transfer data to/from a memory mapped location	32 bit and 64 bit addressing	Only for read and atomic requests
IO – Read and Write	Transfer data to/from an I/O-mapped location	Only 32 bit addressing	For both IO Rd and IO Wr
Config - Read and Write	Used to access configuration registers of Functions within devices	ID routing	For both Cfg Rd and Cfg Wr
Message	From event signaling mechanism to general purpose messaging	Implicit routing	No completion

PCIE DATA FLOW:

When there is data flow, first of all Physical Layer becomes active, then indication goes to Data Link Layer and Data Link Layer becomes active and then Transaction Layer becomes active.

In a data packet, as shown below at TX side, the fields are added by the each layer and at RX side, respective fields are removed by each layer. The layers and the data flow through the layers are described in the diagram below:

^{Click to enlarge}

PCIe in Automotive SoCs:

For over a century automobiles have been evolving as manufacturers constantly find ways to improve vehicle handling, efficiency, safety and performance. One of the recent advancement in automotive industry is Advanced Driver Assistant Systems (ADAS), which are developed for better safety of vehicle and driver as well as better driving. It includes providing features for collision avoidance by either alerting the driver or taking control of the vehicle. These features include automate lighting, adaptive cruise control, automate braking, GPS warnings, alerting driver to dangers, correct lane driving and analyzing blind spots. These systems are generally based upon camera/vision systems, sensor technology etc. In vision systems, we require real time processing of image and video data coming from the surroundings. As the data received is at a very high speed and in huge amounts, we require a fast interface to handle such processing. PCI-Express, with its high speed and throughput, provides a reliable solution to this requirement of handling real time data coming at high speed. PCI-Express communicates at GT/s speeds (varies for different PCIe generations). A typical use case of PCI-Express in vehicles is for connecting rear seat LCD subsystems.

Areas to cover in PCIe SoC verification:

PCIe is a complex IP supporting a huge number of features. As per general trend in the VLSI industry, an IP is verified before it is integrated in the SoCs. So, we need to list down the features which must be verified on SoC level. Verifying each and every internal feature of the IP at SoC level is not recommended for obvious reasons. Now the question arises, what all we need to verify as per SoC’s perspective?

PCIe, being a high speed interconnect, has its own share of problems from functional as well as electrical point of view. We can’t sign off the SoC which has PCI-Express present in it without completely verifying the functional features as well as electrical features of the IP. For high speed interfaces, integrity of physical layer needs to validated and it needs to be taken care that reflections, cross-talk, emissions, and other effects are within allowable limits. They should not degrade the signal level to the extent of making it unrecognizable by the link partner. Details of electrical verification is beyond the scope of this paper so for now, we will be focusing on the functional verification only.

Even from functional verification engineer’s perspective, being a high speed interface, we need to run RTL as well as Gate Level Simulations. RTL simulations help in catching functional issues while Gate Level Simulations help in finding out timing issues. So we need to run the tests in both kinds of simulations. Following list covers the features which we need to test for complete coverage on SoC level from functional point of view.

1. Functionality between application layer (system core/processor and external registers) and the PCIe IP: Generally, application layer is implemented in the SoC, external to the PCIe IP provided by the IP team. We need to verify the features which are controlled by the application layer registers and glue logics. These could include changing the device type (Root Complex to End Point and vice versa); some features of PHY like de-emphasis, voltage swing; some status bits coming from PCIe like link status, LTSSM current state etc.
2. Functionality between system crossbar (interconnect) and the PCIe IP: Support for AXI or AHB bridge is normally given in PCIe IP, so that it can interact with the corresponding system interface. We need to verify both the modes here, PCIe as slave and different masters communicating with it via crossbar; and PCIe as master and communicating with different slaves via crossbar. It is required to perform all kinds of read and write accesses (8-bit, 16-bit, 32-bit, 64-bit, etc.). Different masters can be cores, DMA etc. Different slaves can be DDR RAM, Flash Memory, System RAM, Graphics RAM etc.
3. Interrupt functionality: PCIe IP can support multiple types of interrupts going to interrupt controller. We have error interrupts for uncorrectable, fatal error messages. We have interrupts for INTx messages or MSI packets received as Root Complex. We also have timeouts like completion timeout sent as interrupt to the interrupt controller. We need to check the interrupt generation scenarios and their connectivity to the interrupt controller. We can also have interrupts for different conditions like link up, link down, power down etc.

How to verify PCIe in complex SoCs:

We can verify the above listed functionalities in the SoC environment in two stages:

1. Loopback mode.
2. Using a Verification IP/BFM.

LOOPBACK MODE: In this mode, we connect the transmit lanes of PCIe, which are output from SoC, to the receive lanes of PCIe, which are input to the SoC. This way whatever data we send from our SoC is received back by the SoC. In this mode, we can check only certain features as the PCIe can be either RC or EP at a time. So the features which require both of them can’t be verified in loopback mode. Usually, the features which can be verified using this mode include memory connectivity, system crossbar connectivity and some of the application layer connectivity. The below diagram shows the connection for loopback mode and the path accessed while doing a memory transaction.

The path accessed in above transaction is as follows:

SYSTEM_CORE → CROSSBAR → PCIE → LOOPBACK → PCIE → CROSSBAR → MEMORY

Core finally reads the memory contents to check that the loopback data sent through PCIe Tx is same as received through PCIe Rx.

USING A VERIFICATION IP/BFM: In this mode, we use verification IP which can interact with the SoC and send and receive transactions on the PCIe interface. As we can see in the diagram below, we use a BFM which acts as a link partner for the PCIe integrated in the SoC. The BFM needs to have a driver as well as a monitor. The purpose of the driver is to drive the transactions onto the PCIe interconnect. The purpose of the monitor is to collect the data coming on the receive interface of the BFM and generate data related to the packet received. Using this, we can verify the functionalities which need a remotelink partner. Some of these features include legacy interrupts like INTx, MSI, L1/L2 entry, hot reset etc. The below diagram shows the connection with BFM and the path accessed while doing a transaction.

The path accessed in above transaction is as follows:

SYSTEM_CORE → CROSSBAR → PCIE → BFM (Transaction path)

BFM → PCIE (Response path)

In case the packet type is an interrupt like INTx, PCIE sends an interrupt to the system core and system core services the interrupt.

Conclusion:

With PCIe being a critical part of today’s complex SoCs for high speed and reliable communication, to save resources as well as ensure a quality design, we need to list down certain features for PCIe to be verified at SoC level. We can verify the features like basic master slave connectivity using the loopback mode and the features requiring the involvement of link partner using Verification IP/BFM.