Enter the Inner Sanctum of RapidIO: Part 1

Enter the Inner Sanctum of RapidIO: Part 1
By Greg Shippen, Motorola, Inc., CommsDesign.com
November 24, 2003 (9:42 a.m. EST)
URL: http://www.eetimes.com/story/OEG20031120S0007

The RapidIO specification isn't new to the communication sector. This spec has been long talked about and implemented as an interconnect for linking DSPs, microprocessors, and other components in a communication architecture. But, while not new, the RapidIO specification has been adapted to meet the control plane and data plane requirements of next-generation communication architectures, thus forcing designers to once again take a close look at the key technical elements that make up the RapidIO specification.

To help out, this two-part tutorial provides an inside look at the key technical functionality defined by the current RapidIO specification. In Part 1, we'll examine the three layers that make the specification run, as well as the serial and parallel interface specifications. In Part 2, we'll expand the discussion by detailing the bandwidth requirements and flow control me chanisms. We'll also look at the key requirements for building RapidIO switches and endpoints.

Systems Supported
Two types of devices are defined in a RapidIO system: endpoints and switches (Figure 1). Endpoints source and sink data while switches move data from one port to another (Note: More specifics on endpoints and switches will be provided in Part 2).

Figure 1: The RapidIO architecture consists of both switches and endpoints.
Individual transactions pass between devices across a unidirectional link. Each port has an outgoing transmit and incoming receive link. Transactions cross the link as discrete packets of variable size. A packet consists of a header and optional data payload. The header defines both the operation to be performed and the attributes necessary for the packet to reach its intended destination. The data payload size may be up to 256 bytes.
Both the RapidIO specification and the packet formats themselves are organized in a layered fashion similar to the OSI model. Specific bit fields are allocated to logical, transport, and physical layers as color coded in the diagram below. This layering enables extensibility and backward compatibility as the specification evolves. Let's look at each layer in more detail, starting with the logical layer.
Layer 1: Logical
The logical layer describes request and response transactions. They are used to perform actions at a remote destination endpoint in the system and are defined between a source and destination endpoint. The logical layer is ignorant of how the transaction makes its way to the destination.
Four logical layer specifications define a variety of transaction types. Read, write, and atomic transactions are provided using 34, 50 or 66 bit addressing. Reads and writes may be between 1 and 256 bytes in length.
In addition, separate maintenance transactions provide access to a 16-Mbyte command and status register space associated with each device. These requests may be between 4 and 64 bytes in length.
Messaging allows two endpoints to pass data between each other without being aware of the other's internal address space. The message facility defines hardware segmentation and reassembly (SAR) support for messages up to 4 kbytes. Up to four queues at the destination can be addressed with four messages in flight to each of the queues. A doorbell message is also provided to communicate short 16-bit messages to the destination. This can be used to generate doorbell interrupts, although other uses are not precluded.
Transactions are also described to allow multiple processors to maintain coherent caches among them, as well as end-to-end flow control for congestion management.
Layer 2: Transport
The transport layer describes how a packet finds its destination as it crosses the netw ork and uses destination-based routing. Every endpoint is assigned a device ID during initialization and discovery. Packets carry a source and destination ID. Switches use the destination ID to route a packet to the intended destination endpoint. When necessary, the destination endpoint uses the source ID to return a response back to the original source of a request packet.
Small and large transport type fields are defined. The small field supports up to 256 device IDs while the large field handles up to 65,536 device IDs.
Layer 3: Physical
The physical layer (PHY) describes how a packet is passed from one side of a point-to-point link to the other. Unlike the OSI model, the RapidIO PHY includes both the link protocol and the electrical interface characteristics. The link protocol includes extensive error protection and detection mechanisms to ensure that packets are reliably delivered to the destination.
Two PHY interface technologies are defined based on industry-standard electricals : 8 and 16-bit parallel and 1- and 4-lane serializer/deserializer (serdes)-based serial. Each PHY offers specific advantages. For example, the parallel PHY offers the highest bandwidth and lowest possible latency while the serial PHY offers significantly few pins.
Handling Packets
In RapidIO, packets are classified as either a request or response and are defined at the logical layer. As shown in Figure 2, a packet is constructed of bit fields associated with the logical, transport, and physical layers.

Figure 2: Diagram illustrating several packet formats supported by the RapidIO spec.
Packets are always a multiple of 4 bytes in length and 32-bit aligned. Bit fields are arranged to simplify logic design. For example, PHY bits appear near the beginning of the packet so the link protocol bits are available first as the packet arrives at a link receiver. Trans port bits are near the front of the header and aligned to a 16-bit boundary to allow switches easy access for routing purposes.
In general, the various layers of the packet are added to the packet header as the transaction moves from the logical layer of the endpoint down to the physical layer. In like manner, these fields are removed as the packet data is passed to higher layers of the endpoint logic.
Packets are protected by an end-to-end 16-bit cyclic redundancy check (CRC) that is applied at the originating endpoint. The CRC remains unchanged as the packets traverse the network so correct CRC at the destination ensures the integrity of the packet payload. Note that an early 16-bit CRC is also applied when packets reach 80 bytes or larger in length.
Unlike other packet types, a maintenance packet functions as both request and response. Maintenance requests always have a response to guarantee completion before subsequent operations are initiated at the requester. The only exception to this is the port write request. This special maintenance write type is used to communicate status or error information from devices that do not have an endpoint (such as a switch) and hence cannot accept responses.
Because switches do not have a device ID but do have capability, command, and status registers, maintenance packets include a hop_count field in order to target switches. To target a specific switch, an endpoint issues a maintenance packet with an endpoint destination ID whose route takes the packet through the desired switch. The value of hop_count defines which switch is targeted. Each switch examines the hop_count field and consumes the packet if the value is zero. Otherwise, it decrements the value and sends the packet on.
Control Symbols
The process of initializing and synchronizing the link, performing packet delineation, ensuring reliable delivery, and dealing with error conditions is managed through the use of a short 4-byte packet called a control symbol. Control symbols exist only at the physical layer and are never routed between ports on a switch.
Reliable packet delivery across a link is accomplished by the receiver using a control symbol sent to send the transmitter an explicit acknowledgement of each packet. Should the transmitter fail to receive a packet acknowledgement, it either resends the packet or a link error recovery procedure occurs depending on the circumstances. When errors occur, control symbols are used to send error status and coordinate link error recovery.
During the initialization process, control symbols report status and request transmission of training sequences used to synchronize operation of the link.
Some control symbols are specific to the physical interface. For example, in the parallel interface, the idle control symbol is sent when there are no packets available for transmission or the rate of packet arrival needs to be temporarily slowed.
Control symbols are also used to denote packet boundaries. In the parallel case, a control symbol is used to mark the end of a packet when no packet is available to follow it. For the serial, control symbols mark the start and end of all packets.
Because control symbols are used for link control, minimizing loop latency is important for efficient operation of the protocol. To avoid delaying control symbols while a large packet is transmitted, most control symbols can be embedded within packets.
To conserve bandwidth on the link, some kinds of control symbols can cancel transmission of a packet before the packet has been completely sent should an error be detected early.
The format of an acknowledge control symbol is shown in Figure 2 for the parallel PHY case. The "S" bit begins both packets and control symbols. When asserted, the receiver can immediately classify the incoming packet as a control symbol. Note that control symbols are actually 16-bits long but the value is repeated and inverted for parity protection.
The AckID field is the same value carried by the packet being ack nowledged. Besides acknowledgement, this control symbol carries receiver buffer status for transmitter-based link flow control.
Error Coverage
Because reliable transport is crucial in the control plane, RapidIO technology was designed from the start with robust error detection and management. Before recovery or notification of errors can occur, errors must be detected. Errors are detected using a variety of methods:

CRC-based multibit error detection is applied over the entire packet with the exception of the first few bits of header.

Early CRC is provided for packets greater than 80-bytes in length to allow receivers to check the header early in the transmission.

Control symbols in the parallel PHY are repeated and inverted for parity protection. In the serial case, control symbols are protected by a 5-bit CRC and running disparity.

Protocol-based protection for bits not covered by CRC or parity. For example, the AckID field is covered because packets are assigned an AckID in order and are acknowledged in order. Out-of-order AckIDs in the packet or acknowledgement indicate an error has occurred.

The link is always transmitting either control symbols or packets thus allowing prompt detection of a link failure.

Transaction timers at the logical layer track loss of a response to an outstanding request.

Transaction timers at the PHY track loss of a packet acknowledgement.

When an error is detected, recovery can occur. Most errors are recovered in hardware to reduce the software overhead required for error management and minimize the latency incurred when soft errors occur. For example, when a CRC error is detected, the receiver sends a packet-not-accepted control symbol to the transmitter causing it to resend the packet.
Understanding the Parallel PHY
The parallel interface is differential LVDS, source-synchronous, and double-data rate. The driver and receiver are both 100-ohm differentially terminate d. The specification defines five speed bins from 250 MHz to 1000 MHz applied clock. The interface can be operated outside of these bins when both devices and the intervening board design accommodate the skew budget.
The RapidIO spec defines 8-bit and 16-bit parallel interface widths using 40 and 76 pins respectively. Should an 8-bit port be connected to a 16-bit port, hardware automatically detects the situation and causes the 16-bit port to act as an 8-bit port. In addition to the data, a single non-return-to-zero (NRZ) frame delineation bit is used. It toggles at the beginning of each packet or control symbol.
When initializing a parallel port interface, most implementations require two steps. The receiver must first align the sampling clock to the center of the data eye and then align the interface to a 32-bit boundary. Both functions are performed simultaneously by a specific sequence of control symbols and training patterns sent by both sides of the link. The training pattern is a recognition s equence of two data beats of one and two beats of zero for each data bit.
The initialization sequence begins with each side of the link sending a link-request/send-training control symbol followed by 256 training patterns. This sequence is repeated by each side of the link until the port is able to receive the training pattern. Once properly received, the port sends an idle control symbol in place of the link-request/send-training and continues to send 256 training patters. The idle control symbol indicates to its neighbor that it has synchronized properly to incoming data. Both sides finish sending the training pattern, and transition to ready once they have each received an idle symbol.
Detailing the Serial PHY
The serial PHY also uses differential signaling. It is AC-coupled and IEEE XAUI compatible. One and four lane widths are defined using 4 and 16 pins respectively. No sideband signals are used as clock and framing is embedded in the data. Three data rates are defined at both short and long reach transmitter amplitudes: 1.25, 2.5, and 3.125 GBaud/s.
Before transmission, data is 8B/10B encoded. This process transforms an 8-bit value into a 10-bit quantity for transmission. This larger code space provides several benefits. First, it allows two different encodings (called D-codes) for each 8-bit value. By keeping track of the number of ones and zeros (a running disparity), judicious use of encodings keeps the average number equal and maintains zero DC electrical balance on the data link.
Keeping the number of ones and zeroes constant also has the benefit allowing error detection at the receiver. In addition, unique 10-bit patterns (called K-codes) allow control symbol delineation, synchronization and data alignment across multiple lanes.
In the serial PHY, control symbols are similar to that of the parallel PHY, except they have been enhanced to allow one control symbol to carry both a packet handshake (e.g. accepted or not accepted) and a packet boundary delimiter or maint enance function. This dual function was possible because duplicating and inverting the control symbol as in the parallel is unnecessary. A 5-bit CRC, together with running disparity, provides sufficient error protection.
When packets or control symbols are not being sent, the transmitter sends an idle sequence consisting of three K-code symbols. One of the symbols is a SYNC K-code that contains a unique bit sequence called a comma. The comma is used by hardware to ensure proper clock and lane alignment. An idle link must also send a status control symbol periodically.
Initialization of the serial interface is different from parallel. During initialization, both sides send the idle sequence where the SYNC code is used to align the recovered sampling clock. Once a sufficient number of characters is received without encountering an invalid character, the inbound link is considered operational. The port is allowed to transmit after seven status control symbols have been received from the adjacent device. When both sides reach this state, the link is considered operational.
Initialization of a 4x port includes a process to align the stream of characters to the four lanes such that K-codes that define the start of control symbols are always located in lane 0. In addition, during initialization a 4x port detects when a 1x port is connected the 4x port is adjusted to communicate over the 1x lane.
On To Part 2
That wraps up Part 1 in our two-part series on the inner workings of the RapidIO specification. In Part 2, which will appear online next week, we'll look at bandwidth requirements, flow control mechanisms, and deadlock avoidance. We'll also examine the key elements needed to develop RapidIO endpoints and switches.
Author's Note: To find out more information on the RapidIO specifications, visit the Trade Association's web site at www.rapidio.org.
About the Author
Greg Shippen currently serves as chairman of the Technical Working Group for the RapidIO Trade Association. He is also a system architect for Motorola Semiconductor, where he has been involved in PowerQUICC processor design and other RapidIO-related product definition and development. Greg holds an M.S.C.E. from University of Southern California and a B.S.E.E from Brigham Young University. He can be reached at greg.shippen@motorola.com.