Enter the Inner Sanctum of RapidIO: Part 1

Packets are always a multiple of 4 bytes in length and 32-bit aligned. Bit fields are arranged to simplify logic design. For example, PHY bits appear near the beginning of the packet so the link protocol bits are available first as the packet arrives at a link receiver. Trans port bits are near the front of the header and aligned to a 16-bit boundary to allow switches easy access for routing purposes.

In general, the various layers of the packet are added to the packet header as the transaction moves from the logical layer of the endpoint down to the physical layer. In like manner, these fields are removed as the packet data is passed to higher layers of the endpoint logic.

Packets are protected by an end-to-end 16-bit cyclic redundancy check (CRC) that is applied at the originating endpoint. The CRC remains unchanged as the packets traverse the network so correct CRC at the destination ensures the integrity of the packet payload. Note that an early 16-bit CRC is also applied when packets reach 80 bytes or larger in length.

Unlike other packet types, a maintenance packet functions as both request and response. Maintenance requests always have a response to guarantee completion before subsequent operations are initiated at the requester. The only exception to this is the port write request. This special maintenance write type is used to communicate status or error information from devices that do not have an endpoint (such as a switch) and hence cannot accept responses.

Because switches do not have a device ID but do have capability, command, and status registers, maintenance packets include a hop_count field in order to target switches. To target a specific switch, an endpoint issues a maintenance packet with an endpoint destination ID whose route takes the packet through the desired switch. The value of hop_count defines which switch is targeted. Each switch examines the hop_count field and consumes the packet if the value is zero. Otherwise, it decrements the value and sends the packet on.

Control Symbols
The process of initializing and synchronizing the link, performing packet delineation, ensuring reliable delivery, and dealing with error conditions is managed through the use of a short 4-byte packet called a control symbol. Control symbols exist only at the physical layer and are never routed between ports on a switch.

Reliable packet delivery across a link is accomplished by the receiver using a control symbol sent to send the transmitter an explicit acknowledgement of each packet. Should the transmitter fail to receive a packet acknowledgement, it either resends the packet or a link error recovery procedure occurs depending on the circumstances. When errors occur, control symbols are used to send error status and coordinate link error recovery.

During the initialization process, control symbols report status and request transmission of training sequences used to synchronize operation of the link.

Some control symbols are specific to the physical interface. For example, in the parallel interface, the idle control symbol is sent when there are no packets available for transmission or the rate of packet arrival needs to be temporarily slowed.

Control symbols are also used to denote packet boundaries. In the parallel case, a control symbol is used to mark the end of a packet when no packet is available to follow it. For the serial, control symbols mark the start and end of all packets.

Because control symbols are used for link control, minimizing loop latency is important for efficient operation of the protocol. To avoid delaying control symbols while a large packet is transmitted, most control symbols can be embedded within packets.

To conserve bandwidth on the link, some kinds of control symbols can cancel transmission of a packet before the packet has been completely sent should an error be detected early.

The format of an acknowledge control symbol is shown in Figure 2 for the parallel PHY case. The "S" bit begins both packets and control symbols. When asserted, the receiver can immediately classify the incoming packet as a control symbol. Note that control symbols are actually 16-bits long but the value is repeated and inverted for parity protection.

The AckID field is the same value carried by the packet being ack nowledged. Besides acknowledgement, this control symbol carries receiver buffer status for transmitter-based link flow control.

Error Coverage
Because reliable transport is crucial in the control plane, RapidIO technology was designed from the start with robust error detection and management. Before recovery or notification of errors can occur, errors must be detected. Errors are detected using a variety of methods:

• CRC-based multibit error detection is applied over the entire packet with the exception of the first few bits of header.
• Early CRC is provided for packets greater than 80-bytes in length to allow receivers to check the header early in the transmission.
• Control symbols in the parallel PHY are repeated and inverted for parity protection. In the serial case, control symbols are protected by a 5-bit CRC and running disparity.
• Protocol-based protection for bits not covered by CRC or parity. For example, the AckID field is covered because packets are assigned an AckID in order and are acknowledged in order. Out-of-order AckIDs in the packet or acknowledgement indicate an error has occurred.
• The link is always transmitting either control symbols or packets thus allowing prompt detection of a link failure.
• Transaction timers at the logical layer track loss of a response to an outstanding request.
• Transaction timers at the PHY track loss of a packet acknowledgement.

When an error is detected, recovery can occur. Most errors are recovered in hardware to reduce the software overhead required for error management and minimize the latency incurred when soft errors occur. For example, when a CRC error is detected, the receiver sends a packet-not-accepted control symbol to the transmitter causing it to resend the packet.

Understanding the Parallel PHY
The parallel interface is differential LVDS, source-synchronous, and double-data rate. The driver and receiver are both 100-ohm differentially terminate d. The specification defines five speed bins from 250 MHz to 1000 MHz applied clock. The interface can be operated outside of these bins when both devices and the intervening board design accommodate the skew budget.

The RapidIO spec defines 8-bit and 16-bit parallel interface widths using 40 and 76 pins respectively. Should an 8-bit port be connected to a 16-bit port, hardware automatically detects the situation and causes the 16-bit port to act as an 8-bit port. In addition to the data, a single non-return-to-zero (NRZ) frame delineation bit is used. It toggles at the beginning of each packet or control symbol.

When initializing a parallel port interface, most implementations require two steps. The receiver must first align the sampling clock to the center of the data eye and then align the interface to a 32-bit boundary. Both functions are performed simultaneously by a specific sequence of control symbols and training patterns sent by both sides of the link. The training pattern is a recognition s equence of two data beats of one and two beats of zero for each data bit.

The initialization sequence begins with each side of the link sending a link-request/send-training control symbol followed by 256 training patterns. This sequence is repeated by each side of the link until the port is able to receive the training pattern. Once properly received, the port sends an idle control symbol in place of the link-request/send-training and continues to send 256 training patters. The idle control symbol indicates to its neighbor that it has synchronized properly to incoming data. Both sides finish sending the training pattern, and transition to ready once they have each received an idle symbol.

Detailing the Serial PHY
The serial PHY also uses differential signaling. It is AC-coupled and IEEE XAUI compatible. One and four lane widths are defined using 4 and 16 pins respectively. No sideband signals are used as clock and framing is embedded in the data. Three data rates are defined at both short and long reach transmitter amplitudes: 1.25, 2.5, and 3.125 GBaud/s.

Before transmission, data is 8B/10B encoded. This process transforms an 8-bit value into a 10-bit quantity for transmission. This larger code space provides several benefits. First, it allows two different encodings (called D-codes) for each 8-bit value. By keeping track of the number of ones and zeros (a running disparity), judicious use of encodings keeps the average number equal and maintains zero DC electrical balance on the data link.

Keeping the number of ones and zeroes constant also has the benefit allowing error detection at the receiver. In addition, unique 10-bit patterns (called K-codes) allow control symbol delineation, synchronization and data alignment across multiple lanes.

In the serial PHY, control symbols are similar to that of the parallel PHY, except they have been enhanced to allow one control symbol to carry both a packet handshake (e.g. accepted or not accepted) and a packet boundary delimiter or maint enance function. This dual function was possible because duplicating and inverting the control symbol as in the parallel is unnecessary. A 5-bit CRC, together with running disparity, provides sufficient error protection.

When packets or control symbols are not being sent, the transmitter sends an idle sequence consisting of three K-code symbols. One of the symbols is a SYNC K-code that contains a unique bit sequence called a comma. The comma is used by hardware to ensure proper clock and lane alignment. An idle link must also send a status control symbol periodically.

Initialization of the serial interface is different from parallel. During initialization, both sides send the idle sequence where the SYNC code is used to align the recovered sampling clock. Once a sufficient number of characters is received without encountering an invalid character, the inbound link is considered operational. The port is allowed to transmit after seven status control symbols have been received from the adjacent device. When both sides reach this state, the link is considered operational.

Initialization of a 4x port includes a process to align the stream of characters to the four lanes such that K-codes that define the start of control symbols are always located in lane 0. In addition, during initialization a 4x port detects when a 1x port is connected the 4x port is adjusted to communicate over the 1x lane.

On To Part 2
That wraps up Part 1 in our two-part series on the inner workings of the RapidIO specification. In Part 2, which will appear online next week, we'll look at bandwidth requirements, flow control mechanisms, and deadlock avoidance. We'll also examine the key elements needed to develop RapidIO endpoints and switches.

Author's Note: To find out more information on the RapidIO specifications, visit the Trade Association's web site at www.rapidio.org.

About the Author
Greg Shippen currently serves as chairman of the Technical Working Group for the RapidIO Trade Association. He is also a system architect for Motorola Semiconductor, where he has been involved in PowerQUICC processor design and other RapidIO-related product definition and development. Greg holds an M.S.C.E. from University of Southern California and a B.S.E.E from Brigham Young University. He can be reached at greg.shippen@motorola.com.