PCIe 5.0 vs. Emerging Protocol Standards - Will PCIe 5.0 Become Ubiquitous in Tomorrow's SoCs?

1. Introduction

Over the last 3 years, a number of protocol standards have emerged, aiming to address the growing demand for higher data throughput and more efficient data movement. While CCIX, Gen-Z, and OpenCAPI are relative newcomers, PCIe has been around for almost 2 decades. With the imminent release of version 5.0 of the PCIe Specification, SoC designers have a variety of options for supporting bandwidths in excess of 400 Gbit/s while improving overall communication efficiency.

In this article, we look into the various protocols by providing a brief history, a quick technical comparison, and the latest deployment status. We also offer our perspective on the evolution of these protocols.

The second part of the article focuses on PCIe 5.0. We discuss how PCIe 5.0 currently appeals to SoC designers in specific market segments and for specific use cases, and whether PCIe 5.0 has the potential to become ubiquitous in future generation SoCs.

2. PCIe and Emerging Interconnect Standards

Figure 1 illustrates the evolutions of the PCI Express protocol since it was first standardized in 2003.

Fig. 1 - Evolution of the PCI Express protocol Specification

As shown in Figure 1, the transition from PCIe 3.0 to PCIe 4.0 took 7 years, a much longer cycle than for the previous two generations. The transition from a PCIe roadmap driven by consumer and enterprise applications to a PCIe roadmap driven by HPC and Hyperscale applications may have been be a determining factor.

During that transition, the explosion of internet data traffic and the requirement for more Ethernet bandwidth in data centers prompted for the rapid deployment of larger Ethernet pipes, as illustrated in Figure 2.

Fig. 2 - The Data Center I/O push: PCIe vs. Ethernet

As shown in Figure 2, PCIe 3.0 was becoming a bottleneck in data center architectures right around 2014-2015 coincident with the mainstream deployment of 40G Ethernet.

This context was a fertile ground allowing new high-speed interconnect protocols to emerge. The CCIX Consortium was formed in May 2016, the Gen-Z Consortium in August 2016, and the OpenCAPI Consortium in October 2016.

2.1. Technical Comparison

The promise of CCIX, Gen-Z, and OpenCAPI is, to some extent, to fill the gap left by PCIe, whether in term of pure bandwidth, native coherency support (for tighter coupling between processors and accelerators), support for new topologies, and in the case of Gen-Z a complete paradigm shift towards a memory-centric interconnect fabric.

Table 1 provides a basic technical comparison between the various protocols.

Table 1 - Protocol standards compared

2.2. Deployment Status

Figure 3 provides a timeframe for when we could see SoCs supporting these protocols. Known dates and timeframes are shown in black; dates and timeframes shown in grey are estimated based on our interactions with the various ecosystem members.

Fig. 3 - I/O-enabled SoC deployment timeframe

As shown in Figure 3, OpenCAPI 3.0 is the only new interconnect protocol to be rolling out in silicon, however currently only available in IBM’s POWER9 SoCs, IBM being the main backer of the technology. CCIX 1.0 is expected to be rolling out in select SoCs sometime in H2 2019 and we expect to see early PCIe 5.0 enabled silicon in H1 of 2020. Gen-Z interconnect should make its way into mainstream silicon chips by H1 2021.

2.3. Perspective and Convergence

While many leading edge technology companies are involved in the Consortiums, often with multiple memberships, it is difficult to see any convergence path or anticipate that any of these interconnect protocols will see widespread adoption.

Because of this uncertainty, what we believe to be a possible scenario is that SoC developers will likely want to support a combination of these protocols in their upcoming SoCs, for the following reasons:

As a way to future-proof the SoCs, minimizing the dependency to a particular interconnect protocol
Because the risk and cost factors are minimal:
- Physical Layer (PHY) can be shared among interconnect protocols (e.g. a single multi-lane 25G PHY IP covers PCIe 4.0, CCIX, OpenCAPI, and Gen-Z)
- Controller IP exists for all these interconnect standards and accounts for little in the overall SoC size and cost
Because SoC vendors are often multi-consortium members and as such, have a vested interest in seeing these protocol standards succeed

Interestingly, IBM’s POWER9 family of processors support PCIe 4.0 and OpenCAPI 3.0, in addition to NVLink 2.0 (a NVIDIA proprietary interconnect protocol).

Is there a path towards convergence or a consolidation on the longer term? PCI Express is likely not going away, due to its omnipresence in the x86 ecosystem, its huge software and firmware legacy, and the aggressive roadmap towards PCIe 5.0 pushed by the PCI-SIG Consortium.

OpenCAPI seems very much tied to the IBM POWER ecosystem and there is no indication that it might expand beyond that.

CCIX could see some adoption outside of IBM POWER and Intel x86 ecosystems, however may lose steam as PCIe 5.0 and Gen-Z ramp up, and especially if coherency extensions (ECNs) make their way into the PCI Express 5.0 roadmap.

Gen-Z is the wildcard that could find its way in data centers at the edge for long-haul or rack-to-rack communication in memory-centric composable systems.

3. A Closer Look at PCIe 5.0

PCIe 5.0 is a natural evolution of the PCI Express Specification that primarily doubles the line rate to 32Gbit/s, from 16Gbit/s for PCIe 4.0. No functional changes are introduced at Link and Transaction layer level, the focus being on the Physical layer for improved signal integrity (with an update to EIEOS, SKP, and Equalization sequences, new precoding mechanism, and a new physical connector specification).

One interesting addition to the PCIe 5.0 Specification is the support for “Alternate Protocols” through modified TS1/TS2 sequences. This could signal an opening of the standard, allowing future protocols to leverage some of the proven PCIe firmware and software stacks and extend the reach and capabilities of PCI Express.

3.1. PCIe Market Adoption

Early adopters in specific market segments have started to develop to the PCIe 5.0 Specification, basing their SoC designs on revision 0.7 or 0.9 of the Specification. Backward compatibility to PCIe 4.0 ensures there is a fallback mechanism until PCIe 5.0 is more widely deployed.

Early PCIe 5.0 adoption markets include:

HPC, Cloud and AI: the need for more host-to-accelerator bandwidth and the high cost of manufacturing SoCs at 7nm prompts for future-proofing designs with the latest and greatest I/O technology.
Enterprise Storage SoCs: emerging memory technologies (HBM, HMC, SCM, etc.) coupled with PCIe-based NVMe will demand higher performance within a constrained form factor. PCIe 5.0 can provide higher bandwidth on fewer lanes at lower power.
Test Equipment SoCs: test and measurement systems vendors are at the front line for adopting new I/O technologies as they are key ecosystem enablers.

With PCIe technology deeply anchored in the x86 ecosystem, the natural use model for PCIe 5.0 is high speed communication with x86 processor subsystems, via point-to-point or switched topologies. It remains to be seen whether PCIe can evolve to allow other use models, such as cache coherent communication between processors and accelerators (CCIX-like), coherent accelerator to accelerator communication in a mesh topology (CCIX and OpenCAPI-like), or memory-semantic fabric (Gen-Z like).

4. Closing Statement

The past 2 years have seen the arrival of new high-speed interconnect protocols developed to better enable emerging computing architectures. CCIX, Gen-Z, and OpenCAPI aim to provide higher bandwidth, tighter coupling between compute elements, and more efficient data movement in heterogeneous systems. During the same period, long-awaited PCIe 4.0 was finally released, however without the bandwidth and features necessary to fully address emerging data center architectures. An accelerated PCI-SIG roadmap should see PCIe 5.0 released in the first half of 2019, less than 2 years after the release of PCIe 4.0. Can PCIe 5.0 take over and make its way into SoCs that will power tomorrow’s computing architectures? SoCs that incorporate PCIe 5.0 rev. 0.7 or 0.9 I/O are already being designed, however in very specific market segments. It remains to be seen whether the PCIe Specification can open up to allow for new capabilities and quick evolutions to support the rapidly evolving computing landscape.

PLDA is committed to developing and licensing leading edge semiconductor IP for high-speed interface protocols. Our PCIe 5.0 rev. 0.9 interface IP is available today and is being integrated into SoCs due for tape-out in early 2020. For more information, visit our web site at www.plda.com.

For more information on the protocol standards mentioned in this article, please visit the Consortium web sites:

Disclaimer

The views exposed in this article are those of the author and do not necessarily reflect the official policy or position of any other organization, employer or company. Assumptions made in the analysis are not reflective of the position of any entity other than the author - and, since we are critically-thinking human beings, these views are always subject to change, revision, and rethinking at any time.