The Battle of Data center Interconnect Fabric

By Priyank Shukla, insilico

With the exponential increase in off-chip bandwidth requirement, chip-to-chip interconnect is turning out to be the bottleneck of the information highway. Such bottlenecks invariably drive a new wave of innovation. During 1990s and around the turn of the millennium, rapid adoption of personal computers and development of various IO devices fuelled early innovations in interconnect technology and interfaces such as SATA, SAS, USB, PCIe were defined.

Traditional data center architecture where individual servers employ multi-core CPUs, extremely fast RAM and local storage are connected by a Network Interface Card (NIC) through Ethernet switches to other servers, is neither able to cope with the explosive growth in internet traffic nor provides infrastructure for real-time processing of data needed for deep learning. Also, at the necessary performance levels, when applications communicate with the memory, I/O or accelerator devices the interconnect technologies result in very high processor overhead. These constraints are driving a new wave of interfaces focused to provide high-speed, low-latency and low power options for interconnects.

Before we discuss new protocols, a word on PCIe - for last few years, SSD memory vendors started adopting PCIe extensively and it appeared as if PCIe had found its place in data center fabric. However, the rate of bandwidth increase through PCIe protocol specifications could not keep pace with the bandwidth demand generated by exponential internet data traffic. PCI-SIG came up with base version Gen 3.0; offering 8 Gbps data rate in November 2010 and at the time of writing of this article, its next-generation specification Gen4 with 16 Gbps data rate is still not ratified. At a time when internet traffic doubles in months, this rate of bandwidth augmentation by PCI-SIG has left industry leaders wanting for more. Also, as PCIe is a more generic IO interface, implementers have always been looking for ways to improve latency and of late, incorporating the wish list from the folks who want to focus on reducing latency was turning out to be a difficult proposition for PCI-SIG. These led to the formation of new consortiums to define interconnect standards targeted for data center and cloud computing. This article focuses on the top three contenders in the race for dominance in interconnect fabric - OpenCAPI, GenZ, and CCIX.

Let’s consider competitive landscape of computing ASICs in a typical data center. The CPU socket is seeing competition among processor architectures of Intel’s x86, IBM’s PowerPC and the new entrant - RISC-V whereas the accelerator socket is finding FPGA and custom ASICs at loggerheads for dominance. Through project Catapult, Microsoft has been a vocal advocate of using FPGA to accelerate their Bing search engine, Azure and many other web services. Google, on the other hand, has been a proponent of Tensorflow ASICs. So in 2017, the interconnect technology for data center not only must address pain points of the processor to accelerator interconnect in the best possible way but it should also be able to work with all possible processor architectures.

Today Intel dominates server market and its acquisition of Altera in 2015 added FPGA to its portfolio. So if processor + FPGA as the accelerator is going to be the way forward, Intel is poised to win both processor and accelerator sockets and with their proprietary interface QPI (Quick Path Interconnect) the entry of any other semiconductor vendor will be extremely difficult. This is where CCIX (Cache Coherent Interconnect Accelerator) finds its niche. Proposed by a group of companies including AMD, ARM and Xilinx, it offers a cache coherent interface between processor and accelerator. To promote rapid adoption and leverage existing standard, CCIX offers the use of PCIe PHY and DLL (Data Link Layer) with critical improvements in upper protocol layers to reduce latency and to offer cache coherency between processor and accelerator. CCIX provides options for data rates from 16Gbps to 25 Gbps with a step size of 1 Gbps. The link partners can operate at max possible supported data-rate. The speed flexibility especially helps with early adoption. PCIe4 PHYs running at 16 Gbps are currently available and as CCIX uses PCIe PHY, the industry essentially has CCIX PHY available today!

The next major interconnect found in data center fabric is the processor to the memory interface. The two types of memory employed are Random Access Memory (SRAM/DRAM) and non-volatile memory (NVM). Each of which has its own interface this offers a scope of unification of these interfaces. Focused on this, formed in August 2016, GenZ open systems interconnect enables mix of DRAM and non-volatile memory to be directly accessed by applications at data rates from 32 to 400 Gbps. Keeping up with the requirements of other interconnects in the datacenter, Gen-Z supports cache coherency in point-to-point, meshed, and switch-based topologies. Cache coherency can be used between processors with accelerators, accelerators with accelerators, or accelerators with memory and storage, thereby challenging CCIX’s adoption as data center interconnect. Gen-Z understands that defining a PHY and addressing Card Electromechanical Issues take quite a while and they plan to leverage existing PHY standards. Gen-Z also uses PCIe PHY to ensure rapid adoption of protocol with changes implemented only in the controller.

Formed in October 2016, Open Coherent Accelerator Processor Interface (OpenCAPI) wants to be agnostic to processor architecture and allows coherent user-level accelerators and I/O devices. It not only allows any processor to attach to coherent user-level accelerators and I/O devices but also makes advanced memories accessible via read/write or user-level DMA semantics. The stated target of Open CAPI is to provide DMI like latency (10ns), but with enhanced command set of CAPI. The consortium promoting OpenCAPI conspicuously lacks Intel and has visible PowerPC architecture undercurrent. Setting data rate 25 Gbps, the protocol leverages 802.3 PHY for rapid adoption.

While which of these protocols will pass the test of the time is still to be seen, it is clear that Intel, the current dominant semiconductor company in server/data center space is up for a great competition from new entrants. It's good news for data center users, maybe that is why Google is backing OpenCAPI without intel. However, if the past is any indication, this does not augur very well for the bottom-line of semiconductor companies. But, that’s the nature of semiconductor ecosystem.

If you wish to download a copy of this white paper, click here