Ethernet interconnect outpacing Infiniband at Intel

Intel has licensed technology from Lucent Technologies to help the CPU maker design TCP offload engines (TOEs) for Ethernet that will incorporate the remote direct-memory-access (RDMA) features of Infiniband. Intel said those chips — using a state machine implementation — will boost Gigabit Ethernet to price/performance capabilities beyond Infiniband for use in networking, storage and clustering in computer data centers.

"The ultimate goal is to have one pipe with transport offload for all these applications in one infrastructure," said Anil Vasudevan, a senior architect in Intel's LAN access division.

While much work remains to deliver TCP offload with RDMA, Vasudevan predicted those chips could create Ethernet links with latencies in the 10- to 40-microsecond range. Latency as low as 6 microseconds has been one of the chief selling points for Infiniband.

Intel and Broadcom sources said Gigabit Ethernet chips with TOE and RDMA will emerge sometime next year, with 10-Gigabit versions following. Infiniband chips are expected to begin shipping in production volumes at the 10-Gbit level late this year or early next year. With the economic slowdown tightening purse strings for IT buyers, however, announcements of Infiniband systems have been slow in coming.

"At 1 gigabit you will still see some Infiniband products, but as you go to 10 G the price/performance of Ethernet will drive the convergence of computing and communications in the data center," said Vasudevan.

That appears to mark a major shift in Intel's product plans for computer interconnects, though other parts of the corporation still pledge fealty to Infiniband as an industry initiative.

Intel spearheaded Infiniband, formerly NGIO, as a server interconnect for storage, clustering and networking. But earlier this year, it canceled its 2.5-Gbit/second Infiniband product plans, saying the industry was well-served with emerging 10-Gbit Infiniband chips from companies such as Banderacom, IBM and Mellanox.

Jim Pappas, who runs technology initiatives for Intel's server division, said Infiniband will dominate as an interconnect for high-performance scientific computing clusters and in large database clusters using databases from IBM and Oracle. But whether Infiniband or Ethernet becomes the mainstream data center interconnect is yet to be determined, he said.

"The big question is whether Ethernet can get good enough, fast enough. If the market stalls for years it will be a bigger challenge for Infiniband," Pappas said.

A handful of large storage and computer makers, including IBM, are expected to ship systems using Infiniband late this year or early next year. But even here the support for Infiniband appears to be waning.

"More specialized and scaled back seem to be the words to describe Infiniband now. I don't think anyone would debate that," said Tom Bradicich, chief technology officer for IBM's server group. When 10-Gigabit Ethernet arrives on copper with RDMA and TOE support "that will be very compelling," he added.

"Infiniband has no future. I don't see a critical mass for it," said Peter Glaskowsky, editor in chief of the Microprocessor Report. "Intel mistook the gap between gigabit and 10-G Ethernet as a market window. In fact the markets for Ethernet and servers are slow-moving ones and people don't want the technology to change faster," he added.

IBM Microelectronics remains committed to Infiniband silicon, although to date it has only received orders for sample quantities, said Jerry Rogich, who runs the Infiniband chip group at IBM. "It's not big numbers but it's the start of a curve. We see activity in high-end servers and in emerging server-blade designs," Rogich said.

"These Ethernet products are two to four years away and Infiniband is here now," said Eyal Waldman, chief executive officer of Mellanox Technologies, an Infiniband chip startup.

One storage-system maker will ship an Infiniband product with Mellanox silicon this fall, Waldman said, the company's first major public design win. "The systems announcements have been pushed out two or three quarters," Waldman said.

Mellanox garnered $56 million in what it considers its final round of private financing in February. The company recently laid off 67 of its 214 people, which Waldman said will help that funding last until the company can become profitable.

Infiniband startup Banderacom Inc. has also faced layoffs recently.

Plenty of work remains before good TOE and R DMA features can be delivered for Gigabit Ethernet.

The RDMA Consortium, a group of several large hardware systems makers, is developing a specification for RDMA-over-Ethernet. An initial draft of its work is expected to be made public and delivered to the Internet Engineering Task Force soon.

In part, that work maps the RDMA features of Infiniband onto Ethernet. The group has also added new framing features to the technology. A transmitter can use framing to put headers on Ethernet byte streams so that those byte streams can be more easily decoded and executed in an out-of-order mode at the receiver, boosting performance — especially for iSCSI storage traffic — according to one source in the consortium.

But TOE chips are still largely unproven, said Gary Simpson, a technology analyst for BIOS software maker Phoenix Technologies Ltd. "The state machine implementations are swamped by complex exception handling and those who use embedded processors or multicore ASICs wind up with something as powerful as the host processor they were trying to offload," he said.

Intel's approach will be to build a state machine for bulk TCP processing but push exception handling to a host or embedded processor.

Software support is another hurdle. Microsoft Corp. is expected to create an interconnect-agnostic approach in its next-generation operating system, which is not expected to debut until late 2004. Until then, OEMs and chip makers are expected to create their own intermediary software layers to support the accelerated Ethernet chips.