Myricom at

Reno, Nevada 10-16 November 2007

Myri-10G: 10-Gigabit Ethernet with a Supercomputing Heritage

512-port Myri-10G multi-protocol switch on display
Myri-10G 10-Gigabit Ethernet NICs show great performance with VMware ESX Server
QSFP ports and QSFP-terminated ribbon-fiber cables

Indiana University wins the SC07 Bandwidth Challenge with Myri-10G 10GbE NICs

Myri-10G is 10-Gigabit Ethernet from Myricom, and more. This fourth generation of Myricom networking products is a convergence at 10-Gigabit/s data rates of Myrinet cluster-interconnect technology with mainstream Ethernet.

As shown by several contestants in the SC07 Bandwidth Challenge and by many other SC07 exhibitors, Myri-10G 10-Gigabit Ethernet NICs provide near-wire-speed performance for conventional TCP/IP Ethernet applications. In addition, all of the capabilities of Myrinet, including kernel-bypass operation for low latency and low host-CPU load, are available with Myri-10G NICs over either 10-Gigabit Ethernet or 10-Gigabit Myrinet networks. Finally, Myri-10G switches – multi-protocol 10-Gigabit Ethernet and 10-Gigabit Myrinet – are economically scalable to thousands of ports.

This 4-page flyer (pdf, 1.4MB) provides an overview of Myri-10G technology and applications.

Myricom's exhibit at SC07 emphasized Myri-10G capabilities introduced during the past year. In addition, Myri-10G networking was conspicuous in many other SC07 booths and in the SC07 Bandwidth Challenge.

The 512-port Myri-10G multi-protocol switch shown in the photo to the right drew crowds of visitors almost continuously through SC07. This 21U switch is mounted at the top of a 36U rack with four 2U servers below it, and also a Coraid AoE (ATA over Ethernet) EtherDrive® storage appliance. The Coraid storage appliance uses Myri-10G NICs internally.

The Myri-10G switch provided connectivity to the four servers (10GBase-CX4, blue cables, Myrinet protocols), to the Coraid storage appliance, to 10-Gigabit Ethernet clients (10G-Base-SR, orange fiber cables, Ethernet protocols) in the three other corners of the Myricom booth, and to SCinet (10GBase-LR, orange fiber cable, Ethernet protocols).

Myricom announced this new family of Myri-10G switches at ISC07 (see the Myricom at ISC07 news item), and has subsequently shipped the first twelve 21U switches to customers, both for HPC-cluster applications and for data-storage systems. With the successes of these early installations, Myricom is now expanding production of this new family of Myri-10G switches.

Just prior to SC07, Argonne National Laboratory and IBM announced a 445-Teraflops expansion of the Blue Gene/P system at the Argonne Leadership Computing Facility (ALCF), and the role that Myri-10G switches play in providing a very high performance file system for Blue Gene/P installations. In this ALCF press release, Argonne notes that "Myricom's economical, low-latency modular switches represent the heart of the ALCF's data-management system. The nine-switch complex supports up to 2,048 connections, each of which simultaneously exchanges data at around 1 billion bytes per second."

VMware ESX Server demo. Although virtualization is not yet a hot topic in traditional HPC, a demonstration and benchmarks of Myri-10G 10-Gigabit Ethernet NICs under VMware's ESX 3.5 attracted many visitors interested in higher performance for data centers.

The demonstration and benchmarks showed that a single virtual machine on a modest-performance server was able to send data at 9.85 Gb/s (essentially 10GbE line rate), and was able to receive at 8 Gb/s. Two virtual machines running in parallel could receive at 9.86 Gb/s. This aggregate receive performance is made possible through the use of multiple queues in the Myri-10G NIC, which may each be assigned to individual virtual machines along with the associated interrupt for each queue.

A recent Techworld article reports that many users are finding that running multiple virtual servers are overwhelming Gigabit Ethernet NICs, and 10-Gigabit Ethernet NICs are required to provide the aggregate network bandwidth required. The advent of multi-core processors allows a single server to run far more simultaneous virtual machines than before, driving this demand for bandwidth.

The native I/O virtualization in the Myri-10G NIC allows efficient sharing of NIC resources while minimizing the involvement of a hypervisor to steer I/O to or from the appropriate virtual machine. This approach frees processors to do the work of the virtual machines instead of spending all their cycles on I/O.

QSFP ports and EOE cables. Two acronyms that we recommend that you learn if you don't know them already:

• EOE = "Electrical-Optical-Electrical," referring to optical-fiber cables in which the electrical-optical conversion is performed in the cable end.
• QSFP = "Quad Small-Form-factor Pluggable," a multi-source module format and connector for quad-ribbon-fiber transceivers. (The Multi Source Agreement is published here.)

Myricom was involved at SC07 in demonstrating new EOE cables from three companies. The orange cable hoops in the photo above of the 512-port switch are QSFP-terminated Zlynx cables from Zarlink, which distributed this press release at SC07. Luxtera also announced QSFP-terminated EOE cables, and demonstrated these cables in their SC07 booth with a Myri-10G switch line card with 16 QSFP ports (photo below). Finally, Finisar announced their Laserwire serial-fiber EOE cable for XFP or SFP+ 10-Gigabit Ethernet ports, with the demonstration in the Finisar booth being with Myri-10G 10GBase-R NICs. EOE cables offer economy (improved manufacturing margins due to an end-to-end solution), and the safety and reliability of electrical termination (no Class-1M laser-safety warnings, and no problems with dust in a pluggable optical-fiber port).

In order to take advantage of these new EOE cables, Myricom introduced and demonstrated at SC07 switch line cards with QSFP ports:

These front-panel ports can accommodate either QSFP transceivers to MTP/MPO-terminated ribbon fiber, or QSFP-terminated EOE cables. Myricom will shortly also be offering Myri-10G NICs with QSFP ports, which provide the same alternatives. In fact, QSFP ports may become nearly universal. The port connector provides power, required if a transceiver is plugged into the port, and other companies have announced "active" copper cables with signal-conditioning circuitry in the QSFP cable ends.

The SC07 Bandwidth Challenge. We are pleased that a team led by Indiana University, with partners from the Technische Universität Dresden, Rochester Institute of Technology, Oak Ridge National Laboratory, and the Pittsburgh Supercomputing Center, was awarded first place in the SC07 Bandwidth Challenge. From the IU press release: "Using the IU Data Capacitor, a system designed to store and manipulate massive data sets, the IU team achieved a peak transfer rate of 18.21 Gigabits/second out of a possible maximum of 20 Gigabits/second. This performance was nearly twice the peak rate of the nearest competitor." In fact, the IU team was the only team to employ full-duplex transfers. Corporate sponsors for the IU Bandwidth Challenge effort included Data Direct Networks, Dell, Myricom, and Force 10 Networks. The IU site and installation at SC07 used Myri-10G NICs exclusively.

The complete Bandwidth Challenge results are at https://scinet.supercomp.org/2007/bwc/. Teams other than IU using Myri-10G NICs included the National Center for Data Mining (NCDM) and the University of Delaware. In addition, a Caltech team set a wide-area record outside of the Bandwidth Challenge framework with a sustained bi-directional flow of more than 80 Gbits/s using Myricom and Intel NICs.

Myri-10G in other exhibits. Myricom is pleased that many of our customers and partners displayed Myri-10G networking in their exhibits. For example, to the right is a photo of the innovative new SiCortex multicomputer, which uses Myri-10G 10-Gigabit Ethernet Express Module NICs for I/O. Fujitsu's exhibit of their low-latency, 20-port, XG2000 10-Gigabit Ethernet switches was driven by Myri-10G NICs. Myricom was included as a partner in the Microsoft booth, and remote Windows cluster demonstrations, including at a Myri-10G cluster at the Microsoft Partners Solutions Center, were included in the Myricom booth. In the SC07 Cluster Challenge, the Indiana University team built an Apple Xserve cluster using Myri-10G NICs with MX over Ethernet and a Fujitsu 10-Gigabit Ethernet switch.

Reno was a convenient venue for SC07, with the convention center and hotels being close to each other and to the Reno airport. For readers who are not familiar with the annual SC conference, it is in a different city each year. SC is not only a technical conference and exposition, but is a convention of HPC people. Myricom technical and sales people saw many old friends and made some new friends both on the exposition floor and at numerous off-site visits with our customers and partners.

The Myricom SC07 booth early one morning before the exhibits opened.
The booth was almost always crowded during exhibit hours.

The Exhibit Team. The Myricom team attending and exhibiting at SC07 were Scott Atchley, Member Technical Staff; Susan Blackford, Member Technical Staff; Dr. Nan Boden, Executive Vice President; Bob Brown, Microsoft Business Development; John Daley, Senior Programmer; Reese Faucette, Senior Software Architect; Dr. Markus Fischer, Senior Software Architect; Tom Leinberger, Director of Sales - Central Region; Dr. Patrick Geoffray, Senior Software Architect; Dave PeGan, Vice President, Sales; Justin Pratt, Member Technical Staff; Dr. Loic Prylli, Senior Software Architect; Dr. Wolfgang Rühmer, Director of EMEA Sales; Scott Schweitzer, Director of OEM Business Development; Dr. Chuck Seitz, CEO; Dr. Jakov Seizovic, CTO; Dr. Ruth Sivilotti, Member of the Technical Staff; Marty Stewart, Executive Assistant; and Tim Sticklinski, Director of Sales - Western Region.

We're now looking forward to ISC08, to be held 17-20 June 2007 in Dresden, Germany, and to SC08, to be held in 15-21 November 2008 in Austin, Texas.

Last updated: 3 December 2007