Myri-10G 10-Gigabit Ethernet Solutions

Myri-10G 10-Gigabit Ethernet Solutions
for Linux, Windows, Solaris, Mac OS X, FreeBSD, and VMware ESX

For standard TCP/IP and UDP/IP 10-Gigabit Ethernet applications, Myri-10G Network Interface Cards (NICs) and software deliver excellent performance at low cost, and are fully compliant with Ethernet standards.

Performance. The standard netperf (2.4.5) benchmark results below were performed between one-port Myri-10G "8B" NICs with the Linux 2.6.18-128.1.16.el5 kernel (x86_64) and the Myri10GE 1.5.0 driver between two Asus RS500-E6-PS4 servers with dual Intel quad-core Xeon X5570s @ 2.93GHz (a total of eight 2.93GHz Nehalem cores):

Netperf Test   MTU    BW        TX_CPU %  RX_CPU %
------------   ----   -------   --------  --------
TCP_STREAM     9000   9910.33     4.52      2.84
TCP_SENDFILE   9000   9910.32     2.71      2.82
TCP_STREAM     1500   9477.10     4.62      5.57
TCP_SENDFILE   1500   9452.54     2.56      5.63

Thanks to the offloads in the driver and NIC firmware, the host-CPU load is quite low even at these near-wire-speed throughput rates. See this Myri-10G 10-Gigabit Ethernet Performance Measurements page for additional benchmarks, including with other operating systems.

Some 10-Gigabit Ethernet NIC vendors are ambiguous about the capabilities of two-port ("dual-port") NICs. Myri-10G NICs with two network ports are always listed as either:

Two ports for failover: These NICs appear to the host operating system as one PCI device. The failover between network ports is handled by the NIC firmware with a response time that can be set to as little as ~10µs. Either port can operate at near wire speed, but not both at the same time. The performance listed above is representative of the active port.
Two ports for performance: These NICs appear to the host operating system as two PCI devices. In hosts with sufficient PCI Express and memory bandwidth, both ports can operate concurrently at near wire speed. These NICs can be regarded as two independent network devices, and the performance listed above can be achieved concurrently on both ports. Of course, the two ports can also be used for failover.

Another factor that is important to the near-wire-speed performance of Myri-10G NICs is that they connect to hosts through PCI Express x8. "Gen1" (2.5 GT/s) PCI Express x8 is a 16+16 Gigabit/s full-duplex I/O fabric that is fast enough, even after PCI Express protocol overhead, to keep up with a 10+10 Gigabit/s full-duplex network port. However, to keep up with two 10-Gigabit/s network ports concurrently, the high-end NICs with two ports for performance use "Gen 2" (5 GT/s) PCI Express x8, a 32+32 Gigabit/s IO fabric.

Preferred NICs for 10-Gigabit Ethernet Applications. Myri-10G NICs are available in the standard PCI Express add-in card and other form factors, and with a full choice of PHYs for the network ports. For complete details, see the Myri-10G Product List and this Guide to Myri-10G PHYs (pdf). However, the most popular and cost-effective 10-Gigabit Ethernet PHYs for PCI Express add-in cards are SFP+ ports and CX4 ports. SFP+ ports may be used with optical-fiber transceivers for 10GBase-SR, 10GBase-LR, and other standards over serial fiber, with "Direct Attach" twinax copper cables up to 7m, and with other SFP+-terminated cables. CX4 ports support the 10GBase-CX4 standard for communication over copper cables up to 15m.

**Selection Guide for 10-Gigabit Ethernet NICs**
Network Port ››	SFP+ transceiver socket for 10GBase-SR or 10GBase-LR over serial fiber, or SFP+-terminated "Direct Attach" twinax cables, or other cables	CX4 connector for 10GBase-CX4 over copper cables
PCI Express x8 NICs with one network port (Nearly 10 Gb/s throughput)	10G-PCIE-8B-S NIC Low-profile (half-height) PCI Express add-in card (Standard PCI faceplate version shown) $595 list price (SFP+ transceiver sold separately)	10G-PCIE-8B-C NIC Low-profile (half-height) PCI Express add-in card (Standard PCI faceplate version shown) $495 list price
PCI Express x8 NICs with two network ports for failover (Nearly 10 Gb/s throughput)	10G-PCIE-8B-2S NIC Low-profile (half-height) PCI Express add-in card (Standard PCI faceplate version shown) $795 list price (SFP+ transceivers sold separately)	10G-PCIE-8B-2C NIC Standard-height PCI Express add-in card $595 list price
"Gen 2" (5 GT/s) PCI Express x8 NICs with two network ports for performance and failover (Nearly 20 Gb/s throughput)	10G-PCIE2-8B2-2S NIC Low-profile (half-height) PCI Express add-in card (Standard PCI faceplate version shown) $995 list price (SFP+ transceivers sold separately)	10G-PCIE2-8B2-2C NIC Standard-height PCI Express add-in card $795 list price
Cables for this type of network port	LC-connectorized duplex multimode fiber cables for use with 10GBase-SR SFP+ transceivers, or LC-connectorized single-mode fiber cables with 10GBase-LR SFP+ transceivers, or SFP+-terminated "Direct Attach" twinax or other cables	10GBase-CX4 copper cables

High Speed Expansion Cards (HSECs) for the IBM BladeCenter H. Myricom also produces the fastest, lowest cost, and lowest power Ethernet expansion cards available for the IBM BladeCenter H:

Two ports for failover	Four ports, two for performance, two additional ports for failover

Single PCI Express x8 Device Nearly 10Gb/s throughput 3 Watts typical Product code 10G-PCIE-8B-2I List price $595	Two PCI Express x8 Devices Nearly 20Gb/s throughput 6 Watts typical Product code 10G-PCIE-8B2-4I List price $895

Software. The driver and NIC firmware for 10-Gigabit Ethernet operation is currently available for Linux, Windows, Solaris 10, Mac OS X, FreeBSD, and VMware ESX. You can download the drivers (which include the NIC firmware) from this Myri-10G 10-Gigabit Ethernet Driver Download page. The driver was contributed to and accepted in the Linux kernel, and is included in the 2.6.18 and later kernels and in many Linux distributions. However, the driver available by download will be more up-to-date. The driver is also included in FreeBSD 7.0. The Windows software distributions are WHQL-certified.

Offloads. The driver and NIC firmware implement zero-copy on the send side with all supported operating systems, and, depending on the OS, use a variety of highly effective stateless offloads, including:

Interrupt Coalescing
IP and TCP checksum offload, send and receive
TSO (TCP Segmentation Offload, also known as Large Send Offload)
RSS (Receive-Side Scaling)
LRO (Large Receive Offload)
Multicast filtering

The driver and NIC firmware do not implement “stateful” offloads. These NICs and their software are deliberately not TCP Offload Engines. See, for example, http://linux-net.osdl.org/index.php/TOE for some of the reasons why TCP offload "has little merit."

10-Gigabit iSCSI. These NICs may also be used for 10-Gigabit iSCSI. See the Myri-10G iSCSI installation instructions and iSCSI performance measurements.

Form factors. All of the one-port and most of the two-port Myri-10G NICs are available as low-profile (half-height) PCI Express add-in cards. Low-profile cards allow the option of installation in some 2U servers without the need for a riser card if you specify low-profile PCI faceplates. NICs with 8A, 8B, or 8B2 in their product codes have standard PCI faceplates, which are ~120.5mm high. NICs with 8AL, 8BL, or 8B2L in their product codes have low-profile PCI faceplates, which are ~80mm high. If you compare the photos below, you'll see that the cards are the same but that they have different PCI faceplates.

Low-profile PCI Express add-in card with a standard PCI faceplate	Low-profile PCI Express add-in card with a low-profile PCI faceplate

10G-PCIE-8B-2S	10G-PCIE-8BL-2S

It is not possible to include two CX4 connectors on a low-profile PCI Express add-in card without violating the PCI Express Card Electromechanical Specifications; thus, Myri-10G NICs with two CX4 ports are available from the standard product list only as standard-height PCI Express add-in cards.

As noted above, Myricom also produces Myri-10G NICs in other form factors, including High Speed Expansion Cards (HSECs) for the IBM BladeCenter H. See the full Myri-10G product list for additional information.

Ethernet interoperability has been demonstrated with 10-Gigabit Ethernet switches from Myricom, Brocade (Foundry), Extreme, Fujitsu, HP, Quadrics, SMC, Force10, Cisco, Blade Network, Broadcom, and Arista.

PCI Express interoperability has been verified with all of the common PCI Express chipsets and many commercially available motherboards. This tabulation of test results with "Gen1" (2.5 GT/s) PCI Express chip sets and motherboards shows their PCI Express performance, which can vary significantly between different products. Myri-10G "Gen1" PCI Express NICs are compatible with "Gen2" slots in hosts; the NIC auto-negotiates operation in the widest available mode (x8, x4, x2, or x1) supported by the slot it is plugged into, and at the 2.5 GT/s data rate.

Myri-10G "Gen2" PCI Express NICs similarly auto-negotiate operation in the widest available mode (x8, x4, x2, or x1) supported by the slot it is plugged into, and at the highest data rate (5 or 2.5 GT/s), but these "Gen2" PCI Express NICs cannot achieve full performance in "Gen1" PCI Express slots in a host. This tabulation of test results with "Gen2" (5 GT/s) PCI Express chip sets and motherboards shows the PCI Express performance of the dominant "Gen2" PCI Express host today, the Intel Xeon X5500 series (Nehalem), whose "Gen2" PCI Express x8 slots have nearly 50 Gb/s bidirectional throughput after PCI Express protocol overhead.

The power of programmable NICs. Myri-10G NICs include processors and firmware, and can be used in system solutions that go beyond the standard TCP/IP and UDP/IP over Ethernet. These same NICs can be used with any of the following optional software distributions:

MX (Myrinet Express) software for kernel-bypass operation for low latency and very low host-CPU load over either Ethernet or Myrinet networks. This software is used principally for HPC applications. See this Low-Latency 10-Gigabit Ethernet product brief (pdf).
Video Pump^TM for low-CPU-load UDP streaming for IPTV applications.
Sniffer10G for wire-speed 10-Gigabit Ethernet packet capture.
DBL (Datagram Bypass Layer) for low-latency UDP/IP (pdf) communication. This software is used principally for financial-trading applications.

These optional software distributions may be customized for individual customers, and other optional software is in development for other applications. Contact the Myricom Sales Team for additional information, which must in some cases be under an NDA.

See also these product briefs:

Last updated: 31 January 2010