Product Announcement
14 October 2003
Product
Announcement
14 October
2003
High-End, Two-Port, Myrinet/PCI-X Interface
M3F2-PCIXE-2 Two-Port Myrinet-Fiber/PCI-X Interface
Impressive performance. The M3F2-PCIXE-2 "E card" Myrinet/PCI-X interface pictured above has dual Myrinet-fiber ports for double the throughput performance and for high availability. Also, in comparison with the low-cost, single-port, "D-card," Myrinet/PCI-X interface, the E card operates at a 48% higher internal clock rate, resulting in faster firmware execution for lower user-level latency.
The GM-2 or MX software support for this high-end E-card interface presents exactly the same application-programming interfaces (APIs) as for the popular D-card interface. The firmware executing in the Lanai 2XP distributes packets adaptively across the two ports in sending, and reassembles messages in receiving, so that the two 250+250 MB/s Myrinet ports act as a single 500+500 MB/s port. High-availability operation is instantaneous: If you remove the fiber cable from either port, communication will continue using the remaining port.
In the data-rearrangement phases of distributed computations, hosts send and receive messages simultaneously. Thus, the sustained bidirectional data rate is a critical metric. Microbenchmarks with developmental versions of GM 2.1 on dual-Xeon hosts show:
| User-level sustained bidirectional data rate |
| 950 MBytes/s |
The E card is nearly as fast in sustained bidirectional data rate as any PCI-X NIC can be, and is faster in this microbenchmark than all competing products. The peak PCI-X and bidirectional link data rates are balanced, and the E card provides user-level performance that closely approaches the peak data rates. The PCI-X DMA performance of the dual-Xeon benchmark host, although excellent, limits equal read and write traffic across the PCI-X bus to a total of 985MB/s (an explanation is here). The user-level bidirectional data rate reaches 96% of the PCI-X peak, and 95% of the 1GB/s send+receive peak of the two Myrinet links.
The user-level sustained one-way data rate is 495MB/s, 99% of the aggregate 500MB/s one-way data rate of two links.
The short-message latency is an important metric for many distributed computations.
User-level short-message latency | |
| GM-2 | 5µs |
| MPICH-GM over GM-2 | 5.5µs |
| MX | 3.5µs |
| MPICH-MX over MX | 3.5µs |
When an E card has only one port connected, the throughput performance is similar to that of the one-port D card, but with these smaller user-level latencies.
Myricom software support for the E cards requires GM 2.1 or later, or MX. In addition to supporting the E cards, GM 2.1 and MX provide multi-path dispersive routing in the Myrinet switch fabric, a technique that diffuses "hot spots" and increases the utilization of the network bisection. The release of the GM 2.1 software is scheduled for the end of October, and the first release of MX will occur later this year. Myricom will be demonstrating a cluster running MX at SC2003 in mid-November.
The 64-bit, 133MHz, PCI-X implementation in the E card (Lanai-2XP chip) is the same, superlative, silicon implementation as that in the D card (Lanai-XP chip). A summary of the PCI-X DMA performance of the PCIX-series interfaces in many of today's best cluster hosts is shown in this table.
The E cards can be used with separate, independent, Myrinet fabrics (a "dual-rail network"), or with a single Myrinet fabric. The Myrinet mapping in GM 2.1 and MX provides multiple routes from each port to each other reachable port, and maintains the association of ports to interfaces.
Note: The performance measurements cited above were made at Myricom on software-development clusters that use Supermicro X5DL8-GG dual 2.4GHz Xeon hosts with a 533MHz FSB and the Serverworks GC-LE chip set. The software was developmental versions of GM 2.1 and MX. The performance observed on other hosts or with release versions of the software may vary. Performance is expected to improve slightly over time as these software packages become better optimized.
Price and Availability
The M3F2-PCIXE-2 interface is being introduced at a list price of $1,195, and will be available in volume by the end of October. See this page for specifications and other documentation.
14 October 2003