README - Myricom 10GbE driver for MacOSX

This Myricom 10GbE driver for Myri-10G NICs is intended for use only with MacOSX kernel version 10.4 or later.  The driver is distributed as a Universal binary.

Installation
============

To install the driver, extract the Myri10GE.pkg.tgz file, and click on the resulting Myri10ge.pkg.  The installer will copy the driver to /System/Library/Extensions/Myri10GE.kext.

After rebooting, you should see a new ethernet adaptor. Configure it via System Preferences -> Network -> Show -> Ethernet Adaptor (enX)

Note that "enX" in this document refers to the BSD ethernet name which is automatically assigned to the Myri10GE ethernet adaptor by MacOSX. This will typically be "en2", but may differ on your system depending on the number of ethernet adaptors you have.

Performance Tuning
==================

The driver supports TCP Large Receive Offload (LRO), however it is disabled by default as it may interfere with packet forwarding and 3rd party network interface  filters.   LRO has been shown to roughly double standard frame receive performance.   You may enable LRO at runtime via sysctl.  Open a terminal window, and type:

    sudo sysctl -w net.myri10ge.enX.lro_cnt=$VALUE

Where $VALUE is the number of simultaneous LRO frames which can be constructed in parallel.  The suggested value is 8.

You may adjust the interrupt coalescing parameter at runtime via sysctl.  Open a terminal window, and type:

    sudo sysctl -w net.myri10ge.enX.intr_coal_delay=$VALUE

The default setting is a compromise between latency and cpu overhead. You may wish to reduce intr_coal_delay if latency is more important and you are using a low-latency switch or a point-to-point connection. Similarly, you may wish to increase intr_coal_delay if you are interested in reducing CPU overhead for large transfers.  Note that intr_coal_delay controls both transmit and receive coalescing.

To make your changes permanent, edit /System/Library/Extensions/Myri10GE.kext/Contents/Info.plist and change the corresponding value there.  For example, to make
intr_coal_delay=50 the default, you would change the key/value pair intr_coal_delay to 50:

                                <key>intr_coal_delay</key>
                                <integer>50</integer>

For better TCP performance, it is necessary to increase the TCP window size beyond the default value.  In order to do that, one must increase the maximum socket buffer size via sysctl: 

    sudo sysctl -w kern.ipc.maxsockbuf=2097152

To make this change permanent, edit (or create) the file /etc/sysctl.conf with this line in it:
    kern.ipc.maxsockbuf=2097152

On MacOSX, as with most BSD based stacks, restricting the TCP maximum segment size (MSS) to an even multiple of the mbuf cluster size keeps things nicely aligned, and results in improved performance when using jumbo frames.  Unfortunately, the MacOSX TCP stack does not do this (Apple Bug Id #4919145), and the only way to do this is by adjusting the interface MTU by hand.  The most common TCP packets will have an rfc1323 timestamp option, making for a header size of 52 bytes. Therefore, setting the MTU to 8192 + 52 (= 8244) results in optimal performance.  To set the MTU, go into the Network preferences pane, configure the Myri10GE adaptor, and select the Ethernet pane.  Choose "Advanced".  This allows you to set a custom MTU of 8244.

Troubleshooting
===============

The ability to saturate a 10GbE link depends on having sufficient PCI-Express bandwidth. When loaded, our driver calculates the available bus bandwidth (read DMA, write DMA, and simultaneous read and write DMA) and stores it so that sysctl may retrieve it later. To view your bus bandwidth, use the following command:

   # sysctl net.myri10ge | grep dma

Note that the reported bandwidth is measured in megabytes per second, not megabits.  This means that 10Gb/s corresponds to 1280MB/s.