README - Myricom 10GbE driver for Solaris

This Myricom 10GbE driver for Myri-10G NICs is intended for use only
with Solaris 10 Update 1 (1/06) or later.  It is currently available
only for the AMD64 and Sparc64 platforms.

Installation
============

To install the driver, type

   % su root
   # pkgadd -d .


Select the Myri10GE driver from the menu, and 
continue with the installation.
    
Now you have a new interface, myri10ge0, which you
can treat like any other ethernet interface.

Removal
============

To remove the driver, type:

   % su root
   #  pkgrm Myri10GE

Answer yes, and the driver will be removed.



Performance Tuning
==================

The driver will use a 9000 byte "jumbo" mtu by default.
To force a smaller mtu (and less memory allocation for
receive buffers) you must set myri10ge_mtu_override
to a value between 9000 and 1500 via the driver configuration
file (/kernel/drv/myri10ge.conf).

You may adjust the interrupt coalescing timer at runtime, using
ndd.  For example, to set the delay to 75us you would do:

    ndd -set /dev/myri10ge0 myri10ge_intr_coal_delay 75

This change will be lost at reboot.  You can make this permanent by
editing /kernel/drv/myri10ge.conf, uncommenting and changing
myri10ge_intr_coal_delay.

The default setting is a compromise between latency and cpu overhead.
You may wish to reduce myri10ge_intr_coal_delay if latency is more
important and you are using a low-latency switch or a point-to-point
connection.  Similarly, you may wish to increase
myri10ge_intr_coal_delay if you are interested in reducing CPU
overhead for large transfers.  Note that myri10ge_intr_coal_delay
controls both transmit and receive coalescing.

The driver will attempt to use MSI interrupts on amd64, but not on
sparc64 (see Caveats).  For optimal performance, we suggest you use
MSI interrupts by setting myri10ge_use_msi=1; in
/kernel/drv/myri10ge.conf.

Opteron systems which cannot use MSI interrupts may benefit from
disabling myri10ge_deassert_wait in the driver configuration file or
via ndd.  This allows the interrupt handler to avoid waiting for
confirmation the interrupt line is low before exiting the interrupt
handler and may increase performance.  If you see evidence of spurious
interrupts, this should be disabled.

The interrupt handler will optionally hash different incoming TCP
streams into several taskqs for processing in another context.  The
myri10ge_use_rx_taskq option should be enabled when the NIC is faster
than the CPU, such as on CoolThreads-based UltraSparcs.  This option
is enabled by default only on UltraSparcs.

Caveats
======= 

Solaris requires network interfaces to be plumbed to appear in
the output of ifconfig:

   # ifconfig myri10ge0 plumb

The ability to saturate a 10GbE link depends on having sufficient
PCI-Express bandwidth. When loaded, our driver calculates the
available bus bandwidth (read DMA, write DMA, and simultaneous read
and write DMA) and stores it so that kstat may retrieve it later. To
view your bus bandwidth, use the following command:

   # kstat myri10ge | grep dma

Note that the reported bandwidth is measured in megabytes per second,
not megabits.  This means that 10Gb/s corresponds to 1280MB/s.

Some Sunfire servers will generate "erroneous faults" if you remove or
move a PCI card.  Furthermore, it will generate the fault on every
boot until you clear the fault.

If you see faults in the output of "fmdump -v", do the following:

    pkgrm Myri10GE

    Clear the errors as described in
    "Sun Fire T2000 Server Product Notes", in the chapter
    "Recognizing Erroneous Error Messages".  (Skipping step 4 ("verify
    that there are no faults") will probably save you a lot of time.)
    Make sure you "fmadm repair" all errors listed in "fmadm faulty",
    else the system may re-report the errors on reboot.

    Remove any lines mentioning myri10ge from /etc/path_to_inst

    Execute "reboot -- -r"

    reinstall Myri10GE.

If "fmdump -v" shows a fault.io.sunfire.pci.dev fault, it is related
to a legacy PCI or PCI-X slot, not a PCI Express slot.

Detailed explanations of faults are available at, e.g.,
http://www.sun.com/msg/SUN4-8000-ER.  The useful bit is under "Details".

MSI interrupts are disabled by default on the sparc64 architecture due
a Solaris bug: "6343017 ddi_intr_remove_handler() returns DDI_FAILURE
for MSI interrupts."  This bug prevents our driver from reloading on
sparc64, however it does not affect amd64.  Until this is fixed by
Sun, you will be unable to reload our driver if you use MSI interrupts
on sparc64. This will prevent you from upgrading the driver without
doing a reboot on sparc64.