MX-2G Software
Downloads and Installation Instructions

Home | Mail for Product Information | Documentation | Software Downloads |
Switch Software | Diagnostic Tools | Other Documentation and Tools |
Technical Support | RMA Procedures

To download Myrinet software, you need a login and password.
If you don't already have these, please request them.

Source | Binary | Myrinet roll for Rocks

MX-2G Software

MX-2G (Myrinet Express for Myrinet-2000 PCI-X NICs) is a low-level message-passing system for Myrinet networks. The MX software includes the MX library, drivers, Myrinet-NIC control program, and a network mapping program. MX-2G is available for Myrinet-2000 PCI-X NICs. For Myri-10G PCI-Express NICs, refer to the MX-10G Software Download web page.

MX-2G provides kernel-bypass operation for low latency and low host-CPU utilization over Myrinet-2000 networks. In addition to MX over Myrinet (MXoM), MX-2G includes IP communication (TCP/IP and UDP/IP) over Myrinet (IPoM).

For application or middleware developers who need to develop code using the MX API, refer to the MX API manual. MX-2G for Myrinet-2000 PCI-X NICs and MX-10G for Myri-10G PCI-Express NICs are fully compatible at the API and application levels, and use the same MPI and Sockets middleware.

Complete installation instructions and platform-specific information can be found in the README in the top-level directory of the source distribution.

Source code is available for all distributions. However,

• MX-2G software for Windows is not WHQL-certified but is available pre-compiled in binary form. To install on Windows Vista or Windows Server 2008 (Longhorn), refer to this FAQ entry.
• MX-2G software for Linux, Solaris, Mac OS X, and FreeBSD is distributed only in source form.
• An MX-2G roll for Rocks v4.3 is available.

* Note for recent Linux versions: we try to support newer Linux versions within a few weeks of their official release. We also encourage customers to report any problem encountered with development Linux versions or release candidates.

Note:

• Legacy PCI64-based and PCI32-series NICs are not supported.
• MX is supported on the following Linux distributions: RHEL kernels ≥ RHEL3, SuSE kernels ≥ SuSE-8.0, and any kernel based on a kernel.org version ≥ 2.4.23. We provide support for other kernels on a per-request basis. Contact help@myri.com for details.
• If you would like to compile MX on AMD64 running a Solaris 10 release after s10u1_18, you must apply patches 118344, and 118844. For patch access, refer to the SunSolve Patch Access web page.

Here is a summary of the CHANGES between MX versions.

Abbreviated Installation Instructions

To compile MX on Linux platforms, you will need GNU make and C/C++ compilers. Please be sure to read the README file for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions may also be addressed in the FAQ.

Customer compilation of MX for Windows is not recommended due to the difficulties involved. Users should be comfortable with the Cygwin environment. Contact help@myri.com for instructions.

Current cautions and common problems:

• Legacy PCI32-based and PCI64-based Myrinet NICs are not supported.
• MX must be compiled with the system compiler (gcc for Linux, Solaris/AMD64, Mac OS X, and FreeBSD). For Solaris/UltraSPARC, MX must be compiled with the SunPRO compiler. We do not support third-party compilers.
• If you would like to compile MX on AMD64 running a Solaris 10 release after s10u1_18, you must apply patches 118344, and 118844. For patch access, refer to the SunSolve Patch Access web page.
• If you are installing MX on Linux, you must configure/compile/load MX on a Linux box whose running kernel is configured to match the source kernel tree. Note that some Linux distributions (e.g., SuSE, RedHat, and Fedora) ship a mismatched source kernel tree.
• For optimal performance, write-combining must be enabled on the PCI chipset of the host. The MX driver will automatically attempt to enable write-combining. If MX is unable to enable write-combining at load time, an error message will appear in the kernel log. Refer to the README for details.
• For application or middleware developers who need to develop code using the MX API, refer to the MX API manual.
• If the host is rebooted, you must reload the MX drivers.
• MX interoperates with GM-2 for mapping and ethernet emulation.

MX installation is performed in four easy steps:

1. Configuring and compiling MX.

	     $ gunzip -c mx2g_1.2.5.tar.gz | tar xvf -
	     $ cd mx-1.2.5
	     $ ./configure
	     $ make

By default, we assume that the header and config files of your Linux kernel (required to compile outside modules and either part of a kernel-headers or kernel-source package depending on your distribution) are pointed by /lib/modules/`uname -r`/{source,build}. If your Linux installation is not standard, or you are cross-compiling for a kernel different from the one of the compile node you must configure with the following option:

 
	     $ ./configure --with-linux=<linux-source-dir>

where <linux-source-dir> specifies the directory for the Linux kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options.

Note:

• For Linux 2.6 kernels, the kernel headers/scripts often come in two parts in two different directories, you might need to use both --with-linux and --with-linux-build. For instance to select a specific kernel, you might need something like:

  	     $ ./configure --with-linux=/usr/src/linux-2.6.5-7.151/ \
  		   --with-linux-build=/usr/src/linux-2.6.5-7.151-obj/x86_64/smp/
  	

• By default, the mapper in MX is provided by the Fabric Management System (FMS).

  If you would like to use the diagnostic capabilities of FMS, you need to specify the name of the FMS server (the node on which the fms process will be run) at configure time, using --with-fms-server=<fms_server>.

  	     $ ./configure --with-fms-server=<fms_server>
  	

  To defer this specification until install time, or to override it, you may install MX with make install FMS_SERVER=<fms_server>.

  For detailed installation instructions for MX with FMS diagnostic monitoring, refer to the FMS webpage.

• If you would like to use the previous mapper, mx_mapper, you need to configure with the option --disable-fms.

2. Installing the MX mcp and driver.

Select an installation directory path <install_path>. It is usually best for <install_path> to be the path to an NFS directory available on all machines that are to share this MX installation. The directory must be accessible using <install_path> on all machines that are to share the installation. <install_path> must be an absolute path; it must start with /. However, <install_path> may contain symbolic links.

	     $ make install DESTDIR=<install_path>

If you omit DESTDIR=<install_path>, the mcp and driver will be installed in the directory specified with the configure --prefix option, or the default directory, /opt/mx/. The MX binaries are located in <install_path>/bin and <install_path>/sbin. The 32-bit MX libraries are installed in <install_path>/lib32 and the 64-bit MX libraries are installed in <install_path>/lib64. The <install_path>/lib directory is a symbolic link to either lib64 or lib32 depending on the native wordsize detected by configure. E.g., on most ppc64 distributions, gcc defaults to 32-bit, which means that lib links to lib32. However, on most x86_64 distributions, gcc defaults to 64-bit, so lib links to lib64.

Unless specified on the configure line, MX builds 32-bit libraries on 32-bit architectures (i386, ppc) and 64-bit libraries on 64-bit architectures (ia64, AMD64, ppc64, Alpha). It is possible to build both by using the --enable-32b and --enable-64b configure flags.

For Mac OS X, when the Apple Xcode compiler is 64-bit, MX fat libraries are built which are usable by both 32-bit and 64-bit applications, and libraries are always installed in <install_path>/lib.

For Linux and FreeBSD, add the MX library directory to the system library search path. Otherwise, individual users will have to either manage their LD_LIBRARY_PATH environment variable or link their program with an -rpath option for the dynamic linker to locate the MX shared library.

Next, you must run

	     $ su root
	     $ <install_path>/sbin/mx_local_install
	     $ <install_path>/sbin/mx_start_stop start
	  

on each machine to perform local install steps, to load the modules, and to start a mapper for each Myrinet NIC contained in the machine. If applicable, the mx_start_stop script is also available in /etc/init.d/mx.

Important notes:

• For optimal performance of MX on i386 and x86_64 hosts, write-combining must be enabled on the PCI chipset of the host. MX will enable write combining on i386 and x86_64 hosts when there are no conflicting attribute regions for physical or PCI memory pre-existing at driver load time. If MX is unable to enable write-combining at load time, an error message will appear in the kernel log. Refer to the README for details.
• Details of the installation scripts can be found in the README.
• The MX software provides a separate kernel module for the mcp (firmware) and the driver. If you do not use the mx_start_stop script to load the MX drivers and start the MX mapper, you must ensure that the mcp module is loaded first, as the driver module depends on it. The MX driver and mcp contain a number of tunable parameters which may be adjusted by the user when the MX driver is loaded. These parameters are set in an OS-dependent manner. Refer to the README for details.
• For Linux and FreeBSD, add the MX library directory to the system library search path. Otherwise, individual users will have to either manage the LD_LIBRARY_PATH environment variable or link the program with an -rpath option for the dynamic linker to locate the MX shared library.
• The 32-bit MX libraries are installed in <install_path>/lib32 and the 64-bit MX libraries are installed in <install_path>/lib64. The <install_path>/lib directory is a symbolic link to either lib64 or lib32 depending on the native wordsize detected by configure. E.g., on most ppc64 distributions, gcc defaults to 32-bit, which means that lib links to lib32. However, on most x86_64 distributions, gcc defaults to 64-bit, so lib links to lib64.
• Unless specified on the configure line, MX builds 32-bit libraries on 32-bit architectures (i386, ppc) and 64-bit libraries on 64-bit architectures (ia64, AMD64, ppc64, Alpha). It is possible to build both by using the --enable-32b and --enable-64b configure flags.
• For Mac OS X, when the Apple Xcode compiler is 64-bit, MX fat libraries are built which are usable by both 32-bit and 64-bit applications, and libraries are always installed in <install_path>/lib.
• The installation scripts do not configure the IP device. If you wish to run IP over MX/Myrinet (ethernet emulation), you must configure the device. Refer to step 3.

3. Enabling IPoM (OPTIONAL)

Instructions for Linux and FreeBSD:

If you wish to run IP communication (TCP/IP and UDP/IP) over Myrinet, the command to enable IPoM is as follows:

	   /sbin/ifconfig myri0 <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

Instructions for Solaris:

If you wish to run IPoM, the command to enable IPoM is as follows:

	   ifconfig myri0 plumb <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

• Note for Solaris 10GA: Due to a bug, Solaris 10GA does not support 9000 byte (jumbo) frames as released. In order to obtain full ethernet performance, jumbo frames are critical, so we encourage you to apply Sun's patch 119832-01 for UltraSPARC or patch 119833-01 for AMD64. For patch access, refer to the SunSolve Patch Access web page. If you do not apply this patch, you will need to set the Ethernet MTU to 1500 bytes in /kernel/drv/myri.conf, and specify mx_mtu_override=1500 when loading MX. Refer to the README for details. No jumbo frames will be allocated until after the myri0 interface is plumbed.

Instructions for Mac OS X:

You should configure the MX ethernet emulation interface as you would any other ethernet interface. On most systems, the MX ethernet adaptor will appear as en1. It is possible, if you have additional network cards, that the adaptor will appear as en2, en3, etc.

To verify which ethernet adaptor belongs to MX, you may need to run the Network Utility /Applications/Utilities/Network Utility. Click the Info tab and select Network Interface from the menu until you find the one whose Vendor is Myricom, and whose Hardware Address matches the MAC address printed by mx_info.

Once you have found the correct adaptor, configure it via:

	  System Preferences -> Network -> Show -> Ethernet Adaptor (enX)
	  

4. Testing the MX Installation.

Once the MXoM firmware is running on all of the hosts, and all host-to-switch and switch-to-switch cables have been connected, you should verify that all hosts are mapped in the network (via the mx_info command) before launching jobs on the network. We also recommend that you check the health of all of the Myricom hardware components in the Myrinet installation as instructed in the Troubleshooting section of the FAQ.

Congratulations! You have successfully installed MX.

Refer to the Software and Documentation Page for details of MX-based middleware (MPICH-MX, Sockets-MX). If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.

Binary Installation Instructions

To install on Windows Vista or Windows Server 2008 (Longhorn), refer to this FAQ entry.

The binary Windows driver can be installed on the command line or via the graphical interface. The instructions for installing the driver on the command line can be found in the README. The user interface for installing a device has changed with each new release of Windows. Steps for each OS are given below. We suggest booting into safe mode and uninstalling old drivers before reinstalling new ones.

• Windows 2000 installation instructions
• Windows XP installation instructions

Note: Each Myrinet roll contains MX-2G 1.2.2, MPICH-MX 1.2.7..5, and HPL. Installation instructions are available on the Myrinet FAQ and the Rocks homepage.

Last updated: 01 April 2008

Home | Mail for Product Information | Documentation | Software Downloads |
Switch Software | Diagnostic Tools | Other Documentation and Tools |
Technical Support | RMA Procedures