Supererogatory Penguin Image

Myrinet GM-2 Software for Linux

GM-2 Software Available

Linux Kernel	Processor
	IA32	IA64	AMD64	EM64T	Alpha	PowerPC	PowerPC64	Power4	Power5
2.4
2.6

Note:

• GM-2 does not support Xen or VMware virtualization software.

To download Myrinet software, you need a login and password.
If you don't already have these, please request them.

Source | Myrinet roll for Rocks

The Linux GM-2 software is distributed only in source form. Which GM version you use may depend on which Myrinet NICs are used in your hosts. GM rolls for Rocks are also available.

How to use the table below: Generally, all hosts on a single Myrinet network must run the same version of GM. If you have multiple types of NICs on your network, you'll need to use the higher GM version, e.g., if you have PCI64 and PCIXD NICs, the preferred GM version is 2.0.x. If there are multiple versions in a box in the table, they are listed in order of usual preference. Additional information about the version choices is below the table.

* For Linux 2.6.2{1,2,3,4,5} support, you will need the following GM patch for GM-2.0.28. This patch is already included in GM-2.1.29.

* For Linux 2.6.2{6,7,8} support, you will need to apply two patches to GM-2.0.28, gm-2.x.28_linux-2.6.25.patch and gm-Linux-2.6.28.patch3. For GM-2.1.29, you only need to apply the second patch, gm-Linux-2.6.28.patch3.

* For IA64 support with SLES10, you may need two patches as described in this FAQ entry.

GM-2 does not support Xen or VMware virtualization software.

Choices: GM-2.0.x and GM-2.1.x are compatible at the API level, but employ different protocols in the network, so for interoperability you should use the same GM version on all hosts in a Myrinet cluster.

• GM 2.0.x is your best choice for clusters with PCIXF "F card" or PCIXD "D card" NICs, and can also be used with PCI64-series NICs. GM 2.0.9 and later can be used with the dual-port PCIXE "E cards" if you use only interface port 0.
• GM 2.1.x, an extension of GM 2.0.x, is required to use both ports of PCIXE "E card" NICs, but can also be used with PCIXD "D card" and PCI64-series NICs. GM 2.1.x has an advanced feature of multipath, dispersive routing, which helps avoid "hot spots" in large networks. However, GM 2.1.x has slightly higher latency than GM 2.0.x due to managing multiple routes. The reason why GM 2.1.x and GM 2.0.x cannot interoperate is that GM 2.1.x allows for the out-of-order packets that may occur with multiple interface ports and multiple routes, and GM 2.0.x cannot handle out-of-order packets.

Other version considerations:

• GM 2.0.18 or later is required for use with PCIXF "F card" NICs.
• The Myrinet-2000 Switches for Large Clusters require the use of GM 2.0.15 or later, or GM 2.1.5 or later.
• GM 2.0.11 or later, or GM 2.1.1 or later, are required for ethernet-emulation interoperability with M3-SW16-8E Gigabit-ethernet switch line cards.
• Mapping changes were introduced starting with GM 2.0.11 and GM 2.1.1, such that these versions may not be compatible in mapping with earlier GM 2 versions.
• GM 2.0.11 and GM 2.1.1 are also the first GM releases to operate with both the 2.4 and 2.6 Linux kernels.
• Linux 2.2 and earlier are not supported in GM-2.
• PCI32 NICs are NOT supported in GM-2.0 and higher.
• If you will be running MPI applications over GM-2, you must use MPICH-GM 1.2.5..10 or later.

GM-1 versus GM-2: GM-2 software is required for use with the PCIX-series NICs and for ethernet-emulation interoperability with M3-SW16-8E Gigabit-ethernet switch line cards. GM 2 provides a number of features not available in GM 1, including fully distributed (active) mapping. Although GM-2 was backported to support the PCI64-series NICs for interoperability between successive generations of NICs, it is not as well optimized for performance on PCI64-series NICs as GM 1.6.x. Thus, we do not recommend at this time upgrading clusters using only PCI64-series NICs to use GM-2 software unless you have need for the new features in GM 2.

NOTE:The installation procedure for GM has changed slightly between gm-1.6.4 and gm-2.0 and later. Refer to the abbreviated installation instructions below. Automated installation scripts will need to be updated!

Abbreviated Installation Instructions

To compile GM on Linux platforms, you will only need GNU make and C/C++ compilers. Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions may also be addressed in the FAQ.

These abbreviated instructions presume that you are installing GM 2.0. (Here is the README-linux for GM 2.0.)

Abbreviated instructions for installing GM 1.6.4 are available.
Abbreviated instructions for installing GM 1.5.2.1 are available.

Current cautions and common problems:

• PCI32 NICs are NOT supported in gm-2.0 or later.
• GM is not in the critical performance path so it does not need to be built with specialized compilers and flags. GM should be built with gcc and only built with -O level of optimization.
• You must configure/compile/load GM on a Linux box whose running kernel is configured to match the source kernel tree. Note that some Linux distributions (e.g., SuSE, RedHat, and Fedora) ship a mismatched source kernel tree.
• If you're using SuSE SLES8 with GM-2.0.13 or later, refer to the following FAQ entry.
• When building/linking GM applications, you must do so on a Linux box that matches the OS version of the machine on which you will be running. For instance you cannot compile on a glibc-2.2 based OS and run the executable on a glibc-2.1 based one.
• If you're using a Linux kernel using the first draft of the "Fix get_user_pages() race for write access"/hptc-linux-2.6.9-get_user_pages_race.patch, which was proposed between 2.6.12 and 2.6.13, you will need the following GM patch for fork() support.
• GM_INSTALL fails? Refer to this FAQ entry for troubleshooting.
• AGP (nVidia and ATI) conflicts? Refer to the "Caveats" section of the README-linux.
• APIC IRQ conflicts? Refer to the "Caveats" section of the README-linux.
• If you're using GM-2.0.21/GM-2.1.21 or earlier on PowerPC64 (ppc64), refer to the following FAQ entry for an important patch.
• If you are using GM-2.0.19 (or GM-2.1.9) or earlier on IA64, refer to the following FAQ entry.
• If you are installing GM-2 on AMD64 or EM64T, refer to the following FAQ entry.
• If you are installing GM-2 on PowerPC64 (or Power4 or Power5), refer to the following FAQ entry.
• If you are using an Apple G4 Xserve mother board, please read this FAQ entry for important performance information.
• Do not use the configure flag --enable-directcopy with GM 2.0. This flag is not a valid option to GM 2.0. It will be re-enabled in a future release.
• If the host is rebooted, you must reload the GM driver.
• GM-2 does not interoperate with GM-1.
• If you will be running MPI applications over GM-2, you must use MPICH-GM 1.2.5..10 or later.

GM installation is performed in four easy steps:

1. Configuring and compiling GM.

        gunzip -c gm-2.0.28_Linux.tar.gz | tar xvf -
        cd gm-2.0.28_Linux
        ./configure
        make

By default, we assume that the kernel header files for your Linux installation are located in /lib/modules/`uname -r`/build. If your Linux installation is not located in /lib/modules/`uname -r`/build, you must configure with the following option:

 
        ./configure --with-linux=<linux-source-dir>

where <linux-source-dir> specifies the directory for the Linux kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options.

Note:

• For detailed installation instructions for GM with FMS diagnostic monitoring, refer to the FMS webpage.
• Prior to GM-2.0.15, we assumed that the default location for the Linux header files was in /usr/src/linux/.

2. Installing the GM driver.

Select an installation directory path <install_path>. It is usually best for <install_path> to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using <install_path> on all machines that are to share the installation. <install_path> must be an absolute path; it must start with /. However, <install_path> may contain symbolic links.

        cd binary
        ./GM_INSTALL <install_path>

If you omit <install_path>, the driver will be installed in the default directory, /opt/gm/.

Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /etc/init.d/gm start
        echo <install_path>/lib64 >> /etc/ld.so.conf
        echo <install_path>/lib >> /etc/ld.so.conf && /sbin/ldconfig

on each machine. The ldconfig line is optional, and adds the GM library directory to the system library search path. If you do not do this, individual users will have to either manage their LD_LIBRARY_PATH environment variable or link their programs with an -rpath option for the dynamic linker to locate the GM shared library.

Details of these installation scripts can be found in the README-linux. When the GM module is loaded, a mapper daemon called gm_mapper is started for each Myrinet NIC contained in the machine. The PIDs of the running gm_mappers are stored in /var/run/gm_mapper/pid.<board_id>, and the map files are located in /var/run/gm_mapper/map.<board_id>.

Important notes:

• For AMD64, EM64T, and PowerPC64 systems, the 32-bit GM libraries are installed in <install_path>/lib and the 64-bit GM libraries are installed in <install_path>/lib64. Both 32-bit and 64-bit GM libraries are built by default on AMD64 and EM64T systems. On architectures that only support 64-bit user spaces, like IA64, the 64-bit GM libraries are installed in <install_path>/lib.
• Stopping the mapper while GM is running is not supported. The gm_mapper should be left running at all times, and it will not interfere with the performance of jobs running over Myrinet.
• The installation scripts do not configure the IP device. If you wish to run IP over GM/Myrinet (ethernet emulation), you must configure the device. Refer to step 3. However, if you wish to stop the GM driver, IP over Myrinet will be shut down as part of the gm stop process.
• Running /etc/init.d/gm stop kills all of the gm_mapper daemons, shuts down IP over Myrinet, and unloads the gm module.

If you wish for the driver to auto-load at boot, you must create appropriate links in the /etc/rcN directories to the /etc/init.d/gm and /etc/init.d/myri scripts. Alternatively, you may start and stop the drivers manually using

 
        su root
        /etc/init.d/gm start
        /etc/init.d/gm stop

or

 
        su root
        /etc/init.d/gm restart

to start, stop, or restart the driver, respectively.

For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section of the README-linux.

3. Enabling IP over Myrinet (Ethernet emulation) (OPTIONAL)

If you wish to run IP over Myrinet (ethernet emulation), the Linux command to enable IP over GM is as follows:

   /sbin/ifconfig myri0 <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

4. Testing the GM Installation.

Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests.

• Check the LEDs on each switch port and NIC port
• Run gm_board_info on one host
• Run gm_debug to test the PCI bandwidth
• Run gm_allsize to test the links in the network
• Run gm_stress to test the network

Each of these steps is detailed in the Troubleshooting section of the FAQ.

The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are available in <install_path>/bin in your GM installation. A README describing each of these tests can be found in <install_path>/bin/README. Mute is not included in the GM distribution, but can be downloaded from the SCS webpage.

Congratulations! You have successfully installed GM-2.

Refer to the Software and Documentation Page for details of GM-based software (MPICH-GM, VI-GM, PVM-GM, Sockets-GM). Currently only MPICH-GM, PVM-GM, and Sockets-GM can be used with GM-2. All other middleware requires GM-1. If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-linux and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.

Note: Each Myrinet roll contains GM-2.1.28 (or GM-2.0.28), MPICH-GM 1.2.7p1..20, and Open MPI v1.2.6. Installation instructions are available on the Myrinet FAQ and the Rocks homepage.

Last updated: 13 March 2009