Supererogatory Penguin Image

Myrinet GM-1 Software for Linux

GM-1 Software Available

Linux Kernel	Processor
	IA32	IA64	Alpha	PowerPC
2.4
2.2
2.0

Note:

• GM-1 does not support Xen or VMware virtualization software.
• GM-1 does not support hosts based on PowerPC64, AMD64, and EM64T processors, and PCI64-series Myrinet NICs are not qualified for use on these hosts. If you wish to use hosts with these processors, you must use GM-2 with PCIX-series Myrinet NICs.
• GM-1 does not support Linux 2.6. If you wish to use Linux 2.6, you must use GM-2.
• GM-1 does not support the new M3-CLOS and M3-SPINE Myrinet-2000 switches.

To download Myrinet software, you need a login and password.
If you don't already have these, please request them.

The Linux GM software is distributed only in source form. Which GM version you use may depend on which Myrinet NICs are used in your hosts.

Linux 2.4 requires GM 1.5 or higher. Certain of the legacy PCI32 NICs cannot run GM 1.5.x for lack of sufficient memory, thus, cannot be used with Linux 2.4.

• GM-1 does not support Xen or VMware virtualization software.
• PCI32 NICs are NOT supported in GM 1.6 and higher.
• PCI32 NICs are not supported on IA64.

Important note: GM-1.5.2 or later is required for PCI64C NICs running on machines with 33MHz PCI buses!

VERY IMPORTANT NOTE: The installation procedure for GM has changed significantly between GM-1.5.2.1 and GM-1.6.3 and later. Refer to the abbreviated installation instructions below. Automated installation scripts will need to be updated!

Abbreviated Installation Instructions

To compile GM on Linux platforms, you will only need GNU make and C/C++ compilers. Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions may also be addressed in the FAQ.

These abbreviated instructions presume that you are installing GM 1.6.5 or later. (Here is the README-linux for GM 1.6.5.)

Abbreviated instructions for installing GM 1.5.2.1 are still available. (Here is the README-linux for GM 1.5.2.1.)

Current cautions and common problems:

• GM-1 does not support the new M3-CLOS and M3-SPINE Myrinet-2000 switches.
• GM is not in the critical performance path so it does not need to be built with specialized compilers and flags. GM should be built with gcc and only built with -O level of optimization.
• When building/linking GM applications, you must do so on a Linux box that matches the OS version of the machine on which you will be running. For instance you cannot compile on a glibc-2.2 based OS and run the executable on a glibc-2.1 based one.
• GM_INSTALL fails? Refer to this FAQ entry for troubleshooting.
• AGP (nVidia and ATI) conflicts? Refer to the "Caveats" section of the README-linux.
• APIC IRQ conflicts? Refer to the "Caveats" section of the README-linux.
• If you're using SuSE 8 with highmem/highpte support, you will need gm-1.6.4 or later.
• If you are using an Apple G4 Xserve mother board, please read this FAQ entry for important performance information.
• To install GM on RedHat Linux Advanced Server (also known as Pensacola Gold Master), you will need gm-1.6.3 or later.
• If you have PCI64C NICs running on machines with 33MHz PCI buses, you must use gm-1.5.2 or later.
• If you're installing GM on an Itanium IA64 McKinley (Tiger motherboard), you will need gm-1.5.2 or later.
• If you're installing GM on a PAE-enabled x86 Linux kernel with more than 4GB of physical memory, you will need gm-1.5.2 or later.
• PCI32 NICs are NOT supported in gm-1.6.3 or later.
• PCI32 NICs are NOT supported on IA64.
• For Alphas, if you have 2GB or more of memory, we recommend kernel version 2.4.18 or later. You must use kernel version 2.4.14 or later (2.4.9 also works).
• By default, GM-1.5 and higher has security features enabled. Thus, you must now be root to run the GM Mapper (or gm_simpleroute).
• Do not use the configure flag --enable-directcopy with GM 1.6.5 or later. This flag is not a valid option to GM 1.6.5 or later. It may be re-enabled in a future release.
• If the host is rebooted, you must reload the GM driver and rerun the GM Mapper.
• The GM Mapper must be run before any communication over Myrinet can occur.
• In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications).

GM installation is performed in four easy steps:

1. Configuring and compiling GM.

        gunzip -c gm-1.6.7_Linux.tar.gz | tar xvf -
        cd gm-1.6.7_Linux
        ./configure
        make

By default, we assume that the kernel header files for your Linux installation are located in /usr/src/linux. If your Linux installation is not located in /usr/src/linux, you must configure with the following option:

 
        ./configure --with-linux=<linux-source-dir>

where <linux-source-dir> specifies the directory for the Linux kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options.

Note:

• For detailed installation instructions for GM with FMS diagnostic monitoring, refer to the FMS webpage.

2. Installing the GM driver.

Select an installation directory path <install_path>. It is usually best for <install_path> to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using <install_path> on all machines that are to share the installation. <install_path> must be an absolute path; it must start with /. However, <install_path> may contain symbolic links.

        cd binary
        ./GM_INSTALL <install_path>

You may omit <install_path> to install the driver in /opt/gm/.

Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /etc/init.d/gm start

on each machine to install/copy the drivers on that machine.

If you wish for the driver to auto-load at boot, you must create appropriate links in the /etc/rcN directories to the /etc/init.d/gm and /etc/init.d/myri scripts. Alternatively, you may start and stop the drivers manually using

 
        su root
        /etc/init.d/gm start
        /etc/init.d/gm stop

or

 
        su root
        /etc/init.d/gm restart

For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section of the README-linux.

Note: If the host is rebooted, you must reload the GM driver (and rerun the GM mapper).

3. Running the GM Mapper.

        cd  <install_path>/sbin/
        su root
        ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to the README-linux for details.

4. Testing the GM Installation.

A variety of test scripts are available in bin/ to test your GM installation. We recommend the five tests described in bin/README.

Congratulations! You have successfully installed GM.

Refer to the Software and Documentation Page for details of GM-based software (MPICH-GM, VI-GM, PVM-GM, Sockets-GM). If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-linux and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.

Last updated: 15 May 2008