************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2005 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README-solaris for gm-2.1.27 Supported OS/processors: Solaris 8, 9, and 10 on UltraSPARC. Solaris 9 and 10 on x86. Solaris 10 on AMD64. Supported NICs: PCI64, PCI64A, PCI64B, PCI64C, PCIXD, PCIXE, PCIXF Note: gm-2.0 does not interoperate with gm-2.1.x or gm-1.x. A mixture of hosts with gm-1.x and gm-2.0 or gm-2.1 cannot talk to each other. GM-2 does not support PCI32-based NICs. If you have PCI32{A,B,C} you will need to upgrade your NIC, or use a previous version of GM. Use gm-1.6_Solaris for PCI32 with 1MB (PCI32, SBus), or gm-1.2.3 for PCI32 with 256K or 512K. For installation instructions of an earlier GM version please refer to the respective README and README- files. Important notes: The GM-2 software release is optimized for use with the PCIXD, PCIXE, and PCIXF NICs. Although this GM-2 release also works with PCI64-series Myrinet NICs, it is not yet as well optimized for performance on PCI64-based NICs as gm 1.6.4. Thus, for performance reasons, we do not recommend upgrading PCI64-based clusters to GM-2 unless you wish to take advantage of the new features of GM-2. For best performance, we recommend GM-2.0.x for clusters with PCIXD and PCIXF NICs. Clusters with PCIXE NICs require GM-2.1.x. If you will be running ClusterTools' MPI application over GM-2, you must use CT_Myrinet_PM_0.8 or later. Table of Contents: ----------------- I. Installation Requirements a. Minimum installation requirements b. Choose an appropriate source tarball II. GM Source Installation Instructions a. Configuring and compiling GM b. Loading the GM driver c. Enabling IP over Myrinet (Ethernet emulation) (OPTIONAL) d. Testing the GM Installation III. Verifying GM performance IV. Miscellaneous a. Uninstallation of the GM driver ************************************************************************ If difficulties are encountered, please consult the FAQ http://www.myri.com/scs/FAQ/ and all technical support questions should be directed to help@myri.com. ************************************************************************ ============================ I. Installation Requirements ============================ a. Minimum installation requirements: ------------------------------------ gm-2.1.27_Solaris is released in source form only. The minimum installation requirements are: UltraSPARC: cc -- Sun Forte Developer 7 or later make -- GNU Make 3.80 x86: gcc -- gcc (GCC) 3.3.4 make -- GNU Make 3.80 AMD64: Solaris 10 b_74 or later Sun Studio 10 23_1 or later, or GNU gcc 3.4.3 or later make -- GNU Make 3.80 b. Choose an appropriate source tarball --------------------------------------- The source release of gm-2.1.27 for Solaris is prebuilt with new-features and heart-beat enabled. For performance reasons, there are two source tarballs from which to choose. One source tarball is prebuilt with sparc streaming enabled. This tarball should be used for UltraSPARC machines that are using the "Psycho" or "Schizo" PCI chipset. Examples of such machines are as follows: Machines that use the "Psycho" PCI chipset: Ultra 30/60/80/450, Netra t 1100/1120/1125/1400/1405, Enterprise 220R/250/420R/450 Machines that use the "Schizo" PCI chipset: SunBlade 1000/2000, Netra 20, Sun Fire 280R/V880/3800/4800/4810/6800 The other source tarball is prebuilt with sparc streaming disabled. This tarball should be used on x86-based Solaris machines and for lower latency and better bandwidth on machines using the "Tomotill" PCI chipset. Examples of such machines are as follows: x86-based Solaris machines AMD64-based Solaris machines Machines that use the "Tomotill" PCI chipset: Sun Fire V240 ======================================= II. GM Source Installation Instructions ======================================= a. Configuring and compiling GM: -------------------------------- gunzip -c gm-2.1.27_Solaris.tar.gz | tar xvf - cd gm-2.1.27_Solaris ./configure Add the "--disable-64b" option if you are building a 32-bit driver on a machine running a 64-bit kernel, or "--enable-64b" if you are building a 64-bit driver on a machine running a 32-bit kernel. Without these options, a driver matching the running kernel on the compilation machine will be built. Add the --disable-sparc-streaming configure option if you are building the driver from a sparc-streaming-disabled source tarball. make b. Loading the GM driver ------------------------- Select an installation directory path . It is usually best for to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using on all machines that are to share the installation. must be an absolute path; it must start with "/". However, may contain symbolic links. cd binary ./GM_INSTALL If you omit the , the driver will be installed in the default directory, /opt/gm/. Next, you must run su root /sbin/gm_install_drivers /etc/init.d/gm start on each machine in your cluster. The GM_INSTALL script copies the GM binaries to the specified binary installation directory . The gm_install_drivers script performs the following operations: * Copies gm into /kernel/drv/($SPARCV9)/gm * Removes the previous installation by executing /sbin/gm_uninstall_drivers (rem_drv) * Copies other files from the binary installation directory to an architecture-specific directory (/etc/init.d/). * Creates the devices (/dev/gm* and /dev/gmp*), one device per NIC * Creates the mapper's per-host configuration directory (/etc/gm_mapper) and possibly store configuration files there. The gm "start" script performs the following operations: * Loads the GM module (add_drv) * Starts a mapper daemon called "gm_mapper" for each Myrinet NIC contained in the machine. The PIDs of the running gm_mappers are stored in /var/run/gm_mapper/pid.{board_id}. The gm "stop" script performs the following operations: * Shuts down the gm_mapper daemon * ifconfig's down the myri* ethernet devices * Unloads the GM module (rem_drv) Important note: Stopping the mapper while GM is running is not supported. The gm_mapper should be left running at all times, and it will not interfere with the performance of jobs running over Myrinet. Important note: The installation scripts do not configure the IP device. If you wish to run IP over GM/Myrinet (ethernet emulation), you must configure the device. Refer to step c. If you wish for the driver to auto-load an boot, you can create appropriate links in the /etc/rcN directories to the /etc/init.d/gm and /etc/init.d/myri scripts. Alternatively, you may start and stop the drivers manually using su root /etc/init.d/gm start /etc/init.d/gm stop or su root /etc/init.d/gm restart to start, stop, or restart the driver, respectively. For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section. Note: If the host is rebooted, you must reload the GM driver. c. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL) ----------------------------------------------------------- If you wish to run IP over Myrinet (ethernet emulation), the Solaris command to enable IP over GM is as follows: /sbin/ifconfig myri0 plumb /sbin/ifconfig myri0 up where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host. d. Testing the GM Installation. ------------------------------ Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests. * Check the LEDs on each switch port and NIC port * Run gm_board_info on one host * Run gm_debug to test the PCI bandwidth * Run gm_allsize to test the links in the network * Run gm_stress to test the network Each of these steps is detailed in the Troubleshooting section of the FAQ http://www.myri.com/scs/FAQ/ The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are available in /bin in your GM installation. A README describing each of these tests can be found in /bin/README. ============================= III. Verifying GM Performance ============================= We recommend the following test to verify the GM performance. cd /bin ./gm_debug --no-counters This gm_debug test displays the results of the hardware benchmark test of the PCI bus with the DMA engine of the Myrinet NIC. The output of this command indicates the maximum sustained bandwidth that can be obtained from the PCI bus, and thus provides an upper bound on GM performance. A detailed description of this benchmark can be found in the FAQ (http://www.myri.com/scs/FAQ/). The output of this command also tells you if the Myrinet NIC was correctly detected as 64-bit / 66 MHz, for example. If the NIC was not correctly detected by the BIOS, you should suspect a riser card problem or a PCI slot problem. Performance graphs (http://www.myri.com/scs/solaris/index-gm2.html) for GM-2 on Solaris are available. The performance measurements were obtained by running gm_allsize tests for latency and bandwidth as described in the FAQ entry ("What are the run-time options to gm_allsize?"). Refer to the section entitled "GM Performance" in the /README for complete details on expected GM performance. ================= IV. Miscellaneous ================= ------------------------------------ a. Uninstallation of the GM driver ------------------------------------ The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers, which can be used to uninstall the drivers. The GM_INSTALL script generates the script /sbin/GM_UNINSTALL, which can be used to uninstall GM. ----------- b. patches ----------- If you encounter the following errors, you will need to obtain the corresponding patches from Sun 1. For "peekpoke_ctlops_t not defined" when build on a solaris 10 x84 box, please apply solaris patches 118344-08 and 118844-30. 2. For "ube: error: Bad template number found in tmpl_emit" when built on a solaris 10 x84 box using Sun Studio 11, please apply Studio 11 patch 120759-04.