************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2003 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README-FreeBSD for gm-2.0 Supported platforms: 4.1.1-RELEASE - 4.7-RELEASE for ia32 and alpha, and 4.9-RELEASE for ia32 and alpha. Note: 4.8-RELEASE requires a patch for proper operation. See section VI.c below. Supported interfaces: PCI64, PCI64A, PCI64B, PCI64C, PCIXD Note: gm-2.0 does not interoperate with gm-1.x. A mixture of hosts with gm-1.x and gm-2.0 cannot talk to each other. If you have PCI32{A,B,C} interfaces, you will need to upgrade your interface, or use a previous version of GM (gm-1.2.3 for PCI32A and gm-1.5.2.1 for PCI32B and PCI32C). For installation instructions of an earlier GM version please refer to the respective README and README- files. Table of Contents: ----------------- I. GM Binary Installation a. Unpacking the GM driver b. Installing the GM driver c. Enabling IP over Myrinet (Ethernet emulation) d. Testing the GM installation II. GM Source Installation a. Configuring and compiling GM b. Installing the GM driver c. Enabling IP over Myrinet (Ethernet emulation) d. Testing the GM installation III. Verifying the GM performance IV. Improving IP Performance V. Fork() and System() Support VI. Caveats a. Loading/unloading the GM driver b. Difficulties to unload the GM module? c. Running GM on FreeBSD 4.8-RELEASE VII. Miscellaneous a. Uninstallation of the GM driver ************************************************************************ If difficulties are encountered, please consult the FAQ http://www.myri.com/scs/FAQ/ and all technical support questions should be directed to help@myri.com. ************************************************************************ ========================== I. GM Binary Installation ========================== GM Binary installation is supported for FreeBSD/x86 versions 4.1.1-RELEASE through 4.9-RELEASE. If you're running an older or newer version of FreeBSD (4.0 through 4.1-RELEASE, or 5.x), or if you are running FreeBSD on an alpha, you must build the driver from source. Please skip to section II.( GM Source Installation Instructions). Please note that while the GM driver works with 5.1-CURRENT as of August 20th 2003, the 5.x branch of FreeBSD is still under active development. Therefore interfaces may change between the date of the gm-2.0_FreeBSD release and the time when FreeBSD 5.x becomes the stable branch. Consequently, we recommend Myrinet users who are interesting in running FreeBSD choose FreeBSD 4.x. GM installation is performed in the following four steps. 1. Unpacking the GM driver. -------------------------------------- tar zxvf gm-2.0_FreeBSD-x86-FreeBSD-4.1.1-RELEASE-16ports.tar.gz 2. Installing the GM driver: --------------------------------------------- Select an installation directory path . It is usually best for to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using on all machines that are to share the installation. must be an absolute path; it must start with "/". However, may contain symbolic links. cd gm-2.0_FreeBSD-x86-FreeBSD-4.1.1-RELEASE ./GM_INSTALL If you omit , the driver will be installed in the default directory, /opt/gm/. Next, you must run su root /sbin/gm_install_drivers /usr/local/etc/rc.d/gm start on each machine to install the drivers on that machine. Important note: The gm_install_drivers script does not configure the IP device. If you wish to run IP over GM/Myrinet (ethernet emulation), you must configure the device. Refer to step 3. If you wish for the driver to auto-load at boot, you must make the appropriate symlink in the /usr/local/etc/rc.d directory: su root ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh Alternatively, you may start and stop the drivers manually using: su root /usr/local/etc/rc.d/gm start /usr/local/etc/rc.d/gm stop or su root /usr/local/etc/rc.d/gm restart to start, stop, or restart the driver, respectively. For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section. Note: If the host is rebooted, you must reload the GM driver. 3. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL) ----------------------------------------------------------- If you wish to run IP over Myrinet (ethernet emulation), the FreeBSD command to enable IP over GM is as follows: /sbin/ifconfig myri0 up where you must replace 'myri0' with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet interface per host. For suggestions on improving performance, please refer to section "IV. Improving IP Performance". Consult the "Running IP" section of the FAQ (http://www.myri.com/scs/FAQ/) for other related questions. 4. Testing the GM Installation ------------------------------ Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests. * Check the LEDs on each switch port and interface port * Run gm_board_info on one host * Run gm_debug to test the PCI bandwidth * Run gm_allsize to test the links in the network * Run gm_stress to test the network Each of these steps is detailed in the Troubleshooting section of the FAQ http://www.myri.com/scs/FAQ/ The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are available in /bin in your GM installation. A README describing each of these tests can be found in /bin/README. =========================== II. GM Source Installation =========================== GM installation is performed in the following four steps. 1. Configuring and compiling GM: --------------------------------------------- Install GNU make from the /usr/ports/devel/gmake port or the gmake package. gunzip -c gm-2.0_FreeBSD.tar.gz | tar xvf - cd {GM_HOME} ./configure gmake By default, we assume that your FreeBSD kernel build directory is located in /sys/compile/GENERIC. If your FreeBSD kernel build directory is not located in /sys/compile/GENERIC, you must configure with the following option: ./configure --with-freebsd-kernel= where specifies the directory for the FreeBSD kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options. If you are running the binary "GENERIC" kernel, you must install the kernel sources, configure a "GENERIC" kernel: # cd /sys//conf; config -g GENERIC and run "make depend" in the GENERIC kernel build directory (/sys/compile/GENERIC) # cd /sys/compile/GENERIC; make depend For a complete listing of all options to configure, type: ./configure --help Note: Do not use the configure flag --enable-directcopy. This flag is not a valid option to GM 2.0. It will be re-enabled in a future release. The GM_INSTALL script will unload any existing GM device driver, load the current device driver and create /dev/gm device i-nodes. It does not configure the IP device, nor does it set up any scripts to load the GM driver at boot time. During the GM_INSTALL phase, GM prints messages to the kernel log (dmesg). If the running kernel and the kernel header used for compilation are mismatched, GM will print a warning message to the kernel log. Please be sure to read the {GM_HOME}/README and the {GM_HOME}/README-FreeBSD for further details of operating-system-specific caveats. 2. Installing the GM driver: --------------------------------------------- Select an installation directory path . It is usually best for to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using on all machines that are to share the installation. must be an absolute path; it must start with "/". However, may contain symbolic links. cd binary ./GM_INSTALL If you omit , the driver will be installed in the default directory, /opt/gm/. Next, you must run su root /sbin/gm_install_drivers /usr/local/etc/rc.d/gm start on each machine to install the drivers on that machine. If you wish for the driver to auto-load at boot, you must make the appropriate symlink in the /usr/local/etc/rc.d directory: su root ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh Alternatively, you may start and stop the drivers manually using: su root /usr/local/etc/rc.d/gm start /usr/local/etc/rc.d/gm stop or su root /usr/local/etc/rc.d/gm restart to start, stop, or restart the driver, respectively. For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section. Note: If the host is rebooted, you must reload the GM driver. 3. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL) ----------------------------------------------------------- If you wish to run IP over Myrinet (ethernet emulation), the FreeBSD command to enable IP over GM is as follows: /sbin/ifconfig myri0 up where you must replace 'myri0' with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet interface per host. For suggestions on improving performance, please refer to section "IV. Improving IP Performance". Consult the "Running IP" section of the FAQ (http://www.myri.com/scs/FAQ/) for other related questions. 4. Testing the GM Installation ------------------------------ Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests. * Check the LEDs on each switch port and interface port * Run gm_board_info on one host * Run gm_debug to test the PCI bandwidth * Run gm_allsize to test the links in the network * Run gm_stress to test the network Each of these steps is detailed in the Troubleshooting section of the FAQ http://www.myri.com/scs/FAQ/ The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are available in /bin in your GM installation. A README describing each of these tests can be found in /bin/README. ================================= III. Verifying the GM Performance ================================= We recommend the following test to verify your GM performance. cd /bin ./gm_debug --no-counters This gm_debug test displays the results of the hardware benchmark test of the PCI bus with the DMA engine of the Myrinet interface. The output of this command indicates the maximum sustained bandwidth that can be obtained from the PCI bus, and thus provides an upper bound on GM performance. A detailed description of this benchmark can be found in the FAQ (http://www.myri.com/scs/FAQ/). The output of this command also tells you if the Myrinet interface was correctly detected as 64-bit / 66 MHz, for example. If the interface was not correctly detected by the BIOS, you should suspect a riser card problem or a PCI slot problem. Performance graphs (http://www.myri.com/myrinet/performance) for GM on Linux are available. The performance measurements were obtained by running gm_allsize tests for latency and bandwidth as described in the FAQ entry ("What are the run-time options to gm_allsize?"). Refer to the section entitled "GM Performance" in the /README for complete details on expected GM performance. ============================ IV. Improving IP performance ============================ IP tuning consists of the following: sudo sysctl -w net.inet.tcp.sendspace=65536 sudo sysctl -w net.inet.tcp.recvspace=65536 Some additional gain can sometimes be seen by increasing these values to 262144. If one increases these values to 262144, one must also increase kern.ipc.maxsockbuf to something larger, such as 524288. If you would like to obtain the maximum performance out of IP, you can increase the size of your mbuf clusters to PAGE_SIZE by changing MCLSHIFT in sys/{i386,alpha}/include/param.h to 12 on i386 and 13 on alpha. (eg, PAGE_SIZE == 1 << MCLSHIFT). This requires a kernel rebuild and a GM rebuild. =============================== V. Fork() and System() Support =============================== By default, GM supports the function vfork(). As the function popen() calls vfork(), it is also safe to use at any time with GM. However, GM does not support fork() when a GM port is open, as it is incompatible with some GM OS manipulations. Please contact help@myri.com for further details. ============ VI. Caveats ============ ----------------------------------- a. Loading/unloading the GM driver ----------------------------------- To load the driver, run 'kldload /sbin/gm.ko' To unload the driver, run 'kldunload -n gm.ko' ----------------------------------------- b. Difficulties to unload the GM module? ----------------------------------------- >>Let's say a user left a gm_allsize running and forgot about it. She then >>attempts to unload the module, and receives the following message: % sudo kldunload gm kldunload: can't unload file: Device busy >> At this point, this message will appear on the system console: WARNING: GM: 1 refs to minor unit 8. module cannot be unloaded until all ports are closed >> Now the user has enough information to track down the process & >> kill it. The easiest way to do this is to use fstat (like lsof, >> but in base system) to look for processes with /dev/gm* open: % fstat | grep gm root gm_allsize 601 3 / 8484 crw------- gm8 rw % sudo kill -TERM 601 % sudo kldunload gm % ------------------------------------ c. Running GM on FreeBSD 4.8-RELEASE ------------------------------------ FreeBSD 4.8 contains a minor bug in the VM system which will cause our driver to leak wired memory when GM applications exit uncleanly. This can happen if you ^C an MPICH-GM job. This bug was introduced in version 1.187.2.16 of sys/vm/vm_map.c prior to 4.8-RELEASE, and was corrected in version 1.187.2.21 of sys/vm/vm_map.c, prior to 4.9-RELEASE. To work around this problem, you can apply the patch below. Alternatively, we suggest that you upgrade to FreeBSD 4.9 if possible. Index: vm/vm_map.c =================================================================== RCS file: /home/ncvs/src/sys/vm/vm_map.c,v retrieving revision 1.187.2.20 retrieving revision 1.187.2.21 diff -u -r1.187.2.20 -r1.187.2.21 --- vm/vm_map.c 10 Aug 2003 18:14:11 -0000 1.187.2.20 +++ vm/vm_map.c 1 Oct 2003 01:21:37 -0000 1.187.2.21 @@ -1659,6 +1659,7 @@ * becomes completely unwired, unwire its physical pages and * mappings. */ + entry = start_entry; while ((entry != &map->header) && (entry->start < end)) { KASSERT(entry->eflags & MAP_ENTRY_USER_WIRED, ("expected USER_WIRED on entry %p", entry)); entry->eflags &= ~MAP_ENTRY_USER_WIRED; ================== VII. Miscellaneous ================== ------------------------------------ a. Uninstallation of the GM driver ------------------------------------ The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers, which can be used to uninstall the drivers. The GM_INSTALL script generates the script /sbin/GM_UNINSTALL, which can be used to uninstall GM.