************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2002 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README-MacOSX for gm-1.6.3 Supported platforms: Mac OS X 10.1 and 10.2 (darwin 1.4 - 6.0) for PowerPC Supported interfaces: LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C) Table of Contents: ----------------- I. GM Binary Installation a. Unpacking the GM driver b. Installing the GM driver b. Running the GM Mapper c. Testing the GM installation II. GM Source Installation a. Configuring and compiling GM b. Installing the GM driver c. Running the GM Mapper d. Testing the GM installation III. Verifying the GM performance IV. Running IP over GM V. Improving IP Performance VI. Cautions/Caveats VII. Miscellaneous a. Uninstallation of the GM driver ************************************************************************ If difficulties are encountered, please consult the FAQ http://www.myri.com/scs/GM_FAQ.html and all technical support questions should be directed to help@myri.com. If you have problems on any architecture, please send the output from /var/log/system.log to . ************************************************************************ ====================================== I. GM Binary Installation Instructions ====================================== Current cautions and common problems: * If the host is rebooted, you must rerun the GM Mapper. * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). * Incorrectly written programs which attempt to free() or munmap() memory prior to deregistering it may "hang" indefinitely. For details see the Cautions/Caveats below GM installation is performed in three easy steps: 1. Unpacking the GM driver. -------------------------------------- When reading the instructions below, please replace VERSION with your operating system version (10.1 or 10.2): tar zxvf gm-1.6.3_MacOSX-VERSION-new-features-8ports.tar.gz 2. Installing the GM driver: --------------------------------------------- Select an installation directory path . It is usually best for to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using on all machines that are to share the installation. must be an absolute path; it must start with "/". However, may contain symbolic links. cd gm-1.6.3_MacOSX-VERSION-new-features-8ports ./GM_INSTALL You may omit to install the driver in /opt/gm/. Next, you must run sudo /sbin/gm_install_drivers on each machine to install the drivers on that machine. If you wish for the driver to auto-load an boot, you must copy StartupParameters.plist to /System/Library/StartupItems/GM sudo cp /etc/StartupParameters.plist /System/Library/StartupItems/GM/ Alternatively, you may start and stop the drivers manually using: sudo /System/Library/StartupItems/GM/gm start sudo /System/Library/StartupItems/GM/gm stop or sudo /System/Library/StartupItems/GM/gm restart to start, stop, or restart the driver, respectively. For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section. Note: If the host is rebooted, you must reload the GM driver (and rerun the GM mapper). 3. Running the GM Mapper. ------------------------ cd {GM_HOME}/binary/sbin sudo ./mapper ../etc/gm/map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation. ------------------------------ A variety of test scripts are available in binary/bin/ to test your GM installation. We recommend the five tests described in tests/README. ======================================= II. GM Source Installation Instructions ======================================= To compile GM on Mac OS X platforms, you will need GNU make, and the Apple C/C++ compiler. These tools are provided by Apple as part of the Developer Tools package available through the Apple Developer Connection (http://www.apple.com/developer). The Developer Tools package is also included in the Mac OS X 10.2 retail box. Current cautions and common problems: * If the host is rebooted, you must rerun the GM Mapper. * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). * Incorrectly written programs which attempt to free() or munmap() memory prior to deregistering it may "hang" indefinitely. For details see the Cautions/Caveats below. GM installation is performed in three easy steps: 1. Configuring and compiling the GM driver: ---------------------------------------------------- tar zxvf gm-1.6.3_MacOSX.tar.gz cd gm-1.6.3_MacOSX ./configure make 2. Installing the GM driver: --------------------------------------------- Select an installation directory path . It is usually best for to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using on all machines that are to share the installation. must be an absolute path; it must start with "/". However, may contain symbolic links. cd binary ./GM_INSTALL You may omit to install the driver in /opt/gm/. Next, you must run sudo /sbin/gm_install_drivers on each machine to install the drivers on that machine. If you wish for the driver to auto-load an boot, you must copy StartupParameters.plist to /System/Library/StartupItems/GM sudo cp /etc/StartupParameters.plist /System/Library/StartupItems/GM/ Alternatively, you may start and stop the drivers manually using: sudo /System/Library/StartupItems/GM/gm start sudo /System/Library/StartupItems/GM/gm stop or sudo /System/Library/StartupItems/GM/gm restart to start, stop, or restart the driver, respectively. For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section. Note: If the host is rebooted, you must reload the GM driver (and rerun the GM mapper). 3. Running the GM Mapper. ------------------------ cd {GM_HOME}/binary/sbin sudo ./mapper ../etc/gm/map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation. ------------------------------ A variety of test scripts are available in binary/bin/ to test your GM installation. We recommend the five tests described in tests/README. ================================= III. Verifying the GM Performance ================================= We recommend the following test to verify the GM performance. View the results of the hardware benchmark test of the PCI bus with the DMA engine of the Myrinet adapter. cd {GM_HOME}/binary/bin ./gm_debug --no-counters Note: The output of this command gives the maximum sustained bandwidth that can be obtained from the PCI bus. Refer to the section entitled "GM Performance" in the {GM_HOME}/README for complete details on expected GM performance. ====================== IV. Running IP over GM ====================== The IP device is accessed via ifconfig enX
netmask broadcast up where you must replace 'enX with the appropriate name (en1, en2, etc.) Note that this is different than GM on other platforms, where the device name is myriX. This is due to quirks in the Mac OS X network stack, over which we have no control. The first ethernet device number used by GM is obtained via sysctl: % sysctl net.gm.gm_base_en_unit net.gm.gm_base_en_unit: 1 This means that the first myrinet interface on this host is named "en1", and subsequent interfaces will be named "en2", "en3", etc. Note that the GM network interface is not visible or configurable through the Network control panel. You must use ifconfig to configure it. =========================== V. Improving IP performance =========================== If you don't see the speed that you expect, you might consider adjusting the TCP send and receive window sizes. To adjust the window sizes, modify the kern.ipc.maxsockbuf, net.inet.tcp.sendspace and net.inet.tcp.recvspace like this: % sudo sysctl -w kern.ipc.maxsockbuf=2097152 % sudo sysctl -w net.inet.tcp.sendspace=262144 % sudo sysctl -w net.inet.tcp.recvspace=262144 ==================== VI. Cautions/Caveats ==================== ************************************************************************* You need to run the mapper at least once after loading the driver on all nodes in your network. cd {GM_HOME}/binary/sbin sudo ./mapper ../etc/gm/map_once.args COMPILING from SOURCE: --------------------- You may not use the --enable-maintainer-mode flag on Mac OS X. This is because the lanai cross compiler contains a bug which causes it to crash when run on PowerPC. You must use the prebuilt firmware, or build your firmware on another platform. INCORRECTLY WRITTEN PROGRAMS "HANG" ----------------------------------- Incorrectly written programs which attempt to free() or munmap() memory prior to deregistering it may "hang" indefinitely. This happens because the MacOSX/darwin kernel does not expect drivers to leave memory regions "wired" in memory for long periods of time. When the memory region is free'ed, the VM system blocks uninterruptibly waiting for the GM driver to unwire the memory. The GM driver has no way of knowing that the memory system is waiting for it to unwire a memory region. To recover from this problem, you can unload & reload the module (cd binary; sudo ./GM_INSTALL). At unload time, the GM driver will free all memory associated with all ports and will send kill -9 signals to all processes with open ports. This will allow the hung program to exit. To diagnose this problem, run the program under sc_usage. If the last system call the program made before "hanging" was vm_deallocate or munmap, then your application has a bug which must fixed before it can run on MacOSX/darwin. See the sc_usage(1) man page for instructions on how to run sc_usage. Ethernet Unit Number "Stealing" ------------------------------ Due to a quirk in the way the GM driver interacts with the MacOSX/darwin network stack, users planning to use ethernet emulation must load the GM module *after* loading all other network drivers. If a network driver is loaded after GM, the newly loaded network driver has no way of knowing what unit number the GM ethernet (en) interface is using and it may "steal" the GM unit number. This could make one or both devices unusable until the machine is rebooted. DARWIN on INTEL --------------- Darwin on Intel has not been tested and is not supported. We would not be terribly surprised to hear that it worked, but we offer no support. ************************************************************************* =================== VIII. Miscellaneous =================== ------------------------------------ a. Uninstallation of the GM driver ------------------------------------ The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers, which can be used to uninstall the drivers. The GM_INSTALL script generates the script /sbin/GM_UNINSTALL, which can be used to uninstall GM.