************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2001 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README-Tru64 for gm-1.5 Supported platforms: Tru64 UNIX V4.0e, V5.0, and V5.1 for Alpha. Supported interfaces: LANai4 with 1MB or 512K (PCI32, SBus), LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C) (If you have LANai4 with 256K, you will need to upgrade your interface, or use a previous version of GM (gm-1.2.3). For installation instructions of an earlier GM version please refer to the respective README and README- files. Please also note that Linux 2.4 is not supported on earlier GM versions). Table of Contents: ----------------- I. GM Binary Installation Instructions a. Unpacking and loading the GM driver b. Running the GM Mapper c. Testing the GM Installation II. GM Source Installation Instructions a. Configuring, compiling, and loading the GM driver b. Running the GM Mapper c. Testing the GM Installation III. Verifying the GM performance IV. Running IP over GM V. Improving IP Performance VI. Cautions/Caveats ************************************************************************ If difficulties are encountered, please consult the FAQ http://www.myri.com/scs/GM_FAQ.html and all technical support questions should be directed to help@myri.com. If you have problems on any architectures, please send the output from /var/adm/messages to . ************************************************************************ ====================================== I. GM Binary Installation Instructions ====================================== Current cautions and common problems: * If the host is rebooted, you must rerun the GM Mapper. * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). * We advise you to avoid unloading the GM module. This means one should NEVER run GM_INSTALL twice on any machine! GM installation is performed in three easy steps: 1. Unpacking and loading the GM driver. -------------------------------------- When reading the the instructions below, please replace VERSION with your operating system version (V4.0E, V5.0, V5.1) as returned by sizer -v, and please replace REVISION with your operating system revision as returned by uname -r (V4.0, V5.0, V5.1). gunzip -c gm-1.5pre1_for_Tru64-alpha-OSF1-VERSION-new-features-8ports.tar.gz | tar xvf - cd gm-1.5pre1_for_Tru64-alpha-OSF1-REVISION su root If you have never installed GM on this host: ./GM_INSTALL If you have previously installed GM on this host: ./GM_copy_driver shutdown -r now 2. Running the GM Mapper. ------------------------ cd gm-1.5pre1_for_Tru64-alpha-OSF1-REVISION/sbin/ su root ./mapper map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation. ------------------------------ A variety of test scripts are available in binary/bin/ to test your GM installation. We recommend the five tests described in tests/README. ======================================= II. GM Source Installation Instructions ======================================= To compile GM on Tru64 Alpha platforms, you will need GNU make, the Compaq C compiler, and either the GNU or Compaq C++ compilers. Current cautions and common problems: * If the host is rebooted, you must rerun the GM Mapper. * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). * We advise you to avoid unloading the GM module. This means one should NEVER run GM_INSTALL twice on any machine! GM installation is performed in three easy steps: 1. Configuring, compiling, and loading the GM driver: ---------------------------------------------------- gunzip -c gm-1.5pre1_for_Tru64.tar.gz | tar xvf - cd gm-1.5pre1_for_Tru64 ./configure make cd binary su root If you have never installed GM on this host: ./GM_INSTALL If you have previously installed GM on this host: ./GM_copy_driver shutdown -r now 2. Running the GM Mapper. ------------------------ cd binary/sbin/ su root ./mapper map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation. ------------------------------ A variety of test scripts are available in binary/bin/ to test your GM installation. We recommend the five tests described in tests/README. ================================= III. Verifying the GM Performance ================================= We recommend the following test to verify the GM performance. View the results of the hardware benchmark test of the PCI bus with the DMA engine of the Myrinet adapter. cd {GM_HOME}/binary/bin ./gm_debug --no-counters Note: The output of this command gives the maximum sustained bandwidth that can be obtained from the PCI bus. Refer to the section entitled "GM Performance" in the {GM_HOME}/README for complete details on expected GM performance. ====================== IV. Running IP over GM ====================== The IP device is accessed via ifconfig myri0
netmask broadcast up where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.) if you have more than one Myrinet interface per host. For more information, please refer to the FAQ (http://www.myri.com/scs/GM_FAQ.html). =========================== V. Improving IP performance =========================== If you don't see the speed that you expect, you might consider adjusting the TCP send and receive window sizes. Tru64 5.0 and newer ship with a fairly reasonable set of defaults for the tcp window size. If you're running 4.x, you may want to increase the window size to between 64k and 256k. To adjust the window sizes, modify the inet:tcp_sendspace and inet:tcp_recvspace like this: # sysconfig -r inet tcp_sendspace=262144 # sysconfig -r inet tcp_recvspace=262144 You can make this happen automatically at boot via the sysconfigdb command. Read the sysconfigdb(8) man page for details. ==================== VI. Cautions/Caveats ==================== ************************************************************************* Never run GM_INSTALL twice on any machine. To simply update the driver binary (e.g. after you have run GM_INSTALL once at some point in time in the distant past for a previous GM release), ./GM_copy_driver sync /sbin/reboot run ./GM_INSTALL to install/configure the driver. You need to run the mapper at least once after loading the driver on all nodes in your network. cd sbin mapper active.args You might want to run gm_setname -h `hostname` to write a "useful" name into the card/driver. Otherwise, the gm_name of the machine will be an ASCII-ized version of its 48-bit ID. COMPILING from SOURCE By default, the configure will use cc and c++. To compile from source you'll need to have a /usr/sys/ or a /usr/sys/MYRI directory. We put both of those on the "-I" include path list for building kernel code. If the machine has ever been used for building a kernel, the /usr/sys/ directory is probably already populated. To compile from source using the Tru64 C and C++ compilers #DIGITAL C++ V6.1-027 setenv CC /usr/ccs/bin/cc setenv CFLAGS "-std -msg_disable extrasemi,emptyfile " setenv CXX /usr/bin/cxx setenv CXXFLAGS "-O2 -inline speed -x cxx -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic" rm -f config.cache ./configure #Compaq C++ V6.2-024 setenv CC /usr/ccs/bin/cc setenv CFLAGS "-std -msg_disable extrasemi,emptyfile " setenv CXX /usr/bin/cxx setenv CXXFLAGS "-O2 -inline speed -x cxx -nocxxstd -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic" rm -f config.cache ./configure ************************************************************************* ************************************************************************* The driver is theoretically unloadable, but we have not found an "if_detach" function, so reloading the driver will result in multiple myri0 interfaces showing up in the system. Most likely, any ifconfig or IP traffic will cause a system panic. The driver developer docs at http://www.unix.digital.com/faqs/publications/dev_doc/usr/examples/ddk/lan/README say this about unloading a network driver. NOTES: ------ o LAN drivers can only be loaded. There is currently no method for unloading LAN drivers. We are planning on fixing this for a future release. ************************************************************************* **************************************************************** If you have problems running the mapper and see something like this ./mapper: /sbin/loader: Fatal Error: cannot map libstdc++.so.2.8.0 then set up a symbolic link from libstdc++.so.2.8.1 to libstdc++.so.2.8.0 assuming you can find libstdc++.so.2.8.0 in /usr/local/lib. If not, a copy is at ftp.myri.com cd /pub/GM www% ls -l lib* -rw-r--r-- 1 web 399700 Jul 16 16:57 libstdc++.so.2.8.0.gz -rw-r--r-- 1 web 399613 Aug 16 17:45 libstdc++.so.2.8.1.gz This should let you use the mapper until we release a statically linked version. We're now compiling with libstdc++.so.2.8.1 so both are available on our web site. **************************************************************** To unload the driver you need to do the following /sbin/sysconfigdb -d gm /sbin/reboot The system will boot with the gm driver not loaded. You can then ./GM_INSTALL to re-load it. **************************************************************** If you see errors of this form - gm: LANAI: gm_sdma.h: call handle_send_error(0x0x4f698,18) destid=0 gm: LANAI: handle_send_error: subport 4 handling send error 18 gm: LANAI: handle_send_error: disabling subport 4 then you will need to apply the latest aggregate patch for Tru64 5.0, which is currently T64V50AS0003-20010221. This patch can be found at: http://ftp.support.compaq.com/patches/.new/unix.html t64v50as0003-20010221.README, T64V50AS0003-20010221 Tru64 V5.0 ECO Summary 12-Mar-2001 use uname -a to check your Tru64 version and sys_check -config -nohtml > output_file to check what patches you have currently installed (look in the output_file for "Installed patch kit versions") ******************************************************************