Myrinet received Solaris-Ready certification in July 2001.
for Solaris
Myrinet received Solaris-Ready certification in July 2001. |
Myrinet Sun HPC ClusterTools Software for Solaris |
To download Myrinet software,
you need a login and password.
If you don't already have these, please
request them.
GM Source | ClusterTools Binary | ClusterTools Source | Performance
The Sun HPC ClusterTools software implements MPI communication between processes using loadable protocol modules (PM) across TCP networks, shared memory interfaces (SHM), and remote shared memory (RSM). In order to exploit the low latency and high data rates of Myrinet, Myricom developed the Myrinet protocol module directly over GM. The Myrinet PM is very efficient, both in using GM's OS-bypass communication, and in avoiding memory copying whenever possible.
For UltraSPARC/Solaris 9 users, Myricom supports a Myrinet Protocol Module (Myrinet PM) for the Sun HPC ClusterTools 4 and 5 software.
For Solaris 10 users, Sun HPC ClusterTools 6 is required, and the Myrinet PM for ClusterTools 6 is not yet available. We recommend using MPICH-GM for MPI jobs.
| NIC | |||
|---|---|---|---|
| Operating System/ Processor |
PCIXD (Lanai XP) |
PCI64B and PCI64C (Lanai 9) |
PCI64 and PCI64A (Lanai 7) |
| Solaris 8 and 9 for UltraSPARC |
ClusterTools 5 | ClusterTools 5 | ClusterTools 5 |
Here is a summary of the CHANGES between ClusterTools releases.
Validation
The Myrinet Protocol Module Validation Guide for Sun HPC ClusterTools 4.0 (Adobe Acrobat version) is available and contains a detailed discussion of the installation instructions below, as well as testing and troubleshooting. The Myrinet Protocol Module (PM) has been tested with Sun HPC ClusterTools 4.0 and 5.0 software. ClusterTools 6.0 is not yet supported on Myrinet.
Abbreviated Binary Installation Instructions
The installation of Myrinet PM for Sun HPC ClusterTools involves the following 8 steps:
1. Install Sun HPC ClusterTools.
Install Sun HPC ClusterTools into the directory <CT_install_path> (default is /opt/SUNWhpc).
2. Install the Myrinet PM libraries.
gunzip -c CT-Myrinet-PM-1.2-no-mt.tar.gz | tar xvf -
su root
cp myr*.so* <CT_install_path>/lib
cp sparcv9/myr*.so* <CT_install_path>/lib/sparcv9
3. Set up the GM hostnames and bring up the ethernet driver.
For each node of the cluster, run ip2hostname to obtain the hostname corresponding to myri0's IP address.
ip2hostname x.x.x.x (myri0's IP address)
Edit/create file /etc/gm/hostname.0 and put this hostname into it.
Restart /etc/init.d/gm script on each node.
/etc/init.d/gm restart
This will set the node's GM name to the hostname in /etc/gm/hostname.0 and make the change globally know to all nodes. The name setup may be done just once. Once the file /etc/gm/hostname.0 is in place, the /etc/init.d/gm will pick up the right name to set up for the GM node whenever the deamon is started.
For each node of the cluster, plumb myri0 and set its IP address by the command
ifconfig myri0 plumb x.x.x.x up
4. Construct the configure file <CT_install_path>/conf/gm.conf.
An example of a <CT_install_path>/conf/gm.conf file for a cluster consisting of two nodes is given as follows:
-----------------------------------------
2
# node_name board_num port_num port_ids
u81-t 1 12 4 5 6 7 8 9 10 11 12 13 14 15
u82-t 1 12 4 5 6 7 8 9 10 11 12 13 14 15
-----------------------------------------
In this file, the first line specifies the total number of nodes in the cluster. Starting on the second line, the first column specifies the node's GM name; the second column specifies the number of Myrinet NICs installed on this node; the third column specifies the number of GM ports available for CT on each Myrinet NIC; from the fourth column on are the GM port ids that can be used on each Myrinet NIC.
Set the environment variable MPI_MYR_CONF to <CT_install_path>/conf/gm.conf file.
setenv MPI_MYR_CONF <CT_install_path>/conf/gm.conf
If no MPI_MYR_CONF is set, /opt/SUNWhpc/conf/gm.conf will be used as default.
5. Modify <CT_install_path>/conf/hpc.conf.
On each node of the cluster, modify <CT_install_path>/conf/hpc.conf to provide Myrinet PM with the proper rank.
------------------------------------------------------------ ... # List the available Protocol Modules # PMODULE LIBRARY Begin PMODULES shm () rsm () myr () tcp () End PMODULES # SHM settings # NAME RANK Begin PM=shm shm 5 End PM # RSM settings # NAME RANK AVAIL Begin PM=rsm wrsm 20 1 End PM # MYR settings # NAME RANK Begin PM=myr myr 30 End PM ... # TCP settings # NAME RANK MTU STRIPE LATENCY BANDWIDTH Begin PM=tcp midn 0 16384 0 20 150 idn 10 16384 0 20 150 ... myri 161 4096 0 20 150 ... ------------------------------------------------------------
6. Start the daemon (requires root privileges).
On the master node,
su root
/etc/init.d/sunhpc.cre_master start
/etc/init.d/sunhpc.cre_node start
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node start
7. Initialize the "all" partition (if necessary).
Initialize the "all" partition (requires root privileges) when mpinfo -N does not have all nodes in "all" partition, using
su root
<CT_install_path>/etc/part_initialize
================================================================================ example: mpinfo -N NAME UP PARTITION OS OSREL NCPU FMEM FSWP LOAD1 LOAD5 LOAD15 u81 y all SunOS 5.8 4 3901 3910 0.02 0.04 0.02 u82 y - SunOS 5.8 4 3873 3873 3.05 2.48 1.42 ================================================================================
8. Rerun the daemon (if necessary).
If the <CT_install_path>/conf/hpc.conf is modified (eg. rank changed, NIC added/removed, PM added/removed, etc.), or the status of any NIC in <CT_install_path>/conf/hpc.conf is changed (eg. NIC up/down, etc.), the daemon should be rerun.
Stop the daemon (requires root privileges).
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node stop
On the master node,
su root
/etc/init.d/sunhpc.cre_node stop
/etc/init.d/sunhpc.cre_master stop
Restart the daemon.
Repeat Step 6.
Congratulations! You have successfully installed Myrinet PM for Sun HPC ClusterTools 4 or 5.
If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-clustertools, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.
| NIC | |||
|---|---|---|---|
| Operating System/ Processor |
PCIXD (Lanai XP) |
PCI64B and PCI64C (Lanai 9) |
PCI64 and PCI64A (Lanai 7) |
| Solaris 8 and 9 for UltraSPARC |
ClusterTools 5 | ClusterTools 5 | ClusterTools 5 |
Here is a summary of the CHANGES between ClusterTools releases.
Validation
The Myrinet Protocol Module Validation Guide for Sun HPC ClusterTools 4.0 (Adobe Acrobat version) is available and contains a detailed discussion of the installation instructions below, as well as testing and troubleshooting. The Myrinet Protocol Module (PM) has been tested with Sun HPC ClusterTools 4.0 and 5.0 software. ClusterTools 6.0 is not yet supported on Myrinet.
Abbreviated Source Installation Instructions
The source release installation of Myrinet PM for Sun ClusterTools differs only from the installation of the binary release at Step 2. Substitute Step 2 of the binary installation's instructions with the following commands.
2. Build and Install the Myrinet PM libraries.
gunzip -c CT-Myrinet-PM-1.2-no-mt-src.tar.gz | tar xvf -
cd CT-4.0/mpi/engine/myr
For a normal build:
make lib32/libmt32/lib64/libmt64
su root
cp ../../solaris/release/lib/myr* <CT_install_path>/lib/.
cp ../../solaris/release/lib/sparcv9/myr* <CT_install_path>/lib/sparcv9/.
For a build with debug turned on:
make libg32/libmtg32/libg64/libmtg64
su root
cp ../../solaris/debug/lib/myr* <CT_install_path>/lib/.
cp ../../solaris/debug/lib/sparcv9/myr* <CT_install_path>/lib/sparcv9/.
The remaining steps in the installation procedure are identical to the CT binary installation. Start with Step 3, and follow the steps until the installation is complete.
Congratulations! You have successfully installed Myrinet PM for Sun HPC ClusterTools 4 or 5.
If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-clustertools, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.
Last updated: 01 September 2006