III. Overview

GM is a message-based communication system for Myrinet. Like many messaging systems, GM's design objectives included low CPU overhead, portability, low latency, and high bandwidth. Additionally, GM has several distinguishing characteristics:

• GM has extremely low overhead of about 1 microsecond per packet on all architectures.

• GM can provide simultaneous memory-protected user-level OS-bypass network interface access to several user-level applications simultaneously. (On systems that do not support memory protection, such as VxWorks, no memory protection is provided.)

• GM provides reliable ordered delivery between hosts in the presence of network faults. GM will detect and retransmit lost and corrupted packets. GM will also reroute packets around network faults when alternate routes exist. Catastrophic network errors, such as crashed hosts or disconnected links, are nonfatal; the undeliverable packets are returned to the client with an error indication, although most client programs are unable to adapt in the presence of such severe errors.

• GM supports clusters of over 10,000 nodes.

• GM provides two levels of message priority to allow efficient deadlock-free bounded-memory forwarding.

• GM allows clients to send messages up to 2**31 - 1 bytes long, under operating systems that support sufficient amounts of DMAable memory to be allocated.

• GM automatically maps Myrinet networks.

GM is a light-weight communication layer, and as such has limitations that can be addressed by layering a heavier-weight interface over GM. Some such limitations are the following:

• GM is unable to send messages from or receive messages into nonDMAable memory.

• The GM API does not yet support any gather or scatter operations directly.

From the client's point of view, GM consists of a library, libgm.a, and a header file, gm.h. All externally visible GM identifiers in these files match the regular expression `^_*[Gg][Mm]_' to minimize name space pollution.

Additionally, GM has other parts that system administrators need to be concerned about:

• gm The GM driver provides systems services. It is called ``gm'' under Unix, and is the `Myricom Myrinet Adapter' driver implemented in ``gm.sys'' under Windows NT.

• mapper The Myrinet mapper daemon maps the network. It is called ``sbin/mapper'' under Unix, and is the `Myricom Myrinet Mapper Daemon' service implemented in ``gm_mapper_service.exe'' under Windows NT.

1. Definitions

This document attaches special meaning to a few commonly used words. The meaning of each of these words in the context of this document is defined here. In particular, please note the special meanings of the words size and length. Understanding the special meaning of these terms is critical to understanding this document.

• aligned A value is said to be aligned if it is a multiple of the required GM alignment. The required GM alignment is 1 on LANai7 hardware, 4 on LANai4 hardware, and 8 on LANai5 hardware. Pointers to memory allocated by GM are always automatically aligned.

• client software

• client The client software or simply client is the non-GM software that uses GM to provide a reliable ordered message delivery service. It can be an application, or a higher level networking layer, such as MPI or VI.

• message A message is an aligned array of bytes in DMAable memory.

• buffer A buffer is a contiguous region of DMAable memory into which a message may be copied. All GM buffers must be aligned.

• length The length of a message is the number of bytes of data that comprise the message. There is no alignment restriction on the length of any GM message.

  The length of a receive buffer is the number of bytes that may be safely copied into the buffer.

• packet A packet is an aggregation of bytes sent over the network. Packet lengths are limited to just over gm_mtu()(port) (usually 4096 bytes) to bound the time any packet can monopolize network resource. Note that multiple packets are required to send large messages over the network, but the segmentation of messages into packets and reassembly of packets into messages is performed automatically by GM.

• size The size of the message is any integer greater than or equal to

            log  (LENGTH + 8)
               2
     

  where LENGTH is the length of the message.

  The size of a receive buffer is any positive integer less than or equal to

            log  (LENGTH + 8)
               2
     

  where LENGTH is the length of the buffer. Consequently, a buffer of size SIZE must have a LENGTH of at least 2**SIZE - 8.

  A buffer having a longer length serves no useful purpose in GM, but is allowed.

  The function gm_min_size_for_length()(LENGTH) can be used to compute the minimum size for any length, and the function gm_max_length_for_size()(SIZE) can be used to compute the maximum length for any size.

  • port A port is a GM communication endpoint, and serves as the interface between the client software and the network.

• user A human using an application that uses GM.

• user virtual memory Memory directly accessible by software running in a user application.

• kernel virtual memory Memory directly accessible by the GM driver.

2. Notation

The following terms abbreviations are used in the GM documentation and source code. Some of these abbreviations are obvious to speakers of English, but are included for speakers of other languages. This section does not include architecture-specific abbreviations used in the architecture-specific GM driver code, as those are documented by the architecture's vendor and are not of interest to most GM developers.

• accum accumulator

• addr address

• alloc allocate

• arch(s) architecture(s)

• buf
• buff buffer

• cnt count

• create allocate and then initialize

• destroy finalize and then free

• dma direct memory access

• hash hash table

• hp host pointer (a pointer of the appropriate size for the host architecture in question)

• insn(s) instruction(s)

• intr interrupt

• KVMA kernel virtual memory address

• lookaside lookaside list

• LSB(s) least significant byte(s)

• lsb('s) least significant bit(s)

• MAC Media Access Control. This is a commonly referred to sublayer of the datalink layer in the ISO network reference model.

• MAC Address A 6-byte address unique to a Myrinet interface. It is equivalent to an ethernet address.

• minor device minor number

• num number

• phys physical

• pre prefetch or precompute

• PTE page table entry

• recv receive

• ref reference

• seg segment

• sema semaphore

• UVMA user virtual memory address

• virt virtual

• VM virtual memory

• VMA virtual memory address

• zalloc allocate and clear

---