IV. Programming Model

The GM communication system provides reliable, ordered delivery between communication endpoints, called ports, with two levels of priority. This model is connectionless in that there is no need for client software to establish a connection with a remote port in order to communicate with it: the client software simply builds a message and sends it to any port in the network. (This apparently paradoxical connectionless reliability is achieved by GM maintaining reliable connections between each pair of hosts in the network and multiplexing the traffic between ports over these reliable connections.)

           Host                        Host
  .----------------------.    .---------------------.
  |           Process    |    |  Process            |
  |         .-----------.|    |.--------.           |
  |         |       Port||    ||Port    |           |
  |         |       .--.________.--.    |           |
  |         | Port /|  |||   ,--|  |    |           |
  |         | .--./ `--'|| ,' |/`--'    |           |
  |         | |  |----|---'   /|        |           |
  |         | `--'    | ||   /||        |           |
  |         `----\----|-'|  / |`--------'           |
  |               \   |  | /  |                     |
  |           .----\--|-.|/^  |                     |
  |           |     \ | |/ ^  |                     |
  |           |     .--./| ^  |                     |
  |    Process| Port|  ||| ^  |                     |
  |           |     `--'|| ^  |                     |
  |           |         || ^  |                     |
  |           |         || ^  |                     |
  |           `---------'| ^  |                     |
  `----------------------' ^  `---------------------'
                           ^
                  Reliable Connection

1. GM Endpoints (Ports)

Under operating systems that provide memory protection, GM provides memory protected network access. It should be impossible for any non-privileged GM client application to use GM to access any memory other than the application's own memory, except as explicitly allowed by the GM API. The unforgeable source of each received message is available to the receiver, allowing the receiver to discard messages from untrusted sources.

The largest message GM can send or receive is limited to (2**31)-1 bytes. However, because send and receive buffers must reside in DMAable memory, the maximum message size is limited by the amount of DMAable memory the GM driver is allowed to allocate by the operating system. Most GM applications obtain DMAable memory using the straightforward gm_dma_malloc() and gm_dma_free() calls, but sophisticated applications with large memory requirements may perform DMA memory management using gm_register_memory() and gm_deregister_memory() to pin and unpin memory on operating systems that support memory registration.

Message order is preserved only for messages of the same priority, from the same sending port, and directed to the same receiving port. Messages with differing priority never block each other. Consequently, low priority messages may pass high priority messages, unlike in some other communication systems. Typical GM applications will either use only one GM priority, or use the high priority channel for control messages (such as client-to-client acks) or for single-hop message forwarding.

Both sends and receives in GM are regulated by implicit tokens, representing space allocated to the client in various internal GM queues, as depicted in the following figure. At initialization, the client implicitly possesses gm_num_send_tokens() send tokens, and gm_num_receive_tokens() receive tokens. The client may call certain functions only when possessing an implicit send or receive token, and in calling that function, the client implicitly relinquishes the token(1). The client program is responsible for tracking the number of tokens of each type that it possesses, and must not call any GM function requiring a token when the client does not possess the appropriate token. Calling a GM API function without the required tokens has undefined results, but GM usually reports such errors, and such errors will not cause system security to be violated.

               Send Queue
              +-+-+-+-+-+
   .----------| | | | | |<-------------------.
   |          +-+-+-+-+-+                    |
   |  gm_num_send_tokens() slots             |
   |                                         |
   |       Receive Buffer Pool               |
   |          +-+-+-+-+-+                    |
   | .--------| | | | | |<-----------------. |
   | |        +-+-+-+-+-+                  | |
   | | gm_num_receive_tokens() slots       | |
   | |                                     | |
   | |                                     | |         LANai Memory
- -|-|- - - - - - - - - - - - - - - - - - -|-|- - - - - - - - - - -
   | |                                     | |  User Virtual Memory
   | |     Receive Event Queue             | |
   | `------->+-+-+-+-+-+                +--------+
   |          | | | | | |--------------->| Client |
   `--------->+-+-+-+-+-+                |Software|
       gm_num_receive_tokens() +         +--------+
       gm_num_send_tokens() slots

2. User Token Flow (Sending)

As stated above, sends are token regulated. A client of a port may send a message only when it possesses a send token for that port. By calling a GM API send functions, the client implicitly relinquishes that send token. The client passes a callback and context pointer to the send function. When the send completes, GM calls callback, passing a pointer to the GM port, the client-supplied context pointer, and status code indicating if the send completed successfully or with an error. When GM calls the client's callback function, the send token is implicitly passed back to the client. Most GM programs, which rely on GM's fault tolerance to handle transient network faults, should consider a send completing with a status other than GM_SUCCESS to be a fatal error. However, more sophisticated programs may use the GM fault tolerance API extensions to handle such non-transient errors. These extensions are described in an appendix. It is important to note that the client-supplied callback function will be called only within a client's call to gm_unknown(), the GM unknown event handler function that the client must call when it receives an unrecognized event. The gm_unknown() function is described in more detail below.

                                              Sent Packet
                                                   ^
                                                   |
                                               +-------+
                                               |Send   |
         .-------------------------------------|State  |
         |                                     |Machine|
         |                                     +-------+
         |                                         ^
         |                                         |
         |                                    Send Queue
         |                                +-----------------+
         |                                | | | | | | | | | |
         |                                +-----------------+
         |                                         ^
         |                                         |          LANai Memory
- - - - -|- - - - - - - - - - - - - - - - - - - - -|- - - - - - - - - - - -
         |                                         |   User Process Memory
         V                                         |
Receive Event Queue                                |
+-----------------+                      .---------'
| | | | | | | | | |                      |
+-----------------+                      |
        |                                |
        |                             .--^--.
        |   gm_send_with_callback(...,ptr,len,callback,context)
        |   ...                                  |        |
        `-> event=gm_receive();                  |        |
            switch(event.recv.type){             |        |
              ...                                |        |
              default:                           |        |
                gm_unknown(port,event);          |        |
            }                |     `-------------|--.     |
                             `--------.          |  |     |
                                .-----|----------'  |     |
                                |     |     .-------|-----'
                                |     |     |       |
                         + - - -|- - -|- - -|- - - -|- - - - - - - - - - - +
                         |      V     V     V       V                      |
                         |  callback(port,context,status)                  |
                         |                                                 |
                         |            [behind the scenes in gm_unknown()]  |
                         + - - - - - - - - - - - - - - - - - - - - - - - - +

The following functions require send tokens:

• gm_datagram_send()

• gm_directed_send_with_callback()

• gm_drop_sends()

• gm_resume_sending()

• gm_send_to_peer_with_callback()

• gm_send_with_callback()

The send token is implicitly returned to the client when the function's callback is called or, for the GM-1.0 functions gm_send() and gm_send_to_peer(), a send token is implicitly passed to the client with each pointer returned in a GM_SENT_EVENT. (The legacy GM_SENT_EVENTs are generated if and only if the legacy gm_send() and gm_send_to_peer() functions are called.)

3. User Token Flow (Receiving)

GM receives are also token regulated. After a port is opened, the client implicitly possesses gm_num_receive_tokens() receive tokens, allowing it to provide GM with up to this many receive buffers using gm_provide_receive_buffer(). With each call to gm_provide_receive_buffer(), the client implicitly relinquishes a receive token. With each buffer passed to gm_provide_receive_buffer(), the client passes a corresponding integer SIZE indicating that the length of the receive buffer is at least gm_max_length_for_size() bytes.

Before a client of a port can receive a message of a particular size and priority, the client software must provide GM with a receive token of matching size and priority. The receive token specifies the buffer in which to store the matching receive. When a message of matching size and priority is received, that message will be transferred into the receive buffer specified in the receive token. Note that multiple receive tokens of the same size and priority may be provided to the port.

After providing receive buffers with sizes matching the sizes of all packets that potentially could be received, the client must poll for receive events using a gm_receive(), gm_blocking_receive(), or gm_blocking_receive_no_spin() function. The gm_receive(), gm_blocking_receive(), or gm_blocking_receive_no_spin() function will return a gm_receive_event. The receipt of events of type GM_RECV_EVENT and GM_HIGH_RECV_EVENT describe received packets of low and high priority, respectively. All other events should be simply passed to gm_unknown(). Such events are used internally by GM for sundry purposes, and the client need not be concerned with the contents of unrecognized receive events unless otherwise stated in this document.

      Arriving Packet
            |
            | ,---------------------.
            V V                     |
         +-------+                  |
         |Receive|           Receive Buffer Pool
         |State  |           +-------------------+
         |Machine|           | | | | | | | | | | |
         +-------+           +-------------------+
             |                         ^                     LANai Memory
        - - -|- - - - - - - - - - - - -|- - - - - - - - - - - - - - - - -
             |                         |              User Process Memory
             V                         |
       Receive Event Queue             |
      +-------------------+            |
      | | | | | | | | | | |            |
      +-------------------+            `--- gm_provide_receive_buffer()
                 |                          ...
                 `------------------------- gm_receive()

To avoid deadlock of the port, the client software must ensure that the port is never without a receive token for any acceptable combination of size and priority for more than a bounded amount of time, that the port is informed which combinations of size and priority are not acceptable for receives, and that the client not send to any remote port that does not do likewise.

By convention, when a port runs out of low priority receive tokens for any combination of sizes, the client may defer replacing the receive tokens pending the completion of a bounded number of high priority sends, but must always replace exhausted types of high priority receive tokens without waiting for any sends to complete. Using this technique, reliable, deadlock-free, single-hop forwarding can be achieved.

The following functions require a receive token:

• gm_provide_receive_buffer()

• gm_provide_receive_buffer_with_tag()

A single receive token is passed to the client with each of the following events:

• GM_RAW_RECV_EVENT

• GM_RECV_EVENT

• GM_HIGH_RECV_EVENT

• GM_HIGH_PEER_RECV_EVENT

• GM_FAST_HIGH_RECV_EVENT

• GM_FAST_HIGH_PEER_RECV_EVENT

(However, if the client passes these events to gm_unknown(), then the token is implicitly returned to GM.) Any of the GM receive functions can generate these types of events. These functions are:

• gm_receive()

• gm_blocking_receive()

• gm_blocking_receive_no_spin()

---