5. Programming Model

The GM communication system provides reliable, ordered delivery between communication endpoints, called "ports," with two levels of priority. This model is "connectionless" in that there is no need for client software to establish a connection with a remote port in order to communicate with it: the client software simply builds a message and sends it to any port in the network. (This apparently paradoxical "connectionless reliability" is achieved by GM maintaining reliable connections between each pair of hosts in the network and multiplexing the traffic between ports over these reliable connections.)

Under operating systems that provide memory protection, GM provides memory protected network access. It should be impossible for any non-privileged GM client application to use GM to access any memory other than the application's own memory, except as explicitly allowed by the GM API. The unforgeable source of each received message is available to the receiver, allowing the receiver to discard messages from untrusted sources.

The largest message GM can send or receive is limited to (2**31)-1 bytes. However, because send and receive buffers must reside in DMAable memory, the maximum message size is limited by the amount of DMAable memory the GM driver is allowed to allocate by the operating system. Most GM applications obtain DMAable memory using the straightforward gm_dma_malloc() and gm_dma_free() calls, but sophisticated applications with large memory requirements may perform DMA memory management using gm_register_memory() and gm_deregister_memory() to pin and unpin memory on operating systems that support memory registration.

Message order is preserved only for messages of the same priority, from the same sending port, and directed to the same receiving port. Messages with differing priority never block each other. Consequently, low priority messages may pass high priority messages, unlike in some other communication systems. Typical GM applications will either use only one GM priority, or use the high priority channel for control messages (such as client-to-client acks) or for single-hop message forwarding.

Both sends and receives in GM are regulated by implicit tokens, representing space allocated to the client in various internal GM queues, as depicted in the following figure. At initialization, the client implicitly possesses gm_num_send_tokens() send tokens, and gm_num_receive_tokens() receive tokens. The client may call certain functions only when possessing an implicit send or receive token, and in calling that function, the client implicitly relinquishes the token(1). The client program is responsible for tracking the number of tokens of each type that it possesses, and must not call any GM function requiring a token when the client does not possess the appropriate token. Calling a GM API function without the required tokens has undefined results, but GM usually reports such errors, and such errors will not cause system security to be violated.

As stated above, sends are token regulated. A client of a port may send a message only when it possesses a send token for that port. By calling a GM API send functions, the client implicitly relinquishes that send token. The client passes a callback and context pointer to the send function. When the send completes, GM calls callback, passing a pointer to the GM port, the client-supplied context pointer, and status code indicating if the send completed successfully or with an error. When GM calls the client's callback function, the send token is implicitly passed back to the client. Most GM programs, which rely on GM's fault tolerance to handle transient network faults, should consider a send completing with a status other than GM_SUCCESS to be a fatal error. However, more sophisticated programs may use the GM fault tolerance API extensions to handle such non-transient errors. These extensions are described in an appendix. It is important to note that the client-supplied callback function will be called only within a client's call to gm_unknown(), the GM unknown event handler function that the client must call when it receives an unrecognized event. The gm_unknown() function is described in more detail below.

GM receives are also token regulated. After a port is opened, the client implicitly possesses gm_num_receive_tokens() receive tokens, allowing it to provide GM with up to this many receive buffers using gm_provide_receive_buffer(). With each call to gm_provide_receive_buffer(), the client implicitly relinquishes a receive token. With each buffer passed to gm_provide_receive_buffer(), the client passes a corresponding integer size indicating that the length of the receive buffer is at least gm_max_length_for_size() bytes.

Before a client of a port can receive a message of a particular size and priority, the client software must provide GM with a receive token of matching size and priority. The receive token specifies the buffer in which to store the matching receive. When a message of matching size and priority is received, that message will be transferred into the receive buffer specified in the receive token. Note that multiple receive tokens of the same size and priority may be provided to the port.

After providing receive buffers with sizes matching the sizes of all packets that potentially could be received, the client must poll for receive events using a gm_*receive*() function. (Most developers who think polling is unacceptable in their application find that polling is fine as long as they do it in a separate thread.) The gm_*receive*() function will return a gm_receive_event. The receipt of events of type GM_RECV_EVENT and GM_HIGH_RECV_EVENT describe received packets of low and high priority, respectively. All other events should be simply passed to gm_unknown(). Such events are used internally by GM for sundry purposes, and the client need not be concerned with the contents of unrecognized receive events unless otherwise stated in this document.

To avoid deadlock of the port, the client software must ensure that the port is never without a receive token for any acceptable combination of size and priority for more than a bounded amount of time, that the port is informed which combinations of size and priority are not acceptable for receives, and that the client not send to any remote port that does not do likewise.

By convention, when a port runs out of low priority receive tokens for any combination of sizes, the client may defer replacing the receive tokens pending the completion of a bounded number of high priority sends, but must always replace exhausted types of high priority receive tokens without waiting for any sends to complete. Using this technique, reliable, deadlock-free, single-hop forwarding can be achieved.