8. Sending Messages

In GM, message sends are regulated by a simple token-passing mechanism to prevent GM's bounded-size internal queues from overflowing. The client software must possess a send token before calling gm_send_with_callback(). After initialization, the client software implicitly possesses all gm_num_send_tokens() send tokens, and implicitly passes one token to the GM library with each call to gm_send_with_callback() or gm_send_to_peer_with_callback(). The token is retained by GM until the send completes, at which time GM calls the client-supplied callback, implicitly returning the send token to the client. The contents of the send message should not be modified in the interval between the call to gm_send() and the send completion, because doing so will cause undefined data to be delivered to the receiver.

The gm_send_with_callback() call requires the following parameters:

port

a pointer to the GM port over which the message is to be sent

message

a pointer to the data to be sent

size

the size receive buffer in which to store the message on the remote node

len

the number of bytes to send

priority

the priority with which to send the message (GM_HIGH_PRIORITY or GM_LOW_PRIORITY

target_node_id

the ID of the node to which the message should be sent

target_port_id

the ID of the GM port to which the message should be sent

callback

the client function to call when the send completes

context

a pointer to pass to the callback function when it is called

The order of messages with different priorities or with different destination ports is not preserved. Only the order of messages with the same priority and to the same destination port is preserved.

In the special case that the target_port_id is the same as the sending port ID (as is often the case), the streamlined gm_send_to_peer_with_callback() function may be used instead of gm_send_with_callback(), allowing the target_port_id parameter to be omitted, and slightly improving small-message performance on 32-bit Myrinet interfaces.