4. Architecture

4.3.1.1. Sources Advertise

UM sources help UM receivers discover transport information in the following ways.

• Advertise Active Topics - Each source advertises its active topic first upon its creation and subsequently according to the resolver_advertisement_*_interval configuration options for the Initial and Sustaining Phases. Sources advertise by sending a Topic Information Record (TIR). (You can prevent a source from sending an advertisement upon creation with resolver_send_initial_advertisement.)

• Respond to Topic Queries - Each source responds immediately to queries from receivers about its topic.

Both a topic advertisement and a query response contain the topic's transport session information. Based on the transport type, a receiver can join the appropriate multicast group (for LBT-RM), send a connection request (for LBT-RU), connect to the source (for TCP) or access a shared memory area (for LBT-IPC). The address and port information potentially contained within a TIR includes:

• For a TCP transport, the source address and TCP port.

• For an LBT-RM transport, the unicast UDP port (to which NAKs are sent) and the UDP destination port.

• For an LBT-RU transport, the source address and UDP port.

• For an LBT-IPC transport, the Host ID, LBT-IPC Session ID and Transport ID.

• For the UM Gateway, the context instance and Domain ID of the original source plus the Hop Count and Portal Cost. See Forwarding Costs

• For various UMP options, the store address and TCP port, and the source address and TCP port (to which receivers send delivery confirmations).

• For UMQ, the Queue Name, which allows the receiver to then resolve the Queue in order to receive queued messages.

4.3.1.2. Receivers Query

Receivers can discover transport information in the following ways.

• Search advertisements collected in the resolver cache maintained by the UM context.

• Listen for source advertisements on the resolver_multicast_address:port.

• Send a topic query (TQR).

A new receiver queries for its topic according to the resolver_query_*_interval configuration options for the Initial and Sustaining Phases.

A Topic Query Record (TQR) consists primarily of the topic string. Receivers continue querying on a topic until they discover the number of sources configured by resolution_number_of_sources_query_threshold. However the large default of this configuration option (10,000,000) allows a receiver to continue to query until both the initial and sustaining phase of topic resolution complete.

4.3.1.3. Wildcard Receivers

Wildcard receivers can discover transport information in the following ways.

• Search advertisements collected in the resolver cache maintained by the UM context.

• Listen for source advertisements on the resolver_multicast_address:port.

• Send a wildcard receiver topic query (WC-TQR).

UM implements only one phase of wildcard receiver queries, sending wildcard receiver queries according to wildcard receiver resolver_query_*_interval configuration options until the topic pattern has been queried for the resolver_query_minimum_duration. The wildcard receiver topic query (WC-TQR) contains the topic pattern and the pattern_type.

4.3.2.1. Initial Phase

The initial topic resolution phase for a topic is an aggressive phase that can be used to resolve all topics before sending any messages. During the initial phase, network traffic and CPU utilization might actually be higher. You can completely disable this phase, if desired. See Disabling Aspects of Topic Resolution.

4.3.2.2. Sustaining Phase

The sustaining topic resolution phase follows the initial phase and can be a less active phase in which a new receiver resolves its topic. It can also act as the sole topic resolution phase if you disable the initial phase. The sustaining phase defaults use less network resources than the initial phase and can also be modified or disabled completely. See Disabling Aspects of Topic Resolution.

4.3.2.3. Quiescent Phase

This phase is the absence of topic resolution activity for a given topic. It is possible that some topics may be in the quiescent phase at the same time other topics are in initial or sustaining phases of topic resolution. This phase ends if either of the following occurs.

• A new receiver sends a query.

• Your application calls lbm_context_topic_resolution_request() that provokes the sending of topic queries for any receiver or wildcard receiver in this state.

4.3.3.1. Assigning Different Configuration Options to Individual Topics

You can assign different configuration option values to individual topics by accessing the topic attribute table (lbm_*_topic_attr_t_stct) before creating the source, receiver or wildcard receiver.

4.3.3.2. Multicast Network Options

Essentially, the _incoming and _outgoing versions of resolver_multicast_address/port provide more fine-grained control of topic resolution. By default, the resolver_multicast_address and resolver_multicast_port and the _incoming and _outgoing address and port are set to the same value. If you want your context to listen to a particular multicast address/port and send on another address/port, then you can set the _incoming and _outgoing configuration options to different values.

4.3.4.1. Topic Information Records

Of all topic resolution messages, only the TIR contains address and port information. This tells a receiver how it can get the data being published. Based on the transport type, a receiver can join the appropriate multicast group (for LBT-RM), send a connection request (for LBT-RU), or connect to the source (for TCP).

The TIR contains additional blocks of information to define UMP capabilities, indicating the address and port of the source and store.

The address and port information potentially contained within a TIR includes:

• For a TCP transport, the source address and TCP port.

• For an LBT-RM transport, the unicast UDP port (to which NAKs are sent) and the UDP destination port.

• For an LBT-RU transport, the source address and UDP port.

• For various UMP options, the store address and TCP port, and the source address and TCP port (to which delivery confirmations are sent).

• For UMQ, the Queue Name and TCP port.

For unicast-based transports (TCP and LBT-RU), the TIR source address is 0.0.0.0, not the actual source address. This allows some minimal functionality within a Network Address Translation (NAT) environment.

Topic resolution messages (whether received by the application via multicast, or by the unicast topic resolution daemon via unicast) are always UDP datagrams. They are received via a recvfrom() call, which also obtains the address and port from which the datagrams were received. If the address 0.0.0.0 (INADDR_ANY) appears for one of the addresses, lbmrd replaces it with the address from which the datagram is received. The net effect is as if the actual source address had originally been put into the TIR.

4.3.4.2. Unicast Topic Resolution Resilience

Running multiple instances of lbmrd allows your applications to continue operation in the face of a lbmrd failure. Your applications' sources and receivers send topic resolution messages as usual, however, rather than sending every message to each lbmrd instance, UM directs messages to lbmrd instances in a round-robin fashion. Since the lbmrd does not maintain any resolver state, as long as one lbmrd instance is running, UM continues to forward LBMR packets to all connected clients. UM switches to the next active lbmrd instance every 250-750 ms.

4.3.4.3. Unicast Topic Resolution Across Administrative Domains

If your network architecture includes remote or local LANs that use Network Address Translation (NAT), you can implement an lbmrd configuration file to translate IP addresses/ports across administrative domains. Without translation, lbmrd clients (sources and receivers) across NAT boundaries cannot connect to each other in response to topic advertisements due to NAT restrictions.

By default, topic advertisements forwarded by lbmrd contain the private (or inside) address/port of the source. Routers implementing NAT prevent connection to these private addresses from receivers outside the LAN.

The lbmrd configuration file allows lbmrd to insert a translation or outside address/port for the private address/port of the source in the topic advertisement. This outside or translation address must already be configured in the router's static NAT table. When the receiver attempts to connect to the source by using the source address/port in the topic advertisement, the NAT router automatically translates the outside address/port to the private address/port, thereby allowing the connection.

<?xml version="1.0" encoding="UTF-8" ?>
<lbmrd version="1.0">
    <domains>
     <domain name="domain-name-1">
         <network>network-specification</network>
     </domain>
     <domain name="domain-name-2">
         <network>network-specification</network>
     </domain>
    </domains>
    <transformations>
     <transform source="source-domain-name"
         destination="destination-domain-name">
         <rule>
          <match address="original-address" port="original-port"/>
          <replace address="replacement-address" port="replacement-port"/>
         </rule>
     </transform>
    </transformations>
</lbmrd>

4.4.1.1. Implicit Batching Operation

Implicit Batching buffers messages until:

• the buffer size exceeds the configured implicit_batching_minimum_length or

• the oldest message in the buffer has been in the buffer for implicit_batching_interval milliseconds.

When either condition is met, UM flushes the buffer, pushing the messages onto the network.

It may appear this design introduces significant latencies for low-rate topics. However, remember that Implicit Batching operates on a transport session basis. Typically many low-rate topics map to the same transport session, providing a high aggregate rate. The implicit_batching_interval option is a last resort to prevent messages from becoming stuck in the Implicit Batching buffer. If your UM deployment frequently uses the implicit_batching_interval to push out the data (i.e. if the entire transport session has periods of inactivity longer than the value of implicit_batching_interval (defaults to 200 ms), then either the implicit batching options need to be fine-tuned (reducing one or both), or you should consider an alternate form of batching. See Intelligent Batching.

The minimum value for the implicit_batching_interval is 3 milliseconds. The actual minimum amount of time that data stays in the buffer depends on your Operating System and its scheduling clock interval. For example, on a Solaris 8 machine, the actual time is approximately 20 milliseconds. On Microsoft Windows machines, the time is probably 16 milliseconds. On a Linux 2.6 kernel, the actual time is 3 milliseconds. Using a implicit_batching_interval value of 3 guarantees the minimum possible wait for whichever operating system you are using.

4.4.1.2. Implicit Batching Example

The following example demonstrates how the implicit_batching_minimum_length is actually a trigger or floor, for sending batched messages. It is sometimes misconstrued as a ceiling or upper limit.

  implicit_batching_minimum_length = 2000
       

1. The first send by your application puts 1900 bytes into the batching buffer, which is below the minimum, so UM holds it.

2. The second send fills the batching buffer to 3800 bytes, well over the minimum. UM sends it down to the transport layer, which builds a 3800-byte (plus overhead) datagram and sends it.

3. The Operating System fragments the datagram into packets independently of UM and reassembles them on the receiving end.

4. UM reads the datagram from the socket at the receiver.

5. UM parses out the two messages and delivers them to the appropriate topic levels, which deliver the data.

The proper setting of the implicit batching parameters often represents a tradeoff between latency and efficiency, where efficiency affects the highest throughput attainable. In general, a large minimum length setting increases efficiency and allows a higher peak message rate, but at low message rates a large minimum length can increase latency. A small minimum length can lower latency at low message rates, but does not allow the message rate to reach the same peak levels due to inefficiency. An intelligent use of implicit batching and application-level flushing can be used to implement an adaptive form of batching known as Intelligent Batching which can provide low latency and high throughput with a single setting.

4.4.4.1. Explicit Batching Example

The following example demonstrates explicit batching.

implicit_batching_minimum_length = 8000
       

1. Your application performs 10 sends of 100 bytes each as a single explicit batch.

2. At the 10th send (which completes the batch), UM delivers the 1000 bytes of messages to the implicit batch buffer.

3. Let's assume that the buffer already has 7899 bytes of data in it from other topics on the same transport session

4. UM adds the first 100-byte message to the buffer, bringing it to 7999.

5. UM adds the second 100-byte message, bringing it up to 8099 bytes, which exceeds implicit_batching_minimum_length but is below the 8192 maximum datagram size.

6. UM sends the 8099 bytes (plus overhead) datagram.

7. UM adds the third through tenth messages to the implicit batch buffer. These messages will be sent when either implicit_batching_minimum_length is again exceeded, or the implicit_batching_interval is met, or a message arrives in the buffer with the flush flag (LBM_MSG_FLUSH) set.