2.6.1. Multi-Transport Threads

Part of UM's design is a single threaded model for message data delivery which reduces latency in the receiving CPU. UM, however, also has the ability to distribute data delivery across multiple CPUs by using a receiving thread pool. Receivers created with the configuration option, use_transport_thread set to 1 use a thread from the thread pool instead of the context thread. The option, receive_thread_pool_size controls the pool size.

As receivers discover new sources through Topic Resolution, UM assigns the network sockets created for the receivers to receive data to either the context thread (default) or to a thread from the pool if use_transport_thread is set for the receiver. It is important to understand that thread assignment occurs at the socket level - not the transport level. Transports aggregated on to the same network socket use the same thread.

UM distributes data from different sockets to different threads allowing better process distribution and higher aggregate throughput. Distributing transports across threads also ensures that activity on each transport has no impact on transports assigned to other threads leading to lower latencies in some traffic patterns, e.g. heavy loss conditions.

The following lists restrictions to using multi-transport threads.

• Only LBT-RM, LBT-RU, TCP and TCP-LB transport types may be distributed to threads.

• Multi-Transport threads are not supported under sequential mode.

• UM processes sources using the same transport socket, e.g. multicast address and port, on the same thread (regardless of the use_transport_thread setting. To leverage threading of different sources, assign each source to a different transport destination, e.g. multicast address/port.

• Hot failover sources using LBT-RM on the same topic must not be distributed across threads because they must share the same multicast address and port.

• Hot failover sources using other transport types may not be distributed across threads and must use the context thread.

• Each transport thread has its own Unicast Listener (request) port. Ultra Messaging recommends that you expand the range request_tcp_port_low - request_tcp_port_high to a larger range when using transport threads. When late join is occurring, UM creates a TCP connection from the transport thread to the source.

• Multi-transport threads are not recommended for use over the UM Gateway.

• Multi-Transport Threads do not support persistent stores (UMP) or persistent receivers

• Multi-Transport Threads do not support queues (UMQ) or queing receivers.

• Multi-Transport Threads are not compatible with UMDS Server or UMCache

3.3.1. Message Properties Object

The message properties object allows your application to insert named, typed metadata in topic messages, and to implement functionality that depends on the message properties. UM allows eight property types: boolean, byte, short, int, long, float, double, and string.

To use message properties, create a message properties object with lbm_msg_properties_create(). Then send topic messages with lbm_src_send_ex() (or LBMSource.send() in the Java API or .NET API) passing the message properties object through lbm_src_send_ex_info_t object. Set the LBM_SRC_SEND_EX_FLAG_PROPERTIES flag on the lbm_src_send_ex_info_t object to indicate that it includes properties.

Upon a receipt of a message with properties, your application can access the properties directly through the messages properties field, which is null if no properties are present. You can retrieve individual property values directly by name, or you can iterate over the collection of properties to determine which properties are present at runtime.

The UM message property object supports the standard JMS message properties specification.

3.3.2. Source Configuration and Transport Sessions

As with contexts, a source holds configuration information that is of source scope. This includes network options, operational options and reliability options for LBT-RU and LBT-RM. For example, each source can use a different transport and would therefore configure a different network address to which to send topic messages. See the UM Configuration Guide for source configuration options.

As stated in Transports, many topics (and therefore sources) can be mapped to a single transport. Many of the configuration options for sources actually control or influence transport session activity. If many sources are sending topic messages over a single transport session (TCP, LBT-RU or LBT-RM), UM only uses the configuration options for the first source assigned to the transport.

For example, if the first source to use a LBT-RM transport session sets the transport_lbtrm_transmission_window_size to 24 MB and the second source sets the same option to 2 MB, UMS assigns 24 MB to the transport session's transport_lbtrm_transmission_window_size.

The UM Configuration Guide identifies the source configuration options that may be ignored when UM assigns the source to an existing transport session. Log file warnings also appear when UM ignores source configuration options.

3.3.3. Zero Object Delivery (Source)

The Zero Object Delivery (ZOD) feature for Java and .NET lets sources deliver events to an application with no per-event object creation. (ZOD can also be utilized with context source events.) See Zero Object Delivery (ZOD) for information on how to employ ZOD.

3.4.1. Receiver Configuration and Transport Sessions

A receiver holds configuration information that is of receiver scope. This includes network options, operational options and reliability options for LBT-RU and LBT-RM. See the UM Configuration Guide for receiver configuration options.

As stated above in Source Configuration and Transport Sessions, many topics (and therefore receivers) can be mapped to a single transport. As with source configuration options, many receiver configuration options control or influence transport session activity. If many receivers are receiving topic messages over a single transport session (TCP, LBT-RU or LBT-RM), UM only uses the configuration options for the first receiver assigned to the transport.

For example, if the first receiver to use a LBT-RM transport session sets the transport_lbtrm_nak_generation_interval to 10 seconds and the second receiver sets the same option to 2 seconds, UMS assigns 10 seconds to the transport session's transport_lbtrm_nak_generation_interval.

The UM Configuration Guide identifies the receiver configuration options that may be ignored when UM assigns the receiver to an existing transport session. Log file warnings also appear when UM ignores receiver configuration options.

3.4.2. Wildcard Receiver

A wildcard receiver object is created by calling lbm_wildcard_rcv_create(). Instead of a topic object, the caller supplies a pattern which is used by UM to match multiple topics. Since the application doesn't explicitly lookup the topics, the topic attribute is passed into lbm_wildcard_rcv_create() so that options can be set. Also, wildcarding has its own set of options (e.g. pattern type).

The pattern supplied for wildcard matching is normally a general regular expression. There are two types of supported regular expressions that differ somewhat in the syntax of the patterns (see the wildcard_receiver pattern_type option in the UM Configuration Guide). Those types are:

1. PCRE - (recommended) the same form of regular expressions recognized by Perl; see http://perldoc.perl.org/perlrequick.html for details, or

2. regex - POSIX extended regular expressions; see http://www.freebsd.org/cgi/man.cgi?query=re_format&section=7 for details. Note that regex is not supported on all platforms.

A third type of wildcarding is appcb, in which the application defines its own algorithm to select topic names. When appcb is configured, the pattern parameter of lbm_wildcard_rcv_create() is ignored. Instead, an application callback function is configured (see the wildcard_receiver pattern_callback option in the UM Configuration Guide). UM then calls that application function with a topic name and the function can use whatever method is appropriate to decide if the topic should be included with the receiver.

Be aware that some platforms may not support all of the regular expression wildcard types. For example, UM does not support the use of Unicode PCRE characters in wildcard receiver patterns on any system that communicates with a HP-UX or AIX system. See the UM Knowledgebase article, Platform-Specific Dependencies for details. Also note that if UM topic resolution is configured to turn off source advertisements, then wildcard receivers must be configured for PCRE. The other wildcard types do not support receiver queries for topic resolution.

For an example of wildcard usage, see lbmwrcv.c

Users of TIBCO® SmartSockets™ will want to look at the UM Knowledgebase article, Wildcard Topic Regular Expressions.

3.4.3. Zero Object Delivery (ZOD)

The Zero Object Delivery (ZOD) feature for Java and .NET lets receivers (and sources) deliver messages and events to an application with no per-message or per-event object creation. This facilitates source/receiver applications that would require little to no garbage collection at runtime, producing lower and more consistent message latencies.

To take advantage of this feature, you must call dispose() on a message to mark it as available for reuse. To access data from the message when using ZOD, you use a specific pair of LBMMessage-class methods (see below) to extract message data directly from the message, rather than the standard data() method. Using the latter method creates a byte array, and consequently, an object. It is the subsequent garbage collecting to recycle those objects that can affect performance.

For using ZOD, the LBMMessage class methods are:

• Java: dispose(), dataBuffer(), and dataLength()

• .NET: dispose(), dataPointer(), and length()

On the other hand, you may need to keep the message as an object for further use after callback. In this case, ZOD is not appropriate and you must call promote() on the message, and also you can use data() to extract message data.

For more details see the Java API Overview or the .Net LBMMessage Class description. This feature does not apply to the C API.

3.6.1. Transport TCP

The TCP UMS transport uses normal TCP connections to send messages from sources to receivers. This is the default transport when it's not explicitly set. TCP is a good choice when:

1. Flow control is desired. I.e. when one or more receivers cannot keep up, it is desired to slow down the source. This is a "better late than never" philosophy.

2. Equal bandwidth sharing with other TCP traffic is desired. I.e. when it is desired that the source slow down when general network traffic becomes heavy.

3. There are few receivers listening to each topic. Multiple receivers for a topic requires multiple transmissions of each message, which places a scaling burden on the source machine and the network.

4. The application is not sensitive to latency. Use of TCP as a messaging transport can result in unbounded latency.

5. The messages must pass through a restrictive firewall which does not pass multicast traffic.

3.6.2. Transport TCP-LB

The TCP-LB UMS transport is a variation on the TCP transport which adds latency-bounded behavior. The source is not flow-controlled as a result of network congestion or slow receivers. So, for applications that require a "better never than late" philosophy, TCP-LB can be a better choice.

However, latency cannot be controlled as tightly as with UDP-based transports (see below). In particular, latency can still be introduced because TCP-LB shares bandwidth equally with other TCP traffic. It also has the same scaling issues as TCP when multiple receivers are present for each topic.

3.6.3. Transport LBT-RU

The LBT-RU UMS transport adds reliable delivery to unicast UDP to send messages from sources to receivers. This provides greater flexibility in the control of latency. For example, the application can further limit latency by allowing the use of arrival order delivery. See the UM Knowledgebase FAQ, Why can't I have low-latency delivery and in-order delivery?. Also, LBT-RU is less sensitive to overall network load; it uses source rate controls to limit its maximum send rate.

Since it is based on unicast addressing, LBT-RU can pass through most firewalls. However, it has the same scaling issues as TCP when multiple receivers are present for each topic.

3.6.4. Transport LBT-RM

The LBT-RM UMS transport adds reliable multicast to UDP to send messages. This provides the maximum flexibility in the control of latency. In addition, LBT-RM can scale effectively to large numbers of receivers per topic using network hardware to duplicate messages only when necessary at wire speed. One limitation is that multicast is often blocked by firewalls.

LBT-RM is a UDP-based, reliable multicast protocol designed with the use of UM and its target applications specifically in mind. The protocol is very similar to PGM, but with changes to aid low latency messaging applications.

• Topic Mapping - Several topics may map onto the same LBT-RM session. Thus a multiplexing mechanism to LBT-RM is used to distinguish topic level concerns from LBT-RM session level concerns (such as retransmissions, etc.). Each message to a topic is given a sequence number in addition to the sequence number used at the LBT-RM session level for packet retransmission.

• Negative Acknowledgments (NAKs) - LBT-RM uses NAKs as PGM does. NAKs are unicast to the sender. For simplicity, LBT-RM uses a similar NAK state management approach as PGM specifies.

• Time Bounded Recovery - LBT-RM allows receivers to specify a a maximum time to wait for a lost piece of data to be retransmitted. This allows a recovery time bound to be placed on data that has a definite lifetime of usefulness. If this time limit is exceeded and no retransmission has been seen, then the piece of data is marked as an unrecoverable loss and the application is informed. The data stream may continue and the unrecoverable loss will be ordered as a discrete event in the data stream just as a normal piece of data.

• Flexible Delivery Ordering - LBT-RM receivers have the option to have the data for an individual topic delivered "in order" or "arrival order". Messages delivered "in order" will arrive in sequence number order to the application. Thus loss may delay messages from being delivered until the loss is recovered or unrecoverable loss is determined. With "arrival-order" delivery, messages will arrive at the application as they are received by the LBT-RM session. Duplicates are ignored and lost messages will have the same recovery methods applied, but the ordering may not be preserved. Delivery order is a topic level concern. Thus loss of messages in one topic will not interfere or delay delivery of messages in another topic.

• Session State Advertisements - In PGM, SPM packets are used to advertise session state and to perform PGM router assist in the routers. For LBT-RM, these advertisements are only used when data is not flowing. Once data stops on a session, advertisements are sent with an exponential back-off (to a configurable maximum interval) so that the bandwidth taken up by the session is minimal.

• Sender Rate Control - LBT-RM can control a sender's rate of injection of data into the network by use of a rate limiter. This rate is configurable and will back pressure the sender, not allowing the application to exceed the rate limit it has specified. In addition, LBT-RM senders have control over the rate of retransmissions separately from new data. This allows sending application to guarantee a minimum transmission rate even in the face of massive loss at some or all receivers.

• Low Latency Retransmissions - LBT-RM senders do not mandate the use of NCF packets as PGM does. Because low latency retransmissions is such an important feature, LBT-RM senders by default send retransmissions immediately upon receiving a NAK. After sending a retransmission, the sender ignores additional NAKs for the same data and does not repeatedly send NCFs. The oldest data being requested in NAKs has priority over newer data so that if retransmissions are rate controlled, then LBT-RM sends the most important retransmissions as fast as possible.

3.6.5. Transport LBT-IPC

The LBT-IPC transport is an Interprocess Communication (IPC) UM transport that allows sources to publish topic messages to a shared memory area managed as a static ring buffer from which receivers can read topic messages. Message exchange takes place at memory access speed which can greatly improve throughput when sources and receivers can reside on the same host. LBT-IPC can be either source-paced or receiver-paced.

The LBT-IPC transport uses a "lock free" design that eliminates calls to the Operating System and allows receivers quicker access to messages. An internal validation method enacted by receivers while reading messages from the Shared Memory Area ensures message data integrity. The validation method compares IPC header information at different times to ensure consistent, and therefore, valid message data. Sources can send individual messages or a batch of messages, each of which possesses an IPC header.

$> lbtipc_resource_manager 
Displaying Resources (to reclaim you must type '-reclaim' exactly)

--Memory Resources--
 Memory resource: Process ID: 24441 SessionID: ab569cec XportID: 20001

--Semaphore Resources--
 Semaphore key: 0x68871d75
	Semaphore resource Index 0: reserved
	Semaphore resource: Process ID: 24441 Sem Index: 1
	Semaphore resource: Process ID: 24436 Sem Index: 2 

$> lbtipc_resource_manager -reclaim

Reclaiming Resources
 Process 24441 not found: reclaiming Memory resource (SessionID: ab569cec XPortID: 20001)
 Process 24441 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 1
 Process 24436 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 2 

$> lbtipc_resource_manager -reclaim

Reclaiming Resources
 Process 24441 still active! Memory resource not reclaimed (SessionID: ab569cec XPortID: 20001)
 Process 24441 still active! Semaphore resource not reclaimed (Key: 0x68871d75 Sem Index: 1)
 Process 24436 still active! Semaphore resource not reclaimed (Key: 0x68871d75 Sem Index: 2)

3.6.6. Transport LBT-RDMA

The LBT-RDMA transport is Remote Direct Memory Access (RDMA) UM transport that allows sources to publish topic messages to a shared memory area from which receivers can read topic messages. LBT-RDMA runs across InfiniBand and 10 Gigabit Ethernet hardware.

When you create a source with lbm_src_create() and you've set the transport option to RDMA, UM creates a shared memory area object on the sending machine's Host Channel Adapter (HCA) card. UM assigns one of the RDMA transport ports to this area specified with the UM context configuration options, transport_lbtrdma_port_high and transport_lbtrdma_port_low. You can also specify a shared memory location outside of this range with a source configuration option, transport_lbtrdma_port, to prioritize certain topics, if needed.

When you create a receiver with lbm_rcv_create() for a topic being sent over LBT-RDMA, UM creates a shared memory area on the receiving machine's HCA card. The network hardware immediately copies any new data from the sending HCA to the receiving HCA. UM receivers monitor the receiving shared memory area for new topic messages. You configure receiver monitoring with transport_lbtrdma_receiver_thread_behavior.

4.3.1. Multicast Topic Resolution

The following diagram depicts the UM topic resolution using multicast.

UM performs topic resolution automatically. Your application does not need to call any API functions to initiate topic resolution, however, you can influence topic resolution with Topic Resolution Configuration Options. Moreover, you can set configuration options for individual topics by using the lbm_*_attr_setopt() functions in your application. See Assigning Different Configuration Options to Individual Topics

4.3.2. Topic Resolution Phases

The phases of topic resolution pertain to individual topics. Therefore if your system has 100 topics, 100 different topic resolution advertisement and query phases may be running concurrently. This describes the three phases of Ultra Messaging topic resolution.

• Initial Phase

• Sustaining Phase

• Quiescent Phase

4.3.3. Topic Resolution Configuration Options

Refer to the UM Configuration Guide for specific information about Topic Resolution Configuration Options.

• Resolver Operation Options

• Multicast Resolver Network Options

• Unicast Resolver Network Options

• Wildcard Receiver Options

4.3.4. Unicast Topic Resolution

This section also discusses the following topics.

• Unicast Topic Resolution Resilience

• Unicast Topic Resolution Across Administrative Domains

By default UM expects multicast connectivity between all sources and receivers. When only unicast connectivity is available, you may configure all sources and receivers to use unicast topic resolution. This requires that you run one or more UM unicast topic resolution daemon(s) (Manpage for lbmrd), which perform the same topic resolution activities as multicast topic resolution. You configure each instance of the unicast topic resolution daemon with resolver_unicast_daemon.

The lbmrd can run on any machine, including the source or receiver (enter lbmrd -h for instructions). Of course, sources will also have to select a transport protocol that uses unicast addressing (e.g. TCP, TCP-LB, or LBT-RU). The lbmrd maintains a table of clients (address and port pairs) from which it has received a topic resolution message, which can be any of the following.

• Topic Information Records (TIR) - also known as topic advertisements

• Topic Query Records (TQR)

• keepalive messages, which are only used in unicast topic resolution

After lbmrd receives a TQR or TIR, it forwards it to all known clients. If a client (i.e. source or receiver) is not sending either TIRs or TQRs, it sends a keepalive message to lbmrd according to the resolver_unicast_keepalive_interval. This registration with the lbmrd allows the client to receive advertisements or queries from lbmrd. lbmrd maintains no state about topics, only about clients.

<?xml version="1.0" encoding="UTF-8" ?>
<lbmrd version="1.0">
	<domains>
	 <domain name="domain-name-1">
		 <network>network-specification</network>
	 </domain>
	 <domain name="domain-name-2">
		 <network>network-specification</network>
	 </domain>
	</domains>
	<transformations>
	 <transform source="source-domain-name"
		 destination="destination-domain-name">
		 <rule>
		  <match address="original-address" port="original-port"/>
		  <replace address="replacement-address" port="replacement-port"/>
		 </rule>
	 </transform>
	</transformations>
</lbmrd>

4.4.1. Implicit Batching

UM automatically batches smaller messages into transport session datagrams. The implicit batching options, implicit_batching_interval (default = 200 milliseconds) and implicit_batching_minimum_length (default = 2048 bytes) govern UM implicit message batching. Although these are source options, they actually apply to the transport session to which the source was assigned. See also Source Configuration and Transport Sessions.

UM establishes the implicit batching parameters when it creates the transport session. Any sources assigned to that transport session use the implicit batching limits set for that transport session, and the limits apply to any and all sources subsequently assigned to that transport session. This means that batched transport datagrams can contain messages on multiple topics. See Explicit Batching for information about topic-level message batching.

  implicit_batching_minimum_length = 2000
		

4.4.2. Adaptive Batching

Adaptive Batching is a convenience batching feature that attempts to send messages immediately during periods of low volume and automatically batch messages during periods of higher volume. The goal of Adaptive Batching is to automatically optimize throughput and latency by monitoring such things as the time between calls to lbm_src_send(), the time messages spend in the Implicit Batching queue, the Rate Controller queue, and other sending activities. With this information, Adaptive Batching determines if sending batched messages now or later produces the least latency.

Adaptive Batching will not satisfy everyone's requirements of throughput and latency. You only need to turn it on and determine if it produces satisfactory performance. If it does, you need do nothing more. If you are not satisfied with the results, simply turn it off.

You enable Adaptive Batching by setting implicit_batching_type to adaptive. When using Adaptive Batching, it is advisable to increase the implicit_batching_minimum_length option to a higher value.

4.4.3. Intelligent Batching

Intelligent Batching uses Implicit Batching along with your application's knowledge of the messages it must send. It is a form of dynamic adaptive batching that automatically adjusts for different message rates. Intelligent Batching can provide significant savings of CPU resources without adding any noticeable latency.

For example, your application might receive input events in a batch, and therefore know that it must produce a corresponding batch of output messages. Or the message producer works off of an input queue, and it can detect messages in the queue.

In any case, if the application knows that it has more messages to send without going to sleep, it simply does normal sends to UM, letting Implicit Batching send only when the buffer meets the implicit_batching_minimum_length threshold. However, when the application detects that it has no more messages to send after it sends the current message, it sets the FLUSH flag (LBM_MSG_FLUSH) when sending the message which instructs UM to flush the implicit batching buffer immediately by sending all messages to the transport layer. Refer to lbm_src_send() in the UMS API documentation (UM C API, UM Java API or UM .NET API) for all the available send flags.

When using Intelligent Batching, it is usually advisable to increase the implicit_batching_minimum_length option to 10 times the size of the average message, to a maximum value of 8196. This tends to strike a good balance between batching length and flushing frequency, giving you low latencies across a wide variation of message rates.

4.4.4. Explicit Batching

UM allows you to batch messages for a particular topic with explicit batching. When your application sends a message (lbm_src_send()) it may flag the message as being the start of a batch (LBM_MSG_START_BATCH) or the end of a batch (LBM_MSG_END_BATCH). All messages sent between the start and end are grouped together. The flag used to indicate the end of a batch also signals UM to send the message immediately to the implicit batching buffer. At this point, Implicit Batching completes the batching operation. UM includes the start and end flags in the message so receivers can process the batched messages effectively.

Unlike Intelligent Batching which allows intermediate messages to trigger flushing according to the implicit_batching_minimum_length option, explicit batching holds all messages until the batch is completed. This feature is useful if you configure a relatively small implicit_batching_minimum_length and your application has a batch of messages to send that exceeds the implicit_batching_minimum_length. By releasing all the messages at once, Implicit Batching maximizes the size of the network datagrams.

implicit_batching_minimum_length = 8000
		

4.4.5. Application Batching

In all of the above situations, your application sends individual messages to UM and lets UM decide when to push the data onto the wire (often with application help). With application batching, your application buffers messages itself and sends a group of messages to UM with a single send. Thus, UM treats the send as a single message. On the receiving side, your application needs to know how to dissect the UM message into individual application messages.

This approach is most useful for Java or .NET applications where there is a higher per-message cost in delivering an UM message to the application. It can also be helpful when using an event queue to deliver received messages. This imposes a thread switch cost for each UM message. At low message rates, this extra overhead is not noticeable. However, at high message rates, application batching can significantly reduce CPU overhead.

4.5.1. Sequence Number Order, Fragments Reassembled (Default Mode)

In this mode, a receiver's delivery controller delivers messages in sequence number order (the same order in which they are sent). This feature also guarantees reassembly of fragmented large messages. To enable sequence number ordered delivery, set the ordered_delivery configuration option as shown:

receiver ordered_delivery 1

Please note that ordered delivery can introduce latency when packets are lost.

4.5.2. Arrival Order, Fragments Not Reassembled

This mode allows messages to be delivered to the application in the order they are received. If a message is lost, UM will retransmit the message. In the meantime, any subsequent messages received are delivered immediately to the application, followed by the dropped packet when its retransmission is received. This mode guarantees the lowest latency.

With this mode, the receiver delivers messages larger than the transport's maximum datagram size as individual fragments. (See transport_*_datagram_max_size in the Ultra Messaging Configuration Guide.) The C API function, lbm_msg_retrieve_fragment_info() returns fragmentation information for the message you pass to it, and can be used to reassemble large messages. (In Java and .NET, LBMMessage provides methods to return the same fragment information.) Note that reassembly is not required for small messages.

To enable this no-reassemble arrival-order mode, set the following configuration option as shown:

receiver ordered_delivery 0

When developing message reassembly code, consider the following:

• Message fragments don't necessarily arrive in sequence number order.

• Some message fragments may never arrive (unrecoverable loss), so you must time out partial messages.

4.5.3. Arrival Order, Fragments Reassembled

This mode delivers messages in the order they are received, except for fragmented messages, which UM reassembles before delivering to your application. Your application can then use the sequence_number field of lbm_msg_t objects to order or discard messages.

To enable this arrival-order-with-reassembly mode, set the following configuration option as shown:

receiver ordered_delivery -1

5.1.1. Late Join Overview

The Late Join feature enables newly created receivers to receive previously transmitted messages. Sources configured for Late Join maintain a retention buffer (not to be confused with a transport retransmission window), which holds transmitted messages for late-joining receivers.

A Late Join operation follows the following sequence:

1. A new receiver configured for Late Join with use_late_join completes topic resolution. Topic advertisements from the source indicate that the source is configured for Late Join with late_join.

2. The new receiver sends a Late Join Initiation Request (LJIR) to request previously transmitted messages. The receiver configuration option, retransmit_request_outstanding_maximum, determines the number of messages the receiver requests.

3. The source responds with a Late Join Information (LJI) message containing the sequence numbers for the retained messages that are available for retransmission.

4. The source unicasts the messages.

5. When Configuring Late Join for Large Numbers of Messages, the receiver issues additional requests, and the source retransmits these additional groups of older messages, oldest first.

The source's retention buffer's is not pre-allocated and occupies an increasing amount of memory as the source sends messages and adds them to the buffer. If a retention buffer grows to a size equal to the value of the source configuration option, retransmit_retention_size_threshold, the source deletes older messages as it adds new ones. The source configuration option, retransmit_retention_age_threshold, controls message deletion based on message age.

5.1.2. Late Join With UMP

Late Join can be implemented in conjunction with UMP's persistent store feature, however in this configuration, it functions somewhat differently. After a Late-Join-enabled receiver has been created, resolved a topic, and become registered with a store, it may then request older messages. This request is handled by the store, which unicasts the retransmission messages. If the store does not have these messages, it requests them of the source (assuming option retransmission-request-forwarding is enabled), thus initiating Late Join.

Unlike with a persistent store, a source/topic using UMQ's queue feature will service Late Join requests in the same manner used by UMS.

5.1.3. Late Join Options Summary

Following is a summary of Late join configuration options. Please refer to UM Configuration Guide for full descriptions of these options.

5.1.4. Using Default Late Join Options

To implement Late Join with default options, set the Late Join configuration options to activate the feature on both a source and receiver in the following manner.

1. Create a configuration file with source and receiver Late Join activation options set to 1. For example, file cfg1.cfg containing the two lines:

   	 source late_join 1
   	 receiver use_late_join 1
   	 

2. Run an application that starts a Late-Join-enabled source. For example:

   	 lbmsrc -c cfg1.cfg -P 1000 topicName
   	 

3. Wait a few seconds, then run an application that starts a Late-Join-enabled receiver. For example:

   	 lbmrcv -c cfg1.cfg -v topicName
   	 

The output for each should closely resemble the following.

LBMSRC

	 $ lbmsrc -c cfg1.cfg -P 1000 topicName
	 LOG Level 5: NOTICE: Source "topicName" has no retention settings (1 message retained max)
	 Sending 10000000 messages of size 25 bytes to topic [topicName]
	 Receiver connect [TCP:10.29.3.77:34200]
	 

LBMRCV

	 $ lbmrcv -c cfg1.cfg -v topicName
	 Immediate messaging target: TCP:10.29.3.77:4391
	 [topicName][TCP:10.29.3.76:4371][2]-RX-, 25 bytes
	 1.001 secs. 0.0009988 Kmsgs/sec. 0.1998 Kbps
	 [topicName][TCP:10.29.3.76:4371][3], 25 bytes
	 1.002 secs. 0.0009982 Kmsgs/sec. 0.1996 Kbps
	 [topicName][TCP:10.29.3.76:4371][4], 25 bytes
	 1.003 secs. 0.0009972 Kmsgs/sec. 0.1994 Kbps
	 [topicName][TCP:10.29.3.76:4371][5], 25 bytes
	 1.003 secs. 0.0009972 Kmsgs/sec. 0.1994 Kbps
	 

Note that the source only retained 1 Late Join message (due to default retention settings) and that this message appears as a retransmit (-RX-). Also note that it is possible to sometimes receive 2 RX messages in this scenario (see Retransmitting Only Recent Messages.)

5.1.5. Specifying a Range of Messages to Retransmit

To receive more than one or two Late Join messages, increase the source's retransmit_retention_size_threshold from its default value of 0. Once this threshold is exceeded, the source now allows the next new message entering the retention buffer to bump out the oldest one. Note that this threshold's units are bytes (which includes a small overhead per message).

While the retention threshold endeavors to keep the buffer size close to its value, it does not set hard upper limit for retention buffer size. For this, the retransmit_retention_size_limit configuration option (also in bytes) sets this boundary.

Follow the steps below to demonstrate how a source can retain about 50MB of messages, but no more than 60MB:

1. Create a second configuration file (cfg2.cfg) with the following options:

   	 source late_join 1
   	 source retransmit_retention_size_threshold 50000000
   	 source retransmit_retention_size_limit 60000000
   	 receiver use_late_join 1

2. Run lbmsrc -c cfg2.cfg -P 1000 topicName.

3. Wait a few seconds and run lbmrcv -c cfg2.cfg -v topicName.

The output for each should closely resemble the following.

LBMSRC

	 $ lbmsrc -c cfg2.cfg -P 1000 topicName
	 Sending 10000000 messages of size 25 bytes to topic [topicName]
	 Receiver connect [TCP:10.29.3.76:34444]
	 

LBMRCV

	 $ lbmrcv -c cfg2.cfg -v topicName
	 Immediate messaging target: TCP:10.29.3.76:4391
	 [topicName][TCP:10.29.3.77:4371][0]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][1]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][2]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][3]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][4]-RX-, 25 bytes
	 1.002 secs. 0.004991 Kmsgs/sec. 0.9981 Kbps
	 [topicName][TCP:10.29.3.77:4371][5], 25 bytes
	 1.002 secs. 0.0009984 Kmsgs/sec. 0.1997 Kbps
	 [topicName][TCP:10.29.3.77:4371][6], 25 bytes
	 1.002 secs. 0.0009983 Kmsgs/sec. 0.1997 Kbps
	 [topicName][TCP:10.29.3.77:4371][7], 25 bytes
	 

Note that lbmrcv received live messages with sequence numbers 7, 6, and 5, and RX messages going from 4 all the way back to Sequence Number 0.

5.1.6. Retransmitting Only Recent Messages

Thus far we have worked with only source late join settings, but suppose that you want to receive only the last 10 messages. To do this, configure the receiver option retransmit_request_maximum option to set how many messages to request backwards from the latest message.

Follow the steps below to demonstrate setting this option to 10.

1. Add the following line to cfg2.cfg and rename it cfg3.cfg.

   receiver retransmit_request_maximum 10

2. Run lbmsrc -c cfg3.cfg -P 1000 topicName.

3. Wait a few seconds and run lbmrcv -c cfg3.cfg -v topicName.

The output for each should closely resemble the following.

LBMSRC

	 $ lbmsrc -c cfg3.cfg -P 1000 topicName
	 Sending 10000000 messages of size 25 bytes to topic [topicName]
	 Receiver connect [TCP:10.29.3.76:34448]
	 

LBMRCV

	 $ lbmrcv -c cfg3.cfg -v topicName
	 Immediate messaging target: TCP:10.29.3.76:4391
	 [topicName][TCP:10.29.3.77:4371][13]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][14]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][15]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][16]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][17]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][18]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][19]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][20]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][21]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][22]-RX-, 25 bytes
	 [topicName][TCP:10.29.3.77:4371][23]-RX-, 25 bytes
	 1.002 secs. 0.01097 Kmsgs/sec. 2.195 Kbps
	 [topicName][TCP:10.29.3.77:4371][24], 25 bytes
	 1.002 secs. 0.0009984 Kmsgs/sec. 0.1997 Kbps
	 [topicName][TCP:10.29.3.77:4371][25], 25 bytes
	 1.002 secs. 0.0009984 Kmsgs/sec. 0.1997 Kbps
	 [topicName][TCP:10.29.3.77:4371][26], 25 bytes
	 

Note that 11, not 10, retransmits were actually received. This can happen because network and timing circumstances may have one RX already in transit while the specific RX amount is being processed. (Hence, please note that it is not possible to guarantee one and only one RX message for every possible Late Join recovery.)

5.1.7. Configuring Late Join for Large Numbers of Messages

Suppose you have a receiver that comes up at midday and must gracefully catch up on the large number of messages it has missed. The following discussion explains the relevant Late Join options and how to use them.

5.2.1. OTR Overview

When a transport cannot recover lost messages, OTR engages and looks to the source for message recovery. It does this by accessing the source's retention buffer (used also by the Late Join feature) to re-request messages that no longer exist in a transport's transmission window or other places such as a UMP store or redundant source.

OTR functions in a manner very similar to that of Late Join, but differs mainly in that it activates in message loss situations rather than following the creation of a receiver, and shares only the source late_join option setting.

Upon detecting loss, a receiver using TCP, TCP-LB, LBT-IPC or LBT-RDMA initiates OTR by sending repeated, spaced, OTR requests to the source, until it recovers lost messages or a timeout period elapses.

OTR operates independently from transport-level recovery mechanisms such as NAKs for LBT-RU or LBT-RM. When you enable OTR for a receiver with use_otr, the otr_request_initial_delay starts as soon as the delivery controller detects a sequence gap. OTR recovery initiates if the gap is not resolved by the end of the delay interval. OTR recovery can occur before, during or after transport-level recovery attempts.

When a receiver initiates OTR, the intervals between OTR requests increases twofold after each request, until the maximum interval is reached (assuming the receiver is still waiting to receive the retransmission). You use configuration options otr_request_minimum_interval and otr_request_maximum_intervalto set the initial (minimum) and maximum intervals, respectively.

The source retransmits lost messages to the recovered receiver via unicast.

5.2.2. OTR With UMP

You can implement OTR in conjunction with UMP's persistent store feature, however in this configuration, it functions somewhat differently. If an OTR-enabled receiver registered with a store detects a sequence gap in the live stream and that gap is not resolved by other means within the next otr_request_initial_delay, the receiver requests those messages from the store(s). If the store does not have some of the requested messages, they will be requested from the source. Regardless of whether the messages are recovered from a store or from the source, OTR delivers all recovered messages the LBM_MSG_OTR flag, unlike Late Join, which uses the LBM_MSG_RETRANSMIT flag.

Unlike with a persistent store, a source/topic using UMQ's queue feature services OTR requests in the same manner used by UMS.

5.2.3. OTR Options Summary

The following set of configuration options govern OTR functionality. Please refer to the Ultra Messaging Configuration Guide for full descriptions of these options. You can click the individual links below for each option's description.

5.3.1. Request Message

UM provides three ways to send a request message.

• lbm_send_request() to send a request to a topic via a source object. Uses the standard source-based transports (TCP, LBT-RM, LBT-RU).

• lbm_multicast_immediate_request() to send a request to a topic as a multicast immediate message. See Multicast Immediate Messaging.

• lbm_unicast_immediate_request() to send a request to a topic as a unicast immediate message. See Multicast Immediate Messaging.

The request function returns a request object and defines an application callback for responses that allows the receiving application to send a response directly to the requesting application via a special TCP connection instead of a normal data transport. The requesting application -- not UM -- determines how many responses it needs. Therefore, it must delete the request object when it no longer wants to receive responses by calling lbm_request_delete(). It discards any responses that arrive after the request object has been deleted.

5.3.2. Response Message

An application responds to an UM request message by calling lbm_send_response(). Contained within that request message's header is a response object, which serves as a return address to the requester. UM passes the response object to lbm_send_response(). Since the response object is part of the message header, it is deleted at the same time that the message is deleted. Therefore, if the sending of the response cannot be done within the responder's receive callback, the message must be retained and subsequently deleted.

5.3.3. TCP Management

UM creates and manages the special TCP connections for responses, maintaining a list of active response connections . When an application sends a response, UM scans that list for an active connection to the destination. If it doesn't find a connection for the response, it creates a new connection and adds it to the list. After the lbm_send_response() function returns, UM schedules the response_tcp_deletion_timeout, which defaults to 2 seconds. If a second request comes in from the same application before the timer expires, the responding application simply uses the existing connection and restarts the deletion timer.

It is conceivable that a very large response could take more than the response_tcp_deletion_timeout default (2 seconds) to send to a slow-running receiver. In this case, UM automatically increases the deletion timer as needed to ensure the last message completes.

5.3.4. Configuration

See the UM Configuration Guide for the descriptions of the Request/Response configuration options.

• Request Network Options

• Request Operations Options

• Response Operation Options

5.3.5. Example Applications

UM includes two example applications that illustrate Request/Response.

• lbmreq.c - application that sends requests on a given topic (single source) and waits for responses. See also the Java example, lbmreq.java, and the .NET example, lbmreq.cs.

• lbmresp.c - application that waits for requests and sends responses back on a given topic (single receiver). See also the Java example, lbmresp.java, and the .NET example, lbmresp.cs.

We can demonstrate a series of 5 requests and responses with the following procedure.

1. Run lbmresp -v topicname

2. Run lbmreq -R 5 -v topicname

LBMREQ

Output for lbmreq should resemble the following.

$ lbmreq -R 5 -q topicname
Event queue in use
Using TCP port 4392 for responses
Delaying requests for 1000 milliseconds
Sending request 0
Starting event pump for 5 seconds.
Receiver connect [TCP:10.29.1.78:4958]
Done waiting for responses. 1 responses (25 bytes) received. Deleting request. Sending request 1
Starting event pump for 5 seconds.
Done waiting for responses. 1 responses (25 bytes) received. Deleting request. Sending request 2
Starting event pump for 5 seconds.
Done waiting for responses. 1 responses (25 bytes) received. Deleting request. Sending request 3
Starting event pump for 5 seconds.
Done waiting for responses. 1 responses (25 bytes) received. Deleting request. Sending request 4
Starting event pump for 5 seconds.
Done waiting for responses. 1 responses (25 bytes) received. Deleting request.
Quitting...
		

LBMRESP

Output for lbmresp should resemble the following.

$ lbmresp -v topicname
Request [topicname][TCP:10.29.1.78:14371][0], 25 bytes
Sending response. 1 responses of 25 bytes each (25 total bytes).
Done sending responses. Deleting response.
Request [topicname][TCP:10.29.1.78:14371][1], 25 bytes
Sending response. 1 responses of 25 bytes each (25 total bytes).
Done sending responses. Deleting response.
Request [topicname][TCP:10.29.1.78:14371][2], 25 bytes
Sending response. 1 responses of 25 bytes each (25 total bytes).
Done sending responses. Deleting response.
Request [topicname][TCP:10.29.1.78:14371][3], 25 bytes
Sending response. 1 responses of 25 bytes each (25 total bytes).
Done sending responses. Deleting response.
Request [topicname][TCP:10.29.1.78:14371][4], 25 bytes
Sending response. 1 responses of 25 bytes each (25 total bytes).
Done sending responses. Deleting response.
[topicname][TCP:10.29.1.78:14371], End of Transport Session
		

5.5.1. Typical PDM Usage Patterns

The typical PDM usage patterns can usually be broken down into two categories: sources (which need to serialize a message for sending) and receivers (which need to deserialize a message to extract field values). However, for optimum performance for both sources and receivers, first set up the definition and a single instance of the message only once during a setup or initialization phase, as in the following example workflow:

1. Create a definition and set its id and version.

2. Add field information to the definition to describe the types of fields to be in the message.

3. Create a single instance of a message based on the definition.

Set up a source to do the following:

1. Add field values to the message instance.

2. Serialize the message so that it can be sent.

Likewise, set up a receiver to do the following:

1. Deserialize the received bytes into the message instance.

2. Extract the field values from the message.

5.5.2. Getting Started

PDM APIs are provided in C, Java, and C#, however, the examples in this section are Java based.

private PDMDefinition defn;
private PDMMessage msg;
private PDMFieldInfo fldInfo100;
private PDMFieldInfo fldInfo101;
private PDMFieldInfo fldInfo102;

public void setupPDM() {
	//Create the definition with 3 fields and using int field names
	defn = new PDMDefinition(3, true);
	
	//Set the definition id and version
	defn.setId(1001);
	defn.setMsgVersMajor((byte)1);
	defn.setMsgVersMinor((byte)0);
	
	//Create information for a boolean, int32, and float fields (all required)
	fldInfo100 = defn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	fldInfo101 = defn.addFieldInfo(101, PDMFieldType.INT32, true);
	fldInfo102 = defn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	
	//Finalize the definition and create the message
	defn.finalizeDef();
	msg = new PDMMessage(defn);
}

public void sourceUsePDM() {
	//Call the function to setup the definition and message
	setupPDM();
	
	//Example values for the message
	boolean fld100Val = true;
	int fld101Val = 7;
	float fld102Val = 3.14F;
	
	//Set each field value in the message
	msg.setFieldValue(fldInfo100, fld100Val);
	msg.setFieldValue(fldInfo101, fld101Val);
	msg.setFieldValue(fldInfo102, fld102Val);
	
	//Serialize the message to bytes
	byte[] buffer = msg.toBytes();
}
    

private PDMDefinition defn;
private PDMMessage msg;
private PDMFieldInfo fldInfo100;
private PDMFieldInfo fldInfo101;
private PDMFieldInfo fldInfo102;

public void setupPDM() {
	//Create the definition with 3 fields and using int field names
	defn = new PDMDefinition(3, true);
	
	//Set the definition id and version
	defn.setId(1001);
	defn.setMsgVersMajor((byte)1);
	defn.setMsgVersMinor((byte)0);
	
	//Create information for a boolean, int32, and float field (all required)
	fldInfo100 = defn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	fldInfo101 = defn.addFieldInfo(101, PDMFieldType.INT32, true);
	fldInfo102 = defn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	
	//Finalize the definition and create the message
	defn.finalizeDef();
	msg = new PDMMessage(defn);
}

public void receiverUsePDM(byte[] buffer) {
	//Call the function to setup the definition and message
	setupPDM();
	
	//Values to be retrieved from the message
	boolean fld100Val;
	int fld101Val;
	float fld102Val;
	
	//Deserialize the bytes into a message
	msg.parse(buffer);
	
	//Get each field value from the message
	fld100Val = msg.getFieldValueAsBoolean(fldInfo100);
	fld101Val = msg.getFieldValueAsInt32(fldInfo101);
	fld102Val = msg.getFieldValueAsFloat(fldInfo102);
}
    

5.5.3. Using the PDM API

The following code snippets expand upon the previous examples to demonstrate the usage of additional PDM functionality (but use "..." to eliminate redundant code).

...
public void setupPDM() {
	//Create the definition with 3 fields and using string field names
	defn = new PDMDefinition(3, false);
	...
	//Create information for a boolean, int32, and float field (all required)
fldInfo100 = defn.addFieldInfo("Field100", PDMFieldType.BOOLEAN, true);
	fldInfo101 = defn.addFieldInfo("Field101", PDMFieldType.INT32, true);
	fldInfo102 = defn.addFieldInfo("Field102", PDMFieldType.FLOAT, true);
	...
}
...
    

private PDMDefinition defn;
private PDMMessage msg;

public void setupPDM() {
	...
	//Create information for a boolean, int32, and float field (all required)
	defn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	defn.addFieldInfo(101, PDMFieldType.INT32, true);
	defn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	...
}

public void receiverUsePDM(byte[] buffer) {
	...	
	//Get each field value from the message
	fld100Val = msg.getFieldValueAsBoolean(defn.getFieldInfo(100));
	fld101Val = msg.getFieldValueAsInt32(defn.getFieldInfo(101));
	fld102Val = msg.getFieldValueAsFloat(defn.getFieldInfo(102));
}

...
private PDMFieldInfo fldInfo103;
...
public void setupPDM() {
	...
//Create information for a boolean, int32, and float field (all required)
	// as well as an optional int8 field
fldInfo100 = defn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	fldInfo101 = defn.addFieldInfo(101, PDMFieldType.INT32, true);
	fldInfo102 = defn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	fldInfo103 = defn.addFieldInfo(103, PDMFieldType.INT8, false);
	...
}

public void sourceUsePDM() {
	...
	//Set each field value in the message
	// except do not set the optional field
	msg.setFieldValue(fldInfo100, fld100Val);
	msg.setFieldValue(fldInfo101, fld101Val);
	msg.setFieldValue(fldInfo102, fld102Val);
...
}
    

...
private PDMFieldInfo fldInfo103;
...
public void setupPDM() {
	...
//Create information for a boolean, int32, and float field (all required)
	// as well as an optional int8 field
	fldInfo103 = defn.addFieldInfo(103, PDMFieldType.INT8, false);
	...
}
public void receiverUsePDM(byte[] buffer) {
	...
	byte fld103Val;
	...
	
	if(msg.isFieldValueSet(fldInfo103)) {
		fld103Val = msg.getFieldValueAsInt8(fldInfo103);
	}
}
    

...
private PDMFieldInfo fldInfo104;
...
public void setupPDM() {
	...
	fldInfo104 = defn.addFieldInfo(104, PDMFieldType.FIX_STRING, 12, true);
	...
}

public void sourceUsePDM() {
	...
	String fld104Val = "Hello World!";
	
	//Set each field value in the message
	// except do not set the optional field
	msg.setFieldValue(fldInfo100, fld100Val);
	msg.setFieldValue(fldInfo101, fld101Val);
	msg.setFieldValue(fldInfo102, fld102Val);
	msg.setFieldValue(fldInfo104, fld104Val);
...
}
    

...
private PDMFieldInfo fldInfo104;
...
public void setupPDM() {
	...
	fldInfo104 = defn.addFieldInfo(104, PDMFieldType.FIX_STRING, 12, true);
	...
}
public void receiverUsePDM(byte[] buffer) {
	...
	String fld104Val;
	...
		
	fld104Val = msg.getFieldValueAsString(fldInfo104);
}
    

...
private PDMFieldInfo fldInfo105;
...
public void setupPDM() {
	...
	fldInfo105 = defn.addFieldInfo(105, PDMFieldType.STRING, true);
	...
}

public void sourceUsePDM() {
	...
	String fld105Val = "variable length value";
	...
	msg.setFieldValue(fldInfo105, fld105Val);
...
}
    

...
private PDMFieldInfo fldInfo105;
...
public void setupPDM() {
	...
	fldInfo105 = defn.addFieldInfo(105, PDMFieldType.STRING, true);
	...
}
public void receiverUsePDM(byte[] buffer) {
	...
	String fld105Val;
	...
		
	fld105Val = msg.getFieldValueAsString(fldInfo105);
}
    

...
private PDMFieldInfo fldInfo106;
private PDMFieldInfo fldInfo107;
private PDMFieldInfo fldInfo108;
...
public void setupPDM() {
	...	
	//Create information for a boolean, int32, and float field (all required)
	// as well as an optional int8 field
	...
	//A required, fixed size array of 3 boolean elements
	fldInfo106 = defn.addFieldInfo(106, PDMFieldType.BOOLEAN_ARR, true, 3);
	//An optional, variable size array of int32 elements
	fldInfo107 = defn.addFieldInfo(107, PDMFieldType.INT32_ARR, false);
	//A required, fixed size array of 2 element which are each 5 character strings
	fldInfo108 = defn.addFieldInfo(108, PDMFieldType.FIX_STRING_ARR, 5, true, 2);
	...
}

public void sourceUsePDM() {
...
	
	//Example values for the message
	...
	boolean fld106Val[] = {true, false, true};
	int fld107Val[] = {1, 2, 3, 4, 5};
	String fld108Val[] = {"aaaaa", "bbbbb"};
	
	//Set each field value in the message
	...
	msg.setFieldValue(fldInfo106, fld106Val);
	msg.setFieldValue(fldInfo107, fld107Val);
	msg.setFieldValue(fldInfo108, fld108Val);
	
	...
}
    

...
private PDMFieldInfo fldInfo106;
private PDMFieldInfo fldInfo107;
private PDMFieldInfo fldInfo108;
...
public void setupPDM() {
	...	
	//Create information for a boolean, int32, and float field (all required)
	// as well as an optional int8 field
	...
	//A required, fixed size array of 3 boolean elements
	fldInfo106 = defn.addFieldInfo(106, PDMFieldType.BOOLEAN_ARR, true, 3);
	//An optional, variable size array of int32 elements
	fldInfo107 = defn.addFieldInfo(107, PDMFieldType.INT32_ARR, false);
	//A required, fixed size array of 2 element which are each 5 character strings
	fldInfo108 = defn.addFieldInfo(108, PDMFieldType.FIX_STRING_ARR, 5, true, 2);
	...
}

public void receiverUsePDM(byte[] buffer) {
	...
	
	//Values to be retrieved from the message
	...
	boolean fld106Val[];
	int fld107Val[];
	String fld108Val[];
	
	//Deserialize the bytes into a message
	msg.parse(buffer);
	
	//Get each field value from the message
	...
	fld106Val = msg.getFieldValueAsBooleanArray(fldInfo106);
	if(msg.isFieldValueSet(fldInfo107)) {
		fld107Val = msg.getFieldValueAsInt32Array(fldInfo107);
	}
	fld108Val = msg.getFieldValueAsStringArray(fldInfo108);
	
}
    

private PDMDefinition defn;
private PDMMessage msg;

public void setupPDM() {
	//Create the definition with 3 fields and using int field names
	defn = new PDMDefinition(3, true);
		
	...

	//Finalize the definition and create the message
	defn.finalizeDef();
	msg = new PDMMessage(defn);

//Set the flag to indicate that the definition should also be serialized
	msg.setIncludeDefinition(true);
}

...
    

private PDMDefinition defn;
private PDMMessage msg;

public void setupPDM() {
	//Don.t define a definition

	//Create a message without a definition since the incoming message will have it

	msg = new PDMMessage();
}

...
    

...

public void setupPDM() {
	//Create the definition with 3 fields and using int field names
	defn = new PDMDefinition(3, true);
	
	//Set the definition id and version
	defn.setId(1001);
	defn.setMsgVersMajor((byte)1);
	defn.setMsgVersMinor((byte)0);
	
	//Create information for a boolean, int32, and float field (all required)
	// as well as an optional int8 field
	fldInfo100 = defn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	fldInfo101 = defn.addFieldInfo(101, PDMFieldType.INT32, true);
	fldInfo102 = defn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	fldInfo103 = defn.addFieldInfo(103, PDMFieldType.INT8, false);
	fldInfo104 = defn.addFieldInfo(104, PDMFieldType.FIX_STRING, 12, true);
	fldInfo105 = defn.addFieldInfo(105, PDMFieldType.STRING, true);
	//A required, fixed size array of 3 boolean elements
	fldInfo106 = defn.addFieldInfo(106, PDMFieldType.BOOLEAN_ARR, true, 3);
	//An optional, variable size array of int32 elements
	fldInfo107 = defn.addFieldInfo(107, PDMFieldType.INT32_ARR, false);
	//A required, fixed size array of 2 element which are each 5 character strings
	fldInfo108 = defn.addFieldInfo(108, PDMFieldType.FIX_STRING_ARR, 5, true, 2);
	
	//Finalize the definition and create the message
	defn.finalizeDef();
	msg = new PDMMessage(defn);
}

public void receiveAndIterateMessage(byte[] buffer) {
	msg.parse(buffer);
	PDMFieldIterator iterator = msg.createFieldIterator();
	PDMField field = null;
	while(iterator.hasNext()) {
		field = iterator.next();
		System.out.println("Field set? " +field.isValueSet());
		switch(field.getIntName()) {
			case 100:
				boolean val100 = field.getBooleanValue();
				System.out.println(
						"Field 100's value is: " + val100);
				break;
			case 101:
				int val101 = field.getInt32Value();
				System.out.println(
						"Field 101's value is: " + val101);
				break;
			case 102:
				float val102 = field.getFloatValue(); 
				System.out.println(
						"Field 102's value is: " + val102);
				break;
			default:
				//Casting to object is possible but not recommended
				Object value = field.getValue();
				int name = field.getIntName();
				System.out.println(
						"Field " + name + "'s value is: " + value);
		}
	}
}
    

Field set? true
Field 100's value is: true
Field set? true
Field 101's value is: 7
Field set? true
Field 102's value is: 3.14
Field set? false
Field 103's value is: null
Field set? true
Field 104's value is: Hello World!
Field set? true
Field 105's value is: Variable
Field set? true
Field 106's value is: [Z@527736bd
Field set? true
Field 107's value is: [I@10aadc97
Field set? true
Field 108's value is: [Ljava.lang.String;@4178460d
    

public void createAndStoreDefinition() {
	PDMDefinition myDefn = new PDMDefinition(3, true);
	//Set the definition id and version
	myDefn.setId(2001);
	myDefn.setMsgVersMajor((byte)1);
	myDefn.setMsgVersMinor((byte)0);
	
	//Create information for a boolean, int32, and float field (all required)
	myDefn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	myDefn.addFieldInfo(101, PDMFieldType.INT32, true);
	myDefn.addFieldInfo(102, PDMFieldType.FLOAT, true);

	myDefn.finalizeDef();
	
	PDMDefinitionCache.getInstance().put(myDefn);
}

public void createMessageUsingCache() {
	PDMDefinition myFoundDefn = PDMDefinitionCache.getInstance().get(2001, 1, 0);
	if(myFoundDefn != null) {
		PDMMessage myMsg = new PDMMessage(myFoundDefn);
		//Get FieldInfo from defn and then set field values in myMsg
		//...
	}
}
    

public void createAndStoreDefinition() {
	PDMDefinition myDefn = new PDMDefinition(3, true);
	//Set the definition id and version
	myDefn.setId(2001);
	myDefn.setMsgVersMajor((byte)1);
	myDefn.setMsgVersMinor((byte)0);
	
	//Create information for a boolean, int32, and float field (all required)
	myDefn.addFieldInfo(100, PDMFieldType.BOOLEAN, true);
	myDefn.addFieldInfo(101, PDMFieldType.INT32, true);
	myDefn.addFieldInfo(102, PDMFieldType.FLOAT, true);
	
	myDefn.finalizeDef();
	
	PDMDefinitionCache.getInstance().put(myDefn);
	
	//Create and store other definitions
	//...
}

public void receiveKnownMessages(byte[] buffer) {
	PDMMessage myMsg = new PDMMessage();
	//Set the flag that enables messages to try
	// looking up the definition in the cache automatically
	// when parsing a byte buffer
	myMsg.setTryToLoadDefFromCache(true);
	myMsg.parse(buffer);
	
	if(myMsg.getDefinition().getId() == 2001 
			&& myMsg.getDefinition().getMsgVersMajor() == 1
			&& myMsg.getDefinition().getMsgVersMinor() == 0) {
		
		PDMDefinition myDefn = PDMDefinitionCache.getInstance().get(2001, 1, 0);
		PDMFieldInfo fldInfo100 = myDefn.getFieldInfo(100);
		PDMFieldInfo fldInfo101 = myDefn.getFieldInfo(101);
		PDMFieldInfo fldInfo102 = myDefn.getFieldInfo(102);
		
		boolean fld100Val;
		int fld101Val;
		float fld102Val;
		
		//Get each field value from the message
		fld100Val = myMsg.getFieldValueAsBoolean(fldInfo100);
		fld101Val = myMsg.getFieldValueAsInt32(fldInfo101);
		fld102Val = myMsg.getFieldValueAsFloat(fldInfo102);
		
		System.out.println(fld100Val + " " + fld101Val + " " + fld102Val);
	}
}
    

5.5.4. Migrating from SDM

Applications using SDM with a known set of message fields are good candidates for migrating from SDM to PDM. With SDM, the source typically adds fields to an SDM message without a definition. But, as shown above in the PDM examples, creating/adding a PDM definition before adding field values is fairly straightforward.

However, certain applications may be incapable of building a definition in advance due to the ad-hoc nature of their messaging needs, in which case a self-describing format like SDM may be preferred.

public class Migration {
	
	//SDM Variables
	private LBMSDMessage srcSDMMsg;
	private LBMSDMessage rcvSDMMsg;
	
	//PDM Variables
	private PDMDefinition defn;
	private PDMFieldInfo fldInfo100;
	private PDMFieldInfo fldInfo101;
	private PDMFieldInfo fldInfo102;
	private PDMMessage srcPDMMsg;
	private PDMMessage rcvPDMMsg;
	

public static void main(String[] args) {
		Migration app = new Migration();
		System.out.println("Setting up PDM Definition and Message");
		app.setupPDM();
		System.out.println("Setting up SDM Messages");
		app.setupSDM();
		
		byte[] sdmBuffer;
		sdmBuffer = app.sourceCreateMessageWithSDM();
		app.receiverParseMessageWithSDM(sdmBuffer);
		
		byte[] pdmBuffer;
		pdmBuffer = app.sourceCreateMessageWithPDM();
		app.receiverParseMessageWithPDM(pdmBuffer);
		
	}

	public void setupSDM() {
		rcvSDMMsg = new LBMSDMessage();
		srcSDMMsg = new LBMSDMessage();
	}
	
	public void setupPDM() {
		//Create the definition with 3 fields and using int field names
		defn = new PDMDefinition(3, false);
		
		//Set the definition id and version
		defn.setId(1001);
		defn.setMsgVersMajor((byte)1);
		defn.setMsgVersMinor((byte)0);
		
		//Create information for a boolean, int32, and float field (all required)
		// as well as an optional int8 field
		fldInfo100 = defn.addFieldInfo("Field100", PDMFieldType.INT8, true);
		fldInfo101 = defn.addFieldInfo("Field101", PDMFieldType.INT16, true);
		fldInfo102 = defn.addFieldInfo("Field102", PDMFieldType.INT32, true);
		
		//Finalize the definition and create the message
		defn.finalizeDef();
		srcPDMMsg = new PDMMessage(defn);
		rcvPDMMsg = new PDMMessage(defn);
	}
	
	public byte[] sourceCreateMessageWithSDM() {
		byte[] buffer = null;
		
		LBMSDMField fld100 = new LBMSDMFieldInt8("Field100", (byte)0x42);
		LBMSDMField fld101 = new LBMSDMFieldInt16("Field101", (short)0x1ead);
		LBMSDMField fld102 = new LBMSDMFieldInt32("Field102", 12345);
		LBMSDMFields fset = new LBMSDMFields();

		try {
			fset.add(fld100);
			fset.add(fld101);
			fset.add(fld102);
		} catch (LBMSDMException e) {
			System.out.println ( e );
		}
		
		
		srcSDMMsg.set(fset);
		try {
			buffer = srcSDMMsg.data();
		} catch (IndexOutOfBoundsException e) {
			System.out.println ( "SDM Exception occurred during build of message:" );
			System.out.println ( e.toString() );
		} catch (LBMSDMException e) {
			System.out.println ( e.toString() );
		}
		return buffer;
		
	}
	
	public byte[] sourceCreateMessageWithPDM() {
		//Set each field value in the message
		srcPDMMsg.setFieldValue(fldInfo100, (byte)0x42);
		srcPDMMsg.setFieldValue(fldInfo101, (short)0x1ead);
		srcPDMMsg.setFieldValue(fldInfo102, 12345);
		
		//Serialize the message to bytes
		byte[] buffer = srcPDMMsg.toBytes();
		return buffer;
	}
	
	public void receiverParseMessageWithSDM(byte[] buffer) {
		//Values to be retrieved from the message
		byte fld100Val;
		short fld101Val;
		int fld102Val;
		
		//Deserialize the bytes into a message
		try {
			rcvSDMMsg.parse(buffer);
		} catch (LBMSDMException e) {
			System.out.println(e.toString());
		}
		
		LBMSDMField fld100 = rcvSDMMsg.locate("Field100");
		LBMSDMField fld101 = rcvSDMMsg.locate("Field101");
		LBMSDMField fld102 = rcvSDMMsg.locate("Field102");
		
		//Get each field value from the message
		fld100Val = ((LBMSDMFieldInt8)fld100).get();
		fld101Val = ((LBMSDMFieldInt16)fld101).get();;
		fld102Val = ((LBMSDMFieldInt32)fld102).get();;
		
		
		System.out.println("SDM Results: Field100=" + fld100Val +
				", Field101=" + fld101Val +
				", Field102=" + fld102Val);
		
	}
	
	public void receiverParseMessageWithPDM(byte[] buffer) {		
		//Values to be retrieved from the message
		byte fld100Val;
		short fld101Val;
		int fld102Val;
		
		//Deserialize the bytes into a message
		rcvPDMMsg.parse(buffer);
		
		//Get each field value from the message
		fld100Val = rcvPDMMsg.getFieldValueAsInt8(fldInfo100);
		fld101Val = rcvPDMMsg.getFieldValueAsInt16(fldInfo101);
		fld102Val = rcvPDMMsg.getFieldValueAsInt32(fldInfo102);
		
		
		System.out.println("PDM Results: Field100=" + fld100Val +
				", Field101=" + fld101Val +
				", Field102=" + fld102Val);
		
	}
	
}
    

		byte[] blob = new byte[25];
		LBMSDMRawBlob rawSDMBlob = new LBMSDMRawBlob(blob);
		try {
			LBMSDMField fld103 = new LBMSDMFieldBlob("Field103",rawSDMBlob);
		} catch (LBMSDMException e1) {
			System.out.println(e1);
		}
    

private PDMFieldInfo fldInfo103;	

public void setupPDM() {
		...
		fldInfo103 = defn.addFieldInfo("Field103", PDMFieldType.BLOB, true);
		...
	}
...
		byte[] blob = new byte[25];
		srcPDMMsg.setFieldValue(fldInfo103, blob);
    

PDMDecimal decimal = new PDMDecimal((long)2, (byte)32);
srcPDMMsg.setFieldValue(fldInfo103, decimal);
    

5.6.1. Temporary Transport Session

MIM uses the same reliable multicast algorithms as LBT-RM. When a sending application sends a message with lbm_multicast_immediate_message(), MIM creates a temporary transport session. Note that no topic-level source object is created.

MIM automatically deletes the temporary transport session after a period of inactivity defined by mim_src_deletion_timeout which defaults to 30 seconds. A subsequent send creates a new transport session. Due to the possibility of head-loss in the switch, it is recommended that sending applications use a long deletion timeout if they continue to use MIM after significant periods of inactivity.

MIM forces all topics across all sending applications to be concentrated onto a single multicast address to which ALL applications listen, even if they aren't interested in any of the topics. Thus, all topic filtering must happen in UM.

MIM can also be used to send an UM request message with lbm_multicast_immediate_request(). For example, an application can use MIM to request initialization information right when it starts up. MIM sends the response directly to the initializing application, avoiding the topic resolution delay inherent in the normal source-based lbm_send_request() function.

5.6.2. Receiving Immediate Messages

MIM does not require any special type of receiver. It uses the topic-based publish/subscribe model so an application must still create a receiver for a topic to receive MIM messages.

5.6.3. MIM Configuration

As of UM 3.1, MIM supports ordered delivery. As of UM 3.3.2, the MIM configuration option, mim_ordered_delivery defaults to ordered delivery. A byproduct of MIM ordered delivery is cross-topic ordering, which normal source-based UM senders cannot guarantee.

See the UM Configuration Guide for the descriptions of the MIM configuration options.

• Multicast Immediate Messaging Network Options

• Multicast Immediate Messaging Reliability Options

• Multicast Immediate Messaging Operation Options

5.6.4. MIM Example Applications

UM includes two example applications that illustrate MIM.

• lbmimsg.c - application that sends immediate messages as fast as it can to a given topic (single source). See also the Java example, lbmimsg.java and the .NET example, lbmimsg.cs.

• lbmireq.c - application that sends immediate requests to a given topic (single source) and waits for responses.

Immediate messaging target: TCP:10.29.1.78:14391
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][0], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][1], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][2], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][3], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][4], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][5], 25 bytes
[topicName][LBTRM:10.29.1.78:14390:644c8862:224.10.10.21:14401][6], 25 bytes
  

$ lbmrcv -v topicName
Immediate messaging target: TCP:10.29.1.78:14391
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
[topicName][LBTRM:10.29.1.78:14390:92100885:224.10.10.21:14401][0], Request
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
1     secs.  0     Kmsgs/sec.  0     Kbps
  

$ lbmireq topicName
Using TCP port 4392 for responses
Sending 1 requests of size 25 bytes to target <> topic <topicName>
Sending request 0
Sent request 0. Pausing 5 seconds.
Done waiting for responses. 0 responses (0 bytes) received. Deleting request
Quitting...
Lingering for 5 seconds...
  

5.7.1. Performance Pluses

The use of separate callbacks for different channels improves filtering and also relieves the source application of the task of including filtering information in the message data.

Java and .NET performance also receives a boost because messages not of interest can be discarded before they transition to the Java or .NET level.

5.7.2. Configuration Options

Spectrum's default behavior delivers messages on any channels the receiver has subscribed to on the callbacks specified when subscribing, and all other messages on the receiver's default callback. This behavior can be changed with the following configuration options.

• null_channel_behavior - behavior for messages delivered with no channel information.

• unrecognized_channel_behavior - behavior for messages delivered with channel information but are on a channel for which the receiver has not registered interest.

• channel_map_tablesz - controls the size of the table used by a receiver to store channel subscriptions.

5.8.1. Implementing Hot Failover Sources

You create Hot Failover sources with lbm_hf_src_create(). This returns a source object with internal state information that lets it send HF messages. You delete HF sources with the lbm_src_delete() function.

HF sources send HF messages via lbm_hf_src_send_ex() or lbm_hf_src_sendv_ex(). These functions take a sequence number, supplied via the exinfo object, that HF receivers use to identify the same message sent from different HF sources. The exinfo has an hf_sequence_number, with a flag (LBM_SRC_SEND_EX_FLAG_HF_32 or LBM_SRC_SEND_EX_FLAG_HF_64) that identifies whether it's a 32- or 64-bit number. Each HF source sends the same message content for a given sequence number, which must be coordinated by your application.

If the source needs to restart its sequence number to an earlier value (e.g. start of day; not needed for normal wraparound), delete and re-create the source and receiver objects. Without re-creating the objects, the receiver sees the smaller sequence number, assumes the data is duplicate, and discards it. In (and only in) cases where this cannot be done, use lbm_hf_src_send_rcv_reset().

Please be aware that non-HF receivers created for an HF topic receive multiple copies of each message. We recommend you establish local conventions regarding the use of HF sources, such as including "HF" in the topic name.

For an example source application, see lbmhfsrc in the UM Examples Page.

5.8.2. Implementing Hot Failover Receivers

You create HF receivers with lbm_hf_rcv_create(), and delete them using lbm_hf_rcv_delete() and lbm_hf_rcv_delete_ex().

Incoming messages have an hf_sequence_number field containing the sequence number, and a message flag (LBM_MSG_FLAG_HF_32 or LBM_MSG_FLAG_HF_64) noting the bit size.

For the maximum time period to recover lost messages, the HF receiver uses the minimum of the LBT-RM and LBT-RU NAK generation intervals (transport_lbtrm_nak_generation_interval, transport_lbtru_nak_generation_interval). Each transport protocol is configured as normal, but the lost message recovery timer is the minimum of the two settings.

Some lbm_msg_t objects coming from HF receivers may be flagged as having "passed through" the HF receiver. This means that the message has not been ordered with other HF messages. These messages have the LBM_MSG_FLAG_HF_PASS_THROUGH flag set. UM flags messages sent from HF sources using lbm_src_send() in this manner, as do all non-HF sources. Also, UM flags EOS, no source notification, and requests in this manner as well.

For an example receiver application, see lbmhfrcv in the UM Examples Page.

5.8.3. Implementing Hot Failover Wildcard Receivers

To create an HF wildcard receiver, set option hf_receiver to 1, then create a wildcard receiver with lbm_wildcard_rcv_create(). This actually creates individual HF receivers on a per-topic basis, so that each topic can have its own set of HF sequence numbers. Once the HF wildcard receiver detects that all sources for a particular topic are gone it closes the individual topic HF receivers and discards the HF sequence information (unlike a standard HF receiver). You can extend or control the delete timeout period of individual HF receivers with option resolver_no_source_linger_timeout.

5.8.4. Java and .NET

For information on implement the HF feature in a Java application, go to UM Java API and see the documentation for classes LBMHotFailoverReceiver and LBMHotFailoverSource.

For information on implement the HF feature in a .NET application, go to UM .NET API and navigate to Namespaces -> com.latencybusters.lbm -> LBMHotFailoverReceiver and LBMHotFailoverSource.

5.8.5. Using Hot Failover with UMP

When implementing Hot Failover with UMP, you must consider the following impact on hardware resources:

• Additional storage space required for a UMP disk store

• Higher disk activity

• Higher network activity

• Increased application complexity regarding message filtering

Also note that you must enable UME explicit ACKs and Hot Failover duplicate delivery in each Hot Failover receiving application.

For detailed information on using Hot Failover with UMP, see the Knowledge Base article UMP Hot Failover.

5.8.6. Hot Failover Intentional Gap Support

UM supports intentional gaps in HF message streams. Your HF sources can supply message sequence numbers with number gaps up to 1073741824. HF receivers automatically detect the gaps and consider any missing message sequence numbers as not sent and do not attempt recovery for these missing sequence numbers. See the following example.

HF source 1 sends message sequence numbers: 10, 11, 12, 13, 25, 26, 38
HF source 2 sends message sequence numbers: 10, 11, 12, 13, 25, 26, 38
 
HF receiver 1 receives message sequence numbers in order with no pause between any messages: 
                                            10, 11, 12, 13, 25, 26, 38
		

5.8.7. Hot Failover Optional Messages

Hot Failover sources can send optional messages that HF receivers can be configured to receive or not receive ( hf_optional_messages). HF receivers detect an optional message by checking lbm_msg_t.flags for LBM_MSG_FLAG_HF_OPTIONAL. HF sources indicate an optional message by passing LBM_SRC_SEND_EX_FLAG_HF_OPTIONAL in the lbm_src_send_ex_info_t.flags field to lbm_hf_src_send_ex() or lbm_hf_src_sendv_ex(). In the examples below, optional messages appear with an "o" after the sequence number.

HF source 1 sends message sequence numbers: 10, 11, 12, 13o, 14o, 15, 16o, 17o, 18o, 19o, 20
HF source 2 sends message sequence numbers: 10, 11, 12, 13o, 14o, 15, 16o, 17o, 18o, 19o, 20
 
HF receiver 1 receives:                     10, 11, 12, 13o, 14o, 15, 16o, 17o, 18o, 19o, 20
HF receiver 2, configured to ignore optional messages, receives: 
                                            10, 11, 12,           15,                     20
		

5.8.8. Using Hot Failover with Ordered Delivery

An HF receiver takes some of its operating parameters directly from the receive topic attributes. The ordered_delivery setting indicates the ordering for the HF receiver. Please see Ordered Delivery for information on the different modes of delivery order.

5.8.9. Hot Failover Across Multiple Contexts

If you have a receiving application on a multi-homed machine receiving HF messages from HF sources, you can set up the Hot Failover Across Contexts (HFX) feature. This involves setting up a separate UM context to receive HF messages over each NIC and then creating an HFX Object, which drops duplicate HF messages arriving over all contexts. Your receiving application then receives only one copy of each HF message. The HFX feature achieves the same effect across multiple contexts as the normal Hot Failover feature does within a single context.

The following diagram displays Hot Failover operation across UM contexts.

For each context that receives HF messages, create one HFX Receiver per topic. Each HFX Receiver can be configured independently by passing in a UM Receiver attributes object during creation. A unique client data pointer can also be associated with each HFX Receiver. The HFX Object is a special Ultra Messaging object and does not live in any UM context.

The following outlines the general procedure for HFX.

1. Create an HFX Object for every HF topic of interest with lbm_hfx_create(), passing in an attributes object created with lbm_hfx_attr_create() to specify any attributes desired.

2. Create a context for the first NIC receiving HF messages with lbm_context_create().

3. Create a HFX Receiver for every HF topic with lbm_hfx_rcv_create(), passing in UM Receive Topic Attributes.

4. Repeat steps 2 and 3 for all NICs receiving HF message

5. Receive messages. The HFX Object identifies and drops all duplicates, delivering messages through a single callback (and optional event queue) specified when you created the HFX Object.

Delete each HFX Receiver with lbm_hfx_rcv_delete() or lbm_hfx_rcv_delete_ex(). Delete the HFX Object with lbm_hfx_delete().

• Hot Failover Operation Options for HFX Configuration Options.

• LBMHFX*.java in UM Java API.

• LBMHFX*.cs in UM .NET API.

6.1.1. Why Monitor?

Monitoring can aid different groups within an organization.

• Developers can spot bugs that impact system performance.

• Performance tuning groups can pinpoint under-performing receivers.

• Testing groups can understand the reaction of a system to stresses like random packet loss during pre-production testing.

• Network or middleware management groups can use monitoring to ensure a production system continues to operate within its design criteria.

6.1.2. What to Monitor

Before discussing the monitoring statistics that are built into UM, we mention two things that are probably more important to monitor: connectivity and latency. UM provides some assistance for monitoring these, but the final responsibility rests with your applications.

6.2.1. Context Statistics

Context statistics help you monitor topic resolution activity, along with the number of unknown messages received and the number of sends and responses that were blocked or returned EWOULDBLOCK. Context statistics also contain transport statistics for Multicast Immediate Messaging (MIM) activity and transport statistics for all the sources or receivers in a context.

6.2.2. Event Queue Statistics

Event Queue statistics help you monitor the number of events currently on the queue, how long it takes to service them (maximum, minimum and mean service times) and the total number of events for the monitoring period. These statistics are available for the following types of events.

• Data messages

• Request messages

• Immediate messages

• Wildcard receiver messages

• I/O events

• Timer events

• Source events

• Unblock events

• Cancel events

• Callback events

• Context source events

• Total events

• Age of events

When monitoring Event Queue statistics you must enable the Event Queue UM Configuration Options, queue_age_enabled, queue_count_enabled and queue_service_time_enabled . UM disables these options by default, which produces no event queue statistics.

6.2.3. Transport Statistics

You can retrieve transport statistics for different types of transports (TCP, LBT-RU, LBT-RM, LBT-IPC, LBT-RDMA). In addition, you can limit these transport statistics to a specifc source sending on the particular transport or a specifc receiver receiving messages over the transport. Source statistics for LBT-RM, for example, include the number of messages (datagrams) sent and the number of retransmissions sent. For receiver LBT-RM, statistics include, for example, the number of messages (datagrams) received and number of UM messages received.

6.3.1. UMS Monitoring Process Flow

The overall process flow appears in the diagram below.

1. Your application creates the monitor source controller, specifying the format and transport modules to use. It also calls lbmmon functions to start monitoring an UM context, UM source or UM receiver.

2. The monitor source controller passes those statistics to the format module serialization function.

3. The monitor source controller passes the resulting serialized data to the transport module send function.

4. The transport module transmits the data over some transport medium (such as a network).

5. The monitor receiver controller transport module receives the serialized data. (Your monitoring application has already created the monitor receiver controller specifying the format and transport modules to use, along with the application callback functions to use upon the receipt of UM source or UM receiver statistics data.)

6. The monitor receiver controller calls the format module's de-serialization function.

7. Finally, the monitor receiver controller passes the statistics to your monitoring application via the specified application callback functions.

Your applications only calls functions in the controller modules, which calls the appropriate functions in the transport and format modules.

6.3.2. API Framework Flexibility

The segregation of UM Monitoring into control, format, and transport modules provides flexibility for monitor receivers in two ways.

• Allows you to use languages for which no UM API or binding exists.

• Allows you to use monitoring products which do not integrate with UM.

As an example, assume you have a Perl application which currently gathers statistics from other network applications (or, you are simply most comfortable working in Perl for such tasks). There is no Perl binding for UM. However, Perl can handle UDP packets very nicely, and can pick apart CSV data easily. By implementing a UDP transport module to be used by the monitor sources, your Perl application can read the UDP packets and process the statistics.

6.3.3. Initial Monitoring Questions

If you can answer the following questions, you're already on your way.

1. What format module will you use? LBMMON CSV Format module or a different one.

2. What transport module will you use? One of the 3 LBMMON modules or a different one.

3. Do you want to monitor individual sources/receivers, or an entire context? The difference is in how the statistics are aggregated.

   • Monitoring a context aggregates transport statistics for all sources and receivers associated with a context, by transport. Note that this is not by transport type. The default configuration for TCP, for example, allocates up to 10 ports, forming up to 10 separate transport sessions. Absent any specific instructions, UM allocates sources and receivers to these 10 transports in a round-robin fashion. So the statistics for a specific transport on a context will aggregate all sources and receivers which use that specific transport.

   • Ultra Messaging recommends that you monitor either a context or source/receiver, but not both. For example if Topic1 and Topic2 are mapped to the same transport session (which is the only transport session for the context) and you monitor both the receivers and the context, you will get 3 identical sets of statistics: one for Topic1 reporting the stats for it's transport session, one for Topic2 reporting the stats for the same transport session, and one for the transport session via the context.

   • In the case of wildcard receivers, only the context may be monitored. UM creates wildcard receivers dynamically as it detects topics which match the wildcard pattern. The application does not have access to these dynamically-created receivers. So the only way to monitor a wildcard receiver is to monitor the context on which it was created.

4. Should statistics be sent automatically, or on demand?

   • Automatic sending of statistics is by far the simplest approach. You simply indicate how often the statistics should be gathered and sent. The rest is taken care of.

   • On-demand is somewhat more involved. Your application decides when statistics should be gathered and sent. If you intend to use the arrival of statistics as a type of heartbeat, this is the method you should use.

The following sections present more discussion and sample source code about starting monitor sources, monitor receivers and the LBMMON format and transport modules.

6.3.4. Creating a Monitoring Source

The following examples demonstrate how to use the UM Monitoring API to enable monitoring in your application.

First, create a monitoring source controller:

lbm_context_t * ctx;
lbm_src_t * src;
lbm_rcv_t * rcv;
lbmmon_sctl_t * monctl;

if (lbmmon_sctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), NULL) == -1)
{
		fprintf(stderr, "lbmmon_sctl_create() failed\n");
		exit(1);
}

The above code tacitly assumes that the ctx, src, and rcv variables have been previously assigned via the appropriate UM API calls.

The monitoring source controller object must be passed to subsequent calls to reference a specific source controller. One implication of this is that it is possible to have multiple monitoring source controllers within a single application, each perhaps monitoring a different set of objects.

In the above example, the default CSV format module and default UM transport module are specified via the provided module functions lbmmon_format_csv_module() and lbmmon_transport_lbm_module().

6.3.5. Specifying the Object to Monitor

Once a monitoring source controller is created, the application can monitor a specific context using:

if (lbmmon_context_monitor(monctl, ctx, NULL, 10) == -1)
{
		fprintf(stderr, "lbmmon_context_monitor() failed\n");
		exit(1);
}

The above example indicates that statistics for all transports on the specified context will be gathered and sent every 10 seconds.

A UM source can be monitored using:

if (lbmmon_src_monitor(monctl, src, NULL, 10) == -1)
{
		fprintf(stderr, "lbmmon_src_monitor() failed\n");
		exit(1);
}

Finally, an UM receiver can be monitored using:

if (lbmmon_rcv_monitor(monctl, rcv, NULL, 10) == -1)
{
		fprintf(stderr, "lbmmon_rcv_monitor() failed\n");
		exit(1);
}

The two above examples also request that statistics for all transports on the specified source or receiver be gathered and sent every 10 seconds.

Statistics can also be gathered and sent in an on-demand manner. Passing 0 for the Seconds parameter to lbmmon_context_monitor(), lbmmon_src_monitor(), or lbmmon_rcv_monitor() prevents the automatic gathering and sending of statistics. To trigger the gather/send process, use:

lbmmon_sctl_sample(monctl);

Such a call will perform a single gather/send action on all monitored objects (contexts, sources, and receivers) which were registered as on-demand.

As part of application cleanup, the created monitoring objects should be destroyed. Each individual object can be de-registered using lbmmon_context_unmonitor(), lbmmon_src_unmonitor(), or lbmmon_rcv_unmonitor(). Finally, the monitoring source controller can be destroyed using:

lbmmon_sctl_destroy(monctl);

Any objects which are still registered will be automatically de-registered by lbmmon_sctl_destroy().

6.3.6. Receiving Monitoring Data

To make use of the statistics, an application must be running which receives the monitor data. This application creates a monitoring receive controller, and specifies callback functions which are called upon the receipt of source or receiver statistics data.

Use the following to create a monitoring receive controller:

lbmmon_rctl_t * monctl;
lbmmon_rctl_attr_t * attr;
lbmmon_rcv_statistics_func_t rcvcb = { rcv_statistics_cb };
lbmmon_src_statistics_func_t srccb = { src_statistics_cb };
lbmmon_evq_statistics_func_t evqcb = { evq_statistics_cb };
lbmmon_ctx_statistics_func_t ctxcb = { ctx_statistics_cb };

if (lbmmon_rctl_attr_create(&attr) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_create() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_RECEIVER_CALLBACK, (void *) &rcvcb, sizeof(rcvcb)) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_SOURCE_CALLBACK, (void *) &srccb, sizeof(srccb)) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_EVENT_QUEUE_CALLBACK, (void *) &evqcb, sizeof(evqcb)) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_CONTEXT_CALLBACK, (void *) &sctxcb, sizeof(ctxcb)) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), (void *) 
transport_options, attr) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_create() failed, %s\n", lbmmon_errmsg());
	exit(1);
}
if (lbmmon_rctl_attr_delete(attr) != 0)
{
	fprintf(stderr, "call to lbmmon_rctl_attr_delete() failed, %s\n", lbmmon_errmsg());
	exit(1);
}

As in the earlier example, the default CSV format module and default UM transport module are specified via the provided module functions lbmmon_format_csv_module() and lbmmon_transport_lbm_module().

As an example of minimal callback functions, consider the following example:

void rcv_statistics_cb(const void * AttributeBlock, const lbm_rcv_transport_stats_t * Statistics)
{
	lbm_ulong_t source = LBMMON_ATTR_SOURCE_NORMAL;
	if (lbmmon_attr_get_source(AttributeBlock, &source) != 0)
	{
		source = LBMMON_ATTR_SOURCE_NORMAL;
	}
	switch (Statistics->type)
	{
		case LBM_TRANSPORT_STAT_TCP:
			handle_rcv_tcp_statistics();
			break; 
		case LBM_TRANSPORT_STAT_LBTRM:
			switch (source)
			{
				case LBMMON_ATTR_SOURCE_IM:
					handle_rcv_im_lbtrm_statistics();
					break;
				default:
					handle_rcv_lbtrm_statistics();
					break;
			}
			break;
		case LBM_TRANSPORT_STAT_LBTRU:
			handle_rcv_lbtru_statistics();
			break;
	}
}

void src_statistics_cb(const void * AttributeBlock, const lbm_src_transport_stats_t * Statistics)
{
	lbm_ulong_t source = LBMMON_ATTR_SOURCE_NORMAL;
	if (lbmmon_attr_get_source(AttributeBlock, &source) != 0)
	{
		source = LBMMON_ATTR_SOURCE_NORMAL;
	}
	switch (Statistics->type)
	{
		case LBM_TRANSPORT_STAT_TCP:
			handle_src_tcp_statistics();
			break;
		case LBM_TRANSPORT_STAT_LBTRM:
			switch (source)
			{
				case LBMMON_ATTR_SOURCE_IM:
					handle_src_im_lbtrm_statistics();
					break;
				default:
					handle_src_lbtrm_statistics();
					break;
			}
			break;
		case LBM_TRANSPORT_STAT_LBTRU:
			handle_src_lbtru_statistics();
			break;
	}
}

void ctx_statistics_cb(const void * AttributeBlock, const lbm_context_stats_t * Statistics)
{
	/* Handle context stats */
}

void evq_statistics_cb(const void * AttributeBlock, const lbm_event_queue_stats_t * Statistics)
{
	/* Handle event queue stats */
}

Upon receipt of a statistics message, the appropriate callback function is called. The application can then do whatever is desired with the statistics data, which might include writing it to a file or database, performing calculations, or whatever is appropriate.

Beyond the actual statistics, several additional pieces of data are sent with each statistics packet. These data are stored in an attribute block, and are accessible via the lbmmon_attr_get_*() functions. Currently, these data include the IPV4 address of machine which sent the statistics data, the timestamp (as a time_t) at which the statistics were generated, and the application ID string supplied by the sending application at the time the object was registered for monitoring. See lbmmon_attr_get_ipv4sender(), lbmmon_attr_get_timestamp(), and lbmmon_attr_get_appsourceid() for more information.

6.3.7. The UMS Transport Module

The UM transport module understands several options which may be used to customize your use of the module. The options are passed via the TransportOptions parameter to the lbmmon_sctl_create() and lbmmon_rctl_create() functions, as a null-terminated string containing semicolon-separated name/value pairs.

The following options are available:

• config specifies a configuration file. This file is processed in a manner similar to lbm_config(). However, unlike lbm_config(), the current default attributes are not changed. Instead, the options parsed from the configuration file are applied only to the UM objects created by the module.

• topic specifies the topic name to use for sending and receiving statistics. By default, the topic /29west/statistics is used.

• wctopic specifies (for monitor receivers only) a wildcard pattern to be used to receive statistics.

As an example, assume your application needs to use a special configuration file for statistics. The following call allows your application to customize the UM transport module using the configuration file stats.cfg.

lbmmon_sctl_t * monctl;
const char * tropt = "config=stats.cfg";
if (lbmmon_sctl_create(&smp;monctl, lbmmon_format_csv_module(), NULL, 
lbmmon_transport_lbm_module(), tropt) == -1)
{
		fprintf(stderr, "lbmmon_sctl_create() failed\n");
		exit(1);
}

If your application also needs to use a specific topic for statistics, the following code specifies that, in addition to the configuration file, the topic StatisticsTopic be used for statistics.

lbmmon_sctl_t * monctl;
const char * tropt = "config=stats.cfg;topic=StatisticsTopic";
if (lbmmon_sctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), 
tropt) == -1)
{
		fprintf(stderr, "lbmmon_sctl_create() failed\n");
		exit(1);
}

It is important to use the same topic and configuration for both monitor sources and receivers. Otherwise your applications may send the statistics, but the monitor receiver won't be able to receive them.

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.3.8. The UDP Transport Module

The UDP transport module understands several options which may be used to customize your use of the module. The options are passed via the TransportOptions parameter to the lbmmon_sctl_create() and lbmmon_rctl_create() functions, as a null-terminated string containing semicolon-separated name/value pairs.

The UDP module supports sending and receiving via UDP unicast, UDP broadcast, and UDP multicast. The following options are available.

• address specifies the unicast IP address to which statistics are sent via UDP. Applicable to sender only.

• port is the IP port packets are sent to. Defaults to 2933.

• interface specifies the network interface over which multicast UDP is sent or received.

• mcgroup is the multicast group on which to send and receive UDP packets.

• bcaddress specified the broadcast address to which UDP packets are sent. Applicable to sender only.

• ttl specifies the TTL for each multicast UDP packet. Applicable to sender only.

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.3.9. The SNMP Transport Module

The SNMP transport modules operates in identical fashion to the UMS Transport Module. See The UMS Transport Module

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.5.1. lbmmon.c

The example application lbmmon.c acts as a Monitor Receiver and is provided in both executable and source form. It writes monitoring statistics to the screen and can be used in conjunction with other example applications (which act as the Monitor Sources). The following procedure uses lbmrcv and lbmsrc to create messaging traffic and adds a configuration file in order to specify the LBT-RM transport instead of the TCP default. (The LBT-RM transport displays more statistics than TCP.)

Since UM does not generate monitoring statistics by default, you must activate monitoring in your application. For the example application, use the --monitor-ctx=n option where n is the number of seconds between reports. The following procedure activates monitoring on the receiver, specifying the context (ctx) to create a complete set of receiver statistics. You could activate monitoring in a similar fashion on the source and create source statistics.

To use lbmmon to view statistics from sample application output:

1. Create configuration file with the single option of source transport lbtrm and name it LBTRM.cfg.

2. Run lbmmon --transport-opts="config=LBTRM.cfg"

3. Run lbmrcv -c LBTRM.cfg --monitor-ctx=5 Arizona

4. Run lbmsrc -c LBTRM.cfg Arizona

After lbmsrc completes, the final output for lbmmon should closely resemble the following.

Receiver statistics received from C:\Program Files\29West\UME_1.2.1\Win2k-i386\bin\lbmrcv.exe 
at 10.29.1.78, sent Wed Jan 09 14:25:49 2008
Source: LBTRM:10.29.1.78:4391:323382d8:224.10.10.10:4400
Transport: LBT-RM
LBT-RM messages received                                 : 45455
Bytes received                                           : 370000000
LBT-RM NAK packets sent                                  : 0
LBT-RM NAKs sent                                         : 0
Lost LBT-RM messages detected                            : 0
NCFs received (ignored)                                  : 0
NCFs received (shed)                                     : 0
NCFs received (retransmit delay)                         : 0
NCFs received (unknown)                                  : 0
Loss recovery minimum time                               : 4294967295ms
Loss recovery mean time                                  : 0ms
Loss recovery maximum time                               : 0ms
Minimum transmissions per individual NAK                 : 4294967295
Mean transmissions per individual NAK                    : 0
Maximum transmissions per individual NAK                 : 0
Duplicate LBT-RM data messages received                  : 0
LBT-RM messages unrecoverable (window advance)           : 0
LBT-RM messages unrecoverable (NAK generation expiration): 0
LBT-RM LBM messages received                             : 10000000
LBT-RM LBM messages received with no topic               : 0
LBT-RM LBM requests received                             : 0
		

Notes:

• Since this procedure was done on a single machine. No packets were lost and therefore lbmrcv did not generate any NAKs and lbmsrc did not send any NCFs. If you run this procedure across a network, packets may be lost and you would see statistics for NAKs, NCFs and loss recovery.

• This procedure activates monitoring on the receiver, specifying the context (--monitor-ctx) to create a complete set of receiver transport statistics. You could activate monitoring in a similar fashion on the source and create source statistics. Each set of statistics shows one side of the transmission. For example, source statistics contain information about NAKs received by the source (ignored, shed, retransmit delay, etc.) where receiver statistics contain data about NCFs received. Each view can be helpful.

• Moreover, as explained earlier in Specifying the Object to Monitor, individual receivers or sources can be monitored instead of all transport activity for a context. For this procedure, use --monitor-rcv or --monitor-src.

• You could run this procedure again specifying a different transport (LBT-RU or TCP) in the configuration file and receive a different set of statistics. For descriptions of all the transport statistics, refer to the transport statistics data structures in the C Application Programmer's Interface. Click on the Data Structures tab at the top and click on lbm_rcv_transport_stats_t or lbm_src_transport_stats_t.

6.5.2. lbmmonudp.c and lbmmondiag.pl

The example application, lbmmonudp.c receives UM statistics and forwards them as CSV data over a UDP transport. The Perl script, lbmmondiag.pl, can read UDP packets and process the statistics, reporting Severity 1 and Severity 2 events. This script only reports on LBT-RM transports.

To run lbmmonudp.c with lbmmondiag.pl, use the following procedure.

1. Create configuration file with the single option of source transport lbtrm and name it LBTRM.cfg.

2. Run lbmmonudp -a 127.0.0.1 --transport-opts="config=LBTRM.cfg"

3. Run lbmrcv -c LBTRM.cfg --monitor-ctx=5 Arizona

4. Run lbmsrc -c LBTRM.cfg Arizona

5. Run lbmmondiag.pl

The following sections discuss some of the possible results of this procedure. Your results will vary depending upon conditions in your network or if you run the procedure on a single machine.

	 Sev1: 34 datagrams unrecovered due to NAK generation interval ending
		

	 Sev1: 249 datagrams unrecovered due to transmission window advancement
		

	 Sev2: 10 datagrams queued by data rate controller
	 

	 Sev2: 101 datagrams queued by retransmission rate controller
	 

	 Sev2: 310 datagrams lost
	 Sev2: 112.236 seconds estimated total latency due to repair of 564 losses
		

	 Sev2: 58 NAKs sent
		

	 Sev2: 7478 NAKs received
	 Sev2: 50 retransmissions sent
	 Sev2: 0.015056 seconds estimated total latency due to retransmissions