Concepts Guide
UM Objects

Many UM documents use the term object. Be aware that with the C API, they do not refer to formal objects as supported by C++ (i.e. class instances). The term is used here in an informal sense to denote an entity that can be created, used, and (usually) deleted, has functionality and data associated with it, and is managed through the API. The handle that is used to refer to an object is usually implemented as a pointer to a data structure (defined in lbm.h), but the internal structure of an object is said to be opaque, meaning that application code should not read or write the structure directly.

However, the UM Java JNI and C# .NET APIs are object oriented, with formal Java/C# objects. See the Java API documentation and .NET API documentation for more information.

The UM API is designed around a set of UM objects. The most-often used object types are:

  • Context - a fairly "heavy-weight" parent object for most other UM objects. Also represents an "instance" of UM, with all foreground and background functions.
  • Source - a publisher creates one or more Source objects, one per topic to be published.
  • Receiver - a subscriber creates one or more Receivers, one per topic that is subscribed.
  • Wildcard Receiver - a subscriber can optionally create Wildcard Receivers to subscribe to patterns that can match multiple topics.
  • Hot Failover Receiver - a specialized type of Receiver object that allows for redundant publishing.
  • HFX Receiver - a specialized type of Hot Failover Receiver object that allows redundant sources across multiple Context objects.
  • Event Queue - an active object (requires one or more threads) which is used to queue and deliver UM events (including received messages) to the application using a separate thread. Event queues are normally not required; by default UM delivers events using the Context's own thread. But there are circumstances where it is useful to transfer the received message or event to an independent thread for delivery.

A typical UM application program manages UM objects following certain ordering rules:

  1. If desired, Event Queues are typically created first. (Most UM applications do not use Event Queues.)
  2. Create a Context object. A typical application creates only one context, but there are some specialized use cases in which a small number of contexts (typically less than 5) are useful.
  3. Create one or more Source and Receiver objects. It is not unusual for an application to create thousands of sources or receivers.
  4. The application's main processing phase consists of publishing messages using the Source objects, and receiving messages using the Receiver objects. UM is most efficiently used if those Source and Receiver objects are created during initialization. Dynamic object creation during normal operation is possible, but can require special coding.
  5. When the application is ready to shut down, Sources and Receivers should be deleted.
  6. Contexts are deleted after all Sources and Receivers are deleted. Note that if the event queue uses Sequential Mode, the event processing thread should be unblocked and joined prior to Context deletion.
  7. Event Queues, if used, are deleted last. Note that the Event Queue dispatching thread(s) should be unblocked and joined prior to deletion of the Event Queue.

Note that it is very important to enforce the above deletion order. For example, it can lead to crashes if you delete a context while it still has active Sources or Receivers associated with it. Similarly, crashes can result if you delete an event queue while it still has active Contexts associated with it.

Also, note that deletion of Source objects can affect the reliability of message delivery. UM Receivers are designed to detect lost packets and request retransmission. However, once a Source object is deleted, it can no longer fulfill retransmission requests. It is usually best for an application to delay a few seconds after sending its last messages before deleting the Sources. This is especially important if Implicit Batching is used since outgoing messages might be held in the Batcher.


Context Object  <-

A UM context object conceptually is an environment in which UM runs. An application creates a context, typically during initialization, and uses it for most other UM operations. In the process of creating the context, UM normally starts an independent thread (the context thread) to do the necessary background processing such as the following:

  • Topic resolution
  • Enforce rate controls for sending messages
  • Manage timers
  • Manage state
  • Implement UM protocols
  • Manage Transport Sessions

You create a context with lbm_context_create(). When an application is finished with the context (no more message passing needed), it should delete the context by calling lbm_context_delete().

Warning
Before deleting a context, you must first delete all objects contained within that context (sources, receivers, wildcard receivers).

Your application can give a context a name, which are optional but should be unique across your UM network. You set a context name before calling lbm_context_create() in the following ways:

Context names are optional but should be unique within a process. UM does not enforce uniqueness, rather issues a log warning if it encounters duplicate context names. Application context names are only used to load template and individual option values within an XML UM configuration file.

One of the more important functions of a context is to hold configuration information that is of context scope. See the UM Configuration Guide for options that are of context scope.

Most UM applications create a single context. However, there are some specialized circumstances where an application would create multiple contexts. For example, with appropriate configuration options, two contexts can provide separate topic name spaces. Also, multiple contexts can be used to portion available bandwidth across topic sub-spaces (in effect allocating more bandwidth to high-priority topics).

Attention
Regardless of the number of contexts created by your application, a good practice is to keep them open throughout the life of your application. Do not close them until you close the application.


Topic Object  <-

A UM topic object is conceptually very simple; it is little more than a container for a string (the topic name). However, UM uses the topic object to hold a variety of state information used by UM for internal processing. It is conceptually contained within a context. Topic objects are used by applications in the creation of sources and receivers.

Technically, the user's application does not create or delete topic objects. Their management is handled internally by UM, as needed. The application uses APIs to gain access to topic objects. A publishing application calls lbm_src_topic_alloc() to get a reference to a topic object that it intends to use for creation of a Source Object. A subscribing application calls lbm_rcv_topic_lookup() to get a reference to a topic object that it intends to use for creation of a Receiver Object.

The application does not need to explicitly tell UM when it no longer needs the topic object. The application's reference can simply be discarded.


Source Object  <-

A UM source object is used to send messages to the topic that it is bound to. It is conceptually contained within a context.

You create a source object by calling lbm_src_create(). One of its parameters is a Topic Object. A source object can be bound to only one topic. The application is responsible for deleting a source object when it is no longer needed by calling lbm_src_delete().


Source String  <-

Every source that a publishing application creates has associated with it a unique source string. Note that if multiple publishing UM contexts (applications) create sources for the same topic, each context's source will have its own unique source string. Similarly, if one publishing UM context (application) creates multiple sources for different topics, each topic's source will have its own unique source string. So a source string identifies one specific instance of a topic within a UM context.

The source string is used in a few different ways in the UM API, for example to identify which Transport Session to retrieve statistics for in lbm_rcv_retrieve_transport_stats(). The source string is made available to the application in several callbacks, for example lbm_src_notify_function_cb, or the "source" field of lbm_msg_t_stct of a received message. See also Sending to Sources.

The format of a source string depends on the transport type:

  • TCP:src_ip:src_port:session_id[topic_idx]
    session_id is optional, per configuration option transport_tcp_use_session_id (source)
    example: TCP:192.168.0.4:45789:f1789bcc[1539853954]

  • LBTRM:src_ip:src_port:session_id:mc_group:dest_port[topic_idx]
    example: LBTRM:10.29.3.88:14390:e0679abb:231.13.13.13:14400[1539853954]

  • LBT-RU:src_ip:src_port:session_id[topic_idx]
    session_id is optional, per configuration option transport_lbtru_use_session_id (source)
    example: LBT-RU:192.168.3.189:34678[1539853954]

  • LBT-IPC:session_id:transport_id[topic_idx]
    example: LBT-IPC:6481f8d4:20000[1539853954]

  • LBT-SMX:session_id:transport_id[topic_idx]
    example: LBT-SMX:6481f8d4:20000[1539853954]

  • BROKER
    example: BROKER

Please note that the topic index field (topic_idx) may or may not be present depending on your version of UM and/or the setting for configuration option source_includes_topic_index (context).

See also lbm_transport_source_format() and lbm_transport_source_parse().


Source Strings in a Routed Network  <-

In a UM network consisting of multiple Topic Resolution Domains (TRDs) connected by DROs, a given source will be uniquely identifiable within a TRD by its source string. However, that same source will have different source strings in different TRDs. For receivers in the same TRD as the source, the source string will refer to the source. But in remote TRDs, that same source's source string will refer to the proxy source of the DRO on the shortest path to the source. The IP information contained in the source string will refer to the DRO.

This can lead to a situation where multiple originating sources located elsewhere in the UM network will have source strings with the same IP information in a given TRD. They will differ by Topic Index number, but even that topic index will be different in different TRDs.


Source Configuration and Transport Sessions  <-

As with contexts, a source holds configuration information that is of source scope. This includes network options, operational options and reliability options for LBT-RU and LBT-RM. For example, each source can use a different transport and would therefore configure a different network address to which to send topic messages. See the UM Configuration Guide for source configuration options.

As stated in UM Transports, many topics (and therefore sources) can be mapped to a single transport. Many of the configuration options for sources actually control or influence Transport Session activity. If many sources are sending topic messages over a single Transport Session (TCP, LBT-RU or LBT-RM), UM only uses the configuration options for the first source assigned to the transport.

For example, if the first source to use a LBT-RM Transport Session sets the transport_lbtrm_nak_generation_interval (receiver) to 24 MB and the second source sets the same option to 2 MB, UM assigns 24 MB to the Transport Session's transport_lbtrm_nak_generation_interval (receiver).

The UM Configuration Guide identifies the source configuration options that may be ignored when UM assigns the source to an existing Transport Session. Log file warnings also appear when UM ignores source configuration options.


Zero Object Delivery (Source)  <-

The Zero Object Delivery (ZOD) feature for Java and .NET lets sources deliver events to an application with no per-event object creation. (ZOD can also be utilized with context source events.) See Zero Object Delivery for information on how to employ ZOD.


Receiver Object  <-

A UM receiver object is used to receive messages from the topic that it is bound to. It is conceptually contained within a context. Messages are delivered to the application by an application callback function, specified when the receiver object is created.

You create a receiver object by calling lbm_rcv_create(). One of its parameters is a Topic Object. A receiver object can be bound to only one topic. The application is responsible for deleting a receiver object when it is no longer needed by calling lbm_rcv_delete().

Multiple receiver objects can be created for the same topic within a single context, which can be used to trigger multiple delivery callbacks when messages arrive for that topic.


Receiver Configuration and Transport Sessions  <-

A receiver holds configuration information that is of receiver scope. This includes network options, operational options and reliability options for LBT-RU and LBT-RM. See the UM Configuration Guide for receiver configuration options.

As stated above in Source Configuration and Transport Sessions, multiple topics (and therefore receivers) can be mapped to a single transport. As with source configuration options, many receiver configuration options control or influence Transport Session activity. If multiple receivers are receiving topic messages over a single Transport Session (TCP, LBT-RU or LBT-RM), UM only uses the configuration options for the first receiver assigned to the transport.

For example, if the first receiver to use a LBT-RM Transport Session sets the transport_lbtrm_nak_generation_interval (receiver) to 10 seconds, that value is applied to the Transport Session. If a second receiver sets the same option to 2 seconds, that value is ignored.

The UM Configuration Guide identifies the receiver configuration options that may be ignored when UM assigns the receiver to an existing Transport Session. Log file warnings also appear when UM ignores receiver configuration options.


UM Wildcard Receivers  <-

You create a wildcard receiver object by calling lbm_wildcard_rcv_create(). Instead of a topic object, the caller supplies a pattern which UM uses to match multiple topics. Because the application does not explicitly lookup the topics, UM passes the topic attribute into lbm_wildcard_rcv_create() so that it can set options. Also, wildcard receivers have their own set of options, such as pattern type. The application is responsible for deleting a wildcard receiver object when it is no longer needed by calling lbm_wildcard_rcv_delete().

The wildcard pattern supplied for matching is a PCRE regular expression that Perl recognizes. See http://perldoc.perl.org/perlrequick.html for details about PCRE. See also the pattern_type (wildcard_receiver) option.

Note
Ultra Messaging has deprecated two other wildcard receiver pattern types, regex POSIX extended regular expressions and appcb application callback, as of UM Version 6.1.

Be aware that some platforms may not support all of the regular expression wildcard types. For example, UM does not support the use of Unicode PCRE characters in wildcard receiver patterns on any system that communicates with a HP-UX or AIX system. See the Informatica Knowledge Base article, Platform-Specific Dependencies for details.

For an example of wildcard usage, see lbmwrcv.c

For more information on wildcard receivers, see Wildcard Receiver Topic Resolution, and Wildcard Receiver Options.

TIBCO ™ users see the Informatica Knowledge Base articles, Wildcard topic regular expressions and SmartSockets wildcards and Wildcard topic regular expressions and Rendezvous wildcards.


Transport Services Provider Object  <-

The Transport Services Provider object ("XSP") is introduced with UM version 6.11 and beyond to manage sockets, threads, and other receive-side resources associated with subscribed Transport Sessions. The primary purpose for an XSP object is to allow the programmer to control the threading of received messages, based on the Transport Sessions of those messages.

For more information on XSP, see Transport Services Provider (XSP).


UM Hot Failover Across Contexts Objects  <-

Hot Failover Across Contexts objects ("HFX") provide a form of hot failover that can operate across multiple network interfaces.

For more information, see Hot Failover Across Multiple Contexts.


Zero Object Delivery  <-

The Zero Object Delivery (ZOD) feature for Java and .NET lets receivers (and sources) deliver messages and events to an application with no per-message or per-event object creation. This facilitates source/receiver applications that would require little to no garbage collection at runtime, producing lower and more consistent message latencies.

To take advantage of this feature, you must call dispose() on a message to mark it as available for reuse. To access data from the message when using ZOD, you use a specific pair of LBMMessage-class methods (see below) to extract message data directly from the message, rather than the standard data() method. Using the latter method creates a byte array, and consequently, an object. It is the subsequent garbage collecting to recycle those objects that can affect performance.

For using ZOD, the LBMMessage class methods are:

  • Java: dispose(), dataBuffer(), and dataLength()
  • .NET: dispose(), dataPointer(), and length()

On the other hand, you may need to keep the message as an object for further use after callback. In this case, ZOD is not appropriate and you must call promote() on the message, and also you can use data() to extract message data.

For more details see the Java API Overview or the .NET LBMMessage Class description. This feature does not apply to the C API.


Event Queue Object  <-

A UM event queue object is a serialization queue structure and execution thread for delivery of other objects' events. For example, a Source Object can generate events that the user's application wants to receive via callback. When the source is created, an event queue can be specified as the delivery agent of those events. Multiple UM contexts, sources, and receivers can specify the same event queue, and these events will be delivered in a FIFO manner (first-in, first-out).

Without event queues, these events are delivered via callback from the originating object's context thread, which places the following restrictions on the application callback function being called:

  • The application function is not allowed to make certain API calls (mostly having to do with creating or deleting UM objects).
  • The application function must execute very quickly without blocking.
  • The application does not have control over when the callback executes. It can't prevent callbacks during critical sections of application code.

Some circumstances require the use of UM event queues. As mentioned above, if the receive callback needs to use UM functions that create or delete objects. Or if the receive callback performs operations that potentially block. You may also want to use an event queue if the receive callback is CPU intensive and can make good use of multiple CPU hardware. Not using an event queue provides the lowest latency, however, high message rates or extensive message processing can negate the low latency benefit if the context thread continually blocks.

Of course, your application can create its own queues, which can be bounded, blocking queues or unbounded, non-blocking queues. For transports that are flow-controlled, a bounded, blocking application queue preserves flow control in your messaging layer because the effect of a filled or blocked queue extends through the message path all the way to source. The speed of the application queue becomes the speed of the source.

UM event queues are unbounded, non-blocking queues and provide the following unique features:

  • Your application can set a queue size threshold with queue_size_warning (event_queue) and be warned when the queue contains too many messages.
  • Your application can set a delay threshold with queue_delay_warning (event_queue) and be warned when events have been in the queue for too long.
  • The application callback function has no UM API restrictions.
  • Your application can control exactly when UM delivers queued events with lbm_event_dispatch(). And you can have control return to your application either when specifically asked to do so (by calling lbm_event_dispatch_unblock()), or optionally when there are no events left to deliver.
  • Your application can take advantage of parallel processing on multiple processor hardware since UM processes asynchronously on a separate thread from your application's processing of received messages. By using multiple application threads to dispatch an event queue, or by using multiple event queues, each with its own dispatch thread, your application can further increase parallelism.

You create an UM event queue in the C API by calling lbm_event_queue_create(). When finished with an event queue, delete it by calling lbm_event_queue_delete(). See Event Queue Options for configuration options related to event queues.

Warning
Before deleting an event queue, you must first delete all objects that reference that event queue (sources, receivers, wildcard receivers, contexts).

In the Java API and the .NET API, use the LBMEventQueue class.


Message Object  <-

When an application subscribes to a topic to which publishers are sending messages, the received messages are delivered to the application by an application callback function (see Event Delivery). One of the parameters that UM passes to the application callback is a message object. This object gives the application access to the content of the message, as well as some metadata about the message, such as the topic.

Unlike other objects described above, the user does not create these message objects by API call. UM creates and initializes the objects internally.

The default life-span of a message object is different between C and Java or .NET.


Message Object Deletion  <-

C API

In C, by default, the message object is deleted when the receiver callback returns. No action is necessary by the application to trigger that deletion.

Java or .NET API

In Java or .NET, the passed-in message object is not automatically deleted when the receiver application callback returns. Instead, the message object is fully deleted only when all references to the object are lost and the garbage collector reclaims the object.

However, applications which allow this kind of garbage buildup and collection usually suffer from large latency outliers, and while garbage collection can be tuned to minimize its impact, it is usually recommended that latency-sensitive applications manage their objects more carefully. See Zero Object Delivery.

Also, there are some UM features in which specific actions are triggered by the deletion of messages, and the application designer usually wants to control when those actions are performed (for example, Persistence Message Consumption).

For these reasons, Java and .NET developers are strongly advised to explicitly dispose of a message object when the application is finished with it. It does this by calling the "dispose()" method of the message object. In the simple case, this should be done in the receiver application callback just before returning.


Message Object Retention  <-

Some applications are designed to process received messages in ways that cannot be completed by the time the receiver callback returns. In these cases, the application must extend the life span of the message object beyond the return from the receiver application callback. This is called "message retention".

Note that message retention prevents the recycling of the UM receive buffer. See Receive Buffer Recycling.

C API

To prevent automatic deletion of the message object when the receiver application callback returns, the callback must call lbm_msg_retain(). This allows the application to transfer the message object to another thread, work queue, or control flow.

When a received message is retained, it becomes the application's responsibility to delete the message explicitly by calling lbm_msg_delete(). Failure to delete retained messages can lead to unbounded memory growth.

Java or .NET

The receiver application callback typically calls the "promote()" method of the message object prior to returning. See Retaining Messages.


Security Considerations  <-

UM should generally be used in secure networks where unauthorized users are unable to access UM components and applications. A UM network can be made secure through the use of certificate-based encryption (see Encrypted TCP), but this increases message delivery latency and reduces maximum possible throughput.

In particular, the use of UDP-based protocols (LBT-RM and LBT-RU) cannot secured in the same way that TCP can. In a system that uses UDP-based protocols, there is no mechanism in UM to prevent unauthorized applications to be deployed which can then subscribe and publish topics with complete freedom.


Webmon Security  <-

Of special interest is the use of simple web-based monitoring of the UM daemons: Store, DRO, SRS. UM does not provide any sort of authentication or authorization for the daemons web pages.

Be aware that the use of UM daemon web-based monitoring pages does place a load on the daemon being monitored. For some pages, that load can be significant. An unauthorized user who rapidly accesses pages can disrupt the normal operation of the daemon, potentially leading to message loss.

Users are expected to prevent unauthorized access to the web monitor through normal firewalling methods. Users who are unable to limit access to a level consistent with their overall security needs should disable the daemon web monitors. See:

Daemon

Configuration Element to Disable Web Monitor

Store

<web-monitor>

DRO

<web-monitor>

SRS <debug-monitor>


Configuration Introduction  <-

UM is designed to be highly configurable. Configuration options are generally given default values to support good performance over a wide variety of use cases, but for users who demand the highest levels of performance, many configuration options allow for optimization.

Also, UM's design philosophy is, where practical, to offer new features that can be enabled via configuration only, without requiring changes to the application source code. This can provide higher performance and reliability just by upgrading, without needing to rebuild the application.

Broadly speaking, there are two kinds of components in a UM-based distributed system that need to be configured:

  1. Application programs, linked with the UM library,
  2. Informatica programs, which provide services to the applications.

Application programs are written by users and call UM API functions. There are different ways of providing configuration to application programs, described in Configuration Overview.

Informatica programs include:


xml:space Attribute  <-

Many XML elements throughout UM's configuration files include an attribute named "xml:space". This attribute instructs the XML parser how to deal with whitespace (spaces, tabs, newlines) in the element's value. The attribute defaults to the value "default", which tells the XML parser to trim leading and trailing whitespace, and to compress multiple whitespace into a single space.

For example:

<log>
my_logfile.log
</log>

Note that the value for the "<log>" element contains a leading newline, followed by two spaces, followed by the file name, followed by another newline. Those whitespace characters should be trimmed, which is the default behavior. It is equivalent to:

<log xml:space="default">
my_logfile.log
</log>

However, let's say you really want a space as the first character of the file name. While unusual, it can be done as follows:

<log xml:space="preserve"> my_logfile.log</log>

Note that it had to be combined into a single line to get rid of the newlines.