2. Concepts

2.1.1. Persistent Store

UMP uses a daemon to persist source and receiver state outside the actual sources and receivers themselves. This is the UMP Persistent Store. The store can persist state in memory as well as on disk. State is persisted on a per-topic, per-source basis by the store. UMP stores need not be a single entity. For fault tolerance purposes, it is possible to configure multiple stores in various ways. See Adding the UMP Store to a Source, UMP Stores, Store Configuration Considerations, Man Pages and Configuration Reference for Umestored.

2.1.2. Registration Identifier

UMP identifies sources and receivers with Registration Identifiers, also called Registration IDs or RegIDs. A RegID is a 32-bit number that uniquely identifies a source or a receiver to a store. This means that RegIDs are also specific to a store and can be reused between individual stores, if needed. No two active sources or receivers can share a RegID or use the same RegID at the same time. This point is critical: since UMP enables your application to use and handle RegIDs very freely, you must use RegIDs carefully to avoid destructive results. See Adding Fault Recovery with Registration IDs and Registration Identifiers. RegIDs can also be managed easily with the use of Session IDs. See Managing RegIDs with Session IDs.

2.1.3. Delivery Confirmation

UMP provides feedback to sources upon notification that a receiver has consumed a given piece of data, in other words, that it has received and processed a message. This feedback is called Delivery Confirmation. See also Confirmed Delivery and Source Message Retention and Release.

2.1.4. Release Policy

Sources and persistent stores retain data according to a release policy, which is a set of rules that specifies when a message can be reclaimed. Each rule would allow any message that complies with the rule to be reclaimed. However, a message must comply with all rules before it can be reclaimed. Conversely, any message not complying with all rules will not be reclaimed. A source or store retains messages until its retention policy dictates the message may be removed. Sources and stores use slightly different retention policies based on their individual roles. For more information on retention policies, see Source Message Retention and Release.

2.1.5. Message Stability

Sources send messages to both receivers and to stores. Messages become stable once the message has been persisted at the store or a set of stores. The number of messages that can be sent by a source has no relation to the number of its messages that have been stabilized unless UMP Flight Size is enabled. In addition, UMP informs the application when messages are stabilized, enabling the application to take any desired action. See Source Message Retention and Release.

Publishing messages to a store is a coordinated hand-off between the publishing application and the store. The store assumes responsibility for delivering a message only when the publisher is informed that the message is stable. Stability refers to the store having a copy of the message in memory and/or on disk, depending on configuration. Until the publisher is informed that a message is stable, it may be lost upon restart. Messages may be delivered without the publisher being informed that they were stable. Upon restart, the publisher is expected to send again any messages previously sent that were not known to be stable. Since the store cannot differentiate between new messages and messages sent again upon publisher restart, the application is responsible for marking messages as sent again in some manner (e.g., by setting a flag in message content or properties), if required for downstream duplicate checking.

2.1.6. Round-Robin Store Failover

Stores can also experience failures from which they may or may not recover. A source can be configured to move to a second store if the first store fails and can not recover in time. Round-robin store behavior describes the behavior of a source moving through a list of stores, using a single store at any one time, with several specified backups available to it in case the single store fails.

See also Sources Using Round-Robin Store Configuration and Round-Robin Store Usage

2.1.7. Quorum/Consensus Store Failover

In addition to a source being configured for round-robin store behavior, several stores can be configured for simultaneous operation. In this situation, a single store or even a handful of stores can fail without impacting the source and receivers. As long as a quorum of the configured stores is accessible, messaging operation generally continues uninterrupted. (UMP defines a Quorum as a majority.)

See also Sources Using Quorum/Consensus Store Configuration, Quorum/Consensus Store Usage, Quorum/Consensus - Single Location Groups and Quorum/Consensus - Mixed Location Groups.

2.2.1. Source Streaming

Sources may send and have in flight several messages to the queue at the same time. This provides some significant throughput benefits.

2.2.2. Message Stability

Publishing messages to a queue is a coordinated hand-off between the publishing application and the queue. The queue assumes responsibility for delivering a message only when the publisher is informed that the message is stable. Stability refers to the queue having a copy of the message in memory and/or on disk, depending on configuration. Until the publisher is informed that a message is stable, it may be lost upon restart. Messages may be delivered without the publisher being informed that they were stable. Upon restart, the publisher is expected to send again any messages previously sent that were not known to be stable. Since the queue cannot differentiate between new messages and messages sent again upon publisher restart, the application is responsible for marking messages as sent again in some manner (e.g., by setting a flag in message content or properties), if required for downstream duplicate checking.

2.2.3. Once-and-Only-Once Delivery

Once-and-Only-Once (OAOO) delivery means that each message is assigned to only one receiver at a time. If a message is not acknowledged by the assigned receiver, it can be reassigned and redelivered to either the same or a different receiver (depending on configuration and circumstances), in which case the resent message is flagged as redelivered. Applications are always responsible for detection and proper handling of potentially duplicate messages.

2.2.4. Application Sets

An Application Set is a group of receivers and can be used to load balance queue topics within a receiving application or accommodate multiple processing purposes for a single topic. The OAOO semantic applies to an Application Set. Therefore, you can configure multiple Application Sets for a queue and only one receiver in each set will process a given message. See Application Set Element.

2.2.5. Receiver Portion Size

You can increase the throughput to receivers by increasing their portion size. This increases the number of messages in flight to a receiver. This setting is specified in a Receiver Type ID. See Options for a Receiver Type's ume-attributes Element. A receiver configured with one Receiver Type ID and subscribed to Topic A in one application can have 5 messages in flight, where a Topic A receiver configured with a different Receiver Type ID in another application can have only one message in flight.

2.2.6. Configurable Store and Forward

You can configure a Queue to assign and send data to receivers after the data has been persisted to disk (Store-Then-Forward) or in parallel to being persisted to disk (Store-While-Forwarding). Store-While-Forwarding produces lower end-to-end latency from sources to receivers at the expense of potential message loss under certain multiple failure conditions. See also ...

• Queue Element

• Queuing Architecture

2.2.7. Multiple Dissemination Models

UMQ provides the following load balancing configurations for data dissemination not possible with other queuing products.

• Serial Queue Dissemination (SQD): Queue sends data to each receiver via serial unicasts. Only the receivers assigned to a message receive that message.

• Parallel Queue Dissemination (PQD): Queue sends data and control information to all receivers via UM transport sessions. Control information contains assignment information.

• Source Dissemination (SD): Source sends data to all receivers via UM transport sessions. Queues send control information to all receivers via separate transport sessions. Control information contains assignment information.

2.2.8. Queue Fault Tolerance

Queues may be composed of several actual queue instances that operate with source and receivers in such a way that failure of one queue instance or even several queue instances need not stop or even slow down sources and receivers from performing queuing operations. See Queue Redundancy.

Known Issue: Configurations with multiple queue instances (slaves) can lead to inconsistent state, which can trigger message loss, crashes or restart issues necessitating removal of files resulting in message loss. Therefore, Informatica recommends deploying configurations with only a single queue instance (without slaves). To facilitate failover, set the sinc-log-filename, sinc-data-filename, and sinc-queue-swap-filename to write to a shared file-system, and use external process management (automatic or manual) to start up a secondary queue instance referencing the same files if and only if the active instance fails (i.e. only allow one queue instance to access the files at any time). With this configuration, sinc files will grow over time, so clean restarts (i.e. shut down, delete all files and restart) will be required periodically. Using a shared file-system may impact performance; Informatica strongly recommends holistic system performance characterization prior to any production deployment.

2.2.9. Indexed Queuing

Messages may be sent with an index using an extended send call, lbm_src_sendv_ex, that includes a pointer to lbm_umq_index_info_t in lbm_src_send_ex_info_t. An index is an application-defined 64-bit unsigned number or free-form string.

By default, all receivers are eligible to be assigned indices by the Queue. Once the Queue assigns the first message sent with a particular index to an individual receiver in each Application Set, the Queue assigns subsequent messages (sent from any source) with that same index to those same individual receivers, provided the receivers remain alive and responsive.

You can exert greater control over how a Queue assigns indices to receivers by configuring individual indices and ranges of indices in the Queue's umestored XML configuration file. You can then set rules that allow or deny receivers the permission to process messages with certain indices. See Indices Element.

Note that with indexed queuing, messages with a particular index can be assigned only to the single consumer responsible for that index (i.e., assignment is "sticky"). If an index consumer stops consuming messages (a failure scenario) long enough for its configured portion size to become full while another message with the same index is pending, message delivery to all other consumers within the same Application Set is halted. Therefore, Informatica recommends using indexed queuing only in configurations with message reassignment enabled and set to a relatively brief interval, as this determines the amount of time that message consumption within an Application Set may be halted in the event of receiver failure.

2.2.10. Dead Letter Queue

Queues can be configured to isolate unconsumed messages in a Dead Letter Queue, which prevents these messages from causing application or queuing system problems. These unconsumed messages remain in the Dead Letter Queue for the life of the Queue. Other applications can access these messages for analysis by starting a wildcard receiver for the Dead Letter topic queue.

Configuring a Dead Letter Queue involves the following actions in the Queue's umestored XML configuration file.

1. Configure a dead-letter-topic-prefix and dead-letter-topic-separator for the Queue. UMQ uses this information to compose a Dead Letter topic name. The example below illustrates this step, but may or may not be suitable to include in your umestored XML configuration file. See also General Options for a Queue's ume-attributes Element.

   <queues>
   <queue name="Sample_Queue_with_Dead_Letter_Topic" port="20333" group-index="0">
       <ume-attributes>
           <!-- dead-letter-topic-prefix must be specified for any dead letter queues to be created -->
           <option type="queue" name="dead-letter-topic-prefix" value="dead_letter"/>
           <!-- dead-letter-topic-separator defaults to '/', so the following line isn't necessary -->
           <option type="queue" name="dead-letter-topic-separator" value="/"/>
       </ume-attributes>
   </queue>
          
   

2. In each of the Queue's Application Sets from which you wish to capture unconsumed messages, set the discard-behavior to dead-letter. The example below illustrates this step, but may or may not be suitable to include in your umestored XML configuration file. See also Options for an Application Set's ume-attributes Element.

   <application-sets>
       <application-set name="Set 1">
           <ume-attributes>
               <option type="queue" name="log-audit-trail" value="1"/>
               <!-- discard-behavior set to dead-letter tells the queue that when messages 
               EOL off this appset, they should be placed on a dead letter queue -->
               <option type="queue" name="discard-behavior" value="dead-letter"/>
           </ume-attributes>
       </application-set>
   </application-sets>
          
   

3. Configure a dead-letter-topic for the Queue and assign it to a different application set than the one in step 2 that has its discard-behavior set to drop. The example below illustrates this step, but may or may not be suitable to include in your umestored XML configuration file. See also Options for a Queue Topic's ume-attributes Element.

   <topics>
       <!-- since the dead-letter-topic-prefix is dead_letter, it is necessary to specify a topic 
       pattern that matches dead_letter -->
       <topic pattern="dead_letter" type="PCRE">
           <ume-attributes>
               <!-- this topic is a dead-letter-topic -->
               <!-- you cannot create a source and submit messages directly to a dead 
               letter topic -->
               <option type="queue" name="dead-letter-topic" value="1"/>
           </ume-attributes>
           <application-sets>
               <!-- you must assign this dead-letter-topic to an application set that has its 
               "discard-behavior" set to "drop" -->
               <application-set name="Set 2"/>
           </application-sets>
       </topic>
   </topics>
          
   

2.2.11. Message Lifetimes

You can configure a lifetime period for messages during which the message may be assigned to a receiver. Upon expiration of the message lifetime, the queue cannot assign the message to a receiver. The queue either discards the message from the queue permanently or sends it to the Dead Letter Queue, if configured. A message's total lifetime starts when the queue enqueues the message. Messages lifetimes apply to both UMQ and ULB messages. See Message Lifetimes and Reassignment.

2.2.12. Queuing Terminology

Ultra Messaging Queuing Edition uses the following terms.