UMQ uses multiple techniques to ensure that messages reach their destinations. These techniques include the storage and verification of messages and node redundancy.

Ultra Messaging queuing systems use message persistence to confirm to the source application that a message is stored on a broker queue. The source no longer needs to hold a copy of that message. Message durability holds the message in the queue until the receiver application sends a consumption acknowledgement to the queue. These techniques help ensure messages reach their destination if a connection, source, or receiver fails.

An Ultra Messaging high-availability queuing system uses three broker nodes, which can all reside on separate machines. One broker actively handles message traffic while the other two serve as hot standby brokers. If the active broker or broker host fails, messaging can continue relatively uninterrupted.

Message Reliability <-

To ensure that messages reach receiver applications, Ultra Messaging uses message stability, persistence, and consumption reporting.

Message Stability <-

Message stability is an indication that the broker has received a message from the source.

The broker confirms receipt of each message it receives from the sending client application. When at least two nodes acknowledge message stability, the active broker sends a stability acknowledgment back to the source.

Typically, after a source sends a message, it retains that message because it might need to retransmit the message. When the source receives a stability acknowledgment, it can then discard the message. During the period after the source sends the message until the source receives the stability acknowledgment, the message status is in flight. If a broker or broker connection fails, the source application can retain the in-flight messages for retransmission.

Message Consumption <-

Message consumption is an indication that the receiving client application has processed a message from the broker.

The receiving client application receives and processes each message from the broker. After the callback delivers the message and returns, Ultra Messaging automatically tells the broker that the message is consumed.

After a broker sends a message, it retains that message because it might need to retransmit the message if the receiver application or connection to the receiver application fails. When the broker receives a message consumption acknowledgment, it can then discard the message.

Message Persistence <-

By default, queuing sources set all messages to be persisted. When the broker receives a message that is set for persistence, the broker holds that message in long-term storage until the receiving client application consumes the message.

To set persistence in a message, you must set the JMSDeliveryMode message property for the message. The following example shows a way that you can set a message property to persistent:

/* setup */
memset(&info, 0, sizeof(lbm_src_send_ex_info_t));
info.flags = LBM_SRC_SEND_EX_FLAG_PROPERTIES;
lbm_msg_properties_create(&info.properties);
/* set a user defined message property */
value = 2;
ret = lbm_msg_properties_set(info.properties, "JMSDeliveryMode",
                             (void *) &value, LBM_MSG_PROPERTY_INT,
                             sizeof(int32_t));
assert(ret==LBM_OK);

Node Redundancy <-

A high availability queuing system uses three host machines on which to run the components of a node. This deployment uses redundancy for processes and for persistence storage.

High Availability <-

To use configure an Ultra Messaging queuing system for high availability, you must use the queuing three-node deployment. Each node resides on a different host, to eliminate any broker single point of failure.

The following image shows a high-availability queuing system with one source application, one receiver application, and three brokers:

The source application sends a persistent message to the active broker. For each persistent message, the active broker performs the following operations:

The active broker receives the persistent message from the source application and attempts to persist the message to disk before adding it to its queue in memory.
The active broker requests that its UM persistence adapter persist to disk the status that the message is added to its queue.
The UM persistence adapter replicates an add message event to all UM persistence adapters in the standby brokers and itself, and to all UMP stores.
The active broker waits until it receives stability from a majority of the UMP stores.
The active broker puts the message into its queue and sends a response back to the source application that the message is stable and has been persisted to disk.
The broker assigns the message to an eligible receiver and requests that its UM persistence adapter persist to disk the status that the message is assigned.
The UM persistence adapter replicates an update message event to the standby UM persistence adapters and to all UMP stores.
The active broker waits until it receives stability from a majority of the UMP stores.
The active broker sends the message to the receiver application.
The receiver application processes the message and then tells the broker that the message is consumed.
The active broker requests that its UM persistence adapter persist to disk the status that the message is removed from its queue.
The UM persistence adapter replicates a remove message event to the standby UM persistence adapters and to all UMP stores.
The active broker waits until it receives stability from a majority of the UMP stores.
The active broker removes the message from its queue in memory.

This configuration keeps all three UMP stores up to date with the latest messages and their statuses. If the active node fails, either standby node is ready to continue with the message broker service, and the standby brokers elect a new active broker.

High Availability System Initiation <-

A high availability queuing system follows an initiation procedure when you first start the system.

When a high availability queuing system first starts, it must determine which node is the active node and which nodes are the standby nodes. The brokers do this through an active-broker election process. The queuing system performs the following initiation actions:

All nodes start.
The nodes elect an active broker, which opens a transport connection for client applications. The other brokers become standby brokers and do not open connections.
Client applications start.
Client applications try broker addresses from the list of addresses to determine which broker is active.
Client applications initiate and complete registration with the active broker.

Node Failover <-

If an active broker fails, Ultra Messaging starts a reelection process to elect a new active broker.

The following image shows the change after an active broker fails and the queuing system elects a new active broker:

In this example, Broker1 has failed and disconnected from the client applications. The client applications then try all IP and port addresses on their broker option list until they find an open socket to connect to. If a client application does not find an active broker, it waits and then tries the list again. This cycle continues until the system elects a new active broker.

After Broker1 failure, Ultra Messaging Broker3 to be the new active broker. The clients then connect to, and register with, the newly active broker. After registration, the clients can resume messaging.

Active and standby brokers send a periodic heartbeat message to each other. If standby brokers do not receive any heartbeat message from the active broker then the standby brokers elect a new active broker.

If an active broker does not receive heartbeat messages from any standby brokers, the active broker becomes a standby broker. The queuing system fails when a quorum of brokers no longer exists.