The Ultra Messaging® Guide for Persistence and Queuing

March 2014

Table of Contents
1. Introduction
2. Concepts
3. Architectures
4. Operational View
5. Enabling Persistence
6. Demonstrating Persistence
7. Designing Persistence Applications
8. Enabling Queuing
9. Designing Queuing Applications
10. Fault Tolerance
11. Man Pages
12. Configuration Reference for Umestored
13. Ultra Messaging® Web Monitor

1. Introduction

In addition to high performance streaming, Ultra Messaging® also provides persistence and low latency queuing by implementing a configurable daemon that runs persistent stores , queues or both .

1.1. Persistence

A system implementing UMP persistence comprises any number of sources, receivers, and persistent stores. Ultra Messaging's unique design provides Parallel Persistence®, which refers to the ability of a persistent store or stores to run independently of sources and receivers and in parallel with messaging. The persistence store does not interfere with message delivery to receiving applications. Parallel Persistence adds several key features missing in other messaging solutions.

  • A fault recovery ability

  • The capacity to continue operation during specific types of failures

Fault recovery refers to the system's ability to recover from a failure of any system component (source, receiver or store). Under certain circumstances, Ultra Messaging can even recover from multiple failures and multiple cascading failures.

1.2. Queuing

In addition to the capabilities of UMP and UMS, UMQ supports brokered and brokerless queuing semantics.

Key features of UMQ include the following:

  • Once and only once (OAOO) delivery for applications such as clearing and settlement that require each trade be processed only once.

  • Application Sets for publishing into multiple logical queues with a single send, making it easy to onboard new applications that require copies of the same data.

  • Ultra Load Balancing (ULB) for brokerless non-persistent queuing semantics, with special considerations for message assignment, receiver pacing, and multi-source fairness.

  • Java Messaging Service (JMS) support. See the Ultra Messaging JMS Guide for more details.

Queuing semantics are supported only within the same Topic Resolution Domain (TRD), without Gateways or Dynamic Routers in the data path.

1.3. Semantic Responsibilities

Brokered and brokerless queuing semantics are orthogonal and independent from the reliable non-persistent and durable persistent streaming semantics provided by UMS and UMP. Differences outlined in the following list:

  • Reliable non-persistent streaming (UMS) provides in-order message delivery and gap/loss notification, while applications are running, without load-balanced message delivery.

  • Durable persistent streaming (UMP) provides in-order message delivery and gap/loss notification, while applications are running and across restarts, without load-balanced message delivery.

  • Brokered queuing (UMQ) provides load-balanced message delivery, while applications are running and across restarts, without message ordering and without gap/loss notification.

  • Brokerless queuing (UMQ) provides load-balanced message delivery, while applications are running, without message ordering and without gap/loss notification.

Regardless of the messaging semantic chosen, applications assume the following responsibilities to the extent applicable for their use case:

  • Application resubmission (re-sending) of in-flight messages after source application restart

  • Handling potentially duplicate messages from messaging layer or application resubmission

  • Marking application resubmitted messages as such if needed to support duplicate handling

  • Detecting stale data, for example, by using synchronized timestamps

  • Source and receiver application failover and state reconstruction

Queuing is recommended for cases where load-balanced message delivery is required and out-of-order message delivery and message loss without notification are acceptable, i.e., message delivery order and message loss are either unimportant or are handled by applications (inclusive of application-level sequencing and gap detection, or equivalent). For example, queuing is well-suited for load-balanced request/response with bidirectional topics and application-level retries.

When designing queuing applications, it is important to consider whether message processing is stateless (any receiver can process any message) or requires local state (only certain receivers can process certain messages). If local state is required, the recovery strategy for receiver hardware failure must consider how to rebuild that local state. Receivers with local state may be better served by durable persistent streaming rather than queuing.

Copyright 2007 - 2014 Informatica Corporation.