4. Operational View

4.1.1. Registration

Figure 5 illustrates network flow during the Registration process.

Sources and receivers unicast registrations to the store. The store unicasts responses back to the sources and receivers. Registrations are on a per topic per source basis. Stores use RegIDs to identify sources and receivers. After registration, receivers may handle recovery, sources may send data, and receivers may send acknowledgements

4.1.2. Normal Operation

Figure 6 illustrates the normal operation of data reception and acknowledgement and also shows how UMP attains Parallel Persistence®. The source sends message data to receivers and stores in parallel.

1. Sources transmit data to receivers and stores at the same time over UM multicast or unicast transport protocols.

2. As the store receives and persists messages, the store unicasts acknowledgements, (message stability control messages), to the source letting it know of successful reception and storage.

3. As receivers process and consume messages they unicast acknowledgments to the store letting the store know of successful consumption of data.

4. If the source desires delivery confirmation, the receiver unicasts acknowledgements directly to the source letting the source know of message consumption as well.

Normal operation and recovery can proceed at the same time. In addition, as a receiver consumes retransmitted messages, the receiver sends normal acknowledgements for consumption and confirmed delivery (if requested by the source).

4.1.3. Receiver Recovery

Figure 7 illustrates receiver recovery.

Receivers unicast retransmission requests. If the store has the message, it unicasts the retransmission to the receiver. If it does not have the message and is configured to forward the request to the source (See retransmission-request-forwarding in Options for a Topic's ume-attributes Element), it unicasts the retransmission request to the source. If the source has the message, it unicasts the retransmission directly to the receiver.

UM sends retransmissions from a thread separate from the main context thread so as not to impede live message data processing. The <store> configuration option, retransmission-request-processing-rate, sets the store's capacity to process retransmission requests. The retransmission thread processes requests off a retransmission queue which is set at 4 times the size of retransmission-request-processing-rate. The following UM Web Monitor statistics record retransmission activity. See UM Web Monitor Store Page.

• Retransmission requests received rate

• Retransmission requests served rate

• Retransmission requests dropped rate

• Total retransmission requests dropped since store startup

4.2.1. RPP Registration

If a source sets ume_receiver_paced_persistence, its topic becomes a RPP topic. When the source registers with the store, the source's repository also becomes a RPP repository. Receivers registering with a store on the RPP topic become RPP receivers.

A source registration request includes the following.

• Designation of a RPP topic (LBMC_UME_PREG_FLAG_REGISTER flag)

• Reconfigured repository configuration option values. Possible options are the 3 repository size options, repository-allow-ack-on-reception, repository-disk-write-delay and source-flight-size-bytes-maximum.

• Re-registration must request same configuration options or the store rejects the request.

Receiver registration request includes its designation as a RPP receiver (LBMC_UME_PREG_FLAG_REGISTER flag).

The repository's registration response to both a source and a receiver acknowledges RPP mode.

4.2.2. RPP Normal Operations

Since all RPP receivers must receive all messages, message overruns at the store or receiver must be prevented by regulating the sending pace of the source. The store uses the source's flight size (bytes) to regulate the source's speed, by withholding stability acknowledgements if the repository does not have at least one flight size available.

1. Sources transmit data to receivers and store repositories at the same time over UM multicast or unicast transport protocols.

2. When a disk repository receives a message, it holds the message in memory cache before it writes the message to disk. The repository sends a stability notification to the source after it writes the message to disk. Memory repositories send the stability notice upon reception. See also Acknowledge on Reception and Receiver Acknowledgement and Flight Size below.

   Figure 9. RPP Stability Acknowledgement

   

3. If the source desires delivery confirmation, receivers unicast acknowledgements directly to the source letting the source know of message consumption as well.

The following also affect when a repository sends a stability acknowledgement to the source.

• Acknowledge on Reception - If you configure the repository for repository-allow-ack-on-reception and the source also sets ume_repository_ack_on_reception, the repository sends a stability acknowledgement to the source immediately upon reception. If the disk write has not already been initiated, UMP does not write the message to disk.

  Figure 10. RPP Acknowledge on Reception

  

• Receiver Acknowledgement - If a repository receives acknowledgements from all receivers before writing the message to disk, it immediately sends a stability acknowledgement to the source. If the disk write has not already been initiated, UMP does not write the message to disk.

  Figure 11. RPP Receiver Acknowledgement

  

• Write Delay - The repository option, repository-disk-write-delay, allows the repository to hold messages in memory cache longer before persisting them to disk. This delay increases the probability that all RPP receivers acknowledge message consumption, eliminating the need to persist the message to disk.

• Flight size - A disk repository only sends stability acknowledgement to the source if its memory cache has at least one flight size (in both messages and bytes) available. A memory repository also sends stability acknowledgement if it has at least one flight size (in both messages and bytes) available. A lack of available space in the repository blocks the source until the repository reclaims the necessary storage space and sends a stability acknowledgement.

For memory store repositories, the behaviors Acknowledge on Reception, Receiver Acknowledgement and Write Delay do not apply.

4.2.3. RPP Message Recovery

An RPP source repository retains messages until all RPP receivers acknowledge receipt of the message. Therefore an RPP receiver can only recover messages that have not been consumed by all RPP receivers. It is important to note that an RPP receiver joining after other RPP receivers have already joined and after messages have already been sent can only be guaranteed to recover messages sent subsequent to its joining.

4.2.4. RPP Deregistration

You can deregister either sources or receivers using deregistration APIs, (lbm_src_ume_deregistration(), lbm_rcv_ume_deregistration() and lbm_wrcv_ume_deregistration()). UM deletes the state of deregistered objects. If you deregister a RPP receiver, UMP automatically updates the number of receiver acknowledgements required to maintain RPP behavior. The store issues Deregistration Successful events for every source or receiver that deregisters. See UMP and UMQ Events.

Applications should be cautious about using the deregistration APIs to deregister RPP sources or receivers. These APIs can be disruptive to RPP.

• lbm_src_ume_deregistration() also deletes any persisted RPP messages in the source's repository. A source application should only use lbm_src_ume_deregistration() if it uses delivery confirmation from the receiver and it knows all messages have been delivered. The source is blocked after deregistering and and must restart in order to register again with the RPP store.

• Deregistering an RPP receiver with lbm_rcv_ume_deregistration() removes the receiver from the list of RPP receivers maintained by the repository. It is no longer part of the persistence operation, but is a valid UM receiver, able to receive messages, but unable to acknowledge message consumption to the repository. Any messages not yet confirmed for that receiver are unrecoverable. The receiver must restart in order to register again with the RPP repository.

• Deregistering an RPP wildcard receiver with lbm_wrcv_ume_deregistration() deregisters all individual topic receivers receiving messages on topics that match the wildcard pattern. Individual topic receivers can still receive messages after deregistering, but cannot acknowledge message consumption. The wildcard receiver must restart in order to register again with the RPP store.

4.2.5. Implementing RPP

Follow the procedure below to configure Receiver-paced Persistence.

1. Set ume_receiver_paced_persistence for sources and receivers in a UM configuration file. If only certain sources or receivers in a context are RPP, use lbm_*setopt()l in the source or receiver application or use Ultra Messaging® Manager to specify RPP in an UM XML configuration file.

2. Set repository-allow-receiver-paced-persistence = 1 for the repository in the umestored XML configuration file.

3. Coordinate ume_flight_size_bytes between the repository and the source. Set the maximum flight size with the repository option, source-flight-size-bytes-maximum. Sources can reconfigure the repository's source-flight-size-bytes-maximum to a value less than or equal to the maximum.

4. Optional. Coordinate the ume_repository_ack_on_reception between the repository and the source. If the repository has repository-allow-ack-on-reception enabled (1), the source can choose to keep it enabled or turn it off ( ume_repository_ack_on_reception = 0). If the repository has repository-allow-ack-on-reception disabled (0), the source cannot turn it on.

5. Optional. If the repository is a disk repository (repository-type = disk or reduced-fd), set the maximum write delay with the repository option, repository-disk-write-delay. Sources can reconfigure the repository's repository-disk-write-delay to a value less than or equal to the maximum configured for the repository with ume_write_delay.

6. Optional. Coordinate repository size options between the source and repository. If you wish to use the repository's values, you do not need to configure source configuration values. The repository sets a maximum for these three options. The source can reconfigure the repository's options with values less than or equal to the maximum configured for the repository using the following UM configuration options.

   • ume_repository_size_threshold

   • ume_repository_size_limit

   • ume_repository_disk_file_size_limit

4.2.6. Example RPP Configuration Files

The sample configuration files shown below show how a store configuration file establishes certain RPP option values and the source can reconfigure them via a UM configuration file. Although only two files appear below, this configuration represents two, single-store quorum/consensus groups and one UM context. A second umestored configuration file would be required for the store store1rpp containing options and values identical to store0rpp.

#Sample UM Configuration File, UMP Version 5.3
#Major Options
source transport lbtrm
# in order and reassembled
receiver ordered_delivery 1
#Multicast Resolver Network Options
context resolver_multicast_address 225.8.17.29
context resolver_multicast_interface 10.29.3.0/24
# Transport LBT-RM Netowrk Options
source transport_lbtrm_multicast_address 225.8.17.30
context transport_lbtrm_multicast_address_low 225.12.17.10
context transport_lbtrm_multicast_address_high 225.12.17.14
#Transport LBT-RM Operation Options
context transport_lbtrm_data_rate_limit 10000000
context transport_lbtrm_retransmit_rate_limit 5000000
# Transport LBT-RM Reliability Options
receiver transport_lbtrm_nak_initial_backoff_interval 40000
receiver transport_lbtrm_nak_initial_backoff_interval 500
receiver transport_lbtrm_nak_generation_interval 10000
##Turn off NAKs
receiver transport_lbtrm_send_naks 0
#Request Network Options
context request_tcp_port_low 55000
context request_tcp_port_high 55500

## Persistence Options ###
source ume_store_group 0:1
source ume_store_name store0rpp:2929:0
source ume_store_group 1:1
source ume_store_name store1rpp:2930:1
source ume_store_behavior qc
source ume_flight_size 500
source ume_flight_size_bytes 1000000
source ume_receiver-paced-persistence 1
source ume_repository_size_threshold 104857600
source ume_repository_size_limit 209715200
source ume_repository_disk_file_size_limit 1073741824
source ume_repository_ack_on_reception 1
source ume_write_delay 1000
receiver ume_receiver-paced-persistence 1
receiver ume_explicit_ack_only 1
source ume_proxy_source 1
#source ume_session_id 535353
context ume_source_liveness_timeout 4000
context ume_receiver_liveness_interval 1000
source ume_confirmed_delivery_notification 1
                   

<?xml version="1.0"?>
<ume-store version="1.2">
  <daemon>
    <log>/configs/stores/umestored1/umestored.log</log>
    <lbm-license-file>/bin/umq_exp_license.txt</lbm-license-file> 
    <lbm-license-file>/bin/lbm_ume_umq_udx_rdma_license.txt</lbm-license-file> 
    <lbm-config>/configs/lbm_4_store.cfg</lbm-config>
    <pidfile>/configs/stores/umestored1/umestored.pid</pidfile>
    <web-monitor>*:15404</web-monitor>
  </daemon>
  <stores>
    <store name="rpp-ump-test-store-1" port="14667">
      <ume-attributes>
        <option type="store" name="disk-cache-directory" value="/stores/store1/cache"/>
        <option type="store" name="disk-state-directory" value="/stores/store1/state"/>  
        <option type="store" name="allow-proxy-source" value="0" />
        <option type="store" name="context-name" value="store1rpp"/>
      </ume-attributes>
      <topics>
        <topic pattern="ABC*" type="PCRE">
          <ume-attributes>
             <option type="store" name="repository-allow-receiver-paced-persistence" value="1"/>
             <option type="store" name="repository-type" value="disk"/>
             <option type="store" name="repository-size-threshold" value="104857600"/>
             <option type="store" name="repository-size-limit" value="209715200"/>
             <option type="store" name="repository-disk-file-size-limit" value="1073741824"/>
             <option type="store" name="repository-allow-ack-on-reception" value="1"/>
             <option type="store" name="repository-disk-write-delay" value="1000"/> 
             <option type="store" name="receiver-new-registration-rollback" value="0"/>
             <option type="store" name="source-activity-timeout" value="120000"/>
             <option type="store" name="receiver-activity-timeout" value="30000"/>
             <option type="store" name="retransmission-request-forwarding" value="0"/>
          </ume-attributes>
        </topic>
      </topics>
    </store>
  </stores>
</ume-store>
                   

4.2.7. RPP Cross Feature Functionality

4.3.1. Registration and Assignment

Source and receiver applications in a queuing operation use the same registration and registration response behaviors as sources and receivers in a persistence operation. (See UMP Registration.) The contents of the actual messages for registration and registration response for queues are slightly different, but the message paths are essentially the same.

After a receiver registers with a Queue, the Queue issues an Assignment ID, which identifies the receiver. The Queue assigns this ID to all messages queued for the receiver's topic. The Assignment feature implements the OAOO delivery feature. The Queue assigns messages to a receiver after any one of the following events.

• A message arrives at the Queue. (If no receiver interested in the topic has registered, the Queue holds the message until a receiver interested in the message's topic registers with the Queue.)

• A receiver registers with the Queue. (Obviously, the Queue must possess messages for the receiver's topic.)

• The Queue receives a Consumption Report from a receiver.

4.3.2. Message Paths

Depending on the data dissemination model in use, the messaging paths between sources, receivers, and queue instances may vary quite a bit.

4.3.3. Queue Feedback

Receiver applications as well as queue instances provide various forms of feedback. Queue instances send Stability Acknowledgements directly back to UMQ source applications to indicate successful submission of messages to the queue. Receiver applications signal message consumption by sending Consumption Reports back to the queue instances. See Queue Feedback.

4.3.4. UMQ Flight Size

UMQ supports a flight size mechanism similar to UMP Flight Size that tracks messages in flight from a particular source and responds when a send would exceed the configured flight size (umq_flight_size). You can configure umq_flight_size_behavior to either:

• block any sends that would exceed the flight size or,

• allow the sends while notifying your application.

UMQ considers a sent message in-flight until the source receives the configured number of stability acknowledgements from the queue(s). (No delivery confirmation exists in UMQ.) As with UMP Quorum/Consensus, intragroup and intergroup stability settings affect whether UMQ considers a messages in flight.

UMQ also supports a flight size mechanism for Multicast Immediate Messages (MIM). You configure MIM flight size with the context scope configuration options, (context) umq_flight_size and (context) umq_flight_size_behavior.

4.3.5. Topics and Queues

Sources send messages on topics and receivers listen on topics for messages. Similarly, UMQ source applications submit messages to queues with each message being sent on a specific topic. UMQ receiver applications listen on topics for messages. This means that Application Sets and the once-and-only-once (OAOO) behavior are on a per topic basis. A potentially helpful analogy is a queue may be the name of a mailbox and a topic may be the subject of an individual letter.

4.3.6. Queue Resolution

Unlike UMP persistent messaging, queues use a queue resolution mechanism built upon UM topic resolution for service location. A UMQ source application does not have to know the IP address and TCP port for each queue instance. Instead, all it requires is the name of the queue. This provides some flexibility in deployment for moving queue instances around and requires much less static information to be maintained for configuration of sources and receivers.

4.3.7. Queue Browser

UMQ supports the JMS Queue Browser specification with the following C API and Java API calls.

The following UM configuration options apply to Queue Browsing.

• create_queue_browser_context

• queue_browser_creation_delay

• queue_browser_timeout

<?xml version="1.0"?>
<um-configuration version="1.0">
  <users>
    <user name="userSmith">
      <verifier>69A4aAE70US3NiuZr/TvAbSWztu5na5TbFo8bdHxU5.ILdMu8rLd5ragE3p4Qcuz/nXxAj6kGnwIF2JrKdCf704U/Lxs7jvK.b2uyOQoqKG</verifier>
        <salt>3BmRWUznvzy0n2</salt>
        <roles>
          <role name="admin"></role>
          <role name="normal"></role>
        </roles>
    </user>
    <user name="">
      <verifier/>
      <salt/>
      <roles>
        <role name="admin"/>
      </roles>
    </user>
  </users>
  <roles>
    <role name="admin">
      <action>MSG_LIST</action>
      <action>MSG_RETRIEVE</action>
      <action>TOPIC_LIST</action>
     </role>
    <role name="normal">
      <action>TOPIC_LIST</action>
    </role>
  </roles>
</um-configuration>
           

4.4.1. Parallel Source Dissemination (PSD)

ULB supports the Parallel Source Dissemination (PSD) dissemination model. In the figure below, note that the ULB PSD model operates in a very similar way to the UMQ Parallel Queue Dissemination (PQD) model.

The key difference between ULB's PSD model and UMQ's is that with ULB, the source sends both the data and the assignment/control information on the same transport session. ULB has no need for a control channel, as with UMQ.

ULB receivers, as with UMQ, send unicast Consumption Reports back to the source.

4.4.2. Source Fairness

ULB differs from UMQ somewhat in source "fairness". While UMQ decouples sources from receivers with a queue -- and therefore avoids receiver overload -- ULB does not use a queue or other middleware, so sources could conceivably combine to overload a receiver, because of the real-time nature of low-latency load balancing.

Therefore, ULB governs fairness between sources by using a) receiver feedback information and b) receiver portion size, in concert with c) source configuration option umq_ulb_application_set_load_factor_behavior. For more, see Queue Feedback and Receiver Portion Size.

4.4.3. Ultra Load Balancing Flight Size

ULB supports a flight size mechanism similar to UMP Flight Size and UMQ Flight Size that tracks messages in flight from a particular source and responds when a send would exceed the configured flight size (umq_ulb_flight_size). You can configure umq_ulb_flight_size_behavior to either:

• block any sends that would exceed the flight size or,

• allow the sends while notifying your application.

ULB considers a message in-flight until the source receives the MSG_COMPLETE notification from all application sets. See also umq_ulb_events.

4.4.4. Indexed Ultra Load Balancing (ULB)

Indexed Ultra Load Balancing (ULB) is similar to Indexed Queuing. The source and receiver APIs are exactly the same with only minor differences (e.g., receiver event types remain the same as for indexed queueing, but have a flag set indicating that the source of the event was a ULB source). There are two important differences to note:

• Assignment of indices is done at the source, on a per-source basis, with no global coordination of index assignments, so multiple sources sending on the same index may happen to assign the index to different receivers.

• The advanced allow/deny configuration rules and configuration option inheritance available via the umestored's Index Rules Element for indexed queueing are not available for indexed ULB.

4.5.1. Source Events

The following events and callbacks are available for source applications.

4.5.2. Receiver Events

The following callbacks and messages are available for receiver applications

4.5.3. Context Events

The following events are available for the context of source and receiver applications.