Guide for Persistence
|
Persistence offers the following message recovery mechanisms:
Method | Product | Transports | Description |
---|---|---|---|
Negative Acknowledgments (NAKs) | UMS, UMP, UMQ | LBT-RM, LBT-RU | Recovers lost transport datagrams from the source which may contain many small topic messages or fragments of a large message. Receivers send unicast NAKs to the source for missed transport datagrams. Source retransmits datagrams over the configured UM transport. |
Late Join | UMS, UMP, UMQ | All | Retransmits messages via unicast to receivers joining the stream after the messages were originally sent. See Using Late Join. |
Durable Receiver Recovery | UMP, UMQ | All | Recovers messages persisted while a durable receiver was off line. UM initiates recovery when a durable receiver joins a persistent stream. The receiver then requests retransmission from the store starting with the low sequence number, defined as the last message it acknowledged to the store plus one. The store unicasts retransmissions. See Persistent Receiver Recovery. |
Off Transport Recovery | UMS, UMP, UMQ | All | Recovers lost topic messages. Receiver detects lost sequence number and requests retransmission from the source or persistent stores (if applicable). UM unicasts retransmissions. See Off-Transport Recovery (OTR). |
Proactive Retransmissions | UMP, UMQ | All | Recovers lost messages never received by the store or never acknowledged by the store. Operates independently of any receivers. Source unicasts retransmissions. See Proactive Retransmissions. |
Deployment decisions play a huge role in the success of any persistent system. Configuration in UM has a number of options that aid in performance, fault recovery, and overall system stability. It is not possible, or at least not wise, to totally divorce configuration from application development for high performance systems. This is true not only for persistent systems, but for practically all distributed systems. When designing systems, deployment considerations need to be taken into account for the following:
Source Considerations
Receiver Considerations
Performance of sources is heavily impacted by:
the release policy that the source uses
streaming methods of the source
Source release settings have a direct impact on memory usage. As messages are retained, they consume memory. You reclaim memory when you release messages. Message stability, delivery confirmation and retention size all interact to create your release policies. UM provides a hard limit on the memory usage. When exceeded, UM delivers a Forced Reclamation event. Thus applications that anticipate forced reclamations can handle them appropriately. See also Source Message Retention and Release.
How the source streams data has a direct impact on latency and throughput. One streaming method sets a maximum, outstanding count of messages. Once reached, the source does not send any more until message stability notifications come in to reduce the number of outstanding messages. The umesrc example program uses this mechanism to limit the speed of a source to something a store can handle comfortably. This also provides a maximum bound on recovery that can simplify handling of streaming source recovery.
The throughput and latency requirements of the data are normal UM concerns.
In addition to the following, receiver performance shares the same considerations as receivers during normal operation.
Acknowledgement Generation
Persistent receivers send a message consumption acknowledgement to stores and the message source. Some applications may want to control this acknowledgement explicitly themselves. In this case, ume_explicit_ack_only (receiver) can be used.
Controlling Retransmission
Persistent receivers during fault recovery are another matter entirely. Receivers send retransmission requests and receive and process retransmissions. Control over this process is crucial when handling very long recoveries, such as hundreds of thousands or millions of messages. A receiver only sends a certain number of retransmission requests at a time.
This means that a receiver will not, unless configured to with retransmit_request_outstanding_maximum (receiver), request everything at once. The value of the low sequence number (Persistent Receiver Recovery) has a direct impact on how many requests need to be handled. A receiving application can decide to only handle the last X number of messages instead of recovering them all using the option, retransmit_request_maximum (receiver). The timeout used between requests, if the retransmission does not arrive, is totally controllable with retransmit_request_interval (receiver). And the total time given to recover all messages is also controllable.
Recovery Process
Theoretically, receivers can handle up to roughly 2 billion messages during recovery. This limit is implied from the sequence number arithmetic and not from any other limitation. For recovery, the crucial limiting factor is how a receiver processes and handles retransmissions which come in as fast as UM can request them and a store can retransmit them. This is perhaps much faster than an application can handle them. In this case, it is crucial to realize that as recovery progresses, the source may still be transmitting new data. This data will be buffered until recovery is complete and then handed to the application. It is prudent to understand application processing load when planning on how much recovery is going to be needed and how it may need to be configured within UM.
UM stores have numerous configuration options. See Configuration Reference for Umestored for details.
Configuring Store Usage per Source
A store handles persisted state on a per topic per source basis. Based on the load of topics and sources, it may be prudent to spread the topic space, or just source space, across stores as a way to handle large loads. As configuration of store usage is per source, this is extremely easy to do. It is easy to spread CPU load via multi-threading as well as hard disk usage across stores. A single store process can have a set of virtual stores within it, each with their own thread.
Disk vs. Memory
As mentioned previously in Persistent Store Concept, stores can be memory based or disk based. Disk stores also have the ability to spread hard disk usage across multiple physical disks by using multiple virtual stores within a single store process. This gives great flexibility on a per source basis for spreading data reception and persistent data load.
UM stores provide settings for controlling memory usage and for caching messages for retransmission in memory as well as on disk. All messages in a store, whether in memory or on disk, have some small memory state. This is roughly about 72 bytes per message. For very large caches of messages, this can become non-trivial in size.
Activity Timeouts
UM stores are NOT archives and are not designed for archival. Stores persist source and receiver state with the aim of providing fault recovery. Central to this is the concept that a source or receiver has an activity timeout attached to it. Once a source or receiver suspends operation or has a failure, it has a set time before the store will forget about it. This activity timeout needs to be long enough to handle the recovery demands of sources and receivers. However, it can not and should not be infinite. Each source takes up memory and disk space, therefore an appropriate timeout should be chosen that meets the requirements of recovery, but is not excessively long so that the limited resources of the store are exhausted.
Recommendations for Store Configuration
The following conditions allow sources to continue to send messages:
Quorum - Completed registration of a quorum of stores within at least one group. This is affected by group definitions, plus intragroup and intergroup stability settings. See also Persistent Store Concept.
Configure your stores to address the failure cases you believe are more probable and from which you want to recover. For example, if a particular store group persists topics of higher importance, you may want to increase the number of stores in that group to maintain quorum in the face of a store failure. Or if a particular location has a higher incidence of failures than other locations, you may want to add additional stores in other locations.
Although many different conditions and requirements can apply to the configuration of persistent stores, Informatica recommends the following best practices:
Minimum of 3 stores - Requiring a minimum of 3 stores needed for quorum in a single store group is optimal. Using 5 stores, for example, in a group allows sources to keep sending in the face of the loss of up to 2 stores.
Multiple store groups - When using multiple store groups, Informatica recommends using at least 3 stores in each group.
Store Configuration Practices to Avoid
Informatica does not support the following store configuration practices:
Do not use multiple store groups of one store each. Recovery does not work well in this configuration because it allows sources to resume sending as soon as it has registered with a single store, and if that store is not fully up-to-date, this can lead to message loss for receivers.
By default, UM expects persistent sources to be running concurrently with persistent receivers. If a source exits, any persistent receivers will disconnect from that source's transport and will wait for the source to come back. More significantly, if a new receiver starts while the source is absent, the receiver will be unable to discover the stores where the old source's previous messages are stored. So that late-joining receiver will not recover messages until the source finally restarts.
The Proxy Source feature allows you to configure stores to automatically continue sending the source's topic advertisements which allow new receivers to join the source's transport session and request Source Registration Information (SRI) to register with the store and request retransmissions. After the source returns, the store automatically stops acting as a proxy source. Stores can be located across a UM Router or within the same LAN as the failed source.
Some other features of Proxy Sources include:
Requires a Quorum/Consensus store configuration.
Normal store failover operation also initiates a new proxy source.
A store can be running more than one proxy source if more than one source has failed.
Note that proxy sources do introduce extra network and CPU loading, so proxy sources should only be enabled if their functionality is needed.
The following sequence illustrates the life of a proxy source:
A source configured for Proxy Source sends to receivers and a group of Quorum/Consensus stores.
The source fails.
The source's ume_activity_timeout (source) or the store's source-activity-timeout expires.
The Quorum/Consensus stores elect a single store to run the proxy source.
The elected store creates a proxy source and sends topic advertisements.
The failed source reappears.
If the store running the proxy source fails, the other stores in the Quorum/Consensus group detect a source failure again and elect a new store to initiate a proxy source.
If a loss of quorum occurs, the proxy source can continue to send advertisements, but cannot send messages until a quorum is re-established.
UM provides activity and state lifetime timers for sources and receivers that operate in conjunction with the proxy source option or independently. This section explains how these timers work together and how they work with proxy sources.
The ume_activity_timeout (source) and ume_activity_timeout (receiver) options determine how long a source or receiver must be inactive before a store allows another source or receiver to register using that RegID. This prevents a second source or receiver from stealing a RegID from an existing source or receiver. An activity timeout can be configured for the source/receiver with the UM Configuration Option cited above or with a topic's <ume-attribute> configured in the umestored XML configuration file. The following diagram illustrates the default activity timeout behavior, which uses source-state-lifetime in the umestored XML configuration file.
In addition to the activity timeout, you can also configure sources and receivers with a state lifetime timer using the following options.
The ume_state_lifetime (source) and ume_state_lifetime (receiver) options, when used in conjunction with the ume_activity_timeout (source) and ume_activity_timeout (receiver) options, determines at what point UM removes the source or receiver state files. UM does not check the state lifetime until the activity timeout expires. The following diagram illustrates this behavior:
If you have enabled the Proxy Source option, the ume_activity_timeout (source) triggers the creation of the proxy source. The following diagram illustrates this behavior:
You must configure both the source and the stores to enable the Proxy Source option.
Configure the source in a UM Configuration File with the source configuration option, ume_proxy_source (source).
When multiple stores in a Quorum/Consensus configuration notice the loss of a registered source (expiration of the source's ume_activity_timeout (source)) configured for proxy sources, only one of the stores needs to create a proxy source to continue sending topic advertisements.
The proxy source election process determines which store creates the proxy source. Each store starts by waiting a randomized amount of time based on its proxy-election-interval option setting. The store creates a proxy source if it has not received a persistent registration request (PREG) from a proxy on a different store. The proxy source then sends a PREG containing a unique random value to the other stores. This value determines which store deletes it's proxy source in the case that any two stores independently determine they should create a proxy source. The nature of the random values ensures that only one store within the Q/C group or configuration of groups keeps its proxy source.
Proactive Retransmissions, which is enabled by default, address two types of loss:
loss of message data between the source and a store
The store sends message stability acknowledgments to the source after the store persists the message data.
With Proactive Retransmissions, the source maintains an unstable message queue for those messages sent but not acknowledged by the store. The source checks this queue at the ume_message_stability_timeout (source). If a message in this queue exceeds its ume_message_stability_timeout (source), the source retransmits the message and puts it back on the unstabilized message queue, restarting the message's ume_message_stability_timeout (source).
The source continues to retransmit and check the message's stability timeout until the ume_message_stability_lifetime (source) expires or it receives a stability acknowledgment from the store. If the source has not received a stability acknowledgment when the ume_message_stability_lifetime (source) expires, the source sends a Store Message Not Stable source event notification to the application. When the store discards the message because it has not met stability requirements, the store sends a Store Forced Reclaim source event notification to the application.
To disable Proactive Retransmissions, set ume_message_stability_timeout (source) to 0 (zero). As a result, sources do not create an unstable message queue.
The following applies whether you enable or disable Proactive Retransmissions.
The store does not discard duplicate messages, but rather always responds to duplicate, retransmitted messages by sending stability acknowledgments even if the message is already stable.