|
Guide for Persistence
|
The <option> element is the primary construct for setting Store configuration options. It always appears inside a <ume-attributes> block, which can appear in two places of your Store configuration file: the <store> element, or the <topic> element.
The <option> element is used to set three kinds of Store configuration options
The <option> element can be used to set LBM configuration options. I.e. options that control the UM library. Those options come in different scopes. For the purposes of the Store, only the "context", "receiver", and "source" scopes are used, as follows:
But the valid LBM scopes depend on the "<option>" element's ancestor:
| Ancestor | Valid LBM type Attributes |
|---|---|
| <store> | type="lbm-context" |
| <topic> | type="lbm-receiver" type="lbm-source" |
See the UM Configuration Guide for the full list of LBM configuration options.
Most Store deployments do not make heavy use of the <option> element to set LBM configuration parameters. Instead they use an LBM configuration file (via the <lbm-config> or the UMP Element "<xml-config>" element) for setting most of the desired LBM configuration options. However, it is often desired to override one or more of those settings based on an individual Store, or an individual topic within a Store.
Example
In this hypothetical example, the Store loads a generic project-specific flat LBM configuration file:
The file "generic.cfg" sets the receiver-side LBT-RM socket buffer size to 4 MB using:
Since we want the Stores to avoid loss as much as possible, the receive socket buffers should be made large (32 MB):
Finally, for the specific topic "EventStream", one might want the initial NAK backoff interval to be longer than for application receivers.
For Store configuration options, the <option> element must appear inside the <store> element, as follows:
For example:
These options apply to all repositories in this Store instance.
Pathname for disk Store message cache directory. Must be between 1 and 230 characters long. It is the user's responsibility to create this directory; the Store will not do so.
Default:
umestored-cache
The Store looks for a sub-directory by that name in the current working directory of the stored process.
Example:
Pathname for disk Store state directory. Must be between 1 and 230 characters long. It is the user's responsibility to create this directory; the Store will not do so.
Default
umestored-state
The Store looks for a sub-directory by that name in the current working directory of the Store Process.
Example:
Allows the Store to act as a proxy source in case a registered source terminates.
Default:
0 (Disable)
Example:
Modifies the Store's proxy source election algorithm to better conform to the general UM persistence design.
See Proxy Source Elections for details.
Default:
0 (Disable)
Example:
Name of the Store that can be used by sources to refer to the Store instead of the address:port. Restricted to 128 characters in length, and may contain only alphanumeric characters, hyphens, and underscores.
A Store runs in its own context, so the Store's context name can be used to identify the Store. UM automatically resolves Store context names, which can facilitate persistent operation across the DRO. Store context names must be unique across the entire network.
See also Identifying Persistent Stores
Default:
None (Store must be referenced by address:port).
Example:
Specifies the number of retransmission requests processed by a Store per second across all topics. The Store drops all retransmission requests that exceed this value.
Default:
262144
Example:
For Store repository configuration options, the <option> element must appear inside the <topic> element, as follows:
For example:
These options apply all repositories for sources that match the specified topic(s).
If enabled (value="1"), the Store forwards retransmission requests to sources if and only if the Store does not have the data. If disabled (value="0"), the Store services retransmission requests for data it has, and does not forward requests to sources for data it does not have. (This option should not be enabled if you anticipate using the Request: Mark Stored Message Invalid feature.)
Default:
0 (Store services retransmission requests and does not forward requests)
Example:
Specifies how messages should be retained by the Store.
Possible Values
Default:
memory
Example:
Specifies the minimum number of message bytes retained in the Store's memory cache. The purpose of this option is for disk Stores to keep some messages in memory even after they have been written to disk. This allows for rapid recovery of recent messages.
For SPP Stores, includes message payload, headers, and Store structure overhead. For RPP Stores, only includes message payload.
Note that the Store's memory cache size can fall below this threshold. With RPP Stores, if all required receivers have acknowledged consumption of all messages, all messages will be deleted.
Also for RPP, the source LBM configuration may override this option with ume_repository_size_threshold (source). In that case, the Store's repository-size-threshold value is used as the maximum allowed value for ume_repository_size_threshold. If the source exceeds this value, its registration is rejected.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
1024
Example:
Specifies the maximum number of message bytes retained in the Store's memory cache. For repository-type "memory", this represents the maximum size for the entire repository. For repository-type "disk", the total repository size is limited by Topic Option "repository-disk-file-size-limit".
Note that the design of UM's persistence allows a maximum of 2,147,483,647 messages (2**31 - 1) to be persisted. Do not specify a limit that would allow more than 2,147,483,647 messages to be stored.
For SPP Stores, includes message payload, headers, and Store structure overhead. For RPP Stores, only includes message payload.
Also for RPP, the source LBM configuration may override this option with ume_repository_size_limit (source). In that case, the Store's repository-size-limit value is used the maximum allowed value for ume_repository_size_limit. If the source exceeds this value, its registration is rejected.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
50331648 (48 MB)
Example:
Specifies how long in seconds the repository keeps a message available. Pertains to a memory Store or the memory cache of a disk Store. The repository reclaims space used to store messages that exceed this threshold. Note that if these deleted messages have been persisted to disk, they are available to receivers for recovery.
This is a rarely-used option, typically only useful on a memory-only Store.
A value of 0 means message age is not considered in retention decisions.
Default:
0 (disabled)
Example:
This option is identical to Topic Option "repository-disk-max-read-async-cbs" and sets the same underlying limit to outstanding async reads.
The original intent of this option was to simultaneously set both Topic Option "repository-disk-max-read-async-cbs" and Topic Option "repository-disk-max-write-async-cbs". But Informatica subsequently determined that the async write limit needed to be set to 1. So this option was changed to update only the async read limit.
Default:
10,000 (callbacks)
Example:
For topics with a repository-type "disk", specifies the maximum number of outstanding async I/O callbacks for writing messages to disk. This option is deprecated, and if supplied, must be set equal to 1. (If supplied with any other value, an error will be logged and the value will be ignored.)
Default:
1 (callback)
Example:
For topics with a repository-type "disk", specifies the maximum number of outstanding async I/O callbacks for reading messages from disk. A low value can lead to severely slower message recovery rates by receivers.
Default:
10,000 (callbacks)
Example:
Specifies the maximum amount of disk space, in bytes, that will be used to store retained messages. A minimum value of 196992 is enforced.
Note that the design of UM's persistence allows a maximum of 2,147,483,647 messages (2**31 - 1) to be persisted. Do not specify a limit that would allow more than 2,147,483,647 messages to be stored.
This option only applies for repository-type "disk".
Also for RPP, the source LBM configuration may override this option with ume_repository_disk_file_size_limit (source). In that case, the Store's repository-disk-file-size-limit value is used as the maximum allowed value for ume_repository_disk_file_size_limit. If the source exceeds this value, its registration is rejected.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
104857600 (100 MB)
Example:
For topics with a repository-type "disk", if set to 1, UM pre-allocates a Store's cache files to match their maximum size on disk (as configured by repository-disk-file-size-limit) upon creation, as opposed to growing to that size as the Store receives new messages. For ext3/4 and NTFS file systems, this options creates a sparse file, which does not allocate all of the underlying data blocks. Advantages of pre-allocation include better performance on rotating disks due to less file fragmentation, and knowing that enough disk space exists for any new source that registers. Disadvantage is the time to create the cache files, especially if many sources register at once.
Default:
0 (do not pre-allocate)
Example:
For topics with a repository-type "disk", specifies the size of the buffers that will be used in async I/O operations for reading and writing messages to disk. A minimum value of 65664 is enforced.
Default:
1024000 (bytes)
Example:
For topics with a repository-type "disk", specifies whether the messages saved to disk should include a checksum field for validation if the Store is restarted.
Default:
0 (disabled)
Example:
Establishes the period of time in milliseconds from a source's last activity to the release of the source's RegID. Stores return an error to any new source requesting the source's RegID during this period. If proxy sources are enabled (ume_proxy_source (source)) the Store does not release the source's RegID and UM elects a proxy source. If neither proxy sources nor ume_state_lifetime (source) are configured, the Store also deletes the source's state and cache. Can be overridden by ume_activity_timeout (source). See also Persistence Proxy Sources.
Note that Topic Option "source-activity-timeout" value should be set to at least double the Topic Option "keepalive-interval" value.
Default:
30000 (30 seconds)
Example:
Establishes the period of time in milliseconds from a source's last activity to the deletion of the source's state and cache by the Store, regardless of whether a proxy source has been created or not. You can also configure ume_state_lifetime (source) for the source. The Store uses whichever is shorter. See also Persistence Proxy Sources.
Default:
0 (disabled)
Example:
Establishes the period of time in milliseconds from a receiver's last activity to the release of the receiver's RegID. Stores return an error to any new request for the receiver's RegID during this period. Can be overridden by ume_activity_timeout (receiver). See also Persistence Proxy Sources.
Default:
30000 (30 seconds)
Example:
Establishes the period of time in milliseconds from a receiver's last activity to the deletion of the receiver's state and cache by the Store. You can also configure ume_state_lifetime (receiver) for the receiver. The Store uses whichever is shorter. See also Persistence Proxy Sources.
Default:
0 (disabled)
Example:
Specifies the period in milliseconds a Store will check for activity of sources and receivers.
Default:
750 (milliseconds)
Example:
Specifies the period in milliseconds a Store will generate keepalive traffic to sources and receivers if there has been no traffic required in the normal course of operation.
Note that Topic Option "source-activity-timeout" value should be set to at least double the Topic Option "keepalive-interval" value.
Default:
3000 (3 seconds)
Example:
Specifies an upper limit to the number of messages a registering persistent receiver should recover during "late join".
The receiver can limit the number of messages that it can request for recovery using the LBM configuration option retransmit_request_maximum (receiver). If the request exceeds the limit configured in the Store's topic option receiver-new-registration-rollback, the receiver's recovery will be limited to the latter number of messages.
Note that this limit interferes with the persistence guarantee of delivery. If a limit of 1,000 is configured, and a receiver exits and restarts after the source has sent 4,000 more messages, the receiver might not be able to recover all 4,000 messages. It might be limited to the most-recent 1,000 messages.
Also, note that a persistent receiver can set up a initial sequence number callback using the LBM configuration option ume_recovery_sequence_number_info_function (receiver). This allows the application to specify a desired starting sequence number. If the application is written to use this approach, Store topic option "receiver-new-registration-rollback" and LBM configuration option retransmit_request_maximum should be at their default values.
If used, the value for receiver-new-registration-rollback must be between 1 and 2147483647 (maximum signed 32-bit integer). The default value of 2147483647 effectively disables the limit.
Default:
2147483647
Example:
Specifies the interval, in milliseconds, used when electing a proxy source. When a source, which requested that a proxy source be provided for it, has been detected as no longer active, each Store eligible to provide a proxy source for it waits for an amount of time which is randomized in the range [0.5*proxy-election-interval .. 1.5*proxy-election-interval]. If no other Store has been elected to serve as the proxy source, the Store declares itself as the proxy source.
Default:
60,000 (60 seconds)
Example:
Specifies the maximum amount of time in milliseconds that stability acknowledgments will be batched before being sent to a source. This batching is only enabled if Topic Option "stability-ack-minimum-number" is set to greater than 1.
Batching stability ACKs can increase throughput of Stores (especially memory Stores) significantly, but introduces a delay between when a message is actually stable in the Store and when the source is notified of message stability.
At high message rates, the stability ACKs will normally be triggered by the received messages exceeding stability-ack-minimum-number. However, if the source publishing rate drops significantly, the stability-ack-interval ensures an upper bound to the time required for stability ACKs.
Default:
200 (200 milliseconds)
Example:
Specifies the minimum number of message fragment stability acknowledgments that must accumulate before a stability ACK is sent to a source. With the default value of 1, stability ACKs are sent immediately as soon as messages are stable. Increasing this value causes stability ACKs to be batched, which can increase throughput of Stores (especially memory Stores) significantly, but introduces a delay between when a message is actually stable in the Store and when the source is notified of message stability.
If using a stability ACK-based flight size on a persistent source in combination with this option, it is advisable to make sure stability-ack-minimum-number is set less than the source's flight size. Otherwise, stability ACKs will only be sent upon expiration of the stability-ack-interval timer, resulting in bursty stop-and-go sending.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
1 (fragment; no batching)
Example:
Specifies if the repository allows receiver-paced persistence (RPP). If allowed (value 1), the source may request RPP with ume_receiver_paced_persistence (source).
Note that the Store cannot be directly configured to enable RPP; the source must be configured to request it. Otherwise, the repository defaults to SPP. This option only allows the source to request it.
See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
0 (Store does not allow the source to specify RPP)
Example:
For RPP, specifies if the repository allows "ack on reception" behavior. If allowed (value 1), the source may request "ack on reception" with ume_repository_ack_on_reception (source). See ume_repository_ack_on_reception (source) for more information about "ack on reception" behavior.
Note that the Store cannot be directly configured to enable "ack on reception"; the source must be configured to request it. Otherwise, the repository defaults to acknowledging when messages are written to disk. This option only allows the source to request it.
Also note that for SPP Stores, this option is ignored.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
0 (Store does not allow the source to specify ack on reception behavior)
Example:
Specifies a delay in milliseconds after a message is received before the message is written to disk. This option is for use with RPP, and is intended to allow all required receivers to acknowledge consumption of messages within the write delay time. This deletes the messages from the memory cache before they are written to disk. If all required receivers acknowledge consumption within the delay time, the Store never needs to write messages to disk.
This option only applies for repository-type "disk".
For RPP, the source LBM configuration may override this option with ume_write_delay (source). In that case, the Store's repository-disk-write-delay value is used as the maximum allowed value for ume_write_delay. If the source exceeds this value, its registration is rejected.
For RPP, see RPP Configuration Specifics for interactions between this and other configuration options. See RPP: Receiver-Paced Persistence for general information on RPP.
Default:
0 (milliseconds, no delay)
Example:
For RPP, specifies the maximum number of in-flight payload bytes that the source is allowed to configure with ume_flight_size_bytes (source). If the source exceeds this value, its registration is rejected.
See Persistence Flight Size for more information.
Default:
4194304 (4 MB)
Example: