|
Guide for Persistence
|
The operating parameters for umestored come from a Store configuration file that must be supplied on the command line (see Man Pages for Store). A Store Process contains a UM context and receivers that may be configured with default values through an LBM configuration file referenced in the XML configuration file. Default UM options my be overridden for each configured Store using the Store configuration file.
An overview of the file format can be seen in the umestored Configuration DTD.
You configure umestored to instantiate Stores with the Store configuration file, which Ultra Messaging reads at start up.
The Store configuration file for persistence has the following sections:
Daemon section - holds administrative parameters for such things as the location of log files, the LBM Configuration File, etc.
High Level Store Configuration File:
The XInclude mechanism can be used to merge or share XML files for UM library configuration, Store configuration, and DRO configuration. This is typically done to avoid duplicating groups of configuration options in multiple places.
To include an external file from a Store configuration file, use the following syntax:
Where FILEPATH can be a local file name, or a network path starting with "http:" or "ftp:". For example:
Note that secure forms of network paths ("https:" or "sftp:") are not supported.
Files to be included must be formatted such that all elements are enclosed in a single container element.
Example of an invalid file:
Example of valid file:
Store configuration files do not support templates. It is common that groups of configuration options need to be repeated across many Store configurations.
For example consider the Store configuration file "store_conf.xml":
This can be a lot of repeated content for the stores running in this daemon instance.
The XInclude feature can be used to reduce duplicate content by creating a second file "store_topic_attr.xml":
Now "store_conf.xml" can be coded as:
Container element that holds the configuration for the Persistent Store Process. Also defines the version of the configuration format used by the file.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| version | Version number of Store XML DTD that the configuration file corresponds to. See umestored Configuration DTD for a description of the different versions. Users are encouraged to update their Store configuration files to correspond to the latest version supported by the Store software in use. | string | (no default; must be specified) |
Example:
Container element for one or more <store> elements. A Store Process can run multiple independent Store instances. Some users prefer to run multiple Store instances in a single process to reduce their process management complexity. Other users prefer to run multiple Store Processes, each with a single Store instance, to reduce the impact of a Store Process failing.
There should be little or no performance difference between multiple Store instances running in the same process vs. multiple Store Processes on the same host. However, for maximum Store performance, it is generally easier to pin Store threads to cores when each process is running a single Store instance.
It is NOT recommended for multiple Stores within a QC group to run in the same process, or even the same host, as this defeats the goal of reliability through redundancy.
Example:
Configuration for a Store instance.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| name | Identifies log messages for this Store instance in the Store Process log file, the Store Web Monitor, and Daemon Stats. Note: this is not the name that sources can use instead of a network address (see the context-name option). | attr_name | (no default; must be specified) |
| interface | Specifies the IP address over which Store Process accepts connection requests for this Store. You can specify a single IP address, such as 10.29.3.16, or a range of addresses, 10.29.3.16/25. See also Identifying Persistent Stores. | string | "0.0.0.0" (INADDR_ANY) |
| port | TCP port where Store Process should listen for connection requests to this Store. Starting with UM version 6.8, zero may be supplied. In that case, the Store will choose an available port in the range request_tcp_port_low (context) to request_tcp_port_high (context). This typically requires the use of named stores. | string | (no default; must be specified) |
Example:
Container for <topic> elements. Defines the topics that this Store instance will persist.
Example:
Defines a topic pattern which the Store will use to find sources to persist. Also contains configuration information about those topics.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| pattern | A string that is used to discover sources to be persisted. The string can be a simple topic name (type="direct"), or it can be a regular expression (type="pcre") which can match more than one topic. | string | (no default; must be specified) |
| type | How the pattern attribute should be interpreted. | "direct" - Topic name (exact string match) "PCRE" - Perl regular expression. "regexp" - Posix regular expression. Deprecated; do not use. | direct |
Example:
In this example, the topic "NYSE.xyz" and all topics that start with "alert." are persisted in the "MyStore1" Store instance.
Container for a set of <option> elements.
Example:
In this example, some options are at the Store level and apply to all topics. Other options are specific to the topic "NYSE.xyz".
Set a configuration option of a particular type.
This element is used to set most of the operational parameters of the Store and its repositories.
There are many different options that can be set. See "<option>" Element Details for the full list.
An option element has a type attribute. The valid types depend on whether the option's ancestor element is "<store>" or "<topic>":
| Ancestor | Valid type Attributes |
|---|---|
| <store> | type="store" type="lbm-context" |
| <topic> | type="store" type="lbm-receiver" type="lbm-source" |
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| type | Type of configuration option. | "lbm-receiver" - LBM configuration option of scope "receiver". "lbm-context" - LBM configuration option of scope "context". "lbm-source" - LBM configuration option of scope "source". "store" - Store configuration option. | (no default; must be specified) |
| name | Name of option. | attr_name | (no default; must be specified) |
| value | Value for option. | string | (no default; must be specified) |
Example:
Control how much saved message data is restored when a disk-based Store is restarted.
By default, when a disk-based Store is stopped and restarted, it will restore all messages in the cache file to rebuild its internal index. If the cache file is very large, this can take significant time.
Using this element, you can direct the Store to read only recent data. Note that it can be specific to a Store, or a topic within a Store.
If this element is supplied in both the "<store>" and "<topic>" levels, the "<topic>" setting will override the "<store>" setting.
See Limit Initial Restore with Restore-Last for use case.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| behavior | Define how the value attribute is interpreted. | "hours" - Directs the Store to restore the most recent value number of hours worth of message data. NOTE: the message time is relative to the last (most recent) message in the cache, not the absolute time that the Store is restarted. For example, if the store is restarted on Sunday, but the most-recent message was sent the previous Friday afternoon, then value hours worth of messages sent on Friday will be restored. "none" - Directs the Store to ignore value, disabling the feature, resulting in the entire cache file being restored. | hours |
| value | Number of units worth of message data to restore. The units are defined by the "behavior" attribute. For units of "hours", the valid range is 0 - 336 (14 days). The special value "0" disables the feature, resulting in the entire cache file being restored. | string | 0 (disable) |
Example: on restart, only restore the most recent 8 hours worth of messages.
DEPRECATED: Set how often the Store publishes its Daemon Stats. See daemonstatistics for general information on Daemon Statistics.
Informatica requests users to migrate to using the UM configuration file to enable automatic monitoring with Protocol Buffer monitoring format for Store and DRO by setting monitor_format (context) to "pb". See Automatic Monitoring.
Example:
Daemon Statistics are configured at both the daemon level and at the Store level.
Configures the rate at which one particular grouping of Daemon Statistics messages are published. See daemonstatistics for general information on Daemon Statistics.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| name | Name of statistics group being configured. | "default" - Sets a default interval for all message types. "store" - Sets the interval for messages of type umestore_store_dmon_stat_msg_t. "source" - Sets the interval for messages of type umestore_repo_dmon_stat_msg_t. "receiver" - Sets the interval for messages of type umestore_rcv_dmon_stat_msg_t. "disk" - Sets the interval for messages of type umestore_disk_dmon_stat_msg_t. "config" - Sets the interval for messages of types umestore_*_dmon_config_msg_t. "memory" - Sets the interval for messages of type umestore_smart_heap_dmon_stat_msg_t. | (no default; must be specified) |
| ivl | Time, in seconds, between publishing the statistics group being configured. | string | (no default; must be specified) |
Example:
Daemon Statistics are configured at both the daemon level and at the Store level.
Container element for configuration elements that apply to the entire Store Process.
Example:
DEPRECATED: The daemon-monitor element configures the Store Process for Store Binary Daemon Statistics.
Informatica requests users to migrate to using the UM configuration file to enable automatic monitoring with Protocol Buffer monitoring format for Store and DRO by setting monitor_format (context) to "pb". See Automatic Monitoring.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| topic | Topic name for used to publish daemon statistics. | string | "umestore.monitor" |
Example:
Daemon Statistics are configured at both the daemon level and at the Store level.
Controls if the daemon will respond to requests from monitoring applications. See Daemon Control Requests for general information, Store Daemon Control Requests for Store-specific requests.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| allow | Enables handling requests. | "0" - Disable request handling. "1" - Enable request handling. | "0" |
Example:
Daemon Statistics are configured at both the daemon level and at the Store level.
Controls if the daemon will respond to requests from monitoring applications. See Daemon Control Requests.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| allow | Enables handling requests. | "0" - Disable request handling. "1" - Enable request handling. | "0" |
Example:
Daemon Statistics are configured at both the daemon level and at the Store level.
Pathname for LBM configuration file to be used when the Store creates UM objects (context, receivers, sources).
When used as a child element of <daemon-monitor>, configures the UM objects used for publishing Store Binary Daemon Statistics.
Note that starting with UM version 6.13, if one or more errors are discovered in the LBM configuration file, the errors are written to the log file and the Store continues running. I.e. errors in the LBM configuration file are treated as warnings. See Configuration Error Handling for an explanation.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
Address and port for the Store web-based monitor. Format is "Address:Port", where "Address" is either an IP address of one of the host's interfaces, or is "*" which allows the use of any interface. See Store Web Monitor for more information.
If omitted, the web monitor is disabled.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
Pathname for UM license file. NOTE: starting with UM version 6.8, a license key is no longer required for Store operation. This element is retained for backwards compatibility.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Pathname for LBM configuration file (in XML format) to be used when the Store creates UM objects (context, receivers, sources). I.e. options that control the UM library.
See XML Configuration Files for general information on XML-based LBM configuration files.
See also <lbm-config>.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
| application-name | Allows the user to select an "application name" for the Store Process, which can then used by the LBM XML configuration file to target a configuration to that Store using UM Element "<application>". See XML Application Names for more information on application names. | string | "umestored" |
Example:
Specifies a Group ID (GID) for daemon process (if run as root).
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
Contains the pathname for daemon process ID (PID) file.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
Specifies a User ID (UID) for daemon process (if run as root).
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
Contains the path name of the Store log file. See Store Rolling Logs for more information.
If omitted, log messages are written to standard output.
XML Attributes:
| Attribute | Description | Valid Values | Default Value |
|---|---|---|---|
| type | Where to write log messages. | "file" - Write log messages to a file. "console" - Write log messages to standard output. | "console" |
| frequency | Time-frame by which to roll the log file. | "disable" - Do not roll the log file based on time. "daily" - Roll the log file at midnight. "hourly" - Roll log file after approximately an hour, but is not exact and can drift significantly over a period of time. "test" - For internal Informatica use only. Do not use. | "disable" |
| size | Size (in MB, i.e. 2**20, or 1,048,576) of current log file at which it is rolled. Specify 0 to disable rolling by log file size. | string | "10" |
| xml:space | Specifies how whitespace (tabs, spaces, linefeeds) are handled in the element content. See xml:space Attribute. | "default" - Trim whitespace. "preserve" - Retain whitespace exactly as entered. | default |
Example:
The DTD for UM Store configuration has evolved over time:
| DTD Version | Release Date | Product Version | Supported Features |
|---|---|---|---|
| 1.0 | Feb. 2007 | UME 1.0 | Persistent Stores |
| 1.1 | April 2010 | UME 3.0.1 / UMQ 1.0 | Persistent Stores, Queues and Ultra Load Balancing (ULB) |
| 1.2 | March 2011 | UME 3.2 / UMQ 2.1 | Persistent Stores, Queues, Ultra Load Balancing (ULB), Dead Letter Queue, Indexed Queuing and Indexed ULB |
| 1.3 | November 2016 | UM 6.10 | Addition of '<xml-config>' element (under '<daemon>'). |
To assist the user with upgrades, a more-recent version of the Store will accept an older version of the Store configuration file. For example, if your Store configuration file starts with this:
the Store will parse the file according to DTD 1.0. If you wish to use Store configuration settings that were not available in 1.0, you will need to upgrade to a later DTD version. Contact Informatica Support if you have trouble.
Here is the current DTD version:
Store Process with one Store.