As mentioned in Persistent Store Concept, the Persistent Stores, also just called Stores, actually persist the source and receiver state and use RegIDs to identify sources and receivers. Each source to which a Store provides persistence may have zero or more receivers. The Store maintains each receiver's state along with the source's state and the messages the source has sent.

This document is oriented mostly to programmers. See also the Operations Guide chapters Running Persistent Stores (umestored), Persistent Store Crashed, Persistent Sending Problems, and UM Persistent Store Log Messages.

The Store can be configured with its own set of options to persist this state information on disk or simply in memory. The term disk Store is used to signify a Store that persists state to disk, and the term memory Store is used to signify a Store that persists state only in memory.

A source does not send data to the Store and then have the Store forward it to the receivers. In UM, the source sends to receivers and the Stores in parallel. See Persistence Normal Operation. Thus, UM can provide extremely low latency to receiving applications.

The Store(s) that a source uses are part of the source's configuration settings. Sources must be configured to use specific Store(s) in a Quorum/Consensus arrangement.

Receivers, on the other hand, do not need to be configured with Store information a priori. The source provides Store information to receivers via a Source Registration Information (SRI) message after the source registers with a Store. Thus the receivers learn about Stores from the source, without needing to be configured themselves. Because receivers learn about the Store or Stores with which they must register via a SRI record, the source must be available to receivers. However, the source does not have to be actively sending data to do this.

Limit Initial Restore with Restore-Last <-

The "restore-last" feature limits the initial message restore for a restarting Store.

When a disk-based store is restarted, during its initialization it will open the state and cache files and restore the data. This makes previously saved message data available for recovering subscribers. Note that this message restoration takes some time. For small files of a few megabytes, it might take a few seconds. But for large files of many gigabytes, it could take minutes.

Starting with UM version 6.15, you can configure a restarting Store to only restore a subset of the saved messages. This can greatly speed up the process of initialization.

For example, you might direct the Store to only restore 8 hours' worth of message data when it is restarted. Doing this means that older messages are not available from this Store for recovering subscribers. But if only one Store of a Q/C group is restarted, one or more of the other Stores will continue to have the older messages available.

The UMP Element "<restore-last>" is used to enable this feature.

When the restore-last feature is enabled, the Store will write an additional, per-source "cache index" file. It is written to the configured cache directory and is named the same as the source's normal message cache file with ".idx" appended. For example, your cache directory might contain:

3085235048-cache
3085235048-cache.idx

When the Store is restarted, the cache index file is used to quickly determine which messages are within the restore-last range. As a result of this algorithm, if a deployed Store is re-configured to enable the restore-last feature, the first time it is restarted will not be sped up. I.e. since the index file will not yet exist, the first startup after enabling the feature will restore the entire existing cache file.

See UMP Element "<restore-last>" for implementation details.

Store Log File <-

The Store Process generates log messages that are used to monitor its health and operation. You can configure these to be directed to "console" (standard output) or a specified log "file", via the UMP Element "<log>". Normally "console" is only used during testing, as a persistent log file is preferred for production use. The Store does not over-write log files on startup, but instead appends them.

Store Rolling Logs <-

To prevent unbounded disk file growth, the Store supports rolling log files. When the log file rolls, the file is renamed according to the model:
CONFIGUREDNAME_PID.DATE.SEQNUM
where:

CONFIGUREDNAME - Root name of log file, as configured by user.
PID - Process ID of the Store Process.
DATE - Date that the log file was rolled, in YYYY-MM-DD format.
SEQNUM - Sequence number, starting at 1 when the process starts, and incrementing each time the log file rolls.

For example: umestorelog_9867.2017-08-20.2

The user can configure when the log file is eligible to roll over by either or both of two criteria: size and frequency. The size criterion is in millions of bytes. The frequency criterion can be daily or hourly. Once one or both criteria are met, the next message written to the log will trigger a roll operation. These criteria are supplied as attributes to the UMP Element "<log>".

If both criteria are supplied, then the first one to be reached will trigger a roll. For example, consider the setting:

<log type="file" size="23" frequency="daily">store.log</log>

Let say that the log file grows at 1 million bytes per hour. At 11:00 pm, the log file will reach 23 million bytes, and will roll. Then, at 12:00 midnight, the log file will roll again, even though it is only 1 million bytes in size.

Note: The rolling logs cannot be configured to automatically overwrite old logs. Thus, the amount of disk space consumed by log files will grow without bound. The user must implement a desired process of archiving or deleting older log files according to the user's preference.

Quorum/Consensus Store Usage <-

To provide the highest degree of resiliency in the face of failures, UM provides the Quorum/Consensus failover strategy which allows a source to provide UM with a number of Stores to be used at the same time. Multiple Stores can fail and messaging can continue operation unhindered as long as a majority of configured Stores are operational.

Quorum/Consensus, also called QC, allows a source and the associated receivers to have their persisted state maintained at several Stores at the same time. Central to QC is the concept of a group of Stores, which is a logical grouping of Stores that are intended to signify a single entity of resilience. Within the group, individual Stores may fail but for the group as a whole to be viable and provide resiliency, a quorum must be available. In UM, a quorum is a simple majority. For example, in a group of five Stores, three Stores are required to maintain a quorum. One or two Stores may fail and the group continues to provide resiliency. UM requires a source to have a quorum of Stores available in the group in order to send messages. A group can consist of a single Store.

QC also provides the ability to use multiple groups. The use of multiple QC groups is a special case and should be discussed with Informatica support before using.

Sources Using Quorum/Consensus Store Configuration <-

In the case of Quorum/Consensus Store behavior, a message is considered stable after it has been successfully stored within a group of Stores or among groups of Stores according to the two settings, intergroup behavior and intragroup behavior, described below.

The intragroup behavior specifies the requirements needed to stabilize a message among the Stores within a group. A message is stable for the group once it is successfully stored at a quorum (majority) of the group's Stores or successfully stored in all the Stores in the group.
The intergroup behavior specifies the requirements needed to stabilize a message among groups of Stores. A message is stable among the groups if it is successfully stored at any group, a majority of groups, all groups, or all active groups.

Notice that a message needs to meet intragroup stability requirements before it can meet intergroup stability requirements. These options provide a number of possibilities for retention of messages for the source.