Release Notes

UM Version 6.10

The most significant updates to UM version 6.10 are related to improving performance (latency, jitter, and throughput).

Attention
  There are some special upgrade instructions for UM versions 6.10 and beyond that will affect a small percentage of users upgrading from pre-6.10 versions of UM. See Special Upgrade Instructions for 6.10. For general upgrade instructions, see Upgrade Procedure. You are also advised to inspect the full list of Deprecations.

# Enhancements for 6.10 <-

## Streaming Enhancements for 6.10 <-

The following new features and enhancements apply to UMS, UMP, and UMQ products.

- IPC performance improvement. Changed the receiver-side dequeuing algorithm to avoid semaphore operations when possible. Also see configuration option transport_lbtipc_pend_behavior_linger_loop_count (context).

- NCF reduction. Changed the LBT-RM reliability algorithm to reduce the number of NCF control messages sent during periods of heavy loss. The change reduces the impact of loss on healthy receivers, but might slightly increase the load on the source. See NAK Suppression for details.

- Host names in configuration files. For configuration options that take an IP address, it is now permissible to supply a host name. UM will resolve the name to an IP address when the configuration file is read.

- Non-contending Statistics retrieval. Statistics gathering API functions '`lbm_*_retrieve*stats()`' no longer contend with the message receive path (context thread). Retrieving statistics no longer introduces latency outliers in normal message flow. Note that a given sample of statistics no longer necessarily represents a single snapshot in time. For example, it is possible that between two samples, the field '`bytes_rcved`' could change but '`lbm_msgs_rcved`' does not. NOTE: brokered queuing contexts are not included in this enhancement.

- Smart Sources. A new, somewhat restrictive API for sending messages which eliminates looping, locking, and dynamic memory allocation/freeing, resulting in a significant reduction in send time and elimination of variance in execution time (jitter). Available for C and Java. See Smart Sources.

- Send path performance improvement. For UDP-based protocols (LBT-RM and LBT-RU), an algorithmic optimization was made to eliminate two memory allocations and frees per send call. This is for established sources (i.e. this optimization is not related to Smart Sources). Reduction of memory allocations and deallocations reduces send time, but more importantly reduces latency outliers (jitter).

- Send message direct to a source. Unicast Immediate Messages (UIM) have always allowed the use of IP address and UIM port (also known as "request port") of the destination application. Starting with UM version 6.10, a "source string" can also be used as the destination. This allows a receiving application to send a message directly to a newly subscribed source application. See the Concepts Guide section Sending to Sources.

- Default interface configuration option. This new configuration option simplifies the previously error-prone process of configuring interfaces for different features. If all or most features should use the same interface, it is no longer necessary to configure each interface option. Instead, specify the default_interface (context) once and then optionally specify any exceptions. (See below for affected configuration options.)

- Increased send-side socket buffer for TR. The send-side socket buffers for Topic Resolution has been increased to 1MB. This will reduce the possibility of send-side loss due to EWOULDBLOCK on socket sends.

- HTML documentation. Version 6.10 contains the first phase of a conversion of UM documentation to a different tool chain which produces manuals in 3 forms: multi-page HTML, single-page HTML, and PDF. Manuals converted in 6.10: "Concepts Guide", "Configuration Guide", "Operations Guide", and "Guide to Queuing". Additional manuals will be converted in a future release. In addition, Doxygen is now being used to produce .NET and Java API documentation.

- Context delay warning. A UM context keeps track of how long each loop of its context thread takes. Starting with UM version 6.10, it now logs a warning (`Core-5688-1882` for embedded mode, `Core-5688-1891` for sequential mode) if a loop exceeds 1500 milliseconds (1.5 s). This behavior can be modified by setting the environment variable `LBM_CONTEXT_DELAY_WARNING` to the number of milliseconds desired, or to zero to disable the warning. Note that if you see this warning, it represents a significant source of latency and should be investigated to determine the root cause.

- Jar file names changed. The following shipped Java jar file's names have changed to remove the JDK version number. In the table below, '`<vers>`' is the UM version number.

  Old jar name | New jar name
  --- | ---
  UMS_<vers>_jdk1.5.0_12.jar | UMS_<vers>.jar
  UMSPDM_<vers>_jdk1.5.0_12.jar | UMSPDM_<vers>.jar
  UMSSDM_<vers>_jdk1.5.0_12.jar | UMSSDM_<vers>.jar
  UMMAPI_1.5.0_12.jar | UMMAPI.jar
  UMMD_1.6.0_02.jar | UMMD.jar
  UMMGUI_1.6.0_02.jar | UMMGUI.jar

- The UMSSDM jar file is no longer included in the UMS jar file. Starting with UM 6.10, Java programs that use SDM need to explicitly include UMSSDM_<vers>.jar in the classpath.

- Windows .NET API is now included with base Windows package. It is no longer necessary to download and install .NET separately. The dynamic libraries and example executables are now contained under '`bin\dotnet`'.

- Windows package built with Visual Studio 2012. The UM packages for Windows are now being built with Visual Studio 2012 (VS 2012). The package is no longer shipped with the Windows Runtime library msvcr80.dll. Now UM ships with msvcr110.dll. User applications should be built with VS 2012 or beyond.

- Improved defaults. Many streaming and persistence configuration options have new default values. For systems already in production, please review the options and determine if the improved defaults are appropriate for your application. If not, then you must ensure that the desired values are explicitly configured. For details, see UM 6.10 Improved Defaults.

## Persistence Enhancements for 6.10 <-

The following new features and enhancements apply to UMP and UMQ products.

- Improved UM configuration defaults. Many persistence-related configuration options have new default values. For systems already in production, please review the options and determine if the improved defaults are appropriate for your application. If not, then you must ensure that the desired values are explicitly configured. See Special Upgrade Instructions for 6.10.

## Queuing Enhancements for 6.10 <-

The following new features and enhancements apply to the UMQ product.

- None.

## Dynamic Router Enhancements for 6.10 <-

The following new features and enhancements apply to the Dynamic Routing Option (DRO).

- Smart-Batch. New method of batching within the DRO which significantly increases throughput without increasing latency (as compared to flushing every message). See <smart-batch>.

# Fixed Problems and Limitations for 6.10 <-

## Streaming Fixed Problems and Limitations for 6.10 <-

The following bug fixes apply to UMS, UMP, and UMQ products.

Change Request | Description
--- | ---
10024 | FIXED: The API function lbm_transport_source_parse() incorrectly parsed source strings under a wide variety of circumstances.
9824 | FIXED: When a .NET program made use of ResolverEventCallback, it could result in a crash with the message, '`A callback was made on a garbage collected delegate...`'.
9710, 9788 | FIXED: The API functions lbm_rcv_topic_attr_dump(), lbm_src_topic_attr_dump(), and lbm_context_topic_attr_dump() generated log messages of the form: '`Core-5688-27 WARNING: ... config variable ... is deprecated. Use ... instead.`' One side-effect of this problem is that the Dynamic Router (DRO) logs these messages. This has been corrected since UM version 6.9.1.
9950 | FIXED: Corrected certain log messages generated due to Windows socket errors. Previously, the reason for the error was not printed correctly (always said error code was zero). Now it correctly prints the error code returned by Windows.
9877 | FIXED: If an LBT-RU source's context thread experiences a significant multi-second delay, due perhaps to scheduling delay on a severely overloaded server, and a receiver is configured to receive multiple datagrams via the multiple_receive_maximum_datagrams (context) option, there is a possibility that the receiver will crash with a fatal assert: '`[map != NULL]`'.
9939 | FIXED: Corrected memory alignment crash when using LZ4 compression on 32-bit SPARC machines.
9981, 9977, 9955 | FIXED: UM version 6.9.* have a problem which prevents the optional UM products UMDS, UMCache, and SNMP from working (mul_inet_ntoa).
9722, 9826 | FIXED: When the UMDS server component was executed on UMS or UMP library versions prior to 6.9, the server would log: '`CoreApi-5688-22 UMQ not supported`' Note that the server would function properly, so the log message could be ignored. In UM 6.10 the log is suppressed.
9957 | FIXED: When using Myricom's Datagram Bypass Library (DBL), a fatal assert ('`handle==dmux- >rcvmsock->sock`') and segmentation fault can occur if a receiver is deleted while the receiver is processing incoming datagrams.
9779 | Enhancement: Added configuration option transport_lbtru_nak_initial_backoff_interval (receiver). Setting this greater than zero reduces NAK traffic when UDP datagrams are delivered out of order. However, it also introduces repair latency when a datagram is lost.
4839, 9249, 9950 | Enhancement: Updated a number of error log messages to include the proper Windows return code.
9878 | Enhancement: In the Java API, if a receiver object is closed more than once, we silently ignore any closes after the first. (Previously an error condition was encountered.)
9677 | Enhancement: UM Windows package now built with Visual Studio 2012. The Windows runtime library msvcr110.dll is now included with UM.
9951 | Enhancement: print a log message when UM attempts to send a topic resolution message and the send to the socket fails. Most failures are serious and require attention to determine root cause.
9983 | Enhancement: include .NET libraries with the base Windows package for all products (UMS, UMP, UMQ). Downloading a separate .NET package is no longer necessary. The DLLs and example program binaries are now under '`bin\dotnet`'.
9926 | Enhancement: Topic Resolution send-side socket buffers are now set to 1MB. This should prevent "EWOULDBLOCK" errors and the associated loss of TR packets.
9967, 8556, 7793, 2548 | Enhancement: reduce the number of NCF control messages sent during periods of heavy loss. The change reduces the impact on healthy receivers, but might slightly increase the load on the source.

## Persistence Fixed Problems and Limitations for 6.10 <-

The following bug fixes apply to UMP and UMQ products.

Change Request | Description
--- | ---
9953 | FIXED: Due to an incorrect print specification, 32-bit applications can occasionally crash with illegal memory access when debug logging is enabled and a persistent session ID is being printed. The print specification has been corrected.
9583 | FIXED: When a source requests registration with a store, if is possible for the store's response control message (PREG_RESP) to be lost. The source will re-try the registration, which the store will reject with the log, "RegID xxx is in use by a different source". This has been corrected (a store allows a registered source to re-register).
9920, 9929 | FIXED: Under unknown circumstances, a store can exit with inconsistent data in the state and/or cache files. When the store is restarted, this inconsistent data can result in a store crash during initialization. Defensive checks have been added.
9861 | FIXED: The UMM tool tip for the source configuration option ume_proactive_keepalive_interval (context) was incorrect. Also, if zero was supplied, CPU utilization jumped to 100%. It is fixed such that zero disables proactive keepalives.
9902, 9925 | FIXED: The Store's log file sometimes shows session IDs as negative numbers. These have been corrected to use unsigned.
9458, 9796 | FIXED: A minor race condition in the Store which results in one or more warning logs: '`WARNING: failed assertion [repo->disk_holder->disk->offset >= deletesz] at line ...`'
9938, 9943 | Enhancement: If a 5.x persistent source is configured for named stores and is also configured not to query for the store name (via the environment variable '`UME_SKIP_NAMED_STORE_QUERY`'), and a 6.x store becomes non-responsive due to excessive load, and then returns to normal operation, the source will not be able to resolve the store again. This is because a 6.x store cannot be configured to periodically advertise its name indefinitely. In UM version 6.10, the store can be configured to periodically advertise indefinitely via the UM configuration option compatibility_include_pre_um_6_0_behavior (context). Note that this option is only recommended for use during a limited transition period while customers upgrade from 5.x to 6.x and persistent stores must be configured not to query. When all components are at version 6.x, the use of this configuration option should be discontinued.
9993 | Enhancement: Proactive Keepalive feature modified to reduce disk activity due to page faults during periods of low message traffic.
9689 | Enhancement: Added Store configuration element <xml-config> to allow the use of XML-based UM configuration files. With this addition, Informatica discourages the use of the environment variable '`LBM_XML_CONFIG_FILENAME`'.

## Queuing Fixed Problems and Limitations for 6.10 <-

The following bug fixes apply to the UMQ product.

- None.

## Dynamic Router Fixed Problems and Limitations for 6.10 <-

The following bug fixes apply to the Dynamic Routing Option (DRO).

Change Request | Description
--- | ---
9710, 9788 | The DRO generated log messages of the form, `Core-5688-27 WARNING: ... config variable ... is deprecated. Use ... instead.` This was due to a problem in the underlying UM library.
9810 | A variety of timing-related issues with single-TCP peer links have been resolved.
9800 | If a DRO has a proxy source with queued messages and the proxy source is deleted, the messages get stuck in the queue, which subsequently has less room for future messages. In the worst case, the queue can be stuck in a full state, preventing message forwarding. Proxy sources are deleted as a result of the originating source being deleted, or as a result of all receivers in a remote TRD being deleted, eliminating interest in the topic in that TRD.
9799 | If a DRO has a proxy source with queued messages, there is a small timing window during message flow that can cause messages to get stuck in the queue, which subsequently has less room for future messages. In the worst case, the queue can be stuck in a full state, preventing message forwarding. (Similar to 9800, but does not require deletion of proxy source.)
9916 | If a debug log is being generated, a garbage value is printed in some of the logged messages, making it more difficult for Informatica Support to diagnose problems.

# Special Upgrade Instructions for 6.10 <-

## DRO (Dual) TCP Peer Link <-

UM version 6.10 no longer supports "Dual TCP" peer links ("<tcp>"). Upgrading customers should use <single-tcp> instead.

## UM 6.10 Improved Defaults <-

The following configuration options have had their default values changed. The intention is for these values to be generally better for users, so you may benefit from them. However, some users may have tuned their applications and/or environments to conform to the old defaults. You should examine these options and compare them to values that you may have set, paying special attention to options that you do not set, since those may result in behavior changes in your system. Contact UM Support for assistance.

Streaming Defaults

Note that these affect streaming, persistence, and queuing use cases.

Option | Old default | New default
--- | --- | ---
multiple_receive_maximum_datagrams (context) | 1 | 0
resolver_multicast_interface (context) | INADDR_ANY | Value of default_interface (context)
resolver_unicast_interface (context) | INADDR_ANY | Value of default_interface (context)
request_tcp_interface (context) | INADDR_ANY | Value of default_interface (context)
response_tcp_interface (context) | INADDR_ANY | Value of default_interface (context)
transport_tcp_interface (source) | INADDR_ANY | Value of default_interface (context)
transport_tcp_interface (receiver) | INADDR_ANY | Value of default_interface (context)
transport_lbtru_interface (source) | INADDR_ANY | Value of default_interface (context)
transport_lbtru_interface (receiver) | INADDR_ANY | Value of default_interface (context)
resolver_multicast_receiver_socket_buffer (context) | 0 (OS default) | 8388608 (8MB) Increase in socket buffers reduces loss and latency outliers for UDP-based transports with ordered delivery.
resolver_unicast_receiver_socket_buffer (context) | 0 (OS default) | 8388608 (8MB)
transport_lbtrm_receiver_socket_buffer (context) | 524288 (512KB) | 8388608 (8MB)
transport_lbtrm_source_socket_buffer (context) | 0 (OS default) | 1048576 (1MB)
transport_lbtru_receiver_socket_buffer (context) | 524288 (512KB) | 8388608 (8MB)
transport_lbtru_source_socket_buffer (context) | 0 (OS default) | 1048576 (1MB)
transport_tcp_sender_socket_buffer (source) | Windows 128000 (125KB) | 0 (OS default) The value zero suppresses the setting of the socket buffer, which allows the OS to make use of auto-tuning.
response_session_sender_socket_buffer (context) | Windows 128000 (125KB) | 0 (OS default) The value zero suppresses the setting of the socket buffer, which allows the OS to make use of auto-tuning.
transport_lbtrm_rate_interval (context) | 100 ms (.1 sec) | 10 ms (.01 sec) Reduction of rate interval can improve network stability and prevent loss, but can also increase latency outliers.
otr_request_initial_delay (receiver) | 10 sec | 2 sec
otr_request_message_timeout (receiver) | 20 sec | 60 sec

Persistent Defaults

Note that these affect persistence and queuing use cases.

Option | Old default | New default
--- | --- | ---
ume_use_ack_batching (receiver) | 0 (no) | 1 (yes) Batching of receiver consumption acknowledgements greatly increases system throughput, although it also increases the likelihood of duplicate messages during a recovery operation.
ume_confirmed_delivery_notification (source) | 1 (yes) | 0 (no) Increases system throughput.
ume_message_stability_timeout (source) | 20,000 (20 sec) | 5,000 (5 sec) Should be less than NAK generation interval (for UDP-based protocols).

Improved Store configuration defaults

Option | Old default | New default
--- | --- | ---
repository-size-threshold | 25,165,824 (24MB) | 1,024 (1K)

## Previous Special Instructions <-

If you are upgrading from a UM version prior to 6.8, you must also examine the Special Upgrade Instructions for 6.8.

Also see Deprecations.