Configuration Guide
Transport Acceleration Options

Transport acceleration options enable kernel-bypass acceleration in conjunction with the following vendor solutions:


Myricom Datagram Bypass Layer (DBL)  <-

DBL is a kernel-bypass technology that accelerates sending and receiving UDP traffic and operates with DBL-enabled Myricom 10-Gigabit Ethernet adapter cards for Linux and Microsoft Windows.

DBL does not support fragmentation and reassembly, so do not send messages larger than the MTU size configured on the DBL interface.

DBL acceleration is compatible with the following Ultra Messaging transport types:

  • LBT-RM (UDP-based reliable multicast)
  • LBT-RU (UDP-based reliable unicast)
  • Multicast Immediate Messaging
  • Multicast Topic Resolution

To use DBL Transport Acceleration, perform the following steps:

  1. Install the Myricom 10-Gigabit Ethernet NIC.
  2. Install the DBL shared library.
  3. Update your search path to include the location of the DBL shared library.
  4. Set option transport_*_datagram_max_size and option resolver_datagram_max_size (context) to a value of no more than 28 bytes smaller than the Myricom interface's configured MTU size.

Users of DBL are advised to make use of Dynamic Fragmentation Reduction.


Reference  <-


dbl_lbtrm_acceleration (context)  <-

Flag indicating if DBL acceleration is enabled for LBT-RM transports.
See Myricom Datagram Bypass Layer (DBL).
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 4.0.

Value Description
1

DBL acceleration is enabled for LBT-RM.

0 DBL acceleration is not enabled for LBT-RM. Default for all.


dbl_lbtru_acceleration (context)  <-

Flag indicating if DBL acceleration is enabled for LBT-RU transports.
See Myricom Datagram Bypass Layer (DBL).
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 4.0.

Value Description
1

DBL acceleration is enabled for LBT-RU.

0 DBL acceleration is not enabled for LBT-RU. Default for all.


dbl_mim_acceleration (context)  <-

Flag indicating if DBL acceleration is enabled for MIM.
See Myricom Datagram Bypass Layer (DBL).
See Multicast Immediate Messaging for general information about MIM.
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 4.0.

Value Description
1

DBL acceleration is enabled for MIM.

0 DBL acceleration is not enabled for MIM. Default for all.


dbl_resolver_acceleration (context)  <-

Flag indicating if DBL acceleration is enabled for topic resolution.
See Myricom Datagram Bypass Layer (DBL).
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 4.0.

Value Description
1

DBL acceleration is enabled for topic resolution.

0 DBL acceleration is not enabled for topic resolution. Default for all.


Solarflare Onload  <-

Solarflare Onload is a kernel-bypass technology that accelerates message traffic and operates with Solarflare 10GbE Ethernet NICs.

Note
Onload does not support fragmentation and reassembly, so do not send messages larger than the MTU size configured on the Solarflare interface.
Warning
Onload does not support both accelerated and non-accelerated processes subscribing to the same multicast group on the same host. An attempt to do so will result in the non-accelerated process becoming "deaf" to the shared multicast group. See "Onload User Guide", chapter 9.11 "Multicast Operation and Stack Sharing", sub-section "Multicast Receive - Onload Stack and Kernel Stack".

Ultra Messaging loads the Onload library dynamically during Ultra Messaging initialization on Linux-based (X86) platforms.

Onload default behavior accelerates all sockets. You can access the Onload onload_set_stackname API extension to select the sockets you want to accelerate by using UM configuration options. Selecting sockets with a stackname lets you accelerate data transmission sockets and not sockets for control messages, topic resolution, or responses.

You can select a stackname with the configuration options onload_acceleration_stack_name (receiver) and onload_acceleration_stack_name (source) for the following Ultra Messaging transport types:

  • LBT-RM (UDP-based reliable multicast)
  • LBT-RU (UDP-based reliable unicast)
  • TCP
Note
If you set the LBM_SUPPRESS_ONLOAD environment variable to any value, Ultra Messaging does not dynamically load the Onload library at runtime. In this case, you cannot use the onload_acceleration_stack_name options.

If you use the onload_set_stackname API directly for any other accelerated sockets, note that after Ultra Messaging accelerates a transport socket, Ultra Messaging resets the stackname to the default for all threads by calling:

onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_NOCHANGE, "");

Ultra Messaging resets the stackname during source creation and when a receiver matched topic opens a transport session.

To enable Onload socket acceleration for selected transports, perform the following steps:

  1. Install Onload.
  2. Set the Onload environment variable EF_DONT_ACCELERATE = 1 to disable Onload default behavior.

  3. To enable acceleration for all applications in an environment, export the following environment variable:

    export LD_PRELOAD=libonload.so

  4. To enable acceleration on a per-application basis, start the application as in the following example:

    onload <app_name> [app_options]

  5. Set UM configuration option onload_acceleration_stack_name (source) according to the thread the source uses.

    Note: Disable batching to ensure that it is the application thread that sends the data out.

  6. Set UM configuration option onload_acceleration_stack_name (receiver) according to the thread the receiver uses.

    Note: Receiver transports might not share the same thread if MTT is enabled.

  7. Set option transport_*_datagram_max_size and option resolver_datagram_max_size (context) to a value of no more than 28 bytes smaller than the Solarflare interface's configured MTU size.

Users of Onload are advised to make use of Dynamic Fragmentation Reduction.

For detailed information about onload stack names, refer to the Solarflare Onload User Guide.


Solarflare Tips  <-

Solarflare NICs can be used with in-kernel device drivers, or with the kernel-bypass driver "Onload".

Sometimes the kernel driver is less susceptible to packet loss than Onload. This is because the kernel driver is invoked as needed via interrupt, while Onload is dependent on the application to call the driver via a socket call (like "epoll()"). With the kernel driver, if the application is delayed in making a socket call, the socket buffer can be configured to hold large numbers of datagrams. In contrast, if using Onload, only the NICs rx ring buffer is available. Even short delays can lead to packet loss with Onload.

Informatica is not an expert in tuning Solarflare NICs and Onload. We recommend using the Onload documentation and discussing your use case with Onload support engineers.

However, we can give a few tips based on our own experience and that of our customers.

  1. The number of receive descriptors (size of rx ring buffer) should always be set to the maximum value (probably 4096, but check to be sure).
    • For kernel driver, check the current and maximum settings with:
      ethtool -g sfdevicename
      You can change it using ethtool:
      ethtool -G rx 4096 sfdevicename
      but this will only stay in effect until the next reboot. Different versions of Linux have different methods for making the changes permanent.
    • For Onload, set the environment variable:
      export EF_RXQ_SIZE=4096
  2. Set the maximum size of datagram that UM will generate.
  3. If using Onload, you can get better performance if you configure sources on one stack using onload_acceleration_stack_name (source) and receivers on a different stack using onload_acceleration_stack_name (receiver).
  4. For a wealth of additional information, see Onload documentation, especially Tuning Onload and Eliminating Drops.


Reference  <-


onload_acceleration_stack_name (receiver)  <-

The stackname to use when creating an OpenOnload transport data socket.
The stackname must be eight characters or less. Because this is a transport setting, the first receiver applies its configuration option setting, and other receivers that join the transport inherit the setting of the first source. To disable the stackname, set this option to NULL, which must be all uppercase.
Note: Use of this option requires Solarflare OpenOnload and applies to LBT-RM, LBT-RU, and TCP transports.
Scope: receiver
Type: string
Default value: NULL
When to Set: Can only be set during object initialization.
Version: This option was implemented in UM 6.5.


onload_acceleration_stack_name (source)  <-

The stackname to use when creating an OpenOnload transport data socket.
The stackname must be eight characters or less. Because this is a transport setting, the first source applies its configuration option setting, and other sources that join the transport inherit the setting of the first source. To disable the stackname, set this option to NULL, which must be all uppercase.
Note: Use of this option requires Solarflare OpenOnload and applies to LBT-RM, LBT-RU, and TCP transports.
Scope: source
Type: string
Default value: NULL
When to Set: Can only be set during object initialization.
Version: This option was implemented in UM 6.5.


UD Acceleration for Mellanox Hardware Interfaces  <-

UD (Unreliable Datagram) acceleration is a kernel-bypass technology that accelerates sending and receiving UDP traffic and operates with Mellanox 10-Gigabit Ethernet or InfiniBand adapter cards for 64-bit Linux on X86 platforms.

UD acceleration does not support fragmentation and reassembly, so do not send messages larger than the MTU size configured on the Mellanox interface.

UD acceleration is available for the following Ultra Messaging transport types:

  • LBT-RM (UDP-based reliable multicast)
  • LBT-RU (UDP-based reliable unicast)
  • Multicast Immediate Messaging
  • Multicast Topic Resolution

To use UD acceleration, perform the following steps:

  1. Install the Mellanox NIC.
  2. Install the VMA package, which is part of the UD acceleration option .
  3. Include the appropriate transport acceleration options in your Ultra Messaging Configuration File.
  4. Set option transport_*_datagram_max_size and option resolver_datagram_max_size (context) to a value of no more than 28 bytes smaller than the Mellanox interface's configured MTU size.

Users of UD acceleration are advised to make use of Dynamic Fragmentation Reduction.


Reference  <-


resolver_ud_acceleration (context)  <-

Flag indicating if Accelerated Multicast is enabled for Topic Resolution. Accelerated Multicast requires Mellanox InfiniBand or 10 Gigabit Ethernet hardware.
UD Acceleration of topic resolution relies on hardware-supported loopback, which InfiniBand provides, but which the 10 Gigabit Ethernet ConnectX hardware does not.
Note: If 10 Gigabit Ethernet ConnectX hardware is used and multiple UM contexts are desired on the host, this option must be disabled.
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 5.2.

Value Description
1

Accelerated Topic Resolution is enabled.

0 Accelerated Topic Resolution is not enabled. Default for all.


ud_acceleration (context)  <-

Flag indicating if Accelerated Multicast is enabled for LBT-RM.
Accelerated Multicast requires InfiniBand or 10 Gigabit Ethernet hardware and the purchase and installation of the Ultra Messaging Accelerated Multicast Module. See your Ultra Messaging representative for licensing specifics.
Scope: context
Type: int
When to Set: Can only be set during object initialization.
Version: This option was implemented in LBM 4.1.

Value Description
1

Accelerated Multicast is enabled.

0 Accelerated Multicast is not enabled. Default for all.