Concepts Guide
Transport Types


Transport TCP  <-

The TCP UM transport uses normal TCP connections to send messages from sources to receivers. This is the default transport when it's not explicitly set. TCP is a good choice when:

  • Flow control is desired. For example, when one or more receivers cannot keep up, you wish to slow down the source. This is a "better late than never" philosophy.
  • Equal bandwidth sharing with other TCP traffic is desired. I.e. when it is desired that the source slow down when general network traffic becomes heavy.
  • There are few receivers listening to each topic. Multiple receivers for a topic requires multiple transmissions of each message, which places a scaling burden on the source machine and the network.
  • The application is not sensitive to latency. Use of TCP as a messaging transport can result in unbounded latency.
  • The messages must pass through a restrictive firewall which does not pass multicast traffic.

UM's TCP transport includes a Session ID. A UM source using the TCP transport generates a unique, 32-bit non-zero random Session ID for each TCP transport (IP:port) it uses. The source also includes the Session ID in its Topic Resolution advertisement (TIR). When a receiver resolves its topic and discovers the transport information, the receiver also obtains the transport's Session ID. The receiver sends a message to the source to confirm the Session ID.

The TCP Session ID enables multiple receivers for a topic to connect to a source across a UM Router. In the event of a UM Router failure, UM establishes new topic routes which can cause cached Topic Resolution and transport information to be outdated. Receivers use this cached information to find sources. Session IDs add a unique identifier to the cached transport information. If a receiver tries to connect to a source with outdated transport information, the source recognizes an incorrect Session ID and disconnects the receiver. The receiver can then attempt to reconnect with different cached transport information.

Note
To maintain interoperability between version pre-6.0 receivers and version 6.0 and beyond TCP sources, you can turn off TCP Session IDs with the UM configuration option, transport_tcp_use_session_id (source).


Transport LBT-RU  <-

The LBT-RU UM transport adds reliable delivery to unicast UDP to send messages from sources to receivers. This provides greater flexibility in the control of latency. For example, the application can further limit latency by allowing the use of arrival order delivery. See the Knowledge Base article, FAQ: How do arrival-order delivery and in-order delivery affect latency?. Also, LBT-RU is less sensitive to overall network load; it uses source rate controls to limit its maximum send rate.

Since it is based on unicast addressing, LBT-RU can pass through most firewalls. However, it has the same scaling issues as TCP when multiple receivers are present for each topic.

UM's LBT-RU transport includes a Session ID. A UM source using the LBT-RU transport generates a unique, 32-bit non-zero random Session ID for each transport it uses. The source also includes the Session ID in its Topic Resolution advertisement (TIR). When a receiver resolves its topic and discovers the transport information, the receiver also obtains the transport's Session ID.

The LBT-RU Session ID enables multiple receivers for a topic to connect to a source across a UM Router. In the event of a UM Router failure, UM establishes new topic routes which can cause cached Topic Resolution and transport information to be outdated. Receivers use this cached information to find sources. Session IDs add a unique identifier to the cached transport information. If a receiver tries to connect to a source with outdated transport information, the transport drops the received data and times out. The receiver can then attempt to reconnect with different cached transport information.

Note
To maintain interoperability between version pre-3.3 receivers and version 3.3 and beyond LBT-RU sources, you can turn off LBT-RU Session IDs with the UM configuration option, transport_lbtru_use_session_id (source).
LBT-RU can benefit from hardware acceleration. See Transport Acceleration Options for more information.


Transport LBT-RM  <-

The LBT-RM transport adds reliable multicast to UDP to send messages. This provides the maximum flexibility in the control of latency. In addition, LBT-RM can scale effectively to large numbers of receivers per topic using network hardware to duplicate messages only when necessary at wire speed. One limitation is that multicast is often blocked by firewalls.

LBT-RM is a UDP-based, reliable multicast protocol designed with the use of UM and its target applications specifically in mind. The protocol is very similar to PGM, but with changes to aid low latency messaging applications.

  • Topic Mapping - Several topics may map onto the same LBT-RM session. Thus a multiplexing mechanism to LBT-RM is used to distinguish topic level concerns from LBT-RM session level concerns (such as retransmissions, etc.). Each message to a topic is given a sequence number in addition to the sequence number used at the LBT-RM session level for packet retransmission.
  • Negative Acknowledgments (NAKs) - LBT-RM uses NAKs as PGM does. NAKs are unicast to the sender. For simplicity, LBT-RM uses a similar NAK state management approach as PGM specifies.
  • Time Bounded Recovery - LBT-RM allows receivers to specify a maximum time to wait for a lost piece of data to be retransmitted. This allows a recovery time bound to be placed on data that has a definite lifetime of usefulness. If this time limit is exceeded and no retransmission has been seen, then the piece of data is marked as an unrecoverable loss and the application is informed. The data stream may continue and the unrecoverable loss will be ordered as a discrete event in the data stream just as a normal piece of data.
  • Flexible Delivery Ordering - LBT-RM receivers have the option to have the data for an individual topic delivered "in order" or "arrival order". Messages delivered "in order" will arrive in sequence number order to the application. Thus loss may delay messages from being delivered until the loss is recovered or unrecoverable loss is determined. With "arrival-order" delivery, messages will arrive at the application as they are received by the LBT-RM session. Duplicates are ignored and lost messages will have the same recovery methods applied, but the ordering may not be preserved. Delivery order is a topic level concern. Thus loss of messages in one topic will not interfere or delay delivery of messages in another topic.
  • Session State Advertisements - In PGM, SPM packets are used to advertise session state and to perform PGM router assist in the routers. For LBT-RM, these advertisements are only used when data are not flowing. Once data stops on a session, advertisements are sent with an exponential back-off (to a configurable maximum interval) so that the bandwidth taken up by the session is minimal.
  • Sender Rate Control - LBT-RM can control a sender's rate of injection of data into the network by use of a rate limiter. This rate is configurable and will back pressure the sender, not allowing the application to exceed the rate limit it has specified. In addition, LBT-RM senders have control over the rate of retransmissions separately from new data. This allows sending application to guarantee a minimum transmission rate even in the face of massive loss at some or all receivers.
  • Low Latency Retransmissions - LBT-RM senders do not mandate the use of NCF packets as PGM does. Because low latency retransmissions is such an important feature, LBT-RM senders by default send retransmissions immediately upon receiving a NAK. After sending a retransmission, the sender ignores additional NAKs for the same data and does not repeatedly send NCFs. The oldest data being requested in NAKs has priority over newer data so that if retransmissions are rate controlled, then LBT-RM sends the most important retransmissions as fast as possible.

UM's LBT-RM transport includes a Session ID. A UM source using the LBT-RM transport generates a unique, 32-bit non-zero random Session ID for each transport it uses. The source also includes the Session ID in its Topic Resolution advertisement (TIR). When a receiver resolves its topic and discovers the transport information, the receiver also obtains the transport's Session ID.

Note
LBT-RM can benefit from hardware acceleration. See Transport Acceleration Options for more information.


Transport LBT-IPC  <-

The LBT-IPC transport is an Interprocess Communication (IPC) UM transport that allows sources to publish topic messages to a shared memory area managed as a static ring buffer from which receivers can read topic messages. Message exchange takes place at memory access speed which can greatly improve throughput when sources and receivers can reside on the same host. LBT-IPC can be either source-paced or receiver-paced.

The LBT-IPC transport uses a "lock free" design that eliminates calls to the Operating System and allows receivers quicker access to messages. An internal validation method enacted by receivers while reading messages from the Shared Memory Area ensures message data integrity. The validation method compares IPC header information at different times to ensure consistent, and therefore, valid message data. Sources can send individual messages or a batch of messages, each of which possesses an IPC header.


LBT-IPC Shared Memory Area  <-

The following diagram illustrates the Shared Memory Area used for LBT-IPC:

IPC_Shared_Memory_Layout.png

Header

The Header contains information about the shared memory area resource.

  • Shared Lock - shared receiver pool semaphore (mutex on Microsoft Windows) to ensure mutually exclusive access to the receiver pool.
  • Version - LBT-IPC version number which is independent of any UM product version number.
  • Buffer Length - size of shared memory area.
  • Receiver Map Size - Number of entries available in the Receiver Pool which you configure with the source option, transport_lbtipc_maximum_receivers_per_transport (source).
  • New Client Flag - set by the receiver after setting its Receiver Pool entry and before releasing the Shared Lock. Indicates to the source that a new receiver has joined the transport.
  • Receiver Paced - Indicates if you've configured the transport for receiver-pacing.
  • Old Message Start - pointer indicating messages that may be reclaimed.
  • New Message Start - pointer indicating messages that may be read.
  • New Message End - pointer indicating the end of messages that may be read, which may not be the same as the Old Message Start pointer.

Receiver Pool

The receiver pool is a collection of receiver connections maintained in the Shared Memory Area. The source reads this information if you've configured receiver-pacing to determine if a message can be reclaimed or to monitor a receiver. Each receiver is responsible for finding a free entry in the pool and marking it as used.

  • In Use flag - set by receiver while holding the Shared Lock, which effectively indicates the receiver has joined the Transport Session. Using the Shared Lock ensures mutually exclusive access to the receiver connection pool.
  • Oldest Message Start - set by receiver after reading a message. If you enable receiver-pacing the source reads it to determine if message memory can be reclaimed.
  • Monitor Shared Lock - checked by the source to monitor a receiver (semaphore on Linux, event on Microsoft Windows).
  • Signal Shared Lock - Set by source to notify receiver that new data has been written. (semaphore on Linux, mutex on Microsoft Windows) If you set transport_lbtipc_receiver_thread_behavior (context) to busy_wait, the receiver sets this semaphore to zero and the source does not notify.

Source-to-Receiver Message Buffer

This area contains message data. You specify the size of the shared memory area with a source option, transport_lbtipc_transmission_window_size (source). The size of the shared memory area cannot exceed your platform's shared memory area maximum size. UM stores the memory size in the shared memory area's header. The Old Message Start and New Message Start point to positions in this buffer.


Sources and LBT-IPC  <-

When you create a source with lbm_src_create() and you've set the transport option to IPC, UM creates a shared memory area object. UM assigns one of the transport IDs to this area specified with the UM context configuration options, transport_lbtipc_id_high (context) and transport_lbtipc_id_low (context). You can also specify a shared memory location outside of this range with a source configuration option, transport_lbtipc_id (source), to prioritize certain topics, if needed.

UM names the shared memory area object according to the format, LBTIPC_x_d where x is the hexadecimal Session ID and d is the decimal Transport ID. Example names are LBTIPC_42792ac_20000 or LBTIPC_66e7c8f6_20001. Receivers access a shared memory area with this object name to receive (read) topic messages.

Using the configuration option, transport_lbtipc_behavior (source), you can choose source-paced or receiver-paced message transport. See Transport LBT-IPC Operation Options for more information.

Sending over LBT-IPC

To send on a topic (write to the shared memory area) the source writes to the Shared Memory Area starting at the Oldest Message Start position. It then increments each receiver's Signal Lock if the receiver has not set this to zero.


Receivers and LBT-IPC  <-

Receivers operate identically to receivers for all other UM transports. A receiver can actually receive topic messages from a source sending on its topic over TCP, LBT-RU or LBT-RM and from a second source sending on LBT-IPC with out any special configuration. The receiver learns what it needs to join the LBT-IPC session through the topic advertisement.

The configuration option transport_lbtipc_receiver_thread_behavior (context) controls the IPC receiving thread behavior when there are no messages available. The default behavior, 'pend', has the receiving thread pend on a semaphore for a new message. When the source adds a message, it posts to each pending receiver's semaphore to wake the receiving thread up. Alternatively, 'busy_wait' can be used to prevent the receiving thread going to sleep. In this case, the source does not need to post to the receiver's semaphore. It simply adds the message to shared memory, which the looping receiving thread detects with the lowest possible latency.

Although 'busy_wait' has the lowest latency, it has the drawback of consuming 100% of a CPU core during periods of idleness. This limits the number of IPC data flows that can be used on a given machine to the number of available cores. (If more busy looping receivers are deployed than there are cores, then receivers can suffer 10 millisecond time sharing quantum latencies.)

For application that cannot afford 'busy_wait', there is another configuration option, transport_lbtipc_pend_behavior_linger_loop_count (context), which allows a middle ground between 'pend' and 'busy_wait'. The receiver is still be configured as 'pend', but instead of going to sleep on the semaphore immediately upon emptying the shared memory, it busy loops for the configured number of times. If a new message arrives, it processes the message immediately without a sleep/wakeup. This can be very useful during bursts of high incoming message rates to reduce latency. By making the loop count large enough to cover the incoming message interval during a burst, only the first message of the burst will incur the wakeup latency.

Topic Resolution and LBT-IPC

Topic resolution operates identically with LBT-IPC as other UM transports albeit with a new advertisement type, LBMIPC. Advertisements for LBT-IPC contain the Transport ID, Session ID and Host ID. Receivers obtain LBT-IPC advertisements in the normal manner (resolver cache, advertisements received on the multicast resolver address:port and responses to queries.) Advertisements for topics from LBT-IPC sources can reach receivers on different machines if they use the same topic resolution configuration, however, those receivers silently ignore those advertisements since they cannot join the IPC transport. See Sending to Both Local and Remote Receivers.

Receiver Pacing

Although receiver pacing is a source behavior option, some different things must happen on the receiving side to ensure that a source does not reclaim (overwrite) a message until all receivers have read it. When you use the default transport_lbtipc_behavior (source) (source-paced), each receiver's Oldest Message Start position in the Shared Memory Area is private to each receiver. The source writes to the Shared Memory Area independently of receivers' reading. For receiver-pacing, however, all receivers share their Oldest Message Start position with the source. The source will not reclaim a message until all receivers have successfully read that message.

Receiver Monitoring

To ensure that a source does not wait on a receiver that is not running, the source monitors a receiver via the Monitor Shared Lock allocated to each receiving context. (This lock is in addition to the semaphore already allocated for signaling new data.) A new receiver takes and holds the Monitor Shared Lock and releases the resource when it dies. If the source is able to obtain the resource, it knows the receiver has died. The source then clears the receiver's In Use flag in it's Receiver Pool Connection.


Similarities with Other UM Transports  <-

Although no actual network transport occurs, IPC functions in much the same way as if you send packets across the network as with other UM transports.

  • If you use a range of LBT-IPC transport IDs, UM assigns multiple topics sent by multiple sources to all the Transport Sessions in a round robin manner just like other UM transports.
  • Transport sessions assume the configuration option values of the first source assigned to the Transport Session.
  • Sources are subject to message batching.


Differences from Other UM Transports  <-

  • Unlike LBT-RM which uses a transmission window to specify a buffer size to retain messages in case they must be retransmitted, LBT-IPC uses the transmission window option to establish the size of the shared memory.
  • LBT-IPC does not retransmit messages. Since LBT-IPC transport is essentially a memory write/read operation, messages should not be be lost in transit. However, if the shared memory area fills up, new messages overwrite old messages and the loss is absolute. No retransmission of old messages that have been overwritten occurs.
  • Receivers also do not send NAKs when using LBT-IPC.
  • LBT-IPC does not support ordered_delivery (receiver) options. However, if you set ordered_delivery (receiver) 1 or -1, LBT-IPC reassembles any large messages.
  • LBT-IPC does not support Rate Control.
  • LBT-IPC creates a separate receiver thread in the receiving context.


Sending to Both Local and Remote Receivers  <-

A source application that wants to support both local and remote receivers should create two UM Contexts with different topic resolution configurations, one for IPC sends and one for sends to remote receivers. Separate contexts allows you to use the same topic for both IPC and network sources. If you simply created two source objects (one IPC, one say LBT-RM) in the same UM Context, you would have to use separate topics and suffer possible higher latency because the sending thread would be blocked for the duration of two send calls.

A UM source will never automatically use IPC when the receivers are local and a network transport for remote receivers because the discovery of a remote receiver would hurt the performance of local receivers. An application that wants transparent switching can implement it in a simple wrapper.


LBT-IPC Configuration Example  <-

The following diagram illustrates how sources and receivers interact with the shared memory area used in the LBT-IPC transport:

IPC_Objects.png

In the diagram above, 3 sources send (write) to two Shared Memory Areas while four receivers in two different contexts receive (read) from the areas. The assignment of sources to Shared Memory Areas demonstrate UM's round robin method. UM assigns the source sending on Topic A to Transport 20001, the source sending on Topic B to Transport 20002 and the source sending on Topic C back to the top of the transport ID range, 20001.

The diagram also shows the UM configuration options that set up this scenario:


Required privileges  <-

LBT-IPC requires no special operating system authorities, except on Microsoft Windows Vista and Microsoft Windows Server 2008, which require Administrator privileges. In addition, on Microsoft Windows XP, applications must be started by the same user, however, the user is not required to have administrator privileges. In order for applications to communicate with a service, the service must use a user account that has Administrator privileges.


Host Resource Usage and Limits  <-

LBT-IPC contexts and sources consume host resources as follows:

  • Per Source - 1 shared memory segment, 1 shared lock (semaphore on Linux, mutex on Microsoft Windows)
  • Per Receiving Context - 2 shared locks (semaphores on Linux, one event and one mutex on Microsoft Windows)

Across most operating system platforms, these resources have the following limits.

  • 4096 shared memory segments, though some platforms use different limits
  • 32,000 shared semaphores (128 shared semaphore sets * 250 semaphores per set)

Consult your operating system documentation for specific limits per type of resource. Resources may be displayed and reclaimed using the LBT-IPC Resource Manager. See also the KB article Managing LBT-IPC Host Resources.


LBT-IPC Resource Manager  <-

Deleting an IPC source or deleting an IPC receiver reclaims the shared memory area and locks allocated by the IPC source or receiver. However, if a less than graceful exit from a process occurs, global resources remain allocated but unused. To address this possibility, the LBT-IPC Resource Manager maintains a resource allocation database with a record for each global resource (memory or semaphore) allocated or freed. You can use the LBT-IPC Resource Manager to discover and reclaim resources. See the three example outputs below.

Displaying Resources

$> lbtipc_resource_manager
Displaying Resources (to reclaim you must type '-reclaim' exactly)
--Memory Resources--
Memory resource: Process ID: 24441 SessionID: ab569cec XportID: 20001
--Semaphore Resources-- Semaphore key: 0x68871d75
Semaphore resource Index 0: reserved
Semaphore resource: Process ID: 24441 Sem Index: 1
Semaphore resource: Process ID: 24436 Sem Index: 2

Reclaiming Unused Resources

$> lbtipc_resource_manager -reclaim
Reclaiming Resources
Process 24441 not found: reclaiming Memory resource (SessionID: ab569cec XPortID: 20001)
Process 24441 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 1
Process 24436 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 2


Transport LBT-SMX  <-

The LBT-SMX (shared memory acceleration) transport is an Interprocess Communication (IPC) transport you can use for the lowest latency message Streaming. LBT-SMX is faster than the LBT-IPC transport. Like LBT-IPC, sources can publish topic messages to a shared memory area from which receivers can read topic messages. Unlike LBT-IPC, the native APIs for the LBT-SMX transport are not thread safe and do not support all UM features such as message batching or fragmentation.

You can use either the native LBT-SMX API calls, lbm_src_buff_acquire() and lbm_src_buffs_complete() to send over LBT-SMX or you can use lbm_src_send_*() API calls. The existing send APIs are thread safe with SMX, but they incur a synchronization overhead and thus are slower than the native LBT-SMX API calls.

LBT-SMX operates on the following Ultra Messaging 64-bit packages:

  • SunOS-5.10-amd64
  • Linux-glibc-2.5-x86_64
  • Win2k-x86_64

The example applications, lbmlatping.c and lbmlatpong.c show how to use the C LBT-SMX API calls. For Java, see lbmlatpong.java and lbmlatpong.java. For .NET, see lbmlatpong.cs and lbmlatpong.cs.

Other example applications can use the LBT-SMX transport with the use of a UM configuration flat file containing 'source transport lbtsmx'. You cannot use LBT-SMX with example applications for features not supported by LBT-SMX, such as lbmreq, lbmresp, lbmrcvq or lbmwrcvq.

The LBT-SMX configuration options are similar to the LBT-IPC transport options. See Transport LBT-SMX Operation Options for more information.

You can use Automatic Monitoring, UM API retrieve/reset calls, and LBMMON APIs to access LBT-SMX source and receiver transport statistics. To increase performance, the LBT-SMX transport does not collect statistics by default. Set the UM configuration option transport_lbtsmx_message_statistics_enabled (context) to 1 to enable the collection of transport statistics.


Sources and LBT-SMX  <-

When you create a source with lbm_src_create() and you've set the source's transport configuration option to LBT-SMX, UM creates a shared memory area object. UM assigns one of the transport IDs to this area from a range of transport IDs specified with the UM context configuration options, transport_lbtsmx_id_high (context) and transport_lbtsmx_id_low (context). You can also specify a shared memory location inside or outside of this range with a source configuration option, transport_lbtsmx_id (source), to group certain topics in the same shared memory area, if needed. See Transport LBT-SMX Operation Options in the UM Configuration Guide.

Note
For every context created by your application, UM creates an additional shared memory area for control information. The name for these control information memory areas ends with the suffix, _0, which is the Transport ID.

UM names the shared memory area object according to the format, LBTSMX_x_d where x is the hexadecimal Session ID and d is the decimal Transport ID. Example names are LBTSMX_42792ac_20000 or LBTSMX_66e7c8f6_20001. Receivers access a shared memory area with this object name to receive (read) topic messages.

Sending on a topic with the native LBT-SMX APIs requires the two API calls lbm_src_buff_acquire() and lbm_src_buffs_complete(). A third convenience API, lbm_src_buffs_complete_and_acquire(), combines a call to lbm_src_buffs_complete() followed by a call to lbm_src_buff_acquire() into one function call to eliminate the overhead of an additional function call.

The native LBT-SMX APIs fail with an appropriate error message if a sending application uses them for a source configured to use a transport other than LBT-SMX.

Note
The native LBT-SMX APIs are not thread safe at the source object or LBT-SMX Transport Session levels for performance reasons. Applications that use the native API LBT-SMX calls for either the same source or a group of sources that map to the same LBT-SMX Transport Session must serialize the calls either directly in the application or through their own mutex.


Sending over LBT-SMX with Native APIs  <-

Sending with LBT-SMX's native API is a two-step process.

  1. The sending application first calls lbm_src_buff_acquire(), which returns a pointer into which the sending application writes the message data.

    The pointer points directly into the shared memory region. UM guarantees that the shared memory area has at least the value specified with the len parameter of contiguous bytes available for writing when lbm_src_buff_acquire() returns. If your application set the LBM_SRC_NONBLOCK flag with lbm_src_buff_acquire(), UM returns an LBM_EWOULDBLOCK error condition if the shared memory region does not have enough contiguous space available.

    Because LBT-SMX does not support fragmentation, your application must limit message lengths to a maximum equal to the value of the source's configured transport_lbtsmx_datagram_max_size (source) option minus 16 bytes for headers. In a system deployment that includes the DRO, this value should be the same as the datagram max sizes of other transport types. See Protocol Conversion.

    After the user acquires the pointer into shared memory and writes the message data, the application may call lbm_src_buff_acquire() repeatedly to send a batch of messages to the shared memory area. If your application writes multiple messages in this manner, sufficient space must exist in the shared memory area. lbm_src_buff_acquire() returns an error if the available shared memory space is less than the size of the next message.

  2. The sending application calls one of the two following APIs.


Sending over LBT-SMX with Existing APIs  <-

LBT-SMX supports lbm_src_send_* API calls. These API calls are fully thread-safe. The LBT-SMX feature restrictions still apply, however, when using lbm_src_send_* API calls. The lbm_src_send_ex_info_t argument to the lbm_src_send_ex() and lbm_src_sendv_ex() APIs must be NULL when using an LBT-SMX source, because LBT-SMX does not support any of the features that the lbm_src_send_ex_info_t parameter can enable. See Differences Between LBT-SMX and Other UM Transports.

Since LBT-SMX does not support an implicit batcher or corresponding implicit batch timer, UM flushes all messages for all sends on LBT-SMX transports done with lbm_src_send_* APIs, which is similar to setting the LBM_MSG_FLUSH flag. LBT-SMX also supports the lbm_src_flush() API call, which behaves like a thread-safe version of lbm_src_buffs_complete().

Note
Users should not use both the native LBT-SMX APIs and the lbm_src_send_* API calls in the same application. Users should choose one or the other type of API for consistency and to avoid thread safety problems.

The lbm_src_topic_alloc() API call generates log warnings if the given attributes specify an LBT-SMX transport and enable any of the features that LBT-SMX does not support. The lbm_src_topic_alloc() call succeeds, but UM does not enable the unsupported features indicated in the log warnings. Other API functions that operate on lbm_src_t objects, such as lbm_src_create(), lbm_src_delete(), or lbm_src_topic_dump(), operate with LBT-SMX sources normally.

Because LBT-SMX does not support fragmentation, your application must limit message lengths to a maximum equal to the value of the source's configured transport_lbtsmx_datagram_max_size (source) option minus 16 bytes for headers. Any send API calls with a length parameter greater than this configured value fail. In a system deployment that includes the DRO, this value should be the same as the datagram max sizes of other transport types. See Protocol Conversion.


Receivers and LBT-SMX  <-

Receivers operate identically over LBT-SMX to receivers as all other UM transports. The msg->data pointer of a delivered lbm_msg_t object points directly into the shared memory region.

The lbm_msg_retain() API function operates differently for LBT-SMX. lbm_msg_retain() creates a full copy of the message in order to access the data outside the receiver callback.

Attention
You application should not pass the msg->data pointer to other threads or outside the receiver callback until your application has called lbm_msg_retain() on the message.
Warning
Any API calls documented as not safe to call from a context thread callback are also not safe to call from an LBT-SMX receiver thread.

Topic Resolution and LBT-SMX

Topic resolution operates identically with LBT-SMX as other UM transports albeit with the advertisement type, LBMSMX. Advertisements for LBT-SMX contain the Transport ID, Session ID, and Host ID. Receivers get LBT-SMX advertisements in the normal manner, either from the resolver cache, advertisements received on the multicast resolver address:port, or responses to queries.


Similarities Between LBT-SMX and Other UM Transports  <-

Although no actual network transport occurs, SMX functions in much the same way as if you send packets across the network as with other UM transports.

  • If you use a range of LBT-SMX transport IDs, UM assigns multiple topics sent by multiple sources to all the Transport Sessions in a round robin manner just like other UM transports.
  • Transport sessions assume the configuration option values of the first source assigned to the Transport Session.
  • Source applications and receiver applications based on any of the three available APIs can interoperate with each other. For example, sources created by a C sending application can send to receivers created by a Java receiving application.


Differences Between LBT-SMX and Other UM Transports  <-

  • Unlike LBT-RM which uses a transmission window to specify a buffer size to retain messages for retransmission, LBT-SMX uses the transmission window option to establish the size of the shared memory. LBT-SMX uses transmission window sizes that are powers of 2. You can set transport_lbtsmx_transmission_window_size (source) to any value, but UM rounds the option value up to the nearest power of 2.
  • The largest transmission window size for Java applications is 1 GB.
  • LBT-SMX does not retransmit messages. Since LBT-SMX transport is a memory write-read operation, messages should not be lost in transit. No retransmission of old messages that have been overwritten occurs.
  • Receivers do not send NAKs when using LBT-SMX.

You cannot use the following UM features with LBT-SMX:

  • Arrival Order Delivery
  • Late Join
  • Off Transport Recovery
  • Request and Response
  • Multi-transport Threads
  • Source-side Filtering
  • Hot Failover
  • Message Properties
  • Application Headers
  • Implicit and Explicit Message Batching
  • Fragmentation and Reassembly
  • Immediate Messaging
  • Receiver thread behaviors other than "busy_wait"
  • Persistent sources
  • Queued sources, both brokered and ULB

You also cannot use LBT-SMX to send egress traffic from a UM daemon, such as the Persistent Store, UM Router, UM Cache, or UMDS.


LBT-SMX Configuration Example  <-

The following diagram illustrates how sources and receivers interact with the shared memory area used in the LBT-SMX transport.

SMX_Objects.png

In the diagram above, three sources send (write) to two Shared Memory Areas while four receivers in two different contexts receive (read) from the areas. The assignment of sources to Shared Memory Areas demonstrate UM's round robin method. UM assigns the source sending on Topic A to Transport 30001, the source sending on Topic B to Transport 30002 and the source sending on Topic C back to the top of the transport ID range, 30001.

The diagram also shows the UM configuration options that set up this scenario.

  • The options transport_lbtsmx_id_high (context) and transport_lbtsmx_id_low (context) establish the range of Transport IDs between 30001 and 30002.
  • The option "source transport lbtsmx" sets the source's transport to LBT-SMX.
  • The option transport_lbtsmx_transmission_window_size (source) sets the size of each Shared Memory Area to 33554432 bytes or 32 MB. This option's value must be a power of 2. If you configured the transmission window size to 25165824 bytes (24 MB) for example, UM logs a warning message and then rounds the value of this option up to the next power of 2 or 33554432 bytes or 32 MB.


Java Code Examples for LBT-SMX  <-

The Java code examples for LBT-SMX send and receive one million messages. Start the receiver example application before you start the source example application.

Java Source Example

import java.nio.ByteBuffer; import com.latencybusters.lbm.*;
public class SimpleSrc
{
private LBMContext ctx;
private LBMSource src;
public static void main(String[] args)
{
try
{
SimpleSrc test = new SimpleSrc();
test.sendMessages();
System.out.println("Send Complete");
} catch (LBMException ex)
{
System.err.println(ex.getMessage());
ex.printStackTrace();
}
} /* main */
public SimpleSrc() throws LBMException
{
ctx = new LBMContext();
LBMSourceAttributes sattr = new LBMSourceAttributes();
sattr.setValue("transport", "lbtsmx");
LBMTopic top = ctx.allocTopic("SimpleSmx", sattr);
src = ctx.createSource(top);
}
public void sendMessages() throws LBMException
{
/* Keep a reference to the source buffer, which does not change */
final ByteBuffer srcBuffer = src.getMessagesBuffer();
/* Sends will block waiting for receivers */
final int flags = LBM.SRC_BLOCK;
final int msgLength = 8;
int pos;
/* Delay a second to let topic resolution complete. */
try { Thread.sleep(1000); } catch (Exception ex) { }
for (long i = 0; i < 1000000; i++)
{
/* Acquire a position in the buffer */
pos = src.acquireMessageBufferPosition(msgLength, flags);
/* Place data at acquired position */
srcBuffer.putLong(pos, i);
/* Inform receivers data has been written */
src.messageBuffersComplete();
}
/* Linger for a short while to allow retransmissions, etc. */
try { Thread.sleep(1000); } catch (Exception ex) { }
src.close();
ctx.close();
} /* sendMessages */
} /* SimpleSrc */

The source sends one million messages using the native LBT-SMX Java APIs. The sendMessages() method obtains a reference to the source's message buffer, which does not change for the life of the source. The method acquireMessageBufferPosition(int, int) contains the requested message length of 8 bytes. When this call returns, it gives an integer position into the previously obtained messages buffer, which is the position of the message data. UM guarantees that you can safely write the value of the counter i into the buffer at this position.

Java Receiver Example

import java.nio.ByteBuffer;
/* Extend LBMReceiver to avoid onReceive synchronization */
public class SimpleSmxRcv extends LBMReceiver
{
protected SimpleSmxRcv(LBMContext lbmctx, LBMTopic lbmtopic) throws LBMException
{
super(lbmctx, lbmtopic);
}
long lastReceivedValue = -1;
/* Override LBMReceiver onReceive method */
protected int onReceive(LBMMessage lbmmsg)
{
if (lbmmsg.type() == LBM.MSG_DATA)
{
/* New API gets byte buffer with position and limit set */
ByteBuffer msgsBuffer = lbmmsg.getMessagesBuffer();
/* Get the message data directly from the buffer */
lastReceivedValue = msgsBuffer.getLong();
}
return 0;
} /* on Receive */
public static void main(String[] args)
{
LBMContext ctx = null;
SimpleSmxRcv rcv = null;
try
{
ctx = new LBMContext();
LBMTopic top = ctx.lookupTopic("SimpleSmx");
rcv = new SimpleSmxRcv(ctx, top);
} catch (LBMException ex)
{
System.out.println(ex.getMessage());
ex.printStackTrace();
System.exit(1);
}
while (rcv.lastReceivedValue < 999999) {
try { Thread.sleep(250); } catch (Exception ex) {}
}
try
{
rcv.close();
ctx.close();
System.out.println("Last Received Value: " + rcv.lastReceivedValue);
} catch (LBMException ex)
{
System.out.println(ex.getMessage());
ex.printStackTrace();
}
} /* main */
}

The receiver reads messages from an LBT-SMX Source using the new API on LBMMessage. The example extends the LBMReceiver class so that you can overwrite the onReceive() method, which bypasses synchronization of multiple receiver callbacks. As a result, the addReceiver() and removeReceiver() methods do not work with this class, but we don't want them anyway. In the overridden onReceive() callback, we call getMessagesBuffer(), which returns a ByteBuffer view of the underlying transport. This allows the application to do zero copy reads directly from the memory that stores the message data. The returned ByteBuffer position and limit is set to the beginning and end of the message data. The message data does not start at position 0. The application reads a long out of the buffer, which is the same long that was placed by the source example.

Batching Example

public void sendMessages() throws LBMException
{
...
for (long i = 0; i < 1000000; i += 2)
{
/* Acquire a position in the buffer */
pos = src.acquireMessageBufferPosition(msgLength, flags);
/* Place data at acquired position */
srcBuffer.putLong(pos, i);
pos = src.acquireMessageBufferPosition(msgLength, flags); srcBuffer.putLong(pos, i+1);
/* Inform receivers two messages have been written */
src.messageBuffersComplete();
}
...
}

You can implement a batching algorithm at the source by doing multiple acquires before calling complete. When receivers notice that there are new message available, they deliver all new messages in a single loop.

Blocking and Non-blocking Sends Example

public void sendMessages() throws LBMException
{
...
/* Acquire will return -1 if need to wait for receivers */
final int flags = LBM.SRC_NONBLOCK;
...
for (long i = 0; i < 1000000; i++)
{
/* Acquire a position in the buffer */
pos = src.acquireMessageBufferPosition(msgLength, flags);
while (pos == -1)
{
/* Implement a backoff algorithm here */
try { Thread.sleep(1); } catch (Exception ex) { }
pos = src.acquireMessageBufferPosition(msgLength, flags);
/* Place data at acquired position */
srcBuffer.putLong(pos, i);
/* Inform receivers data has been written */
src.messageBuffersComplete();
}
...
}
}

By default, acquireMessageBufferPosition() waits for receivers to catch up before it writes the requested number of bytes to the buffer. The resulting spin wait block happens only if you did not set the flags argument to LBM.SRC_NONBLOCK. If the flags argument sets the LBM.SRC_NONBLOCK value, then the function returns -1 if the call would have blocked. For performance reasons, acquireMessageBufferPosition() does not throw new LBMEWouldBlock exceptions like standard send APIs.

Complete and Acquire Function

public void sendMessages() throws LBMException
{
...
for (long i = 0; i < 1000000; i++)
{
/* Mark previous acquires complete and reserve space */
pos = src.messageBuffersCompleteAndAcquirePosition(msgLength, flags);
/* Place data at acquired position */
srcBuffer.putLong(pos, i);
}
/* final buffers complete after loop */
src.messageBuffersComplete();
...
}

The function, messageBuffersCompleteAndAcquirePosition(), is a convenience function for the source and calls messageBuffersComplete() followed immediately by acquireMessageBufferPosition(), which reduces the number of method calls per message.


.NET Code Examples for LBT-SMX  <-

The .NET code examples for LBT-SMX send and receive one million messages. Start the receiver example application before you start the source example application.

.NET Source Example

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Runtime.InteropServices;
namespace UltraMessagingApplication.SimpleSrc
{
class SimpleSrc
{
LBMContext ctx;
LBMSource src;
static void Main(string[] args)
{
SimpleSrc test = new SimpleSrc(); test.sendMessages(); Console.WriteLine("Send Complete");
}
public SimpleSrc()
{
ctx = new LBMContext();
LBMSourceAttributes sattr = new LBMSourceAttributes();
sattr.setValue("transport", "lbtsmx");
LBMTopic top = ctx.allocTopic("SimpleSmx", sattr);
src = ctx.createSource(top);
}
private void sendMessages()
{
IntPtr writePtr;
// Sends will block waiting for receivers
int flags = LBM.SRC_BLOCK;
uint msgLength = 8; Thread.Sleep(1000);
for (long i = 0; i < 1000000; i++) {
// Acquire a position in the buffer src.
buffAcquire(out writePtr, msgLength, flags);
// Place data at acquired position Marshal.
WriteInt64(writePtr, i);
// Inform receivers data has been written.
src.buffsComplete();
}
Thread.Sleep(1000);
src.close();
ctx.close();
}
}
}

You can access the shared memory region directly with the IntPtr structs. The src.buffAcquire() API modifies writePtr to point to the next available location in shared memory. When buffAcquire() returns, you can safely write to the writePtr location up to the length specified in buffAcquire(). The Marshal.WriteInt64() writes 8 bytes of data to the shared memory region. The call to buffsComplete() signals new data to connected receivers.

.NET Receiver Example

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Runtime.InteropServices;
namespace UltraMessagingApplication.SimpleRcv
{
class SimpleRcv
{
private LBMContext ctx;
private LBMReceiver rcv;
private long lastReceivedValue = -1;
static void Main(string[] args)
{
SimpleRcv simpleRcv = new SimpleRcv();
while (simpleRcv.lastReceivedValue < 999999)
{
Thread.Sleep(250);
}
simpleRcv.rcv.close();
simpleRcv.ctx.close();
Console.WriteLine("Last Received Value: {0}", simpleRcv.lastReceivedValue);
}
public SimpleRcv()
{
ctx = new LBMContext();
LBMTopic top = new LBMTopic(ctx, "SimpleSmx");
rcv = new LBMReceiver(ctx, top, new LBMReceiverCallback(onReceive), null);
}
public int onReceive(Object obj, LBMMessage msg)
{
if (msg.type() == LBM.MSG_DATA)
{
// Read data out of shared memory
lastReceivedValue = Marshal.ReadInt64(msg.dataPointerSafe());
}
// dispose the message so the LBMMessage object can be re-used msg.
dispose();
return 0;
}
}
}

The application calls the simpleRcv::onReceive callback after the source places new data in the shared memory region. The msg.dataPointerSafe() API returns an IntPtr to the data, which does not create any new objects. The Marshal.ReadInt64 API then reads data directly from the shared memory.

Batching

private void sendMessages()
{
...
for (int i = 0; i < 1000000; i += 2) {
// Acquire a position in the buffer src.
buffAcquire(out writePtr, msgLength, flags);
// Place data at acquired position Marshal.
WriteInt32(writePtr, i);
// Acquire a position in the buffer src.
buffAcquire(out writePtr, msgLength, flags);
// Place data at acquired position Marshal.
WriteInt32(writePtr, i);
// Inform receivers two messages has been written src.
buffsComplete();
}
...
}

You can implement a batching algorithm at the source by doing multiple acquires before calling complete. When receivers notice that new message are available, they deliver all new messages in a single loop.

Blocking and Non-blocking Sends

private void sendMessages()
{
...
// buffAcquire will return -1 if need to wait for receivers int flags = LBM.SRC_NONBLOCK;
...
for (long i = 0; i < 1000000; i++)
{
// Acquire a position in the buffer
int rc = src.buffAcquire(out writePtr, msgLength, flags);
while (rc == -1)
{
// Implement a backoff algorithm here Thread.Sleep(0);
rc = src.buffAcquire(out writePtr, msgLength, flags);
}
// Place data at acquired position Marshal.
WriteInt64(writePtr, i);
// Inform receivers that a message has been written src.
buffsComplete();
...
}
}

By default, buffAcquire() waits for receivers to catch up before it writes the requested number of bytes to the buffer. The resulting spin wait block happens only if you did not set the flags argument to LBM.SRC_NONBLOCK. If the flags argument sets the LBM.SRC_NONBLOCK value, then the function returns -1 if the call would have blocked. For performance reasons, buffAcquire() does not throw new LBMEWouldBlock exceptions like standard send APIs.

Complete and Acquire Function

private void sendMessages()
{
...
for (long i = 0; i < 1000000; i++) {
// Acquire a position in the buffer src.
buffsCompleteAndAcquire(out writePtr, msgLength, flags);
// Place data at acquired position Marshal.
WriteInt64(writePtr, i);
}
// final buffsComplete after loop src.buffsComplete();
...
}

The function, buffsCompleteAndAcquire(), is a convenience function for the source and calls buffsComplete() followed immediately by buffAcquire(), which reduces the number of method calls per message.

Reduce Synchronization Overhead

public SimpleRcv()
{
ctx = new LBMContext();
LBMReceiverAttributes rattr = new LBMReceiverAttributes();
// Set the enableSingleReceiverCallback attribute to 'true' rattr.
enableSingleReceiverCallback(true);
LBMTopic top = new LBMTopic(ctx, "SimpleSmx", rattr);
// With enableSingleReceiverCallback, a callback must be specified in the ver constructor.
rcv = new LBMReceiver(ctx, top, new LBMReceiverCallback(onReceive), null);
// rcv.addReceiver and rcv.removeReceiver will result in log warnings.
}

Delivery latency to an LBMReceiver callback can be reduced with a single callback. Call LBMReceiverAttributes::enableSingleReceiverCallback on the attributes object used to create the LBMReceiver. The addReceiver() and removeReceiver() APIs become defunct, and your application calls the application receiver callback without any locks taken. The enableSingelReceiverCallback() API eliminates callback related synchronization overhead.

Note
In Java, inheriting from LBMReceiver and overriding the onReceive can achieve the same thing.

Increase Performance with unsafe Code Constructs

for (long i = 0; i < 1000000; i++) {
// Acquire a position in the buffer src.buffAcquire(out writePtr, msgLength, flags);
// Place data at acquired position
unsafe
{
*((long*)(writePtr)) = i;
}
// Inform receivers data has been written src.buffsComplete();
}
public int onReceive(Object obj, LBMMessage msg)
{
if (msg.type() == LBM.MSG_DATA)
{
unsafe
{
lastReceivedValue = *((long*)msg.dataPointer());
}
}
// dispose the message so the object can be re-used msg.dispose();
return 0;
}

Using .NET unsafe code constructs can increase performance. By manipulating pointers directly, you can eliminate calls to external APIs, resulting in lower latencies.


LBT-SMX Resource Manager  <-

Deleting an SMX source or deleting an SMX receiver reclaims the shared memory area and locks allocated by the SMX source or receiver. However, if an ungraceful exit from a process occurs, global resources remain allocated but unused. To address this possibility, the LBT-SMX Resource Manager maintains a resource allocation database with a record for each global resource (memory or semaphore) allocated or freed. You can use the LBT-SMX Resource Manager to discover and reclaim resources. See the three example outputs below.

Displaying Resources

$> lbtsmx_resource_manager
Displaying Resources (to reclaim you must type '-reclaim' exactly)
--Memory Resources--
Memory resource: Process ID: 24441 SessionID: ab569cec XportID: 20001
--Semaphore Resources-- Semaphore key: 0x68871d75
Semaphore resource Index 0: reserved
Semaphore resource: Process ID: 24441 Sem Index: 1
Semaphore resource: Process ID: 24436 Sem Index: 2

Reclaiming Unused Resources

Warning
This operation should never be done while SMX-enabled applications or daemons are running. If you have lost or unused resources that need to be reclaimed, you should exit all SMX applications prior to running this command.
$> lbtsmx_resource_manager -reclaim
Reclaiming Resources
Process 24441 not found: reclaiming Memory resource (SessionID: ab569cec XPortID: 20001)
Process 24441 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 1
Process 24436 not found: reclaiming Semaphore resource: Key: 0x68871d75 Sem Index: 2


Transport Broker  <-

With the UMQ product, you use the 'broker' transport to send messages from a source to a Queuing Broker, or from a Queuing Broker to a receiver.

When sources or receivers connect to a Queuing Broker, you must use the 'broker' transport. You cannot use the 'broker' transport with UMS or UMP products.