6. Monitoring UMS

6.1.1. Why Monitor?

Monitoring can aid different groups within an organization.

• Developers can spot bugs that impact system performance.

• Performance tuning groups can pinpoint under-performing receivers.

• Testing groups can understand the reaction of a system to stresses like random packet loss during pre-production testing.

• Network or middleware management groups can use monitoring to ensure a production system continues to operate within its design criteria.

6.1.2. What to Monitor

Before discussing the monitoring statistics that are built into UM, we mention two things that are probably more important to monitor: connectivity and latency. UM provides some assistance for monitoring these, but the final responsibility rests with your applications.

6.2.1. Context Statistics

Context statistics help you monitor topic resolution activity, along with the number of unknown messages received and the number of sends and responses that were blocked or returned EWOULDBLOCK. Context statistics also contain transport statistics for Multicast Immediate Messaging (MIM) activity and transport statistics for all the sources or receivers in a context.

6.2.2. Event Queue Statistics

Event Queue statistics help you monitor the number of events currently on the queue, how long it takes to service them (maximum, minimum and mean service times) and the total number of events for the monitoring period. These statistics are available for the following types of events.

• Data messages

• Request messages

• Immediate messages

• Wildcard receiver messages

• I/O events

• Timer events

• Source events

• Unblock events

• Cancel events

• Callback events

• Context source events

• Total events

• Age of events

When monitoring Event Queue statistics you must enable the Event Queue UM Configuration Options, queue_age_enabled, queue_count_enabled and queue_service_time_enabled . UM disables these options by default, which produces no event queue statistics.

6.2.3. Transport Statistics

You can retrieve transport statistics for different types of transports (TCP, LBT-RU, LBT-RM, LBT-IPC, LBT-RDMA). In addition, you can limit these transport statistics to a specifc source sending on the particular transport or a specifc receiver receiving messages over the transport. Source statistics for LBT-RM, for example, include the number of messages (datagrams) sent and the number of retransmissions sent. For receiver LBT-RM, statistics include, for example, the number of messages (datagrams) received and number of UM messages received.

6.3.1. UMS Monitoring Process Flow

The overall process flow appears in the diagram below.

1. Your application creates the monitor source controller, specifying the format and transport modules to use. It also calls lbmmon functions to start monitoring an UM context, UM source or UM receiver.

2. The monitor source controller passes those statistics to the format module serialization function.

3. The monitor source controller passes the resulting serialized data to the transport module send function.

4. The transport module transmits the data over some transport medium (such as a network).

5. The monitor receiver controller transport module receives the serialized data. (Your monitoring application has already created the monitor receiver controller specifying the format and transport modules to use, along with the application callback functions to use upon the receipt of UM source or UM receiver statistics data.)

6. The monitor receiver controller calls the format module's de-serialization function.

7. Finally, the monitor receiver controller passes the statistics to your monitoring application via the specified application callback functions.

Your applications only calls functions in the controller modules, which calls the appropriate functions in the transport and format modules.

6.3.2. API Framework Flexibility

The segregation of UM Monitoring into control, format, and transport modules provides flexibility for monitor receivers in two ways.

• Allows you to use languages for which no UM API or binding exists.

• Allows you to use monitoring products which do not integrate with UM.

As an example, assume you have a Perl application which currently gathers statistics from other network applications (or, you are simply most comfortable working in Perl for such tasks). There is no Perl binding for UM. However, Perl can handle UDP packets very nicely, and can pick apart CSV data easily. By implementing a UDP transport module to be used by the monitor sources, your Perl application can read the UDP packets and process the statistics.

6.3.3. Initial Monitoring Questions

If you can answer the following questions, you're already on your way.

1. What format module will you use? LBMMON CSV Format module or a different one.

2. What transport module will you use? One of the 3 LBMMON modules or a different one.

3. Do you want to monitor individual sources/receivers, or an entire context? The difference is in how the statistics are aggregated.

   • Monitoring a context aggregates transport statistics for all sources and receivers associated with a context, by transport. Note that this is not by transport type. The default configuration for TCP, for example, allocates up to 10 ports, forming up to 10 separate transport sessions. Absent any specific instructions, UM allocates sources and receivers to these 10 transports in a round-robin fashion. So the statistics for a specific transport on a context will aggregate all sources and receivers which use that specific transport.

   • Ultra Messaging recommends that you monitor either a context or source/receiver, but not both. For example if Topic1 and Topic2 are mapped to the same transport session (which is the only transport session for the context) and you monitor both the receivers and the context, you will get 3 identical sets of statistics: one for Topic1 reporting the stats for it's transport session, one for Topic2 reporting the stats for the same transport session, and one for the transport session via the context.

   • In the case of wildcard receivers, only the context may be monitored. UM creates wildcard receivers dynamically as it detects topics which match the wildcard pattern. The application does not have access to these dynamically-created receivers. So the only way to monitor a wildcard receiver is to monitor the context on which it was created.

4. Should statistics be sent automatically, or on demand?

   • Automatic sending of statistics is by far the simplest approach. You simply indicate how often the statistics should be gathered and sent. The rest is taken care of.

   • On-demand is somewhat more involved. Your application decides when statistics should be gathered and sent. If you intend to use the arrival of statistics as a type of heartbeat, this is the method you should use.

The following sections present more discussion and sample source code about starting monitor sources, monitor receivers and the LBMMON format and transport modules.

6.3.4. Creating a Monitoring Source

The following examples demonstrate how to use the UM Monitoring API to enable monitoring in your application.

First, create a monitoring source controller:

lbm_context_t * ctx;
lbm_src_t * src;
lbm_rcv_t * rcv;
lbmmon_sctl_t * monctl;

if (lbmmon_sctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), NULL) == -1)
{
        fprintf(stderr, "lbmmon_sctl_create() failed\n");
        exit(1);
}

The above code tacitly assumes that the ctx, src, and rcv variables have been previously assigned via the appropriate UM API calls.

The monitoring source controller object must be passed to subsequent calls to reference a specific source controller. One implication of this is that it is possible to have multiple monitoring source controllers within a single application, each perhaps monitoring a different set of objects.

In the above example, the default CSV format module and default UM transport module are specified via the provided module functions lbmmon_format_csv_module() and lbmmon_transport_lbm_module().

6.3.5. Specifying the Object to Monitor

Once a monitoring source controller is created, the application can monitor a specific context using:

if (lbmmon_context_monitor(monctl, ctx, NULL, 10) == -1)
{
        fprintf(stderr, "lbmmon_context_monitor() failed\n");
        exit(1);
}

The above example indicates that statistics for all transports on the specified context will be gathered and sent every 10 seconds.

A UM source can be monitored using:

if (lbmmon_src_monitor(monctl, src, NULL, 10) == -1)
{
        fprintf(stderr, "lbmmon_src_monitor() failed\n");
        exit(1);
}

Finally, an UM receiver can be monitored using:

if (lbmmon_rcv_monitor(monctl, rcv, NULL, 10) == -1)
{
        fprintf(stderr, "lbmmon_rcv_monitor() failed\n");
        exit(1);
}

The two above examples also request that statistics for all transports on the specified source or receiver be gathered and sent every 10 seconds.

Statistics can also be gathered and sent in an on-demand manner. Passing 0 for the Seconds parameter to lbmmon_context_monitor(), lbmmon_src_monitor(), or lbmmon_rcv_monitor() prevents the automatic gathering and sending of statistics. To trigger the gather/send process, use:

lbmmon_sctl_sample(monctl);

Such a call will perform a single gather/send action on all monitored objects (contexts, sources, and receivers) which were registered as on-demand.

As part of application cleanup, the created monitoring objects should be destroyed. Each individual object can be de-registered using lbmmon_context_unmonitor(), lbmmon_src_unmonitor(), or lbmmon_rcv_unmonitor(). Finally, the monitoring source controller can be destroyed using:

lbmmon_sctl_destroy(monctl);

Any objects which are still registered will be automatically de-registered by lbmmon_sctl_destroy().

6.3.6. Receiving Monitoring Data

To make use of the statistics, an application must be running which receives the monitor data. This application creates a monitoring receive controller, and specifies callback functions which are called upon the receipt of source or receiver statistics data.

Use the following to create a monitoring receive controller:

lbmmon_rctl_t * monctl;
lbmmon_rctl_attr_t * attr;
lbmmon_rcv_statistics_func_t rcvcb = { rcv_statistics_cb };
lbmmon_src_statistics_func_t srccb = { src_statistics_cb };
lbmmon_evq_statistics_func_t evqcb = { evq_statistics_cb };
lbmmon_ctx_statistics_func_t ctxcb = { ctx_statistics_cb };

if (lbmmon_rctl_attr_create(&attr) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_create() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_RECEIVER_CALLBACK, (void *) &rcvcb, sizeof(rcvcb)) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_SOURCE_CALLBACK, (void *) &srccb, sizeof(srccb)) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_EVENT_QUEUE_CALLBACK, (void *) &evqcb, sizeof(evqcb)) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_attr_setopt(attr, LBMMON_RCTL_CONTEXT_CALLBACK, (void *) &sctxcb, sizeof(ctxcb)) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_setopt() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), (void *) 
transport_options, attr) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_create() failed, %s\n", lbmmon_errmsg());
    exit(1);
}
if (lbmmon_rctl_attr_delete(attr) != 0)
{
    fprintf(stderr, "call to lbmmon_rctl_attr_delete() failed, %s\n", lbmmon_errmsg());
    exit(1);
}

As in the earlier example, the default CSV format module and default UM transport module are specified via the provided module functions lbmmon_format_csv_module() and lbmmon_transport_lbm_module().

As an example of minimal callback functions, consider the following example:

void rcv_statistics_cb(const void * AttributeBlock, const lbm_rcv_transport_stats_t * Statistics)
{
    lbm_ulong_t source = LBMMON_ATTR_SOURCE_NORMAL;
    if (lbmmon_attr_get_source(AttributeBlock, &source) != 0)
    {
        source = LBMMON_ATTR_SOURCE_NORMAL;
    }
    switch (Statistics->type)
    {
        case LBM_TRANSPORT_STAT_TCP:
            handle_rcv_tcp_statistics();
            break; 
        case LBM_TRANSPORT_STAT_LBTRM:
            switch (source)
            {
                case LBMMON_ATTR_SOURCE_IM:
                    handle_rcv_im_lbtrm_statistics();
                    break;
                default:
                    handle_rcv_lbtrm_statistics();
                    break;
            }
            break;
        case LBM_TRANSPORT_STAT_LBTRU:
            handle_rcv_lbtru_statistics();
            break;
    }
}

void src_statistics_cb(const void * AttributeBlock, const lbm_src_transport_stats_t * Statistics)
{
    lbm_ulong_t source = LBMMON_ATTR_SOURCE_NORMAL;
    if (lbmmon_attr_get_source(AttributeBlock, &source) != 0)
    {
        source = LBMMON_ATTR_SOURCE_NORMAL;
    }
    switch (Statistics->type)
    {
        case LBM_TRANSPORT_STAT_TCP:
            handle_src_tcp_statistics();
            break;
        case LBM_TRANSPORT_STAT_LBTRM:
            switch (source)
            {
                case LBMMON_ATTR_SOURCE_IM:
                    handle_src_im_lbtrm_statistics();
                    break;
                default:
                    handle_src_lbtrm_statistics();
                    break;
            }
            break;
        case LBM_TRANSPORT_STAT_LBTRU:
            handle_src_lbtru_statistics();
            break;
    }
}

void ctx_statistics_cb(const void * AttributeBlock, const lbm_context_stats_t * Statistics)
{
    /* Handle context stats */
}

void evq_statistics_cb(const void * AttributeBlock, const lbm_event_queue_stats_t * Statistics)
{
    /* Handle event queue stats */
}

Upon receipt of a statistics message, the appropriate callback function is called. The application can then do whatever is desired with the statistics data, which might include writing it to a file or database, performing calculations, or whatever is appropriate.

Beyond the actual statistics, several additional pieces of data are sent with each statistics packet. These data are stored in an attribute block, and are accessible via the lbmmon_attr_get_*() functions. Currently, these data include the IPV4 address of machine which sent the statistics data, the timestamp (as a time_t) at which the statistics were generated, and the application ID string supplied by the sending application at the time the object was registered for monitoring. See lbmmon_attr_get_ipv4sender(), lbmmon_attr_get_timestamp(), and lbmmon_attr_get_appsourceid() for more information.

6.3.7. The UMS Transport Module

The UM transport module understands several options which may be used to customize your use of the module. The options are passed via the TransportOptions parameter to the lbmmon_sctl_create() and lbmmon_rctl_create() functions, as a null-terminated string containing semicolon-separated name/value pairs.

The following options are available:

• config specifies a configuration file. This file is processed in a manner similar to lbm_config(). However, unlike lbm_config(), the current default attributes are not changed. Instead, the options parsed from the configuration file are applied only to the UM objects created by the module.

• topic specifies the topic name to use for sending and receiving statistics. By default, the topic /29west/statistics is used.

• wctopic specifies (for monitor receivers only) a wildcard pattern to be used to receive statistics.

As an example, assume your application needs to use a special configuration file for statistics. The following call allows your application to customize the UM transport module using the configuration file stats.cfg.

lbmmon_sctl_t * monctl;
const char * tropt = "config=stats.cfg";
if (lbmmon_sctl_create(&smp;monctl, lbmmon_format_csv_module(), NULL, 
lbmmon_transport_lbm_module(), tropt) == -1)
{
        fprintf(stderr, "lbmmon_sctl_create() failed\n");
        exit(1);
}

If your application also needs to use a specific topic for statistics, the following code specifies that, in addition to the configuration file, the topic StatisticsTopic be used for statistics.

lbmmon_sctl_t * monctl;
const char * tropt = "config=stats.cfg;topic=StatisticsTopic";
if (lbmmon_sctl_create(&monctl, lbmmon_format_csv_module(), NULL, lbmmon_transport_lbm_module(), 
tropt) == -1)
{
        fprintf(stderr, "lbmmon_sctl_create() failed\n");
        exit(1);
}

It is important to use the same topic and configuration for both monitor sources and receivers. Otherwise your applications may send the statistics, but the monitor receiver won't be able to receive them.

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.3.8. The UDP Transport Module

The UDP transport module understands several options which may be used to customize your use of the module. The options are passed via the TransportOptions parameter to the lbmmon_sctl_create() and lbmmon_rctl_create() functions, as a null-terminated string containing semicolon-separated name/value pairs.

The UDP module supports sending and receiving via UDP unicast, UDP broadcast, and UDP multicast. The following options are available.

• address specifies the unicast IP address to which statistics are sent via UDP. Applicable to sender only.

• port is the IP port packets are sent to. Defaults to 2933.

• interface specifies the network interface over which multicast UDP is sent or received.

• mcgroup is the multicast group on which to send and receive UDP packets.

• bcaddress specified the broadcast address to which UDP packets are sent. Applicable to sender only.

• ttl specifies the TTL for each multicast UDP packet. Applicable to sender only.

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.3.9. The SNMP Transport Module

The SNMP transport modules operates in identical fashion to the UMS Transport Module. See The UMS Transport Module

To view the source code for all LBMMON transport modules, see LBMMON Example Source Code found on the Related Pages tab in the C Application Programmer's Interface.

6.5.1. lbmmon.c

The example application lbmmon.c acts as a Monitor Receiver and is provided in both executable and source form. It writes monitoring statistics to the screen and can be used in conjunction with other example applications (which act as the Monitor Sources). The following procedure uses lbmrcv and lbmsrc to create messaging traffic and adds a configuration file in order to specify the LBT-RM transport instead of the TCP default. (The LBT-RM transport displays more statistics than TCP.)

Since UM does not generate monitoring statistics by default, you must activate monitoring in your application. For the example application, use the --monitor-ctx=n option where n is the number of seconds between reports. The following procedure activates monitoring on the receiver, specifying the context (ctx) to create a complete set of receiver statistics. You could activate monitoring in a similar fashion on the source and create source statistics.

To use lbmmon to view statistics from sample application output:

1. Create configuration file with the single option of source transport lbtrm and name it LBTRM.cfg.

2. Run lbmmon --transport-opts="config=LBTRM.cfg"

3. Run lbmrcv -c LBTRM.cfg --monitor-ctx=5 Arizona

4. Run lbmsrc -c LBTRM.cfg Arizona

After lbmsrc completes, the final output for lbmmon should closely resemble the following.

Receiver statistics received from C:\Program Files\29West\UME_1.2.1\Win2k-i386\bin\lbmrcv.exe 
at 10.29.1.78, sent Wed Jan 09 14:25:49 2008
Source: LBTRM:10.29.1.78:4391:323382d8:224.10.10.10:4400
Transport: LBT-RM
LBT-RM messages received                                 : 45455
Bytes received                                           : 370000000
LBT-RM NAK packets sent                                  : 0
LBT-RM NAKs sent                                         : 0
Lost LBT-RM messages detected                            : 0
NCFs received (ignored)                                  : 0
NCFs received (shed)                                     : 0
NCFs received (retransmit delay)                         : 0
NCFs received (unknown)                                  : 0
Loss recovery minimum time                               : 4294967295ms
Loss recovery mean time                                  : 0ms
Loss recovery maximum time                               : 0ms
Minimum transmissions per individual NAK                 : 4294967295
Mean transmissions per individual NAK                    : 0
Maximum transmissions per individual NAK                 : 0
Duplicate LBT-RM data messages received                  : 0
LBT-RM messages unrecoverable (window advance)           : 0
LBT-RM messages unrecoverable (NAK generation expiration): 0
LBT-RM LBM messages received                             : 10000000
LBT-RM LBM messages received with no topic               : 0
LBT-RM LBM requests received                             : 0
       

Notes:

• Since this procedure was done on a single machine. No packets were lost and therefore lbmrcv did not generate any NAKs and lbmsrc did not send any NCFs. If you run this procedure across a network, packets may be lost and you would see statistics for NAKs, NCFs and loss recovery.

• This procedure activates monitoring on the receiver, specifying the context (--monitor-ctx) to create a complete set of receiver transport statistics. You could activate monitoring in a similar fashion on the source and create source statistics. Each set of statistics shows one side of the transmission. For example, source statistics contain information about NAKs received by the source (ignored, shed, retransmit delay, etc.) where receiver statistics contain data about NCFs received. Each view can be helpful.

• Moreover, as explained earlier in Specifying the Object to Monitor, individual receivers or sources can be monitored instead of all transport activity for a context. For this procedure, use --monitor-rcv or --monitor-src.

• You could run this procedure again specifying a different transport (LBT-RU or TCP) in the configuration file and receive a different set of statistics. For descriptions of all the transport statistics, refer to the transport statistics data structures in the C Application Programmer's Interface. Click on the Data Structures tab at the top and click on lbm_rcv_transport_stats_t or lbm_src_transport_stats_t.

6.5.2. lbmmonudp.c and lbmmondiag.pl

The example application, lbmmonudp.c receives UM statistics and forwards them as CSV data over a UDP transport. The Perl script, lbmmondiag.pl, can read UDP packets and process the statistics, reporting Severity 1 and Severity 2 events. This script only reports on LBT-RM transports.

To run lbmmonudp.c with lbmmondiag.pl, use the following procedure.

1. Create configuration file with the single option of source transport lbtrm and name it LBTRM.cfg.

2. Run lbmmonudp -a 127.0.0.1 --transport-opts="config=LBTRM.cfg"

3. Run lbmrcv -c LBTRM.cfg --monitor-ctx=5 Arizona

4. Run lbmsrc -c LBTRM.cfg Arizona

5. Run lbmmondiag.pl

The following sections discuss some of the possible results of this procedure. Your results will vary depending upon conditions in your network or if you run the procedure on a single machine.

    Sev1: 34 datagrams unrecovered due to NAK generation interval ending
       

    Sev1: 249 datagrams unrecovered due to transmission window advancement
       

    Sev2: 10 datagrams queued by data rate controller
    

    Sev2: 101 datagrams queued by retransmission rate controller
    

    Sev2: 310 datagrams lost
     Sev2: 112.236 seconds estimated total latency due to repair of 564 losses
       

    Sev2: 58 NAKs sent
       

    Sev2: 7478 NAKs received
     Sev2: 50 retransmissions sent
     Sev2: 0.015056 seconds estimated total latency due to retransmissions