Chapter 9. Customizing and Extending PCP Services

Performance Co-Pilot (PCP) has been developed to be fully extensible. The following sections summarize the various facilities provided to allow you to extend and customize PCP for your site:

PMDA Customization

The generic procedures for installing and activating the optional PMDAs has been described in “Managing Optional PMDAs”. In some cases, these procedures prompt the user for information based upon the local system or network configuration, application deployment, or processing profile to customize the PMDA and hence the performance metrics it exports.

The summary PMDA is a special case that warrants further discussion.

Customizing the Summary PMDA

The summary PMDA exports performance metrics derived from performance metrics made available by other PMDAs. It is described completely in the pmdasummary(1) reference page.

The summary PMDA consists of two processes:

pmie process 

A dedicated instance of the PCP pmie inference engine is used to periodically sample the base metrics and compute values for the derived metrics. This pmie process is launched with special command-line arguments by the main process. See the “Introduction to pmie” for a complete discussion of the pmie feature set.

main process 

The main process reads and buffers the values computed by the pmie process and makes them available to the Performance Metrics Collector Daemon (PMCD).

All of the metrics exported by the summary PMDA have a singular instance and the values are “instantaneous”; the exported value is the correct value as of the last time the corresponding expression was evaluated by the pmie process.

The summary PMDA resides in the /usr/pcp/pmdas/summary directory and may be installed with a default configuration by following the steps described in the section “PMDA Installation”.

Alternatively, the summary PMDA may be customized to export your own derived performance metrics, by following the procedure described below:

  1. Choose Performance Metric Name Space (PMNS) names for the new metrics. These must begin with summary and follow the rules described in pmns(4). For example, you might use summary.fs.cache_write and summary.fs.cache_hit.

  2. Edit the file pmns in the /usr/pcp/pmdas/summary directory to add the new metric names in the format described in pmns(4). You must choose a unique Performance Metric Identifier (PMID) for each metric. In the pmns file, these appear as SYSSUMMARY:0:x. The value of x is arbitrary in the range 0 to 1023 and unique in this file. Refer to “PMNS Management” for a further explanation of the rules governing PMNS updates.

    For example:

    summary {
           cpu
           disk
           netif
           fs              /*new*/
    }
    summary.fs {
           cache_write     SYSSUMMARY:0:10
           cache_hit       SYSSUMMARY:0:11
    }
    

  3. Use the local test PMNS root and validate that the PMNS changes are correct.

    For example, enter this command:

    pminfo -n root -m summary.fs 
    

    You see output similar to the following:

    summary.fs.cache_write PMID: 27.0.10
    summary.fs.cache_hit PMID: 27.0.11
    

  4. Edit the file /usr/pcp/pmdas/summary/expr.pmie to add new pmie expressions. If the name to the left of the assignment operator (=) is one of the PMNS names, then the pmie expression to the right will be evaluated and returned by the summary PMDA. The expression must return a numeric value; additional description of the pmie expression syntax may be found in “Specification Language for pmie”.

    For example, consider this expression:

    // filesystem buffer cache hit percentages
    prefix = "irix.kernel.all.io";          // macro, not exported
    summary.fs.cache_write =
                 100 - 100 * $prefix.bwrite / $prefix.lwrite;
    summary.fs.cache_hit =
                 100 - 100 * $prefix.bread / $prefix.lread;
    

  5. Run pmie in debug mode to verify that the expressions are being evaluated correctly, and the values make sense.

    For example, enter this command:

    pmie -t 2sec -v expr.pmie 
    

    You see output similar to the following:

    summary.fs.cache_write:      ?
    summary.fs.cache_hit:      ?
    summary.fs.cache_write:  45.83
    summary.fs.cache_hit:   83.2
    summary.fs.cache_write:  39.22
    summary.fs.cache_hit:  84.51
    

  6. Install the new PMDA.

    From the /usr/pcp/pmdas/summary directory, use this command:

    ./Install 
    

    You see the following output:

    You need to choose an appropriate configuration for installation of
    the “summary” Performance Metrics Domain Agent (PMDA).
    collector collect performance statistics on this system
    monitor   allow this system to monitor local and/or remote systems
    both      collector and monitor configuration for this system
    Please enter c(ollector) or m(onitor) or b(oth) [b] both 
    Interval between summary expression evaluation (seconds)? [10] 10
    Updating the Performance Metrics Name Space...
    Installing pmchart view(s) ...
    Terminate PMDA if already installed ...
    Installing files ..
    Updating the PMCD control file, and notifying PMCD ...
    Wait 15 seconds for the agent to initialize ...
    Check summary metrics have appeared ... 8 metrics and 8 values
    

  7. Check the metrics.

    For example, enter this command:

    pmval -t 5sec -s 8 summary.fs.cache_write 
    

    You see a response similar to the following:

    metric:    summary.fs.cache_write
    host:      localhost
    semantics: instantaneous value
    units:     none
    samples:   8
    interval:  5.00 sec
    63.60132158590308
    62.71878646441073
    62.71878646441073
    58.73968492123031
    58.73968492123031
    65.33822758259046
    65.33822758259046
    72.6099706744868
    

    Note that the values are being sampled here by pmval every 5 seconds, but pmie is passing only new values to the summary PMDA every 10 seconds. Both rates could be changed to suit the dynamics of your new metrics.

  8. You may now create pmchart views, pmview scenes, and pmlogger configurations to monitor and archive your new performance metrics.

PCP Tool Customization

The PCP has been designed and implemented with a philosophy that embraces the notion of “toolkits” and encourages extensibility.

In most cases the PCP tools provide orthogonal services, based on external configuration files. It is the creation of new and modified configuration files that enables PCP users to customize tools quickly and meet the needs of the local environment, in many cases allowing personal preferences to be established for individual users on the same PCP monitor system.

The material in this section is intended to act as a checklist of pointers to detailed documentation found elsewhere in this guide, in the reference pages, and in the files that are made available as part of the PCP installation.

Stripchart Customization

The PCP tool pmchart produces stripchart displays of performance metrics. Refer to “The pmchart Tool” for an extensive description of the capabilities of pmchart.

Customization is centered on PCP views that may be created interactively and saved via the View Save option in the Views menu.

When pmchart is loading a view, the following directories are searched:

. 

The current directory.

$HOME/.pcp 

Views for each user.

/var/pcp/config/pmchart  


The system-wide catalog of views. Any view installed here becomes visible to every pmchart user.

The X11 application resources for pmchart are in /usr/lib/X11/app- defaults/PmChart, and these may be edited to customize the appearance of the display; the default update interval and other attributes are described in the pmchart(1) reference page.

Archive Logging Customization

The PCP archive logger is presented in Chapter 7, “Archive Logging,” and documented in the pmlogger(1) reference page.

The following global files and directories influence the behavior of pmlogger:

/etc/config/pmlogger  


Enable/disable state for the primary logger facility using this command:

chkconfig pmlogger on

/etc/config/pmlogger.options  


Command-line options passed to the primary logger if it is launched from /etc/init.d/pcp.

/var/pcp/config/pmlogger/config.default  


The default pmlogger configuration file that is used for the primary logger when this facility is enabled.

/var/pcp/config/pmlogger/config.view.*  


Every pmchart view also provides a pmlogger configuration file that includes each of the performance metrics used in the view, for example, /var/pcp/config/pmlogger/config.LoadAvg for the LoadAvg view.

/var/pcp/config/pmlogger/config.*  


Every PCP tool with a fixed group of performance metrics contributes a pmlogger configuration file that includes each of the performance metrics used in the tool, for example, /var/pcp/config/pmlogger/config.dkvis for dkvis.

/var/pcp/config/pmlogger/control  


Defines which PCP collector hosts require pmlogger to be launched on the local host, where the configuration file comes from, where the archive log files should be created, and pmlogger start-up options.

/var/pcp/config/pmlogger/crontab  


Prototype crontab entries that may be merged with the crontab for root to schedule the periodic execution of the archive log management scripts, for example, cron.pmdaily.

/var/adm/pcplog/somehost  


The default behavior of the archive log management scripts create archive log files for the host somehost in this directory.

/var/adm/pcplog/somehost/Latest  


A PCP archive folio for the most recent archive for the host somehost. This folio is created and maintained by the cron-driven periodic archive log management scripts, for example, cron.pmcheck. Archive folios may be processed with the pmafm tool.

Inference Engine Customization

The PCP inference engine is presented in Chapter 6, “Performance Metrics Inference Engine,” and documented in the pmie(1) reference page.

The following global files and directories influence the behavior of pmie:

/var/pcp/config/pmie/control.master  


Control file that defines how many concurrent invocations of the various pmie action methods are allowed.

/var/pcp/demos/pmie/*  


Example pmie rules that may be used as a basis for developing local rules.

/usr/pcp/bin/pmie/*  


Scripts that implement the default pmie action methods.

/var/pcp/config/pmrules/*  


The template masters used by pmrules.

Snapshot Customization

The PCP snapshot production facility is presented in “Making Snapshot Images From Archive Logs” and documented in the cron.pmsnap(1) reference page.

The following global files and directories influence the behavior of cron.pmsnap:

/var/pcp/config/pmsnap/control  


Defines how to produce a snapshot, including the output filename, the PCP archive folio name to be used as input, the pmchart configuration file, and command-line arguments to pmchart.

/var/pcp/config/pmsnap/Summary  


A pmchart configuration file to produce a sample summary snapshot in conjunction with cron.pmsnap.

/var/pcp/config/pmlogger/config.Summary  


A pmlogger configuration file that can produce an archive containing performance metrics required by the sample summary snapshot.

/var/pcp/config/pmlogger/crontab  


Prototype crontab entries that may be merged with the crontab for root schedule the periodic execution of the archive log management scripts, for example, cron.pmsnap.

/var/pcp/config/pmsnap/Summary.html  


An example HTML page suitable for publishing images from the pmsnap examples via a Web server.

Icon Control Panel Customization

The gadget specification language of pmgadgets supports the creation of arbitrary gadget layouts and bindings to hosts and performance metrics. See “The pmgadgets Command” and the pmgadgets(1) reference page.

3D Visualization Customization

The 3D scene specification language of pmview supports the creation of block layouts and bindings to hosts and performance metrics. See Chapter 5, “System Performance Visualization Tools,” and the pmview(1) reference page.

PMNS Management

This section describes the syntax, semantics, and processing framework for the external specification of a Performance Metrics Name Space (PMNS) as it might be loaded by the PMAPI routine pmLoadNameSpace; see pmLoadNameSpace(3).

The PMNS specification is a simple ASCII source file that can be edited easily. For reasons of efficiency, a binary format is also supported; the utility pmnscomp translates the ASCII source format into binary format; see pmnscomp(1).

PMNS Processing Framework

The PMNS specification is initially passed through cpp. This means the following facilities may be used in the specification:

  • C-style comments

  • #include directives

  • #define directives and macro substitution

  • conditional processing with #if, #endif, and so on

When cpp is executed, the “standard” include directories are the current directory and /var/pcp/pmns, where some standard macros and default specifications may be found.

PMNS Syntax

Every PMNS is tree structured. The paths to the leaf nodes are the performance metric names. The general syntax for a non-leaf node in PMNS is as follows:

pathname {
name	[pmid]
...
}

Here pathname is the full pathname from the root of the PMNS to this non-leaf node, with each component in the path separated by a period. The root node for the PMNS has the special name root, but the prefix string root. must be omitted from all other pathnames.

For example, refer to the PMNS shown in Figure 9-1. The correct pathname for the rightmost non-leaf node is cpu.util, not root.cpu.util.

Each component in the pathname must begin with an alphabetic character and be followed by zero or more alphanumeric characters or the underscore (_) character. For alphabetic characters in a component, uppercase and lowercase are significant.

Non-leaf nodes in the PMNS may be defined in any order desired. The descendent nodes are defined by the set of names, relative to the pathname of their parent non-leaf node. For descendent nodes, leaf nodes have a pmid specification, but non-leaf nodes do not.

Figure 9-1. Small Performance Metrics Name Space (PMNS)


The syntax for the pmid specification was chosen to help manage the allocation of Performance Metric IDs (PMIDs) across disjoint and autonomous domains of administration and implementation. Each pmid consists of three integers separated by colons, for example, 14:27:11. This is intended to mirror the implementation hierarchy of performance metrics. The first integer identifies the domain in which the performance metric lies. Within a domain, related metrics are often grouped into clusters. The second integer identifies the cluster, and the third integer, the metric within the cluster.

For complete documentation of the PMNS and associated utilities, see the pmns(4), pmnscomp(1), pmnsadd(1), pmnsdel(1), and pmnsmerge(1) reference pages.

Example PMNS Specification

The PMNS specification for Figure 9-1 is as follows:

/*
 * PMNS Specification for Figure 4-1
 */
#include <stdpmid>
root {
    network
    cpu
}
#define NETWORK 26
network {
    intrate    IRIX:NETWORK:1
    packetrate
}
network.packetrate {
    in     IRIX:NETWORK:35
    out    IRIX:NETWORK:36
}
#define CPU 10
cpu {
    syscallrate    IRIX:CPU:10
    util
}
#define USER 20
#define KERNEL 21
#define IDLE 22
cpu.util {
    user    IRIX:CPU:USER
    sys     IRIX:CPU:KERNEL
    idle    IRIX:CPU:IDLE
}

PMDA Development

The PCP is designed to be extensible at the collector site.

Application developers are encouraged to create new PMDAs to export performance metrics from the applications and service layers that are particularly relevant to a specific site, application suite, or processing environment.

These PMDAs use the routines of the libpcp_pmda library, which is discussed in detail by the Performance Co-Pilot Programmer's Guide.

Source code for several PMDAs (simple, trivial, and txmon) is provided in the pcp.sw.demo subsystem, and when installed all of the relevant files reside in directories (one per PMDA) below the /var/pcp/pmdas directory.

PCP Tool Development

The PCP is designed to be extensible at the monitor site.

Application developers are encouraged to create new PCP client applications to monitor or display performance metrics in a manner that is particularly relevant to a specific site, application suite, or processing environment.

Client applications use the routines of the PMAPI (performance metrics application programming interface) described in the Performance Co-Pilot Programmer's Guide.

Source code for a sample PMAPI client (pmclient) is provided in the pcp.sw.demo subsystem, and when installed all of the relevant file reside in /var/pcp/demos/pmclient.