Chapter 9. Customizing and Extending PCP Services

Performance Co-Pilot (PCP) has been developed to be fully extensible. The following sections summarize the various facilities provided to allow you to extend and customize PCP for your site:

PMDA Customization

The generic procedures for installing and activating the optional PMDAs have been described in “Managing Optional PMDAs” in Chapter 2. In some cases, these procedures prompt the user for information based upon the local system or network configuration, application deployment, or processing profile to customize the PMDA and hence the performance metrics it exports.

The summary PMDA is a special case that warrants further discussion.

Customizing the Summary PMDA

The summary PMDA exports performance metrics derived from performance metrics made available by other PMDAs. It is described completely in the pmdasummary(1) man page.

The summary PMDA consists of two processes:

pmie process 

Periodically samples the base metrics and compute values for the derived metrics. This dedicated instance of the PCP pmie inference engine is launched with special command line arguments by the main process. See “Introduction to pmie” in Chapter 6, for a complete discussion of the pmie feature set.

main process 

Reads and buffers the values computed by the pmie process and makes them available to the Performance Metrics Collection Daemon (PMCD).

All of the metrics exported by the summary PMDA have a singular instance and the values are instantaneous; the exported value is the correct value as of the last time the corresponding expression was evaluated by the pmie process.

The summary PMDA resides in the /usr/pcp/pmdas/summary directory and may be installed with a default configuration by following the steps described in “PMDA Installation” in Chapter 2.

Alternatively, you may customize the summary PMDA to export your own derived performance metrics by following the steps in Procedure 9-1:

Procedure 9-1. Customizing the Summary PMDA

  1. Check that the symbolic constant SYSSUMMARY is defined in the /var/pcp/pmns/stdpmid file. If it is not, perform the postinstall update of this file, as superuser:

    # cd /var/pcp/pmns
            # ./Make.stdpmid

  2. Choose Performance Metric Name Space (PMNS) names for the new metrics. These must begin with summary and follow the rules described in the pmns(4) man page. For example, you might use summary.fs.cache_write and summary.fs.cache_hit.

  3. Edit the pmns file in the /usr/pcp/pmdas/summary directory to add the new metric names in the format described in the pmns(4) man page. You must choose a unique Performance Metric Identifier (PMID) for each metric. In the pmns file, these appear as SYSSUMMARY:0:x. The value of x is arbitrary in the range 0 to 1023 and unique in this file. Refer to “PMNS Management”, for a further explanation of the rules governing PMNS updates.

    For example:

    summary {
              cpu
              disk
              netif
              fs              /*new*/
    }
    summary.fs {
              cache_write     SYSSUMMARY:0:10
              cache_hit       SYSSUMMARY:0:11
    }

  4. Use the local test PMNS root and validate that the PMNS changes are correct.

    For example, enter this command:

    pminfo -n root -m summary.fs

    You see output similar to the following:

    summary.fs.cache_write PMID: 27.0.10
    summary.fs.cache_hit PMID: 27.0.11

  5. Edit the /usr/pcp/pmdas/summary/expr.pmie file to add new pmie expressions. If the name to the left of the assignment operator (=) is one of the PMNS names, then the pmie expression to the right will be evaluated and returned by the summary PMDA. The expression must return a numeric value. Additional description of the pmie expression syntax may be found in “Specification Language for pmie” in Chapter 6.

    For example, consider this expression:

    // filesystem buffer cache hit percentages
    prefix = "kernel.all.io";          // macro, not exported
    summary.fs.cache_write =
                    100 - 100 * $prefix.bwrite / $prefix.lwrite;
    summary.fs.cache_hit =
                    100 - 100 * $prefix.bread / $prefix.lread;

  6. Run pmie in debug mode to verify that the expressions are being evaluated correctly, and the values make sense.

    For example, enter this command:

    pmie -t 2sec -v expr.pmie

    You see output similar to the following:

    summary.fs.cache_write:      ?
    summary.fs.cache_hit:      ?
    summary.fs.cache_write:  45.83
    summary.fs.cache_hit:   83.2
    summary.fs.cache_write:  39.22
    summary.fs.cache_hit:  84.51

  7. Install the new PMDA.

    From the /usr/pcp/pmdas/summary directory, use this command:

    ./Install

    You see the following output:

    You need to choose an appropriate configuration for installation of
    the “summary” Performance Metrics Domain Agent (PMDA).
    
    collector collect performance statistics on this system
    monitor   allow this system to monitor local and/or remote systems
    both      collector and monitor configuration for this system
    
    Please enter c(ollector) or m(onitor) or b(oth) [b] both
    Interval between summary expression evaluation (seconds)? [10] 10
    Updating the Performance Metrics Name Space...
    Installing pmchart view(s) ...
    Terminate PMDA if already installed ...
    Installing files ..
    Updating the PMCD control file, and notifying PMCD ...
    Wait 15 seconds for the agent to initialize ...
    Check summary metrics have appeared ... 8 metrics and 8 values

  8. Check the metrics.

    For example, enter this command:

    pmval -t 5sec -s 8 summary.fs.cache_write

    You see a response similar to the following:

    metric:    summary.fs.cache_write
    host:      localhost
    semantics: instantaneous value
    units:     none
    samples:   8
    interval:  5.00 sec
    63.60132158590308
    62.71878646441073
    62.71878646441073
    58.73968492123031
    58.73968492123031
    65.33822758259046
    65.33822758259046
    72.6099706744868

    Note that the values are being sampled here by pmval every 5 seconds, but pmie is passing only new values to the summary PMDA every 10 seconds. Both rates could be changed to suit the dynamics of your new metrics.

  9. You may now create pmchart views, pmview scenes, and pmlogger configurations to monitor and archive your new performance metrics.

    PCP Tool Customization

    Performance Co-Pilot (PCP) has been designed and implemented with a philosophy that embraces the notion of toolkits and encourages extensibility.

    In most cases, the PCP tools provide orthogonal services, based on external configuration files. It is the creation of new and modified configuration files that enables PCP users to customize tools quickly and meet the needs of the local environment, in many cases allowing personal preferences to be established for individual users on the same PCP monitor system.

    The material in this section is intended to act as a checklist of pointers to detailed documentation found elsewhere in this guide, in the man pages, and in the files that are made available as part of the PCP installation.

    Stripchart Customization

    The PCP tool pmchart produces stripchart displays of performance metrics. Refer to “The pmchart Tool” in Chapter 4, for an extensive description of the capabilities of pmchart.

    Customization is centered on PCP views that may be created interactively and saved via the Save View option in the File menu.

    When pmchart is loading a view, the following directories are searched:

    .

    The current directory.

    $HOME/.pcp

    Views for each user.

    /var/pcp/config/pmchart

    The system-wide catalog of views. Any view installed here becomes visible to every pmchart user.

    The X11 application resources for pmchart are in /usr/lib/X11/app-defaults/PmChart, and these may be edited to customize the appearance of the display. The default update interval and other attributes are described in the pmchart(1) man page.

    Archive Logging Customization

    The PCP archive logger is presented in Chapter 7, “Archive Logging”, and documented in the pmlogger(1) man page.

    The following global files and directories influence the behavior of pmlogger:

    /etc/config/pmlogger
     

    Enable/disable state for the primary logger facility using this command:

    chkconfig pmlogger on

    /etc/config/pmlogger.options
     

    Command line options passed to the primary logger if it is launched from /etc/init.d/pcp.

    /var/pcp/config/pmlogger/config.default
     

    The default pmlogger configuration file that is used for the primary logger when this facility is enabled.

    /var/pcp/config/pmlogger/config.view.*
     

    Every pmchart view also provides a pmlogger configuration file that includes each of the performance metrics used in the view, for example, /var/pcp/config/pmlogger/config.LoadAvg for the LoadAvg view.

    /var/pcp/config/pmlogger/config.*
     

    Every PCP tool with a fixed group of performance metrics contributes a pmlogger configuration file that includes each of the performance metrics used in the tool, for example, /var/pcp/config/pmlogger/config.dkvis for dkvis.

    /var/pcp/config/pmlogger/control
     

    Defines which PCP collector hosts require pmlogger to be launched on the local host, where the configuration file comes from, where the archive log files should be created, and pmlogger startup options.

    /var/pcp/config/pmlogger/crontab
     

    Prototype crontab entries that may be merged with the crontab entries for root to schedule the periodic execution of the archive log management scripts, for example, pmlogger.daily.

    /var/adm/pcplog/somehost
     

    The default behavior of the archive log management scripts create archive log files for the host somehost in this directory.

    /var/adm/pcplog/somehost/Latest
     

    A PCP archive folio for the most recent archive for the host somehost. This folio is created and maintained by the cron-driven periodic archive log management scripts, for example, pmlogger.check. Archive folios may be processed with the pmafm tool.

    Inference Engine Customization

    The PCP inference engine is presented in Chapter 6, “Performance Metrics Inference Engine”, and documented in the pmie(1) man page.

    The following global files and directories influence the behavior of pmie:

    /etc/config/pmie
     

    Controls the pmie daemon facility. Enable using this command:

    chkconfig pmie on

    /var/pcp/demos/pmie/*
     

    Example pmie rules that may be used as a basis for developing local rules.

    /var/pcp/config/pmie/config.default
     

    The default pmie configuration file that is used when the pmie daemon facility is enabled.

    /var/pcp/config/pmieconf/*/*
     

    Each pmieconf rule definition can be found below one of these subdirectories.

    /var/pcp/config/pmie/control
     

    Defines which PCP collector hosts require a daemon pmie to be launched on the local host, where the configuration file comes from, where the pmie log file should be created, and pmie startup options.

    /var/pcp/config/pmlogger/crontab
     

    Prototype crontab entries that may be merged with the crontab entries for root to schedule the periodic execution of the pmie_check script, for verifying that pmie instances are running.

    /var/adm/pmielog/
     

    The default behavior of the /etc/init.d/pmie startup scripts create pmie log files for the host in this directory.

     

    The pmcd PMDA exports information about executing pmie instances and their progress in terms of rule evaluations and action execution rates.

    pmie_check 

    This command is similar to the pmlogger support script, pmlogger_check.

    /etc/init.d/pmie 

    This control file supports the starting and stopping of multiple pmie instances that are monitoring one or more hosts.

    /var/tmp/pmie 

    The statistics that pmie gathers are maintained in binary data structure files. These files are in the /var/tmp/pmie directory.

    pmcd.pmie metrics 

    If pmie is running on a system with a PCP collector deployment, the pmcd PMDA exports these metrics via the pmcd.pmie group of metrics.

    Snapshot Customization

    The PCP snapshot production facility is presented in “Making Snapshot Images from Archive Logs” in Chapter 7, and documented in the pmsnap(1) man page.

    The following global files and directories influence the behavior of pmsnap:

    /var/pcp/config/pmsnap/control
     

    Defines how to produce a snapshot, including the output filename, the PCP archive folio name to be used as input, the pmchart configuration file, and command line arguments to pmchart

    /var/pcp/config/pmsnap/Summary
     

    A pmchart configuration file to produce a sample summary snapshot in conjunction with pmsnap

    /var/pcp/config/pmlogger/config.Summary
     

    A pmlogger configuration file that can produce an archive containing performance metrics required by the sample summary snapshot

    /var/pcp/config/pmlogger/crontab
     

    Prototype crontab entries that may be merged with the crontab entries for root schedule the periodic execution of the archive log management scripts, for example, pmsnap

    /var/pcp/config/pmsnap/Summary.html
     

    An example HTML page suitable for publishing images from the pmsnap examples via a Web server

    Icon Control Panel Customization

    The gadget specification language of pmgadgets supports the creation of arbitrary gadget layouts and bindings to hosts and performance metrics. See “The pmgadgets Command” in Chapter 4, and the pmgadgets(1) man page.

    3D Visualization Customization

    The 3D scene specification language of pmview supports the creation of block layouts and bindings to hosts and performance metrics. See Chapter 5, “System Performance Visualization Tools” and the pmview(1) man page.

    PMNS Management

    This section describes the syntax, semantics, and processing framework for the external specification of a Performance Metrics Name Space (PMNS) as it might be loaded by the PMAPI routine pmLoadNameSpace; see the pmLoadNameSpace(3) man page.

    The PMNS specification is a simple ASCII source file that can be edited easily. For reasons of efficiency, a binary format is also supported; the utility pmnscomp translates the ASCII source format into binary format; see the pmnscomp(1) man page.

    PMNS Processing Framework

    The PMNS specification is initially passed through cpp. This means the following facilities may be used in the specification:

    • C-style comments

    • #include directives

    • #define directives and macro substitution

    • Conditional processing with #if, #endif, and so on

    When cpp is executed, the standard include directories are the current directory and /var/pcp/pmns, where some standard macros and default specifications may be found.

    PMNS Syntax

    Every PMNS is tree structured. The paths to the leaf nodes are the performance metric names. The general syntax for a non-leaf node in PMNS is as follows:

    pathname {
    name	[pmid]
    ...
    }

    Here pathname is the full pathname from the root of the PMNS to this non-leaf node, with each component in the path separated by a period. The root node for the PMNS has the special name root, but the prefix string root. must be omitted from all other pathnames.

    For example, refer to the PMNS shown in Figure 9-1. The correct pathname for the rightmost non-leaf node is cpu.util, not root.cpu.util.

    Figure 9-1. Small Performance Metrics Name Space (PMNS)

    Each component in the pathname must begin with an alphabetic character and be followed by zero or more alphanumeric characters or the underscore (_) character. For alphabetic characters in a component, uppercase and lowercase are significant.

    Non-leaf nodes in the PMNS may be defined in any order desired. The descendent nodes are defined by the set of names, relative to the pathname of their parent non-leaf node. For descendent nodes, leaf nodes have a pmid specification, but non-leaf nodes do not.

    The syntax for the pmid specification was chosen to help manage the allocation of Performance Metric IDs (PMIDs) across disjoint and autonomous domains of administration and implementation. Each pmid consists of three integers separated by colons, for example, 14:27:11. This is intended to mirror the implementation hierarchy of performance metrics. The first integer identifies the domain in which the performance metric lies. Within a domain, related metrics are often grouped into clusters. The second integer identifies the cluster, and the third integer, the metric within the cluster.

    The PMNS specification for Figure 9-1 is shown in Example 9-1:

    Example 9-1. PMNS Specification

    /*
        * PMNS Specification
        */
    #include <stdpmid>
    root {
           network
           cpu
    }
    #define NETWORK 26
    network {
           intrate    `:NETWORK:1
           packetrate
    }
    network.packetrate {
           in     IRIX:NETWORK:35
           out    IRIX:NETWORK:36
    }
    #define CPU 10
    cpu {
           syscallrate    IRIX:CPU:10
           util
    }
    #define USER 20
    #define KERNEL 21
    #define IDLE 22
    cpu.util {
           user    IRIX:CPU:USER
           sys     IRIX:CPU:KERNEL
           idle    IRIX:CPU:IDLE
    }

    For complete documentation of the PMNS and associated utilities, see the pmns(4), pmnscomp(1), pmnsadd(1), pmnsdel(1), and pmnsmerge(1) man pages.

    PMDA Development

    Performance Co-Pilot (PCP) is designed to be extensible at the collector site.

    Application developers are encouraged to create new PMDAs to export performance metrics from the applications and service layers that are particularly relevant to a specific site, application suite, or processing environment.

    These PMDAs use the routines of the libpcp_pmda library, which is discussed in detail by the Performance Co-Pilot Programmer's Guide.

    Source code for several PMDAs (simple, trivial, and txmon) is provided in the pcp.sw.demo subsystem. When it is installed, all of the relevant files reside in directories (one per PMDA) below the /var/pcp/pmdas directory.

    PCP Tool Development

    Performance Co-Pilot (PCP) is designed to be extensible at the monitor site.

    Application developers are encouraged to create new PCP client applications to monitor or display performance metrics in a manner that is particularly relevant to a specific site, application suite, or processing environment.

    Client applications use the routines of the PMAPI (performance metrics application programming interface) described in the Performance Co-Pilot Programmer's Guide.

    Source code for a sample PMAPI client (pmclient) is provided in the pcp.sw.demo subsystem, and when installed all of the relevant files reside in /var/pcp/demos/pmclient.