Performance Co-Pilot (PCP) has been developed to be fully extensible. The following sections summarize the various facilities provided to allow you to extend and customize PCP for your site:
“PMDA Customization” describes the general process of installing and removing a PMDA at both a PCP collector and/or a PCP monitor host.
“Customizing the Summary PMDA” describes the procedure for customizing the summary PMDA to export derived metrics formed by aggregation of base PCP metrics from one or more collector hosts.
“PCP Tool Customization” describes the various options available for customizing and extending the basic PCP tools.
“PMNS Management” covers the concepts and tools provided for updating the PMNS (Performance Metrics Name Space).
“PMDA Development” details where to find further information to assist in the development of new PMDAs to extend the range of performance metrics available through the PCP infrastructure.
“PCP Tool Development” outlines how new tools may be developed to process performance data from the PCP infrastructure.
The generic procedures for installing and activating the optional PMDAs has been described in “Managing Optional PMDAs”. In some cases, these procedures prompt the user for information based upon the local system or network configuration, application deployment, or processing profile to customize the PMDA and hence the performance metrics it exports.
The summary PMDA is a special case that warrants further discussion.
The summary PMDA exports performance metrics derived from performance metrics made available by other PMDAs. It is described completely in the pmdasummary(1) reference page.
The summary PMDA consists of two processes:
| pmie process | A dedicated instance of the PCP pmie inference engine is used to periodically sample the base metrics and compute values for the derived metrics. This pmie process is launched with special command-line arguments by the main process. See the “Introduction to pmie” for a complete discussion of the pmie feature set. | |
| main process | The main process reads and buffers the values computed by the pmie process and makes them available to the Performance Metrics Collector Daemon (PMCD). |
All of the metrics exported by the summary PMDA have a singular instance and the values are “instantaneous”; the exported value is the correct value as of the last time the corresponding expression was evaluated by the pmie process.
The summary PMDA resides in the /usr/pcp/pmdas/summary directory and may be installed with a default configuration by following the steps described in the section “PMDA Installation”.
Alternatively, the summary PMDA may be customized to export your own derived performance metrics, by following the procedure described below:
Choose Performance Metric Name Space (PMNS) names for the new metrics. These must begin with summary and follow the rules described in pmns(4). For example, you might use summary.fs.cache_write and summary.fs.cache_hit.
Edit the file pmns in the /usr/pcp/pmdas/summary directory to add the new metric names in the format described in pmns(4). You must choose a unique Performance Metric Identifier (PMID) for each metric. In the pmns file, these appear as SYSSUMMARY:0:x. The value of x is arbitrary in the range 0 to 1023 and unique in this file. Refer to “PMNS Management” for a further explanation of the rules governing PMNS updates.
For example:
summary {
cpu
disk
netif
fs /*new*/
}
summary.fs {
cache_write SYSSUMMARY:0:10
cache_hit SYSSUMMARY:0:11
}
|
Use the local test PMNS root and validate that the PMNS changes are correct.
For example, enter this command:
pminfo -n root -m summary.fs |
You see output similar to the following:
summary.fs.cache_write PMID: 27.0.10 summary.fs.cache_hit PMID: 27.0.11 |
Edit the file /usr/pcp/pmdas/summary/expr.pmie to add new pmie expressions. If the name to the left of the assignment operator (=) is one of the PMNS names, then the pmie expression to the right will be evaluated and returned by the summary PMDA. The expression must return a numeric value; additional description of the pmie expression syntax may be found in “Specification Language for pmie”.
For example, consider this expression:
// filesystem buffer cache hit percentages
prefix = "irix.kernel.all.io"; // macro, not exported
summary.fs.cache_write =
100 - 100 * $prefix.bwrite / $prefix.lwrite;
summary.fs.cache_hit =
100 - 100 * $prefix.bread / $prefix.lread;
|
Run pmie in debug mode to verify that the expressions are being evaluated correctly, and the values make sense.
For example, enter this command:
pmie -t 2sec -v expr.pmie |
You see output similar to the following:
summary.fs.cache_write: ? summary.fs.cache_hit: ? summary.fs.cache_write: 45.83 summary.fs.cache_hit: 83.2 summary.fs.cache_write: 39.22 summary.fs.cache_hit: 84.51 |
Install the new PMDA.
From the /usr/pcp/pmdas/summary directory, use this command:
./Install |
You see the following output:
You need to choose an appropriate configuration for installation of the “summary” Performance Metrics Domain Agent (PMDA). collector collect performance statistics on this system monitor allow this system to monitor local and/or remote systems both collector and monitor configuration for this system Please enter c(ollector) or m(onitor) or b(oth) [b] both Interval between summary expression evaluation (seconds)? [10] 10 Updating the Performance Metrics Name Space... Installing pmchart view(s) ... Terminate PMDA if already installed ... Installing files .. Updating the PMCD control file, and notifying PMCD ... Wait 15 seconds for the agent to initialize ... Check summary metrics have appeared ... 8 metrics and 8 values |
Check the metrics.
For example, enter this command:
pmval -t 5sec -s 8 summary.fs.cache_write |
You see a response similar to the following:
metric: summary.fs.cache_write host: localhost semantics: instantaneous value units: none samples: 8 interval: 5.00 sec 63.60132158590308 62.71878646441073 62.71878646441073 58.73968492123031 58.73968492123031 65.33822758259046 65.33822758259046 72.6099706744868 |
Note that the values are being sampled here by pmval every 5 seconds, but pmie is passing only new values to the summary PMDA every 10 seconds. Both rates could be changed to suit the dynamics of your new metrics.
You may now create pmchart views, pmview scenes, and pmlogger configurations to monitor and archive your new performance metrics.
The PCP has been designed and implemented with a philosophy that embraces the notion of “toolkits” and encourages extensibility.
In most cases the PCP tools provide orthogonal services, based on external configuration files. It is the creation of new and modified configuration files that enables PCP users to customize tools quickly and meet the needs of the local environment, in many cases allowing personal preferences to be established for individual users on the same PCP monitor system.
The material in this section is intended to act as a checklist of pointers to detailed documentation found elsewhere in this guide, in the reference pages, and in the files that are made available as part of the PCP installation.
The PCP tool pmchart produces stripchart displays of performance metrics. Refer to “The pmchart Tool” for an extensive description of the capabilities of pmchart.
Customization is centered on PCP views that may be created interactively and saved via the View Save option in the Views menu.
When pmchart is loading a view, the following directories are searched:
| . | The current directory. | |
| $HOME/.pcp | Views for each user. | |
| /var/pcp/config/pmchart | |
The X11 application resources for pmchart are in /usr/lib/X11/app- defaults/PmChart, and these may be edited to customize the appearance of the display; the default update interval and other attributes are described in the pmchart(1) reference page.
The PCP archive logger is presented in Chapter 7, “Archive Logging,” and documented in the pmlogger(1) reference page.
The following global files and directories influence the behavior of pmlogger:
| /etc/config/pmlogger |
| ||
| /etc/config/pmlogger.options |
| ||
| /var/pcp/config/pmlogger/config.default |
| ||
| /var/pcp/config/pmlogger/config.view.* |
| ||
| /var/pcp/config/pmlogger/config.* |
| ||
| /var/pcp/config/pmlogger/control |
| ||
| /var/pcp/config/pmlogger/crontab |
| ||
| /var/adm/pcplog/somehost |
| ||
| /var/adm/pcplog/somehost/Latest |
|
The PCP inference engine is presented in Chapter 6, “Performance Metrics Inference Engine,” and documented in the pmie(1) reference page.
The following global files and directories influence the behavior of pmie:
| /var/pcp/config/pmie/control.master |
| |
| /var/pcp/demos/pmie/* |
| |
| /usr/pcp/bin/pmie/* |
| |
| /var/pcp/config/pmrules/* |
|
The PCP snapshot production facility is presented in “Making Snapshot Images From Archive Logs” and documented in the cron.pmsnap(1) reference page.
The following global files and directories influence the behavior of cron.pmsnap:
| /var/pcp/config/pmsnap/control |
| |
| /var/pcp/config/pmsnap/Summary |
| |
| /var/pcp/config/pmlogger/config.Summary |
| |
| /var/pcp/config/pmlogger/crontab |
| |
| /var/pcp/config/pmsnap/Summary.html |
|
The gadget specification language of pmgadgets supports the creation of arbitrary gadget layouts and bindings to hosts and performance metrics. See “The pmgadgets Command” and the pmgadgets(1) reference page.
The 3D scene specification language of pmview supports the creation of block layouts and bindings to hosts and performance metrics. See Chapter 5, “System Performance Visualization Tools,” and the pmview(1) reference page.
This section describes the syntax, semantics, and processing framework for the external specification of a Performance Metrics Name Space (PMNS) as it might be loaded by the PMAPI routine pmLoadNameSpace; see pmLoadNameSpace(3).
The PMNS specification is a simple ASCII source file that can be edited easily. For reasons of efficiency, a binary format is also supported; the utility pmnscomp translates the ASCII source format into binary format; see pmnscomp(1).
The PMNS specification is initially passed through cpp. This means the following facilities may be used in the specification:
C-style comments
#include directives
#define directives and macro substitution
conditional processing with #if, #endif, and so on
When cpp is executed, the “standard” include directories are the current directory and /var/pcp/pmns, where some standard macros and default specifications may be found.
Every PMNS is tree structured. The paths to the leaf nodes are the performance metric names. The general syntax for a non-leaf node in PMNS is as follows:
pathname {
name [pmid]
...
}
|
Here pathname is the full pathname from the root of the PMNS to this non-leaf node, with each component in the path separated by a period. The root node for the PMNS has the special name root, but the prefix string root. must be omitted from all other pathnames.
For example, refer to the PMNS shown in Figure 9-1. The correct pathname for the rightmost non-leaf node is cpu.util, not root.cpu.util.
Each component in the pathname must begin with an alphabetic character and be followed by zero or more alphanumeric characters or the underscore (_) character. For alphabetic characters in a component, uppercase and lowercase are significant.
Non-leaf nodes in the PMNS may be defined in any order desired. The descendent nodes are defined by the set of names, relative to the pathname of their parent non-leaf node. For descendent nodes, leaf nodes have a pmid specification, but non-leaf nodes do not.
The syntax for the pmid specification was chosen to help manage the allocation of Performance Metric IDs (PMIDs) across disjoint and autonomous domains of administration and implementation. Each pmid consists of three integers separated by colons, for example, 14:27:11. This is intended to mirror the implementation hierarchy of performance metrics. The first integer identifies the domain in which the performance metric lies. Within a domain, related metrics are often grouped into clusters. The second integer identifies the cluster, and the third integer, the metric within the cluster.
For complete documentation of the PMNS and associated utilities, see the pmns(4), pmnscomp(1), pmnsadd(1), pmnsdel(1), and pmnsmerge(1) reference pages.
The PMNS specification for Figure 9-1 is as follows:
/*
* PMNS Specification for Figure 4-1
*/
#include <stdpmid>
root {
network
cpu
}
#define NETWORK 26
network {
intrate IRIX:NETWORK:1
packetrate
}
network.packetrate {
in IRIX:NETWORK:35
out IRIX:NETWORK:36
}
#define CPU 10
cpu {
syscallrate IRIX:CPU:10
util
}
#define USER 20
#define KERNEL 21
#define IDLE 22
cpu.util {
user IRIX:CPU:USER
sys IRIX:CPU:KERNEL
idle IRIX:CPU:IDLE
}
|
The PCP is designed to be extensible at the collector site.
Application developers are encouraged to create new PMDAs to export performance metrics from the applications and service layers that are particularly relevant to a specific site, application suite, or processing environment.
These PMDAs use the routines of the libpcp_pmda library, which is discussed in detail by the Performance Co-Pilot Programmer's Guide.
Source code for several PMDAs (simple, trivial, and txmon) is provided in the pcp.sw.demo subsystem, and when installed all of the relevant files reside in directories (one per PMDA) below the /var/pcp/pmdas directory.
The PCP is designed to be extensible at the monitor site.
Application developers are encouraged to create new PCP client applications to monitor or display performance metrics in a manner that is particularly relevant to a specific site, application suite, or processing environment.
Client applications use the routines of the PMAPI (performance metrics application programming interface) described in the Performance Co-Pilot Programmer's Guide.
Source code for a sample PMAPI client (pmclient) is provided in the pcp.sw.demo subsystem, and when installed all of the relevant file reside in /var/pcp/demos/pmclient.