The pmchart utility supports interactive selection and plotting of trends over time for arbitrarily selected performance metrics from one or more hosts and one or more domains of performance metrics. First, you enter the following command:

pmchart 

You then see the Performance Co-Pilot Chart window shown in Figure 4-1.

Figure 4-1. pmchart Performance Co-Pilot Chart Window

Normally, pmchart operates in live mode where performance metrics are fetched in real time and plotted against a time axis. The user can choose performance metrics and monitor the current values for these metrics from any host that is accessible on the network and has the PMCD server running.

When launched with the -a command line option, pmchart can also replay PCP archive logs of performance metrics created by pmlogger.

The man page for pmchart explains how to configure charts based on performance metrics, using either the Open View option of the File menu or the New Plot option of the File menu. Once charts have been configured and applied, the charts are placed in an expanded Performance Co-Pilot Chart window, as shown in Figure 4-2.

Figure 4-2. Two Charts and Metrics from Three Hosts in pmchart

All metrics in the Performance Metrics Name Space (PMNS) with numeric value semantics can be graphed. By default, pmchart initially allows the user to select metrics to be plotted from the local host. However, the graphical user interface allows other hosts or archives to be chosen at any time as alternate sources of performance metrics and all metrics (independent of their source) are plotted on a common time axis.

For horizontal lines at major tick marks, see “Displaying Horizontal Lines”.

The -h command line option nominates an alternate default host to be used in preference to the local host.

The -a command line option may be used to start pmchart processing performance metrics from one or more PCP archive logs. The first named archive becomes the default source of performance metrics. This mode is particularly useful for retrospective comparisons and for postmortem analysis of performance problems, where a remote system is not directly accessible or a performance analyst is not available on site.

The pmchart utility examines the semantics of selected metrics, and where sensible, uses the metadata provided by the Performance Metrics Collection Subsystem (PMCS) to convert fetched metric values to a rate before plotting. In the case where different metrics are plotted in the same chart (for example, against a common Y-axis), the metrics must have the same dimension (taking into account any automatic rate conversion), but pmchart may scale metric values where necessary, to produce comparable values with common units and scale.

When replaying archive logs, the user may interactively control the current replay time, direction of replay, and replay rate, using the PCP time control dialog, as described in the “Time Duration and Control” in Chapter 3.

pmchart Select Performance View

A view in pmchart is a predefined collection of charts, typically constructed to display some common performance scenario. Default views are included in the PCP distribution, others are part of the various PCP add-on products, and others may be created by the pmchart end user. The Open View... option in the File menu launches a Select Performance View dialog box similar to Figure 4-3.

Figure 4-3. pmchart Select Performance View Dialog

You may use this dialog to select one of the available views. The default PCP views include the following:

BufferCache		Cumulative amount of data read and written between system buffers and user memory or block devices.
CPU		Processor utilization (user, system, memory break, interrupt, I/O wait, and idle time) aggregated over all CPUs.
NUMAlinks		Usage of NUMAlink node connectors, if this hardware is present.
Disk		Cumulative number of read and write transfers for all disk devices.
DiskCntrls		Cumulative number of read and write transfers for all drives attached to each disk controller on the system.
FileSystem		Percentage of each filesystem in use (percent full).
LoadAvg		System load averaged over intervals of 1, 5, and 15 minutes.
Memory		Memory used by the kernel, filesystems, user processes, and free space.
NetBytes		Network interface activity--octets transmitted on various interfaces.
NetConnDrop		TCP drops, connection drops, timeout drops, and TCP accepts.
NetPackets		Rate of TCP and UDP packets received and sent.
NetTCPcongestion		TCP packets retransmitted, retransmit timeouts, and TCP packets sent.
NFS2, NFS3		Client and server NFS operation rates.
Overview		Composite charts of CPU, LoadAvg, Memory, Disk, and NetBytes.
Paging		Page-in and page-out rates from the virtual memory subsystem.
PMCD		Message rates and CPU time used by PMCD or associated PMDAs.
Swap		System swap space allocated, reserved, and unused.
Syscalls		Rate of exec, fork, read, write, and total system calls.

You can create your own custom views using the metric selection facilities, and save your views for later using the Save View... option of the File menu.

% pmchart -xrm "PmChart*xrtYGridUseDefault: True" 

PmChart*xrtYGridUseDefault: True

pmchart Metric Selection

The pmchart Metric Selection window, shown in Figure 4-4, allows interactive navigation of the Performance Metrics Name Space (PMNS) to create new chart configurations.

Figure 4-4. pmchart Metric Selection Dialog

You can choose metrics, display information about metrics, change the current host or archive, select metric instances, and plot metric values on a common time axis. You bring up this window by choosing New Plot... from the File menu of pmchart.

Metric selection proceeds by navigating through the tree-structured PMNS. If you enter a partial metric specification in the Path field in the Metric Selection dialog, you can avoid having to navigate through the PMNS for the metrics you need. For example, if you enter network.interface, the window changes dynamically, as shown in Figure 4-5.

Figure 4-5. Further Metric Selection

You can continue the selection process by choosing non-leaf nodes from the Nodes list, and finally a leaf node from the Metrics list. At this stage, the Path corresponds to a leaf node in the PMNS, as shown in Figure 4-6.

Figure 4-6. Selecting a Leaf Node in the PMNS (Performance Metric)

Once a metric has been selected, the Info button in the Metric Selection dialog launches the Metric Information dialog, as shown in Figure 4-7.

Figure 4-7. Metric Information Dialog

This dialog displays the name, unit, and semantics for the currently selected metric, along with the verbose help text that describes the metric, and optionally a description of the underlying instance domain.

Finally, you may have to select from several instances of a metric. In the example shown in Figure 4-7, you wish to monitor the input packet rate for some network interface(s). For the current source of performance metrics, there are two network interfaces configured. You must select one or more instances, as shown in Figure 4-8.

Figure 4-8. Selecting a Metric Instance

You can select multiple instances either by clicking and dragging up and down the list with the left mouse button, or by selecting the first instance and then using the Shift key (or Ctrl key) with the left mouse button to select one or more other instances.

Creating a PCP Archive from a pmchart Session

From the File menu of pmchart when running in live mode, the Record (Stop Recording) option may be used to start (or stop) the creation of a PCP archive log. The archive log is created using pmlogger and includes the update interval and all of the performance metrics in the current pmchart configuration when recording begins.

---

Note: Any changes made to the pmchart configuration after recording has been started will not be reflected in the archive log. For these to take effect, the recording must be stopped and restarted (thereby creating a second PCP archive log).

---

When recording is started, a File Chooser dialog is launched, and the user must provide the name of a new file to be used as the PCP archive folio for the new archive (see “PCP Archive Folios” in Chapter 7). The recording session produces multiple files in the same directory as the archive folio.

If necessary, pmchart creates directories on the path to the named archive folio.

It is often convenient to maintain one directory for each new folio, or else one directory for each group of folios related by collector host(s), service type, or chart selection.

When recording is active, a small red indicator appears in the time control button, as shown at the bottom left of Figure 4-9.

Figure 4-9. pmchart Display When Recording

If you choose File > Stop Recording, logging stops immediately. The red light in the lower left turns gray.

To start recording again, chose File > Record and specify a new archive folio name.

If you exit pmchart by choosing File > Quit, an Archive Recording Session-pmchart dialog similar to that shown in Figure 4-10 appears to remind you where the archive folio was created, and to confirm that recording should be terminated.

Figure 4-10. Archive Recording Session-pmchart Dialog

If you select Yes, recording stops immediately.

If you select No, recording continues. This is a useful way to continue archive logging without keeping pmchart active.

PmChart*xrtForegroundColor: "green" 
PmChart*xrtBackgroundColor: "black" 
PmChart*xrtGraphForegroundColor: "rgb:00/b0/00" 
PmChart*xrtGraphBackgroundColor: "black" 
PmChart*xrtHeaderForegroundColor: "green" 
PmChart*xrtHeaderBackgroundColor: "black" 
PmChart*pmDefaultColors: rgb:ff/ff/00 rgb:00/ff/00 rgb:00/00/ff \ 
                           rgb:ff/ff/00 rgb:00/ff/ff rgb:ff/00/ff 

xrdb -merge $HOME/.xrdb 

file:/var/pcp/Tutorial/pmchart.html

The pmgadgets tool creates a small window containing a collection of graphical gadgets driven by performance metrics supplied by the PCP framework. Any numeric metric supported by PCP can be displayed.

---

Note: In the current PCP release, pmgadgets is constrained to process performance metrics from real-time sources (and not PCP archive logs), although metrics from several different hosts may be displayed simultaneously in the same window.

---

The layout of the gadgets and the performance metrics that lie behind them are specified in a configuration file, and pmgadgets is typically run on an existing configuration file or in conjunction with an application that automatically generates a configuration file. For example, pmgsys generates a configuration file for various IRIX performance metrics and feeds it to pmgadgets. The resulting display depends on the host configuration, but the display shown in Figure 4-11 is representative of a system with four CPUs, eleven disks on three controllers, and four network interfaces.

Figure 4-11. Representative pmgadgets Display Using pmgsys

Other pmgadget front end tools such as pmgcluster, pmgevctr, pmgcisco, and pmgshping are not described in this chapter. For information about these tools, see the pmgcluster(1), pmgevctr(1), pmgcisco(1), and pmgshping(1) man pages.

The pmgadgets tool displays much the same information as pmchart, but more compactly, and with less historical information.

The pmgadgets specification language provides the ability to define the following gadgets and components:

Each visible gadget must be assigned a Cartesian position in the pmgadgets display.

By way of an example, the pmgadgets specification shown in Example 4-1 includes CPU, disk, and load average information from two hosts, and produces a customized pmgadgets display like the one shown in Figure 4-12.

_colourlist cpuColours (blue3 red3 yellow3 cyan3 green3)
_legend diskLegend (
       _default  green3
       15        yellow
       40        orange
       75        red
)
# host moomba
_label 70 12 "moomba"
_multibar 5 5 30 6
       _metrics (
           moomba:kernel.all.cpu.user
           moomba:kernel.all.cpu.sys
           moomba:kernel.all.cpu.intr
           moomba:kernel.all.cpu.wait.total
           moomba:kernel.all.cpu.idle
       )
       _maximum 0.0
       _colourlist cpuColours
_bargraph 40 5 25 20
       _metric moomba:kernel.all.load["1 minute"]
       _max 1.0
_led 12 16 6 6
       _metric moomba:disk.all.read _legend diskLegend
_led 25 16 6 6
       _metric moomba:disk.all.write _legend diskLegend
# host gonzo
_label 70 39 "gonzo"
_multibar 5 32 30 6
       _metrics (
           gonzo:kernel.all.cpu.user
           gonzo:kernel.all.cpu.sys
           gonzo:kernel.all.cpu.intr
           gonzo:kernel.all.cpu.wait.total
           gonzo:kernel.all.cpu.idle
       )
       _maximum 0.0
       _colourlist cpuColours
_bargraph 40 32 25 20
       _metric gonzo:kernel.all.load["1 minute"]
       _max 1.0
_led 12 43 6 6
       _metric gonzo:disk.all.read _legend diskLegend
_led 25 43 6 6
       _metric gonzo:disk.all.write _legend diskLegend

Figure 4-12. Customized pmgadgets Display

In addition to the drill-down capabilities of pmgadgets, positioning the cursor over a gadget and entering a space character causes an information dialog to be exposed. This dialog tracks the current values of the performance metrics that are associated with the gadget as illustrated by the pmgadgets information dialog in Figure 4-13.

Figure 4-13. pmgadgets Dialog

The pmgadgets(1) man page provides a complete description of the gadget specification language and the user interface controls of pmgadgets.

The pmkstat command provides a periodic, one-line summary of system performance. This command is intended to monitor system performance at the highest level, after which other tools may be used for examining subsystems to observe potential performance problems in greater detail. After entering the pmkstat command, you see output similar to the following, with successive lines appearing periodically:

pmkstat
# hostname load avg: 0.26, interval: 5 sec, Thu Jan 19 12:30:13 1995
runq    | memory    |      system      | disks|   cpu 
mem swp | free page | scall ctxsw  intr| rd wr|usr sys idl wt
0   0     16268 0     64    19     2396  0  0  0   1   99  0
0   0     16264 0     142   45     2605  0  8  0   2   97  0
0   0     16268 0     308   62     2532  0  1  1   1   98  0
0   0     16268 0     423   88     2643  0  0  1   1   97  0

An additional line of output is added every five seconds. The update interval may be varied using the -t interval option.

The output from pmkstat is directed to standard output, and the columns in the report are interpreted as follows:

As with most PCP utilities, real-time metric, and archive logs are interchangeable.

For example, the following command uses the PCP archive log foo and the timezone of the host (tokyo) from which performance metrics in the archive were collected:

pmkstat -a foo -z
Note: timezone set to local timezone of host "tokyo"
# tokyo load avg: 1.06, interval: 5 sec, Thu Feb  2 08:42:55 1995
 runq  |     memory |     system       |  disks  |      cpu
mem swp|   free page| scall ctxsw  intr|  rd   wr|usr sys idl  wt
  0   0    4316    0    195    64  2242   32   21   0   3   8  89
  0   0    3976    0    279    86  2143   50   17   0   5   8  87
  1   0    3448    0    186    63  2304   35   14   0   4   9  87
  0   0    4364    0    254    81  2385   35    0   0   4   9  87
  0   0    3696    0    266    92  2374   41    0   0   3   9  88
  0   0    2668   42    237    81  2400   44    2   1   4   7  89
  0   0    4644  100    206    68  2590   25    1   0   3   5  91
  0   0    5384    0    174    63  2296   32   22   0   2   8  89
  0   0    4736    0    189    65  2197   31   28   0   3   8  89
pmFetch: End of PCP archive log

For complete information on pmkstat usage and command line options, see the pmkstat(1) man page.

The pmdumptext command displays performance metrics in ASCII tables, suitable for export into databases or report generators. It is a flexible command. For example, the following command provides continuous memory statistics on a host named serv:

pmdumptext -imu -h serv -f `%H:%M:%S' mem.util
Metric        kernel  fs_ctl  _dirty  _clean    free    user 
      Units             b       b       b       b       b       b 
20:14:28        99.14M   6.03M   0.85M  98.42M   0.17G   0.16G 

See the pmdumptext(1) man page for more information.

The pmval command dumps the current values for the named performance metrics. For example, the following command reports the value of performance metric proc.nprocs once per second (by default), and produces output similar to this:

pmval proc.nprocs
metric:    proc.nprocs
host:      localhost
semantics: instantaneous value
units:     none
samples:   indefinite
interval:  1.00 sec
            73
            72
            70
            75
            75

In this example, the number of running processes was reported once per second.

Where the semantics of the underlying performance metrics indicate that it would be sensible, pmval reports the rate of change or resource utilization.

For example, the following command reports idle processor utilization for each of four CPUs on the remote host moomba, each five seconds apart, producing output of this form:

pmval -h moomba -t 5sec -s 4 kernel.percpu.cpu.idle
metric:    kernel.percpu.cpu.idle
host:      moomba
semantics: cumulative counter (converting to rate)
units:     millisec (converting to time utilization)
samples:   4
interval:  5.00 sec
        cpu0        cpu1        cpu2        cpu3 
        0.8193      0.7933      0.4587      0.8193 
        0.7203      0.5822      0.8563      0.7303 
        0.6100      0.6360      0.7820      0.7960 
        0.8276      0.7037      0.6357      0.6997 

Similarly, the following command reports disk I/O read rate every minute for just the disk /dev/dsk/dks0d1, and produces output similar to the following:

pmval -t 1min -i dks0d1 disk.dev.read
metric:    disk.dev.read
host:      localhost
semantics: cumulative counter (converting to rate)
units:     count (converting to count / sec)
samples:   indefinite
interval:  60.00 sec
        dks0d1 
         33.67 
         48.71 
         52.33 
         11.33 
         2.333 

The -r flag may be used to suppress the rate calculation (for metrics with counter semantics) and display the raw values of the metrics.

When used in conjunction with a PCP archive, the -g option may be used to associate a PCP time control dialog (see “Time Duration and Control” in Chapter 3) with the execution of pmval to support temporal navigation within the archive. In the example below, manipulation of the time within the archive is achieved by the exchange of time control messages between pmval and pmtime.

pmval -g -a /var/adm/pcplog/myserver/960801

The pmval command is documented by the pmval(1) man page, and annotated examples of the use of pmval are in the PCP Tutorial.

The pmem command reports per-process memory usage statistics within the PCP framework.

Both virtual size and prorated physical memory usage are reported. The virtual memory usage statistics represent the total virtual size of each process, irrespective of how many pages are valid (resident). Prorated physical memory statistics indicate real memory usage (only valid pages are counted) and are prorated on a per-page basis between all processes that reference each page. Thus the prorated physical memory counts reflect the real memory demands for individual processes in the context of the current process mix.

The output of pmem can be very large. Here is an abbreviated example of pmem output:

Host: gonzo Configured: 65536  Free:18380  Tue Jul 9 16:45:08 1996
     pid  ppid   user  vtxt  ptxt  vdat  pdat vshm pshm command
       1     0   root   232   144    84    76    0    0 /etc/init
     832   827   root  3204  1013  5796  3096    0    0 /usr/bin/X11/Xsg
     221     1   root  1424    54   156   84     0    0 /usr/lib/saf/sad
     838   827   root  2948    36   268    75    0    0 /usr/bin/X11/xdm
      86     1   root  1264    32   144    76    0    0 /usr/etc/syslogd
     182     1   root  1476   129   596   387    0    0 /usr/etc/rpcbind
     827     1   root  2948    13   252    22    0    0 /usr/bin/X11/xdm
     172     1   root  1276    52   148   100    0    0 /usr/etc/routed
Total     vtxt  ptxt   vdat  pdat  vshm  pshm  77 user processes
             121M        36256           0        =  157M virtual
                   13982        20194           0  = 34176 physical

The columns report the following information:

For complete information on pmem usage and command line options, see the pmem(1) man page.

The pminfo command displays various types of information about performance metrics available through the Performance Co-Pilot (PCP) facilities.

The -T option is extremely useful; it provides help text about performance metrics:

pminfo -T mem.util.fs_dirty
mem.util.fs_dirty
Help:
The amount of memory in Kbytes that is holding file system data.

The -t option displays the one-line help text associated with the selected metrics. The -T option prints more verbose help text.

Without any options, pminfo verifies that the specified metrics exist in the Name Space, and echoes those names. Metrics may be specified as arguments to pminfo using their full metric names. For example, this command returns the following response:

pminfo hinv.ncpu network.interface.total.bytes
hinv.ncpu 
network.interface.total.bytes 

A group of related metrics in the Name Space may also be specified. For example, to list all of the hinv metrics you would use this command:

pminfo hinv
hinv.ncpu
hinv.cpuclock
hinv.dcache
hinv.icache
hinv.secondarycache
hinv.physmem
hinv.pmeminterleave
hinv.ndisk

If no metrics are specified, pminfo displays the entire collection of metrics. This can be useful for searching for metrics, when only part of the full name is known. For example, this command returns the following response:

pminfo | grep nfs
nfs.client.badcalls
nfs.client.badcalls
nfs.client.calls
nfs.client.nclget
nfs.client.nclsleep
nfs.client.reqs
nfs.server.badcalls
nfs.server.calls
nfs.server.reqs
nfs.client.badcalls
nfs.client.calls
nfs.client.nclget
nfs.client.nclsleep
nfs.client.reqs
nfs.server.badcalls
nfs.server.calls
nfs.server.reqs

The -d option causes pminfo to display descriptive information about metrics (refer to the pmLookupDesc(3) man page for an explanation of this metadata information). The following command and response show use of the -d option:

pminfo -d proc.nprocs disk.dev.read filesys.free
    proc.nprocs
       Data Type: 32-bit int  InDom: PM_INDOM_NULL 0xffffffff
       Semantics: instant  Units: none
disk.dev.read
       Data Type: 32-bit unsigned int  InDom: 1.2 0x400002
       Semantics: counter  Units: count
filesys.free
       Data Type: 32-bit int  InDom: 1.7 0x400007
       Semantics: instant  Units: Kbyte

The -f option to pminfo forces the current value of each named metric to be fetched and printed. In the example below, all metrics in the group hinv are selected:

pminfo -f hinv
hinv.ncpu
       value 1
hinv.cpuclock
       value 100
hinv.dcache
       value 8192
hinv.icache
       value 8192
hinv.secondarycache
       value 1048576
hinv.physmem
       value 64
hinv.pmeminterleave
       value 0
hinv.ndisk
       value 1

The -h option directs pminfo to retrieve information from the specified host. If the metric has an instance domain, the value associated with each instance of the metric is printed:

pminfo -h babylon.engr.sgi.com -f filesys.mountdir
filesys.mountdir 
       inst [1 or "/dev/root"] value "/"
       inst [2 or "/dev/dsk/dks1d3s7"] value "/usr2"
       inst [3 or "/dev/dsk/dks3d1s7"] value "/dbv"
       inst [4 or "/dev/dsk/dks3d4s7"] value "/dbv/d4"
       inst [5 or "/dev/dsk/dks3d2s7"] value "/dbv/d2"
       inst [6 or "/dev/dsk/dks3d3s7"] value "/dbv/d3"
       inst [7 or "/dev/dsk/dks2d4s7"] value "/vicepb"
       inst [8 or "/dev/dsk/xlv/build9"] value "/build9"
       inst [9 or "/dev/dsk/xlv/build8"] value "/build8"
       inst [10 or "/dev/dsk/xlv/lv9.xfs"] value "/lv9"
       inst [11 or "/dev/dsk/dks2d5s7"] value "/usenet"
       inst [12 or "/dev/dsk/xlv/work"] value "/usr/work"
       inst [13 or "/dev/dsk/xlv/build10"] value "/build10"
       inst [14 or "/dev/dsk/xlv/dist"] value "/usr/dist"
       inst [15 or "/dev/dsk/xlv/people"] value "/usr/people"
       inst [16 or "/dev/dsk/xlv/build12"] value "/build12"
       inst [17 or "/dev/dsk/xlv/build11"] value "/build11"

The -m option prints the Performance Metric Identifiers (PMIDs) of the selected metrics. This is useful for finding out which PMDA supplies the metric. For example, the output below identifies the PMDA supporting domain 4 (the leftmost part of the PMID) as the one supplying information for the metric environ.extrema.mintemp:

pminfo -m environ.extrema.mintemp 
environ.extrema.mintemp PMID: 4.0.3 

The -v option verifies that metric definitions in the PMNS correspond with supported metrics, and checks that a value is available for the metric. Descriptions and values are fetched, but not printed. Only errors are reported.

Some instance domains are not enumerable. That is, it is not possible to ask for all of the instances at once. Only explicit instances may be fetched from such instance domains. This is because instances in such a domain may have a very short lifetime or the cost of obtaining all of the instances at once is very high. The proc metrics are an example of such an instance domain. The -f option is not able to fetch metrics with non-enumerable instance domains; however, the -F option tells pminfo to obtain a snapshot of all of the currently available instances in the instance domain and then to retrieve a value for each.

Complete information on the pminfo command is found in the pminfo(1) man page. There are examples of the use of pminfo in the PCP Tutorial.

From time to time you may wish to change the value of a particular metric. Some metrics are counters that may need to be reset, and some are simply control variables for agents that collect performance metrics. When you need to change the value of a metric for any reason, the command to use is pmstore.

---

Note: For obvious reasons, the ability to arbitrarily change the value of a performance metric is not supported. Rather, the PMCS selectively allows some metrics to be modified in a very controlled fashion.

---

The basic syntax of the command is as follows:

pmstore metricname value 

There are also command line flags to further specify the action. For example, the -i option restricts the change to one or more instances of the performance metric.

The value may be in one of several forms, according to the following rules:

The following example illustrates the use of pmstore to enable performance metrics collection in the txmon PMDA (see /usr/pcp/pmdas/txmon for the source code of the txmon PMDA). When the metric txmon.control.level has the value 0, no performance metrics are collected. Values greater than 0 enable progressively more verbose instrumentation.

pminfo -f txmon.count
txmon.count
No value(s) available!
pmstore txmon.control.level 1
txmon.control.level old value=0 new value=1
pminfo -f txmon.count
txmon.count
       inst [0 or "ord-entry"] value 23
       inst [1 or "ord-enq"] value 11
       inst [2 or "ord-ship"] value 10
       inst [3 or "part-recv"] value 3
       inst [4 or "part-enq"] value 2
       inst [5 or "part-used"] value 1
       inst [6 or "b-o-m"] value 0

For complete information on pmstore usage and syntax, see the pmstore(1) man page.