The pmchart utility supports interactive selection and plotting of trends over time for arbitrarily selected performance metrics from one or more hosts and one or more domains of performance metrics. When you enter the command

pmchart 

you see the window shown in Figure 4-1.

Figure 4-1. pmchart Window

Normally, pmchart operates in “live” mode where performance metrics are fetched in real time and plotted against a time axis. The user can choose performance metrics and monitor the current values for these metrics from any host that is accessible on the network and has the pmcd server running.

When launched with the -a command-line option, pmchart can also replay PCP archive logs of performance metrics created by pmlogger.

The reference page for pmchart explains how to configure charts based on performance metrics, using either the View Selector option of the Views menu or the New Chart option of the Edit menu. Once charts have been configured and applied, the charts are placed in an expanded window, as shown in Figure 4-2.

Figure 4-2. Two Charts and Metrics From Three Hosts in pmchart

All metrics in the Performance Metrics Name Space (PMNS) with numeric value semantics can be graphed. By default, pmchart initially allows the user to select metrics to be plotted from the local host. However, the graphical user interface allows other hosts or archives to be chosen at any time as alternate sources of performance metrics and all metrics (independent of their source) are plotted on a common time axis.

The -h command-line option nominates an alternate default host to be used in preference to the local host.

The -a command-line option may be used to start pmchart processing performance metrics from one or more PCP archive logs. The first named archive becomes the default source of performance metrics. This mode is particularly useful for retrospective comparisons and for post-mortem analysis of performance problems, where a remote system is not directly accessible or a performance analyst is not available on site.

The pmchart utility examines the semantics of selected metrics, and where sensible, uses the metadata provided by the Performance Metrics Collection Subsystem (PMCS) to convert fetched metric values to a rate before plotting. In the case where different metrics are plotted in the same chart (for example, against a common Y-axis), the metrics must have the same dimension (taking into account any automatic rate conversion), but pmchart may scale metric values where necessary, to produce comparable values with common units and scale.

When replaying archive logs, the user may interactively control the current replay time, direction of replay, and replay rate, using the PCP time control dialog, as described in the section “Time Duration and Control”.

pmchart View Selection

A “view” in pmchart is a predefined collection of charts, typically constructed to display some common performance scenario. Default views are included in the PCP distribution, others are part of the various PCP add-on products, and others may be created by the pmchart end user. The View Selector option in the Views menu launches a dialog box as shown in Figure 4-3, which may be used to select one of the available views. The default PCP views include the following:

BufferCache		Cumulative amount of data read and written between system buffers and user memory or block devices.
CPU		Processor utilization (user, system, memory break, interrupt, I/O wait, and idle time) aggregated over all CPUs.
CrayLinks		Usage of CrayLink node connectors, if this hardware is present.
Disk		Cumulative number of read and write transfers for all disk devices.
DiskCntrls		Cumulative number of read and write transfers for all drives attached to each disk controller on the system.
FileSystem		Percentage of each filesystem in use (percent full).
LoadAvg		System load averaged over intervals of 1, 5, and 15 minutes.
Memory		Memory used by the kernel, filesystems, user processes, and free space.
NetBytes		Network interface activity—octets transmitted on various interfaces.
NetConnDrop		TCP drops, connection drops, timeout drops, and TCP accepts.
NetPackets		Rate of TCP and UDP packets received and sent.
NetTCPcongestion		TCP packets retransmitted, retransmit timeouts, and TCP packets sent.
NFS		Client and server NFS operation rates.
Overview		Composite charts of CPU, LoadAvg, Memory, Disk, and NetBytes.
Paging		Page in and page out rates from the virtual memory subsystem.
PMCD		Message rates and CPU time used by pmcd or associated PMDAs.
Swap		System swap space allocated, reserved, and unused.
Syscalls		Rate of exec, fork, read, write and total system calls.

The View Selection dialog looks similar to Figure 4-3.

Figure 4-3. pmchart View Selection Dialog

You can create your own custom views using the metric selection facilities described below. Then use the View Save option in the Views menu to save your setup as a custom view, displayed in the User Views list.

pmchart Metric Selection

The Metric Selection dialog window allows interactive navigation of the PMNS (Performance Metrics Name Space) to create new chart configurations. The dialog gives you the ability to choose metrics, display information about performance metrics, change the current host or archive, select metric instances, and then plot metric values on a common time axis. You obtain this dialog by selecting the New Chart option from the Edit menu of pmchart. This dialog is shown in Figure 4-4.

Figure 4-4. Metric Selection Dialog

Metric selection proceeds by navigating through the tree-structured PMNS.

If you enter a partial metric specification in the Path field in the Metric Selection dialog, you can avoid having to navigate through the PMNS to select the metrics you need. For example, if you enter the path irix.network.interface, the window changes dynamically, as shown in Figure 4-5.

Figure 4-5. Further Metric Selection

You can continue the selection process by choosing non-leaf nodes from the Nodes list, and finally a leaf node from the Metrics list. At this stage, the Path corresponds to a leaf node in the PMNS, as shown in Figure 4-6.

Figure 4-6. Selecting a Leaf Node in the PMNS (Performance Metric)

Once a metric has been selected, the Info button in the Metric Selection dialog launches the Metric Information dialog, as shown in Figure 4-7. This dialog displays the name, unit, and semantics for the currently selected metric, along with the verbose help text that describes the metric, and optionally a description of the underlying instance domain.

Figure 4-7. Metric Information Dialog

Finally, you may have to select from several instances of a metric. In the example shown in Figure 4-7, you wish to monitor the input packet rate for some network interface(s). For the current source of performance metrics, there are two network interfaces configured. You must select one or more instances, as shown in Figure 4-8.

Figure 4-8. Selecting a Metric Instance

You can select multiple instances either by clicking and dragging up and down the list with the left mouse button, or by selecting the first instance and then using the Shift key (or Ctrl key) with the left mouse button to select one or more other instances.

Creating a PCP Archive From a pmchart Session

From the File menu of pmchart when running in “live” mode, the Record (Stop Recording) option may be used to start (or stop) the creation of a PCP archive log. The archive log is created using pmlogger and includes the update interval and all of the performance metrics in the current pmchart configuration when recording begins.

---

Note: Any changes made to the pmchart configuration after recording has been started will not be reflected in the archive log. For these to take effect, the recording must be stopped and re-started (thereby creating a second PCP archive log).

---

When recording is started, a File Chooser dialog is launched, and the user must provide the name of a new file to be used as the PCP archive folio for the new archive (see the section “PCP Archive Folios”). The recording session produces multiple files in the same directory as the archive folio.

If necessary, pmchart creates directories on the path to the named archive folio.

It is often convenient to maintain one directory for each new folio, or else one directory for each group of folios related by collector host(s), service type, or chart selection.

When recording is active, a small red indicator appears in the time control button, as shown at the bottom left of Figure 4-9.

Figure 4-9. pmchart Display When Recording

If you choose File > Stop Recording, logging stops immediately. The red light in the lower left turns gray.

To start recording again, chose File > Record and specify a new archive folio name.

If you exit pmchart by choosing File > Quit, a dialog similar to that shown in Figure 4-10 appears to remind you where the archive folio was created, and to confirm that recording should be terminated.

Figure 4-10. pmchart Stop Recording Dialog

If you select Yes, recording stops immediately.

If you select No, recording continues. This is a useful way to continue archive logging without keeping pmchart active.

PmChart*xrtForegroundColor: "green" 
PmChart*xrtBackgroundColor: "black" 
PmChart*xrtGraphForegroundColor: "rgb:00/b0/00" 
PmChart*xrtGraphBackgroundColor: "black" 
PmChart*xrtHeaderForegroundColor: "green" 
PmChart*xrtHeaderBackgroundColor: "black" 
PmChart*pmDefaultColors: rgb:ff/ff/00 rgb:00/ff/00 rgb:00/00/ff \ 
                         rgb:ff/ff/00 rgb:00/ff/ff rgb:ff/00/ff 

xrdb -merge $HOME/.xrdb 

file:/var/pcp/Tutorial/pmchart.html

The pmgadgets tool creates a small window containing a collection of graphical gadgets driven by performance metrics supplied by the PCP framework. Any numeric metric supported by PCP can be displayed.

---

Note: In the current PCP release, pmgadgets is constrained to process performance metrics from real-time sources (and not PCP archive logs), although metrics from several different hosts may be displayed simultaneously in the same window.

---

The layout of the gadgets and the performance metrics that lie behind them are specified in a configuration file, and pmgadgets is typically run on an existing configuration file or in conjunction with an application that automatically generates a configuration file. For example, pmgirix generates a configuration file for various IRIX performance metrics and feeds it to pmgadgets. The resulting display depends on the host configuration, but the following is representative of a system with four CPUs, eleven disks on three controllers, and four network interfaces.

The pmgadgets tool displays much the same information as pmchart, but more compactly, and without so many historical graphs.

Figure 4-11. Representative pmgadgets Display Using pmgirix

The pmgadgets specification language provides the ability to define the following gadgets and components:

Each visible gadget must be assigned a Cartesian position in the pmgadgets display.

By way of an example, the pmgadgets specification shown in Example 4-1 includes CPU, disk, and load average information from two hosts, and produces a display like the one shown in Figure 4-12.

_colourlist cpuColours (blue3 red3 yellow3 cyan3 green3)
_legend diskLegend (
    _default  green3
    15        yellow
    40        orange
    75        red
)
# host moomba
_label 70 12 "moomba"
_multibar 5 5 30 6
    _metrics (
        moomba:irix.kernel.all.cpu.user
        moomba:irix.kernel.all.cpu.sys
        moomba:irix.kernel.all.cpu.intr
        moomba:irix.kernel.all.cpu.wait.total
        moomba:irix.kernel.all.cpu.idle
    )
    _maximum 0.0
    _colourlist cpuColours
_bargraph 40 5 25 20
    _metric moomba:irix.kernel.all.load["1 minute"]
    _max 1.0
_led 12 16 6 6
    _metric moomba:irix.disk.all.read _legend diskLegend
_led 25 16 6 6
    _metric moomba:irix.disk.all.write _legend diskLegend
# host gonzo
_label 70 39 "gonzo"
_multibar 5 32 30 6
    _metrics (
        gonzo:irix.kernel.all.cpu.user
        gonzo:irix.kernel.all.cpu.sys
        gonzo:irix.kernel.all.cpu.intr
        gonzo:irix.kernel.all.cpu.wait.total
        gonzo:irix.kernel.all.cpu.idle
    )
    _maximum 0.0
    _colourlist cpuColours
_bargraph 40 32 25 20
    _metric gonzo:irix.kernel.all.load["1 minute"]
    _max 1.0
_led 12 43 6 6
    _metric gonzo:irix.disk.all.read _legend diskLegend
_led 25 43 6 6
    _metric gonzo:irix.disk.all.write _legend diskLegend

Figure 4-12. Customized pmgadgets Display

In addition to the “drill-down” capabilities of pmgadgets, positioning the cursor over a gadget and entering a space character causes an information dialog to be exposed, to track the current values of the performance metrics that are associated with the gadget. For example, see Figure 4-13.

Figure 4-13. pmgadgets Information Dialog

The pmgadgets(1) reference page provides a complete description of the gadget specification language and the user interface controls of pmgadgets.

The pmkstat command provides a periodic, one-line summary of system performance. This command is intended to monitor system performance at the highest level, after which other tools may be used for examining subsystems to observe potential performance problems in greater detail. After entering the pmkstat command, you see output similar to the following, with successive lines appearing periodically:

pmkstat 
# hostname load avg: 0.26, interval: 5 sec, Thu Jan 19 12:30:13 1995
runq    | memory    |      system      | disks|   cpu 
mem swp | free page | scall ctxsw  intr| rd wr|usr sys idl wt
0   0     16268 0     64    19     2396  0  0  0   1   99  0
0   0     16264 0     142   45     2605  0  8  0   2   97  0
0   0     16268 0     308   62     2532  0  1  1   1   98  0
0   0     16268 0     423   88     2643  0  0  1   1   97  0

An additional line of output is added every five seconds. The update interval may be varied using the -t interval option.

The output from pmkstat is directed to standard output, and the columns in the report are interpreted as follows:

As with most PCP utilities, real-time metrics and archive logs are interchangeable.

For example, the following command uses the PCP archive log foo and the timezone of the host (tokyo) from which performance metrics in the archive were collected:

pmkstat -a foo -z 

Note: timezone set to local timezone of host "tokyo"
# tokyo load avg: 1.06, interval: 5 sec, Thu Feb  2 08:42:55 1995
 runq  |     memory |     system       |  disks  |      cpu
mem swp|   free page| scall ctxsw  intr|  rd   wr|usr sys idl  wt
  0   0    4316    0    195    64  2242   32   21   0   3   8  89
  0   0    3976    0    279    86  2143   50   17   0   5   8  87
  1   0    3448    0    186    63  2304   35   14   0   4   9  87
  0   0    4364    0    254    81  2385   35    0   0   4   9  87
  0   0    3696    0    266    92  2374   41    0   0   3   9  88
  0   0    2668   42    237    81  2400   44    2   1   4   7  89
  0   0    4644  100    206    68  2590   25    1   0   3   5  91
  0   0    5384    0    174    63  2296   32   22   0   2   8  89
  0   0    4736    0    189    65  2197   31   28   0   3   8  89
pmFetch: End of PCP archive log

For complete information on pmkstat usage and command-line options, see the pmkstat(1) reference page.

This flexible command displays performance metrics in ASCII tables, suitable for export into databases or report generators. For example, the following command provides continuous memory statistics on a host named serv:

pmdumptext -imu -h serv -f `%H:%M:%S' irix.mem.util 
Metric        kernel  fs_ctl  _dirty  _clean    free    user 
   Units             b       b       b       b       b       b 
20:14:28        99.14M   6.03M   0.85M  98.42M   0.17G   0.16G 

See pmdumptext(1) for more information.

The pmval command dumps the current values for the named performance metrics. For example, the following command reports the value of performance metric proc.nprocs once per second (by default), and produces output similar to this:

pmval proc.nprocs 

metric:    proc.nprocs
host:      localhost
semantics: instantaneous value
units:     none
samples:   indefinite
interval:  1.00 sec
         73
         72
         70
         75
         75

In this example, the number of running processes was reported once per second.

Where the semantics of the underlying performance metrics indicate that it would be sensible, pmval reports the rate of change or resource utilization.

For example, the following command reports idle processor utilization for each of four CPUs on the remote host moomba, each five seconds apart, producing output of this form:

pmval -h moomba -t 5sec -s 4 irix.kernel.percpu.cpu.idle 

metric:    irix.kernel.percpu.cpu.idle
host:      moomba
semantics: cumulative counter (converting to rate)
units:     millisec (converting to time utilization)
samples:   4
interval:  5.00 sec
     cpu0        cpu1        cpu2        cpu3 
     0.8193      0.7933      0.4587      0.8193 
     0.7203      0.5822      0.8563      0.7303 
     0.6100      0.6360      0.7820      0.7960 
     0.8276      0.7037      0.6357      0.6997 

Similarly, the following command reports disk I/O read rate every minute for just the disk /dev/dsk/dks0d1, and produces output similar to the following:

pmval -t 1min -i dks0d1 irix.disk.dev.read 

metric:    irix.disk.dev.read
host:      localhost
semantics: cumulative counter (converting to rate)
units:     count (converting to count / sec)
samples:   indefinite
interval:  60.00 sec
     dks0d1 
      33.67 
      48.71 
      52.33 
      11.33 
      2.333 

The -r flag may be used to suppress the rate calculation (for metrics with counter semantics) and display the raw values of the metrics.

When used in conjunction with a PCP archive, the -g option may be used to associate a PCP time control dialog (see the section “Time Duration and Control”) with the execution of pmval to support temporal navigation within the archive. In the example below, manipulation of the “time” within the archive is achieved by the exchange of time control messages between pmval and pmtime.

pmval -g -a /var/adm/pcplog/myserver/960801 

The pmval command is documented by the pmval(1) reference page. There are annotated examples of the use of pmval in the PCP Tutorial.

The pmem command reports per-process memory usage statistics within the PCP framework.

Both virtual size and pro-rated physical memory usage are reported. The virtual memory usage statistics represent the total virtual size of each process, irrespective of how many pages are valid (resident). Prorated physical memory statistics indicate real memory usage (only valid pages are counted) and are prorated on a per-page basis between all processes that reference each page. Thus the prorated physical memory counts reflect the real memory demands for individual processes in the context of the current process mix.

The output of pmem can be very large. Here is an abbreviated example of pmem output:

Host: gonzo Configured: 65536  Free:18380  Tue Jul 9 16:45:08 1996
  pid  ppid   user  vtxt  ptxt  vdat  pdat vshm pshm command
    1     0   root   232   144    84    76    0    0 /etc/init
  832   827   root  3204  1013  5796  3096    0    0 /usr/bin/X11/Xsg
  221     1   root  1424    54   156   84     0    0 /usr/lib/saf/sad
  838   827   root  2948    36   268    75    0    0 /usr/bin/X11/xdm
   86     1   root  1264    32   144    76    0    0 /usr/etc/syslogd
  182     1   root  1476   129   596   387    0    0 /usr/etc/rpcbind
  827     1   root  2948    13   252    22    0    0 /usr/bin/X11/xdm
  172     1   root  1276    52   148   100    0    0 /usr/etc/routed
Total     vtxt  ptxt   vdat  pdat  vshm  pshm  77 user processes
          121M        36256           0        =  157M virtual
               13982        20194           0  = 34176 physical

The columns report the following information:

For complete information on pmem usage and command line options, see the pmem(1) reference page.

The pminfo command displays various types of information about performance metrics available through the facilities of the Performance Co-Pilot.

The -T option is extremely useful—it provides help text about performance metrics:

pminfo -T irix.mem.util.fs_dirty 
irix.mem.util.fs_dirty
Help:
The amount of memory in Kbytes that is holding file system data.

The -t option displays the one-line help text associated with the selected metrics. The -T option prints more verbose help text.

Without any options, pminfo verifies that the specified metrics exist in the name space, and echoes those names. Metrics may be specified as arguments to pminfo using their full metric names. For example, this command returns the following response:

pminfo hinv.ncpu irix.network.interface.total.bytes 
hinv.ncpu 
irix.network.interface.total.bytes 

A group of related metrics in the name space may also be specified. For example, to list all of the hinv metrics you would use this command:

pminfo hinv 
hinv.ncpu
hinv.cpuclock
hinv.dcache
hinv.icache
hinv.secondarycache
hinv.physmem
hinv.pmeminterleave
hinv.ndisk

If no metrics are specified, pminfo displays the entire collection of metrics. This can be useful for searching for metrics, when only part of the full name is known. For example, this command returns the following response:

pminfo | grep nfs 

irix.nfs.client.badcalls
irix.nfs.client.badcalls
irix.nfs.client.calls
irix.nfs.client.nclget
irix.nfs.client.nclsleep
irix.nfs.client.reqs
irix.nfs.server.badcalls
irix.nfs.server.calls
irix.nfs.server.reqs
irix.nfs.client.badcalls
irix.nfs.client.calls
irix.nfs.client.nclget
irix.nfs.client.nclsleep
irix.nfs.client.reqs
irix.nfs.server.badcalls
irix.nfs.server.calls
irix.nfs.server.reqs

The -d option causes pminfo to display descriptive information about metrics (refer to the pmLookupDesc(3) reference page for an explanation of this metadata information). The following command and response show use of the -d option:

pminfo -d proc.nprocs irix.disk.dev.read irix.filesys.free 

proc.nprocs
    Data Type: 32-bit int  InDom: PM_INDOM_NULL 0xffffffff
    Semantics: instant  Units: none
irix.disk.dev.read
    Data Type: 32-bit unsigned int  InDom: 1.2 0x400002
    Semantics: counter  Units: count
irix.filesys.free
    Data Type: 32-bit int  InDom: 1.7 0x400007
    Semantics: instant  Units: Kbyte

The -f option to pminfo forces the current value of each named metric to be fetched and printed. In the example below, all metrics in the group hinv are selected:

pminfo -f hinv 

hinv.ncpu
    value 1
hinv.cpuclock
    value 100
hinv.dcache
    value 8192
hinv.icache
    value 8192
hinv.secondarycache
    value 1048576
hinv.physmem
    value 64
hinv.pmeminterleave
    value 0
hinv.ndisk
    value 1

The -h option directs pminfo to retrieve information from the specified host. If the metric has an instance domain, the value associated with each instance of the metric is printed:

pminfo -h babylon.engr.sgi.com -f irix.filesys.mountdir 
irix.filesys.mountdir 
    inst [1 or "/dev/root"] value "/"
    inst [2 or "/dev/dsk/dks1d3s7"] value "/usr2"
    inst [3 or "/dev/dsk/dks3d1s7"] value "/dbv"
    inst [4 or "/dev/dsk/dks3d4s7"] value "/dbv/d4"
    inst [5 or "/dev/dsk/dks3d2s7"] value "/dbv/d2"
    inst [6 or "/dev/dsk/dks3d3s7"] value "/dbv/d3"
    inst [7 or "/dev/dsk/dks2d4s7"] value "/vicepb"
    inst [8 or "/dev/dsk/xlv/build9"] value "/build9"
    inst [9 or "/dev/dsk/xlv/build8"] value "/build8"
    inst [10 or "/dev/dsk/xlv/lv9.xfs"] value "/lv9"
    inst [11 or "/dev/dsk/dks2d5s7"] value "/usenet"
    inst [12 or "/dev/dsk/xlv/work"] value "/usr/work"
    inst [13 or "/dev/dsk/xlv/build10"] value "/build10"
    inst [14 or "/dev/dsk/xlv/dist"] value "/usr/dist"
    inst [15 or "/dev/dsk/xlv/people"] value "/usr/people"
    inst [16 or "/dev/dsk/xlv/build12"] value "/build12"
    inst [17 or "/dev/dsk/xlv/build11"] value "/build11"

The -m option prints the Performance Metric Identifiers (PMIDs) of the selected metrics. This is useful for finding out which PMDA supplies the metric. For example, the output below identifies the PMDA supporting domain 4 (the leftmost part of the PMID) as the one supplying information for the metric environ.extrema.mintemp:

pminfo -m environ.extrema.mintemp 
environ.extrema.mintemp PMID: 4.0.3 

The -v option verifies that metric definitions in the name space correspond with supported metrics, and checks that a value is available for the metric. Descriptions and values are fetched, but not printed. Only errors are reported.

Some instance domains are not enumerable. That is, it is not possible to ask for all of the instances at once. Only explicit instances may be fetched from such instance domains. This is because instances in such a domain may have a very short lifetime or the cost of obtaining all of the instances at once is very high. The proc metrics are an example of such an instance domain. The -f option is not able to fetch metrics with non-enumerable instance domains; however, the -F option tells pminfo to obtain a snapshot of all of the currently available instances in the instance domain and then to retrieve a value for each.

Complete information on the pminfo command is found in the pminfo(1) reference page. There are examples of the use of pminfo in the PCP Tutorial.

From time to time you may wish to change the value of a particular metric. Some metrics are counters that may need to be reset, and some are simply control variables for agents that collect performance metrics. When you need to change the value of a metric for any reason, the command to use is pmstore.

---

Note: For obvious reasons, the ability to arbitrarily change the value of a performance metric is not supported. Rather, the PMCS selectively allows some metrics to be modified in a very controlled fashion.

---

The basic syntax of the command is as follows:

pmstore metricname value 

There are also command-line flags to further specify the action. For example, the -i option restricts the change to one or more instances of the performance metric. The value may be in one of several forms, according to the following rules:

The following example illustrates the use of pmstore to enable performance metrics collection in the txmon PMDA (see /usr/pcp/pmdas/txmon for the source code of the txmon PMDA). When the metric txmon.control.level has the value 0, no performance metrics are collected. Values greater than 0 enable progressively more verbose instrumentation.

pminfo -f txmon.count 
txmon.count
No value(s) available!
pmstore txmon.control.level 1 
txmon.control.level old value=0 new value=1
pminfo -f txmon.count 
txmon.count
    inst [0 or "ord-entry"] value 23
    inst [1 or "ord-enq"] value 11
    inst [2 or "ord-ship"] value 10
    inst [3 or "part-recv"] value 3
    inst [4 or "part-enq"] value 2
    inst [5 or "part-used"] value 1
    inst [6 or "b-o-m"] value 0

For complete information on pmstore usage and syntax, see pmstore(1).