Performance monitoring and management in complex systems demands the ability to accurately capture performance characteristics for subsequent review, analysis, and comparison. Performance Co-Pilot provides extensive support for the creation and management of archive logs that capture a user-specified profile of performance information to support retrospective performance analysis.
The following major sections are included in this chapter:
“Introduction to Archive Logging” presents the concepts and issues involved with creating and using archive logs.
“Using Archive Logs and Performance Visualization Tools” describes the interaction of the PCP tools with archive logs.
“Cookbook for Archive Logging” shows some shortcuts for setting up useful PCP archive logs.
“Archive Logging Troubleshooting” presents helpful direction if your archive logging implementation is not functioning correctly.
Within the Performance Co-Pilot, the pmlogger utility may be configured to collect archives of performance metrics. The archive creation process is easy and very flexible, incorporating the following features:
Archive log creation at either a PCP collector (typically a server) or a PCP monitor system (typically a workstation), or at some designated PCP archive logger host.
Concurrent independent logging, both local and remote—the performance analyst can activate a private pmlogger instance to collect only the metrics of interest for the problem at hand, independent of other logging on the workstation or remote host.
Record mode in oview, pmchart, and pmview to create archives as needed from the current visualization.
Independent determination of logging frequency for individual metrics or metric instances. For example, you could log the “5 minute” load average every half hour, the write I/O rate on the DBMS log spindle every 10 seconds, and aggregate I/O rates on the other disks every minute.
Dynamic adjustment of what is to be logged, and how frequently, via pmlc. This may be used to disable logging or to increase the sample interval during periods of low activity or chronic high activity (to minimize logging overhead and intrusion). A local pmlc may interrogate and control a remote pmlogger, subject to the access control restrictions implemented by pmlogger.
Self-contained logs that include all system configuration and metadata required to interpret the values in the log. These logs can be kept for analysis at a much later time, potentially after the hardware or software has been reconfigured, and the logs have been stored as discrete, autonomous files for remote analysis.
cron-based scripts to expedite the operational management, for example, log rotation, consolidation and culling.
Archive folios as a convenient aggregation of multiple archive logs. Archive folios may be created with the mkafm utility, and processed with the pmafm tool.
Critical to the success of the PCP archive logging scheme, library routines provide access to real-time feeds of performance metrics also provide access to the archive logs.
Live feeds and archives are literally interchangeable, with a single Performance Metrics Application Programming Interface (PMAPI) that preserves the same semantics for both styles of metric source. In this way, applications and tools developed against the PMAPI can automatically process either current or historical performance data.
One of the most important applications of archive logging services provided by PCP is in the area of retrospective analysis. In many cases, understanding today's performance problems can be assisted by side-by-side comparisons with yesterday's performance. With routine creation of performance archive logs, you can concurrently replay pictures of system performance for two or more periods in the past.
Archive logs are also an invaluable source of intelligence when trying to diagnose what went wrong, as in a performance post-mortem. Because the PCP archive logs are entirely self-contained, this analysis can be performed off-site if necessary.
Each archive log contains metric values from only one host. However, many PCP tools can simultaneously visualize values from multiple archives on different hosts.
The archives can be replayed against the inference engine (pmie is an application that uses the PMAPI). This allows you to automate the regular, first-level analysis of system performance.
Such analysis can be performed by constructing suitable expressions to capture the essence of common resource saturation problems, then periodically creating an archive and playing it against the expressions. For example, you may wish to create a daily performance audit (run by cron) to detect performance regressions.
For more about pmie, see Chapter 6, “Performance Metrics Inference Engine.”
Periodic snapshot images of recent performance, activity levels, and resource utilization can be extracted from the PCP archive logs and published via a World Wide Web (WWW) server. These are high-quality images generated from pmchart that provide an excellent vehicle for publishing performance summary information for users, system and network administrators, or managers. The cron.snap services may be used to automate snapshots.
By collecting performance archives with relatively long sampling periods, or by reducing the daily archives to produce summary logs, the capacity planner can collect the base data required for forward projections, and can estimate resource demands and explore “what if” scenarios by replaying data using visualization tools and the inference engine.
Most PCP tools default to real-time display of current values for performance metrics from PCP collector host(s). However, most PCP tools also have the capability to display values for performance metrics retrieved from PCP archive log(s). The following sections describe plans, steps, and general issues involving archive logs and the PCP tools.
Most commonly, a PCP tool would be invoked with a -a option to process an archive log some time after pmlogger had finished creating the archive. However, a tool such as pmchart that uses a Time Control dialog (see the section “Time Duration and Control”) stops when the end of archive is reached, but could resume if more data is written to the PCP archive log.
 | Note: pmlogger uses buffered I/O to write the archive log, so the end of the archive may be aligned with an I/O buffer boundary, rather than with a logical archive log record, and this might confuse the PCP tool. These problems may be avoided by sending pmlogger a SIGUSR1 signal, or using the flush command of pmlc to force pmlogger to flush its output buffers. |
The IRIX operating system supports the standard cron process scheduling system. Complete information on cron is available in the appropriate reference page and in the IRIX Admin: System Configuration and Operation guide.
Performance Co-Pilot supplies shell scripts to use the cron functionality to help manage your archive logs. The following scripts are supplied:
| cron.pmdaily | This script performs a daily “housecleaning” of archive logs and notices. | |
| cron.pmlogmerge | This script is used to merge archive logs and is called by cron.pmdaily. | |
| cron.pmcheck | This script checks to see that all desired pmlogger processes are running on your system, and invokes any that are missing for any reason. | |
| cron.pmsnap | This script generates graphic image snapshots of pmchart performance charts at regular intervals. |
These scripts can be edited to suit your particular needs, and are generally controlled by the /var/pcp/config/pmlogger/control file (cron.pmsnap has an additional control file). Complete information on these scripts is available in the cron.pmdaily(1) and cron.pmsnap(1) reference pages.
Performance Co-Pilot archive log files can occupy a great deal of disk space, and management of archive logs can be a large task in itself. The upcoming sections provide information to assist you in PCP archive log file management.
When a PCP archive is created by pmlogger, an archive basename must be specified and several physical files are created, as shown in Table 7-1.
Table 7-1. Filenames for PCP Archive Log Components (archive.*)
Filename | Contents |
|---|---|
archive . index | Temporal index for rapid access to archive contents. |
archive . meta | Metadata descriptions for performance metrics and instance domains appearing in the archive. |
archive.N | Volumes of performance metrics values, for N = 0,1,2,... |
A single PCP archive may be partitioned into a number of volumes. These volumes may expedite management of the archive; however, all volumes must typically be present before a PCP tool can process the archive.
You can control the size of an archive log volume by using the -v command-line option to pmlogger. This option takes a numeric argument specifying the number of performance metric samples to be written to each log volume before pmlogger starts a new volume. Archive log volumes retain the same base filename as other files in the archive log, and are differentiated by a numeric suffix that is incremented with each volume change. For example, you might have a log volume sequence that looks like this:
netserver.irix.log.0 netserver.irix.log.1 netserver.irix.log.2 |
You can also cause an existing log to be closed and a new one to be opened by sending a SIGHUP signal to pmlogger, or by using the pmlc command to change the pmlogger instructions dynamically, without interrupting pmlogger operation. Complete information on log volumes is found in the pmlogger(1) reference page.
The PCP archive management tools support a consistent scheme for selecting the basenames for the files in a collection of archives, and mapping these files to a suitable directory hierarchy.
Once configured, the PCP tools that manage archive logs employ a consistent scheme for selecting the basename for an archive each time pmlogger is launched, namely the current date and time in the format YYMMDD.HH.MM. Typically, at the end of each day, all archives for a particular host on that day would be merged to produce a single archive with a basename constructed from the date, namely YYMMDD. The cron.pmdaily script performs this action and a number of other routine housekeeping chores.
If you are using a deployment of PCP tools and daemons to collect metrics from a variety of hosts and storing them all at a central location, you should develop an organized strategy for storing and naming your log files.
| Note: There are many possible configurations of pmlogger, as described in “PCP Archive Logger Deployment”. The directory organization described in this section is recommended for any system on which pmlogger is configured for permanent execution (as opposed to short-term executions, for example, as launched from pmchart to record some performance data of current interest). |
Typically, the IRIX filesystem structure can be used to reflect the number of hosts for which a pmlogger instance is expected to be running locally, obviating the need for lengthy and cumbersome filenames. It makes considerable sense to place all logs for a particular host in a separate directory named after that host. Because each instance of pmlogger can only log metrics fetched from a single host, this also simplifies some of the archive log management and administration tasks.
For example, consider the filesystem and naming structure shown in Figure 7-1.
The specification of where to place the archive log files for particular pmlogger instances is encoded in the configuration file /var/pcp/config/pmlogger/control, and this file should be customized on each host running an instance of pmlogger.
If many archives are being created, and the associated PCP collector systems form peer classes based upon service type (for example, Web servers, DBMS servers, NFS servers, and so on), then it may be appropriate to introduce another layer into the directory structure, or use symbolic links to group together hosts providing similar service types.
The configuration files used by pmlogger describe which metrics are to be logged. Groups of metrics may be logged at different intervals to other groups of metrics. Two states, mandatory and advisory, also apply to each group of metrics, defining whether metrics definitely should be logged or not logged, or whether a later advisory definition may change that state.
The mandatory states are on, off, and maybe. The advisory states, which only affect metrics that are mandatory maybe, are on and off. Therefore, a metric that is mandatory maybe in one definition and advisory on in another definition would be logged at the advisory interval. Metrics are mandatory maybe and advisory off by default, so are not logged.
A complete description of the pmlogger configuration format can be found on the pmlogger(1) reference page.
Once a PCP archive log has been created, the pmdumplog utility may be used to display various information about the contents of the archive. For example, the command
pmdumplog -l /var/adm/pcplog/www.sgi.com/960731 |
might produce the following output:
Log Label (Log Format Version 1) Performance metrics from host www.sgi.com commencing Wed Jul 31 00:16:34.941 1996 ending Thu Aug 1 00:18:01.468 1996 |
The simplest way to discover what performance metrics are contained within an archive is to use pminfo; for example:
pminfo -a /var/adm/pcplog/www.sgi.com/960731 irix.network.mbuf irix.network.mbuf.alloc irix.network.mbuf.typealloc irix.network.mbuf.clustalloc irix.network.mbuf.clustfree irix.network.mbuf.failed irix.network.mbuf.waited irix.network.mbuf.drained |
| Note: In PCP version 2.0 archives, the collector's PMNS is stored in the meta file. |
This section presents a checklist of tasks that may be performed to enable PCP archive logging with minimal effort. For a complete explanation, refer to the other sections in this chapter and the reference pages for pmlogger and related tools.
Assume you wish to activate primary archive logging on the PCP collector host pluto. Execute all of the following tasks while logged into pluto as the superuser (root).
Create the directory to hold the archive logs:
mkdir /var/adm/pcplog/pluto
Choose a suitable pmlogger configuration file. Here are some examples:
The default configuration: /var/pcp/config/pmlogger/config.default.
A broad summary configuration, sufficient to be used with dkvis, mpvis, nfsvis and pmkstat: /var/pcp/config/pmlogger/config.Summary.
One of the other config.* files in the /var/pcp/config/pmlogger directory, tailored for an application, a PCP add-on product, a pmchart view, or a PCP monitor tool.
Copy the chosen configuration file to /var/adm/pcplog/pluto/config.default (possibly after some customization).
Edit /var/pcp/config/pmlogger/control. Using the line for the “local primary logger” as a template, add the following line to the file:
pluto y n /var/adm/pcplog/pluto -c config.keep
Make sure pmcd and pmlogger are enabled and running:
chkconfig pmcd on chkconfig pmlogger on /etc/init.d/pcp start Performance Co-Pilot PMCD started (logfile is .... /pmcd.log) Performance Co-Pilot Primary Logger started
Verify that the primary pmlogger instance is running:
pmlc pmlc> connect primary pmlc> status pmlogger [primary] on host pluto is logging metrics from host pluto log started Thu Aug 8 14:33:01 1996 (times in local time) last log entry Thu Aug 8 14:34:11 1996 current time Thu Aug 8 14:36:54 1996 log volume 0 log size 284
Verify that the archive files are being created in the correct place:
ls -1 /var/adm/pcplog/pluto 960808.14.33.0 960808.14.33.index 960808.14.33.meta Latest pmlogger.log
Assume you wish to create archive logs on the local host for performance metrics collected from the remote host bert. Execute all of the following tasks while logged into the local host as the superuser (root).
Create the directory to hold the archive logs:
mkdir /var/adm/pcplog/bert
Choose a suitable pmlogger configuration file. Here are three examples:
The default configuration: /var/pcp/config/pmlogger/config.default.
A broad summary configuration, sufficient to be used with dkvis, mpvis, nfsvis and pmkstat: /var/pcp/config/pmlogger/config.Summary.
One of the other config.* files in the /var/pcp/config/pmlogger directory, tailored for an application, a PCP add-on product, a pmchart view, or a PCP monitor tool.
Copy the chosen configuration file to /var/adm/pcplog/bert/config.default (possibly after some customization).
Edit /var/pcp/config/pmlogger/control. Using the line for remote as a template, add the following line to the file:
bert n n /var/adm/pcplog/bert -c ./config.default
Start pmlogger:
/usr/pcp/bin/cron.pmcheck Restarting pmlogger for host "bert" ..... done
Verify that the pmlogger instance is running:
pmlc pmlc> show loggers The following pmloggers are running on bert: primary (19144) pmlc> connect 19144 pmlc> status pmlogger [19144] on host ernie is logging metrics from host bert log started Thu Aug 8 10:10:10 1996 (times in local time) last log entry Thu Aug 8 14:50:54 1996 current time Thu Aug 8 14:55:48 1996 log volume 0 log size 256
Assume the local host has been set up to create archive logs of performance metrics collected from one or more hosts (which may be either the local host or a remote host).
To activate the maintenance and housekeeping scripts for a collection of archive logs, execute the following tasks while logged into the local host as the superuser (root):
Augment the crontab file for root. For example:
crontab -l >/tmp/foo
Edit /tmp/foo, adding lines similar to those from /var/pcp/config/pmlogger/crontab for cron.pmdaily and cron.pmcheck; for example:
# daily processing of archive logs 10 0 * * * /usr/pcp/bin/cron.pmdaily # every 30 minutes, check pmlogger instances are running 25,55 * * * * /usr/pcp/bin/cron.pmcheck
Make these changes permanent with this command:
crontab </tmp/foo
You may also choose to enable periodic snapshot images of performance data to be produced from the archive logs using the facilities of cron.pmsnap; instructions for this operation can be found in “Taking Snapshots of pmchart Displays and Value Dialogs”.
Assume the local host has been set up to create archive logs of performance metrics collected from the host oscar (which may be either the local host or a remote host). Execute all of the following tasks while logged into the local host as the superuser (root).
Make sure the optional subsystem pcp_gifts.sw.pmsnap has been installed.
Use the /var/pcp/config/pmsnap/Summary snapshot as an example (you may wish to customize this later).
Ensure that the pmlogger that is collecting performance metrics from the host oscar includes all of the metrics named in the /var/pcp/config/pmlogger/config.Summary configuration file (you may wish to simply use this as the configuration file for this pmlogger instance). If necessary, reconfigure this pmlogger instance as follows:
kill -INT PID-of-pmlogger-instance
Edit the configuration file as required. Restart pmlogger with this command:
/usr/pcp/bin/cron.pmcheck
Check the two Summary lines in the /var/pcp/config/pmsnap/control file. You must replace LOCATHOSTNAME with oscar in both lines (unless oscar is the local host, in which case the change is optional), and you may wish to change the directory for the output files (the default is /var/www/htdocs/snapshots).
Augment the crontab file for root to allow cron.pmsnap to be run periodically. For example:
crontab -l >/tmp/foo
Edit /tmp/foo, adding lines similar to those from /var/pcp/config/pmlogger/crontab for cron.pmsnap; for example:
# every 30 minutes, generate performance snapshot images 30,0 * * * * /usr/pcp/bin/cron.pmsnap -d :0
The snapshots are produced using pmchart, and this tool requires connection to an X server. If the local host is not running a X server, then you must locate a system with an active X server, and ensure that this X server will accept connections from remote X clients; see xhost(1) for details. If this host is grover, then replace
-d :0
in the line above with
-d grover:0
Make these changes permanent with this command:
crontab </tmp/foo
After 30 minutes or so (time enough for cron to complete), check that the GIF files have been created:
ls -l /var/www/htdocs/snapshots
Create a Web page that includes the images. A sample file of HTML source is provided in /var/pcp/config/pmsnap/Summary.html.
A collection of one or more PCP archive logs may be combined with a control file to produce a PCP archive folio. Archive folios are created using either mkaf or the interactive “record mode” services of PCP clients like pmchart and pmview.
The automated archive log management services also create an archive folio named Latest for each managed pmlogger instance, to provide a symbolic name to the most recent archive log. With reference to Figure 7-1, this would mean the creation of the folios /var/adm/pcplog/one/Latest and /var/adm/pcplog/two/Latest.
The pmafm utility is completely described in the pmafm(1) reference page, and provides the interactive commands (single commands may also be executed from the command line) for the following services:
Checking the integrity of the archives in the folio.
Displaying information about the component archives.
Executing PCP tools with their source of performance metrics assigned concurrently to all of the component archives (where the tool supports this), or serially executing the PCP tool once per component archive.
If the folio was created by a single PCP monitoring tool, replaying all of the archives in the folio with that monitoring tool.
Restricting the processing to particular archives, or the archives associated with particular hosts.
The pmlogextract tool takes a number of PCP archive logs from a single host and
merges the archives into a single log, while maintaining the correct timestamps for all values
extracts all metric values within a temporal window that could encompass several archive logs
extracts only a configurable subset of metrics from the archive logs
See the pmlogextract(1) reference page for full information on this command, which replaces functionality of the pmlogmerge tool as of PCP release 2.0.
On each system for which pmcd is active (each PCP collector system), there is an option to have a distinguished instance of the archive logger pmlogger (the “primary” logger) launched each time pmcd is started. This may be used to ensure the creation of minimalist archive logs required for ongoing system management and capacity planning in the event of failure of a system where a remote pmlogger may be running, or because the preferred archive logger deployment is to activate pmlogger on each PCP collector system.
Run the following command as root on each PCP collector system where you want to activate the primary pmlogger:
chkconfig pmlogger on |
The primary logger launches the next time pmcd is started. If you wish this to happen immediately, follow up with this command:
/etc/init.d/pcp start |
When started in this fashion, the file /etc/config/pmlogger.options provides command-line options for pmlogger. In the default setup, this in turn means that the initial logging state and configuration is specified in the file /var/pcp/config/pmlogger/config.default. Either one or both of these files may be modified to tailor pmlogger operation to the local requirements.
You may tailor pmlogger dynamically with the pmlc command. Normally the pmlogger configuration is read at startup, and if you choose to modify the config file to change the parameters under which pmlogger operates, you must stop and restart the program for your changes to have effect. Alternatively, you may change parameters whenever required by using the pmlc interface.
To run the pmlc tool, enter:
pmlc |
By default, pmlc acts on the primary instance of pmlogger on the current host. See the pmlc(1) reference page for a description of command-line options. When invoked, pmlc presents you with a prompt:
pmlc> |
You may obtain a listing of the available commands by entering a question mark (?) and pressing Enter. You see output similar to the following:
show loggers [@<host>] display <pid>s of running pmloggers
connect _logger_id [@<host>] connect to designated pmlogger
status information about connected pmlogger
query metric-list show logging state of metrics
new volume start a new log volume
flush flush the log buffers to disk
log { mandatory | advisory } on <interval> _metric-list
log { mandatory | advisory } off _metric-list
log mandatory maybe _metric-list
timezone local|logger|'<timezone>' change reporting timezone
help print this help message
quit exit from pmlc
_logger_id is primary | <pid> | port <n>
_metric-list is _metric-spec | { _metric-spec ... }
_metric-spec is <metric-name> | <metric-name> [ <instance> ... ]
|
Here is an example
pmlc
pmlc> show loggers @babylon
The following pmloggers are running on babylon:
primary (1892)
pmlc> connect 1892 @babylon
pmlc> log advisory on 2 secs irix.disk.dev.read
pmlc> query irix.disk.dev
irix.disk.dev.read
adv on nl 5 min [131073 or “dks0d1”]
adv on nl 5 min [131074 or “dks0d2”]
pmlc> quit
|
Refer to the pmlc(1) and pmlogger(1) reference pages for complete details.
The following issues concern the creation and use of logs using pmlogger.
| Symptom: | The pmdumplog utility, or any tool that can read an archive log, displays this message:
| |||
| Cause: | An archive consists of at least three physical files. If the base name for the archive is mylog, then the archive actually consists of the physical files mylog.index, mylog.meta, and at least one file mylog.N, where N is in the range 0, 1, 2, 3, and so on. The above message is produced if one or more of the files is missing. | |||
| Resolution: | Use this command to check which files the utility is trying to open:
Turn on the internal debug flag DBG_TRACE_LOG (-D 128) to see which files are being inspected by routine _pmOpenLog. For example:
Locate the missing files and move them all to the same directory, or remove all of the files that are part of the archive, and recreate the archive log. |
| Symptom: | The primary pmlogger cannot be started. A message like the following appears:
| ||
| Cause: | There is either a primary pmlogger already running, or the previous primary pmlogger was terminated unexpectedly before it could perform its cleanup operations. | ||
| Resolution: | If there is already a primary pmlogger running and you wish to replace it with a new pmlogger, use the show command in pmlc to determine the process ID of the primary pmlogger. The process ID of the primary pmlogger appears in parentheses after the word “primary.” Send an SIGINT signal to the process to shut it down (use the kill command). If the process does not exist, proceed to the manual cleanup described in the paragraph below. If the process did exist, it should now be possible to start the new pmlogger. If pmlc's show command displays a process ID for a process that does not exist, a pmlogger process was terminated before it could clean up. If it was the primary pmlogger, the corresponding control files must be removed before one can start a new primary pmlogger. It is a good idea to clean up any spurious control files even if they aren't for the primary pmlogger. The control files are kept in /var/tmp/pmlogger. A control file with the process ID of the pmlogger as its name is created when the pmlogger is started. In addition the primary pmlogger creates a symbolic link named primary to its control file. For the primary pmlogger, remove both the symbolic link and the file (corresponding to its process ID) to which the link points. For other pmloggers, remove just the process ID file. Do not remove any other files in the directory. If the control file for an active pmlogger is removed, pmlc is not able to contact it. |
| Symptom: | You have a PCP archive log that is demonstrably growing, but do not know the identify of the associated pmlogger process. | ||
| Cause: | The PID is not obvious from the log, or the archive name may not be obvious from the output of ps. | ||
| Resolution: | If the archive basename is foo, run the following commands:
All of the information describing the creator of the archive is revealed, and in particular, the instance identifier for the pmcd metrics (10728 in the example above) is the PID of the pmlogger instance, which may be used to control the process via pmlc. |
| Symptom: | PCP tools report:
| ||
| Cause: | Either you are attempting to read a Version 2 archive with a PCP 1.x tool, or the archive log has become corrupted. | ||
| Resolution: | By default, pmlogger in PCP release 2.0 and later generates Version 2 archives that PCP 1.0 to 1.3 tools cannot interpret. If you must use older tools, pass the -V1 option to pmlogger, forcing it to generate Version 1 archives. If the PCP tools are from PCP 2.0 or later, then the archive log may have been corrupted, which can be verified with pmlogcheck(1). |