Chapter 11. Performance Co-Pilot for FailSafe
Prev |  | Next

## Chapter 11. Performance Co-Pilot for FailSafe

This chapter tells you how to use Performance Co-Pilot (PCP) for FailSafe to monitor the availability of an IRIX FailSafe cluster. For information about installing PCP for FailSafe, see “Install Performance Co-Pilot (PCP) Software” in Chapter 3.

PCP provides the following:

- An agent for exporting FailSafe heartbeat and resource monitoring statistics to the PCP framework

- 3-D visualization tools for displaying these statistics in an intuitive presentation

The visualization of statistics provides valuable information about the availability of nodes and resources monitored by FailSafe. For example, it can highlight a reduction in monitoring response times that may indicate problems in availability of services provided by the cluster.

Because PCP for FailSafe is an extension to the PCP framework, you can use other PCP tools to analyze or present FailSafe monitoring statistics, and record PCP for FailSafe metrics as archives for deferred analysis. You can also use PCP to gather statistics about CPU and memory utilization, network and disk activity, and other performance metrics for each node in the cluster.

## Using the Visualization Tools

To view statistics about the FailSafe cluster, use the `hbvis(1)` and `rmvis(1)` commands.

The `hbvis(1)` command constructs a display showing the distribution of heartbeat response times for every node in the cluster. Figure 11-1 shows an example display.

Figure 11-1. Heartbeat Response Statistics

Key features of the display include the frequency of heartbeat responses that arrive at particular intervals within the timeout period and the frequency of heartbeat responses that have been missed (determined not to have arrived). The bar representing the frequency of missed heartbeat responses changes color to indicate the urgency of problems with availability of a node.

The `rmvis(1)` command constructs a display of the resource monitoring response times for resources monitored on every node of the cluster. Figure 11-2 shows an example display.

Figure 11-2. Resource Monitoring Statistics

The display is similar in concept to that of `hbvis(1)`, showing the frequency of resource monitoring responses that arrive within the timeout period, and the frequency of responses that have timed out. The bar representing the frequency of resource responses that have timed out also changes color to indicate the urgency of problems with the availability of particular resources.

If a node has failed or a resource has failed over, its statistics will disappear from the display.

To run a visualization tool on the monitor host, use the `-h` option to specify an available collector host in the cluster (host):

    % hbvis -h host

or

    % rmvis -h host

The collector host specified can be any collector host that is a member of the cluster for which you wish to view statistics.

There are various options available to alter the display provided by `hbvis(1)` and `rmvis(1)`:

`-H hostfile` | Provides a file that lists the nodes that are to appear in the visualization. This is useful in limiting the number of nodes in the display, because it takes more time to construct the display for clusters with more nodes.
`-t interval` | Assigns the sampling time of the visualization. There may be circumstances where extending the period of the sampling time may provide better application responsiveness, particularly for clusters with many nodes. Because FailSafe maintains the statistics, `hbvis(1)` and `rmvis(1)` will always show the latest statistics available for the sampling time selected. For details about the interval option, see the `pmview(1)` and `PCPIntro(1)` man pages.
`-r` | Selects the FailSafe metrics that present a sampling of statistics taken from the time of the last statistical reset. This enables `hbvis(1)` and r`mvis(1)` to improve the sensitivity of the visualization when abrupt changes appear in the FailSafe monitoring statistics. Without the `-r` option, the statistics presented are from a sampling of FailSafe metrics collected from the time `ha_cmsd(1m)` and/or `ha_srmd(1m)` was last restarted.
`-R` | Starts a new statistical sampling.
`-v` | (`hbvis(1)` only) Provides a visualization of heartbeat statistics for each node in the cluster, from the point of view of the selected collector host only. (The collector host is selected using the `-h` option). There is a graphical representation of heartbeat statistics for each node in the cluster as observed by the selected collector host.
`-w` | (`hbvis(1)` only) Provides a visualization of the aggregate of heartbeat statistics for all nodes in the cluster, from the point of view of the selected collector host only. (The collector host is selected using the `-h` option). There is a only one graphical representation of heartbeat statistics for the entire cluster as observed by the selected collector host.

For a complete description of options, see the `hbvis(1)` and `rmvis(1)` man pages.

`hbvis(1)` and `rmvis(1)` use the command `pmview(1)` to display the 3-D visualization of FailSafe performance metrics. For a description of the various menu commands and controls in the visualization window, consult the man pages for `pmview(1)`.

## PCP for FailSafe Performance Metrics

PCP tools such as `pmlogger(1)`, `pmchart(1)`, and `pminfo(1)` can use the metrics exported by PCP for FailSafe.

Appendix C, “Metrics Exported by PCP for FailSafe”, provides a description of PCP for FailSafe metrics. You can also display a description of metrics by using the following command:

    % pminfo -tT -h host

(If you are logged in to a collector host, you can leave out the `-h` option).

## Troubleshooting

A gray display (that is, no colored rectangle bars appear on the node's gray baseplane) when using `hbvis(1)` or `rmvis(1)` may indicate one of the following:

- The node is down.

  If you wish to see only the nodes that are up, create a file containing a list of nodes that are to be displayed and pass it as an option to `hbvis(1)`/`rmvis(1)` using the `-H` option (or the environment variable `PCP_FSAFE_NODES`) so that a new picture of the cluster can be generated. Please refer to the `hbvis(1)`/`rmvis(1)` man pages for more details on the `-H` option.

- The collector daemons have been killed on that node.

  To solve this problem, restart `pmdafsafe(1)` in one of the following ways:

  - If `pmcd(1)` is still running, send `pmcd(1)` the `SIGHUP` signal by entering the following:

        # killall -HUP pmcd

  - If `pmcd(1)` is not running, restart PCP by entering the following:

        # /etc/init.d/pcp start

- The timeout and sampling settings are too short.

  To change the sampling time, use the time controls available in the `pmview(1)` window. By default, this is two seconds; you may need to lengthen the sampling period if you are getting an unsatisfactory display.

  Alternatively, there may be timeout issues between `pmdafsafe(1)` and `pmcd(1)`, or between `pmcd(1)` and `pmview(1)`. Refer to the man pages for `pmcd(1)` and `PCPIntro(1)` for information on how to change the timeout settings for the various PCP tools.

- The resource has failed over (for `rmvis(1)`).

  In this case, restart `rmvis(1)` so that a new picture of the cluster can be generated.

Prev | Table of Contents | Next
Chapter 10. Upgrading and Maintaining Active Clusters |  | Appendix A. Updating from IRIS FailSafe 1.2 to IRIS FailSafe 2.1.x