Chapter 8. Testing the Configuration
Prev |  | Next

## Chapter 8. Testing the Configuration

This chapter explains how to test the IRIS FailSafe system configuration using the FailSafe Manager GUI and the `cmgr(1M)` command. For general information on using these tools, see Chapter 4, “Administration Tools”.

The sections in this chapter are as follows:

- “Overview of FailSafe Diagnostic Commands”

- “Performing Diagnostic Tasks with the GUI”

- “Performing Diagnostic Tasks with cmgr”

## Overview of FailSafe Diagnostic Commands

Table 8-1 shows the tests you can perform with FailSafe diagnostic commands:

Table 8-1. FailSafe Diagnostic Test Summary

Diagnostic Test | Description
--- | ---
Resource | Checks that: Resource type parameters are set Parameters are syntactically correct Parameters exist
Resource group | Tests all resources defined in the resource group
Failover policy | Checks that: Failover policy exists Failover domain contains a valid list of hosts
Network connectivity | Checks that: The control interfaces are on the same network The nodes can communicate with each other
Serial connection | Checks that the nodes can reset each other

All transactions are logged to the diagnostics file `diags_`nodename in the log directory.

You should test resource groups before starting FailSafe HA services or starting a resource group. These tests are designed to check for resource inconsistencies that could prevent the resource group from starting successfully.

## Performing Diagnostic Tasks with the GUI

This section describes how to perform diagnostic tasks with the GUI.

### Test Connectivity with the GUI

This task requires `root` `rsh(1)` access between nodes. To test connectivity, do the following from the FailSafe Manager:

| Note: The Test Node Connectivity screen requires `rsh(1)` access between hosts. The `/.rhosts` file must contain the hosts and local host between which you want to test connectivity.

- Choose whether to test by network or serial connectivity by clicking on the appropriate radio button.

- Choose a node to be tested from the pull-down list and add it to the test list by clicking on Add.

  To delete a node from the list of nodes to be tested, click on the logical name to select it and then click on Delete.

- To start the tests, click on Start Tests. To stop the tests, click on Stop Tests.

- To run another test, click on Clear Output to clear the status screen and start over with step 3.

- To exit from the window, click on Close.

### Test Resources with the GUI

The Test Resources task lets you test the resources on the nodes in your cluster by entering the requested inputs. You can test resources by type and by group. You can test the resources of a resource type or in a resource group on all of the nodes in the cluster at one time, or you can specify an individual node to test. Resource tests are performed only on nodes in the resource group's application failover domain.

### Test Failover Policies with the GUI

The Test Failover Policy task lets you test whether a failover policy is defined correctly. This test checks the failover policy by validating the policy script and failover attributes, and whether the application failover domain consists of valid nodes from the cluster.

## Performing Diagnostic Tasks with `cmgr`

The following subsections described how to perform diagnostic tasks on your system using the `cmgr` command.

### Test the Serial Connections with `cmgr`

You can use the `cmgr` command to test the serial connections between the FailSafe nodes. This test performs a `ping(1)` on each specified node through the serial line and produces an error message if the `ping` is not successful.

| Note: Do not execute this command while FailSafe is running.

Use the following command to test the serial connections for the machines in a cluster:

    test serial in cluster C_name [on node node1 node node2 ...]

For example, to test multiple nodes:

    cmgr> test serial in cluster test-cluster on node blue node green

The serial test yields an error message when it encounters its first error, indicating the node that did not respond. If you receive an error message after executing this test, verify the cable connections of the serial cable from the indicated node's serial port to the remote power control unit or the system controller port of the other nodes and run the test again.

For example:

    cmgr> test serial in cluster eagan on node cm1
    Success: testing serial...
    Success: Ensuring Node Can Get IP Addresses For All Specified Hosts
    Success: Number of IP addresses obtained for <cm1> = 1
    Success:        The first IP address for <cm1> = 128.162.19.34
    Success: Checking serial lines via crsd (crsd is running)
    Success: Successfully checked serial line
    Success: Serial Line OK
    Success: overall exit status:success, tests failed:0, total tests executed:1

The following shows an example of an attempt to run the `test serial` command while FailSafe is running (causing the command to fail to execute):

    cmgr> test serial in cluster eagan on node cm1
    Error: Cannot run the serial tests, diagnostics has detected FailSafe (ha_cmsd) is running

    Failed to execute FailSafe tests/diagnostics ha

    test command failed
    cmgr>

### Test Network Connectivity with `cmgr`

You can use the `cmgr` command to test the network connectivity in a cluster. This test checks if the specified nodes can communicate with each other through each configured interface in the nodes. This test will not run if FailSafe is running.

Use the following command to test the network connectivity for the machines in a cluster:

    test connectivity in cluster C_name [on node node1 node node2 ...]

The following shows an example of the `test connectivity` command:

    cmgr> test connectivity in cluster eagan on node cm1
    Success: testing connectivity...
    Success: checking that the control IP_addresses are on the same networks
    Success: pinging address cm1-priv interface ef0 from host cm1
    Success: pinging address cm1 interface ef1 from host cm1
    Success: overall exit status:success, tests failed:0, total tests
    executed:1

This test yields an error message when it encounters its first error, indicating the node that did not respond. If you receive an error message after executing this test, verify that the network interface has been configured up, using the `ifconfig` command, for example:

# `/usr/etc/ifconfig ec3`

    ec3: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,FILTMULTI,MULTICAST>
            inet 190.0.3.1 netmask 0xffffff00 broadcast 190.0.3.255

The `UP` in the first line of output indicates that the interface is configured up.

If the network interface is configured up, verify that the network cables are connected properly and run the test again.

### Test Resources with `cmgr`

You can use the `cmgr` command to test any configured resource by resource name or by resource type.

Use the following to test a resource by name:

    test resource resourcename of resource_type RT_name  in cluster C_name [on node node1  node node2 ...]

For example:

    cmgr> test resource /disk1 of resource_type filesystem in cluster eagan on machine cm1
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all filesystem resources on node cm1 ***
    Success: testing resource /disk1 of resource type filesystem on node cm1
    Success: overall exit status:success, tests failed:0, total tests executed:1

Use the following to test a resource by resource type:

    test resource_type RT_name in cluster C_name [on node node1 node node2 ...]

For example:

    cmgr> test resource_type filesystem in cluster eagan on machine cm1
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all filesystem resources on node cm1 ***
    Success: testing resource /disk4 of resource type filesystem on node cm1
    Success: testing resource /disk5 of resource type filesystem on node cm1
    Success: testing resource /disk2 of resource type filesystem on node cm1
    Success: testing resource /disk3 of resource type filesystem on node cm1
    Success: testing resource /disk1 of resource type filesystem on node cm1
    Success: overall exit status:success, tests failed:0, total tests executed:5

You can use `cmgr` to test volume and filesystem resources in `destructive` mode. This provides a more thorough test of filesystems and volumes. `cmgr` tests will not run in destructive mode if FailSafe is running.

Use the following to test resources in destructive mode:

    test resource resourcename of resource_type RT_name  in cluster C_name  [on node node1 node node2 ...] destructive

The following sections describe the diagnostic tests available for resources.

#### Test Logical Volumes with `cmgr`

You can use the `cmgr` command to test the logical volumes in a cluster. This test checks if the specified volume is configured correctly.

Use the following command to test a logical volume:

    test resource resourcename of resource_type volume on cluster C_name[on node node1 node node2  ...]

For example:

    cmgr> test resource alternate of resource_type volume on cluster eagan
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all volume resources on node cm1 ***
    Success: running resource type volume tests on node cm1
    Success: *** testing node resources on node cm2 ***
    Success: *** testing all volume resources on node cm2 ***
    Success: running resource type volume tests on node cm2
    Success: overall exit status:success, tests failed:0, total tests executed:2

The following example tests a logical volume in destructive mode:

    cmgr> test resource alternate of resource_type volume on cluster eagan destructive
    Warning: executing the tests in destructive mode
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all volume resources on node cm1 ***
    Success: running resource type volume tests on node cm1
    Success: successfully assembled volume: alternate
    Success: *** testing node resources on node cm2 ***
    Success: *** testing all volume resources on node cm2 ***
    Success: running resource type volume tests on node cm2
    Success: successfully assembled volume: alternate
    Success: overall exit status:success, tests failed:0, total tests executed:2

#### Test Filesystems with `cmgr`

You can use `cmgr` to test the filesystems configured in a cluster. This test checks if the specified filesystem is configured correctly and if the volume the filesystem will reside on is configured correctly.

Use the following command to test a filesystem:

    test resource resourcename of resource_type filesystems on cluster C_name [on node node1 node node2 ...]

The following example displays the filesystems that have been defined in a cluster and tests one of them:

    cmgr> show resources of resource_type filesystem in cluster eagan
    /disk4 type filesystem
    /disk5 type filesystem
    /disk2 type filesystem
    /disk3 type filesystem
    /disk1 type filesystem
    cmgr> test resource /disk4 of resource_type filesystem in cluster eagan on node cm1
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all filesystem resources on node cm1 ***
    Success: successfully mounted filesystem: /disk4
    Success: overall exit status:success, tests failed:0, total tests executed:1

The following example tests a filesystem in destructive mode:

    cmgr> test resource /disk4 of resource_type filesystem in cluster eagan on node cm1 destructive
    Warning: executing the tests in destructive mode
    Success: *** testing node resources on node cm1 ***
    Success: *** testing all filesystem resources on node cm1 ***
    Success: successfully mounted filesystem: /disk4
    Success: overall exit status:success, tests failed:0, total tests executed:1

#### Test Resource Groups with `cmgr`

You can use `cmgr` to test a resource group. This test cycles through the resource tests for all of the resources defined for a resource group. Resource tests are performed only on nodes in the resource group's application failover domain.

Use the following to test resource groups:

    test resource_group RG_name in cluster C_name [on node node1 node node2 ...]

The following displays the resource groups that have been defined in a cluster and test one of them:

    cmgr> show resource_groups in cluster eagan
    Resource Groups:
            nfs2
            informix
    cmgr> test resource_group nfs2 in cluster eagan on machine cm1
    Success: *** testing node resources on node cm1 ***
    Success: testing resource /disk4 of resource type NFS on node cm1
    Success: testing resource /disk3 of resource type NFS on node cm1
    Success: testing resource /disk3/statmon of resource type statd on node cm1
    Success: testing resource 128.162.19.45 of resource type IP_address on node cm1
    Success: testing resource /disk4 of resource type filesystem on node cm1
    Success: testing resource /disk3 of resource type filesystem on node cm1
    Success: testing resource dmf1 of resource type volume on node cm1
    Success: testing resource dmfjournals of resource type volume on node cm1
    Success: overall exit status:success, tests failed:0, total tests executed:16

### Test Failover Policies with `cmgr`

You can use `cmgr` to test whether a failover policy is defined correctly. This test checks the failover policy by validating the policy script, failover attributes, and whether the application failover domain consists of valid nodes from the cluster.

Uses the following to test a failover policy:

    test failover_policy FP_name in cluster C_name [on node node1 node node2 ...]

The following example uses a `show` command to display the failover policies that have been defined in a cluster and tests one of them:

    cmgr> show failover_policies
    Failover Policies:
            reverse
            ordered-in-order
    cmgr> test failover_policy reverse in cluster eagan
    Success: *** testing node resources on node cm1 ***
    Success: testing policy reverse on node cm1
    Success: *** testing node resources on node cm2 ***
    Success: testing policy reverse on node cm2
    Success: overall exit status:success, tests failed:0, total tests executed:2

Prev | Table of Contents | Next
Chapter 7. IRIS FailSafe System Operation |  | Chapter 9. System Recovery