Chapter 9. System Recovery
Prev |  | Next

## Chapter 9. System Recovery

This chapter provides information on FailSafe system recovery, and includes sections on the following topics:

- “Overview of System Recovery”

- “FailSafe Log Files”

- “FailSafe Membership and Resets”

- “Status Monitoring”

- “Cluster Database Sync Failure”

- “Dynamic Control of FailSafe Services”

- “Recovery Procedures”

## Overview of System Recovery

When a FailSafe system experiences problems, you can use some of the FailSafe features and commands to determine where the problem is located.

FailSafe provides the following tools to evaluate and recover from system failure:

- Log files

- Commands to monitor status of system components

- Commands to start, stop, and fail over highly available services

Keep in mind that the FailSafe logs may not detect system problems that do not translate into FailSafe problems. For example, if a CPU goes bad, or hardware maintenance is required, FailSafe may not be able to detect and log these failures.

In general, when evaluating system problems of any nature on a FailSafe configuration, you should determine whether you need to shut down a node to address those problems. When you shut down a node, perform the following steps:

1. Stop FailSafe HA services on that node

2. Shut down the node to perform needed maintenance and repair

3. Start up the node

4. Start FailSafe HA services on that node

It is important that you explicitly stop FailSafe services before shutting down a node, where possible, so that FailSafe does not interpret the node shutdown as node failure. If FailSafe interprets the service interruption as node failure, there could be unexpected ramifications, depending on how you have configured your resource groups and your application failover domain.

When you shut down a node to perform maintenance, you may need to change your FailSafe configuration to keep your system running.

## FailSafe Log Files

FailSafe maintains system logs for each of the FailSafe daemons. You can customize the system logs according to the level of logging you wish to maintain. Table 9-1 shows the levels of messages.

For information on setting logging for `cad`, `cmond`, and `fs2d`, see “Configure System Files” in Chapter 3. For information on setting up log configurations, see “Set Log Configuration” in Chapter 5 in Chapter 5, “Configuration”.

Table 9-1. Message Levels

Message Level | Description
--- | ---
Normal | Normal messages report on the successful completion of a task. An example of a normal message is as follows (`<N` notation indicates a normal message):
| `Wed Sep 2 11:57:25.284 <N ha_gcd cms 10185:0> Delivering TOTAL membership (S# 1, GS# 1)`
Error/Warning | Error or warning messages indicate that an error has occurred or may occur soon. These messages may result from using the wrong command or improper syntax. An example of a warning message is as follows (`<W` notation indications a warning. `<E` indicates an error.):
| `Wed Sep 2 13:45:47.199 <W crsd crs 9908:0 crs_config.c:634> CI_ERR_NOTFOUND, safer - no such node`
`SYSLOG` | All normal and error messages are also logged to `syslog`. Syslog messages include the symbol `<CI>` in the header to indicate they are cluster-related messages. An example of a syslog message is as follows:
| `Wed Sep 2 12:22:57 6X:safe syslog: <<CI> ha_cmsd misc 10435:0> CI_FAILURE, I am not part of the enabled cluster anymore`
Debug | Debug messages appear in the log group file when the logging level is set to `debug0` or higher (using the GUI) or `10` or higher (using `cmgr`). The following message is logged at `debug0` (see `D0` in the message) or log level `10`:
    Thu Sep 27 14:43:24.233 <D0 ha_fsd fsd 57540:0 fs_failsafe.c:1471> Determine
    oldest state: coordinator: perf22/0x10001

Examining the log files should enable you to see the nature of the system error. Noting the time of the error and looking at the log files to observe the activity of the various daemons immediately before error occurred, you may be able to determine what situation existed that caused the failure.

| Note: Many megabytes of disk space can be consumed on the server when debug levels are used in a log configuration.

## FailSafe Membership and Resets

In looking over the actions of a FailSafe system on failure to determine what has gone wrong and how processes have transferred, it is important to consider the concept of FailSafe membership. When failover occurs, the runtime failover domain can include only those nodes that are in the FailSafe membership.

### FailSafe Membership and Tie-Breaker Node

Nodes can enter into the FailSafe membership only when they are not disabled and they are in a known state. This ensures that data integrity is maintained because only nodes within the FailSafe membership can access the shared storage. If nodes that are outside the membership and are not controlled by FailSafe were able to access the shared storage, two nodes might try to access the same data at the same time; this situation would result in data corruption. For this reason, disabled nodes do not participate in the membership computation.

| Note: No attempt is made to reset nodes that are configured disabled before confirming the FailSafe membership.

FailSafe membership in a cluster is based on a quorum majority. For a cluster to be enabled, more than 50% of the nodes in the cluster must be in a known state, able to talk to each other, using heartbeat control networks. This quorum determines which nodes are part of the FailSafe membership that is formed.

If there are an even number of nodes in the cluster, it is possible that there will be no majority quorum; there could be two sets of nodes, each consisting of 50% of the total number of node, unable to communicate with the other set of nodes. In this case, FailSafe uses the node that has been configured as the tie-breaker node when you configured your FailSafe parameters. If no tie-breaker node was configured, FailSafe uses the node with the lowest ID number where HA services have been started.

The nodes in a quorum attempt to reset the nodes that are not in the quorum. Nodes that can be reset are declared `DOWN` in the membership, nodes that could not be reset are declared `UNKNOWN`. Nodes in the quorum are `UP`.

If a new majority quorum is computed, a new membership is declared whether any node could be reset or not.

If at least one node in the current quorum has a current membership, the nodes will proceed to declare a new membership if they can reset at least one node.

If all nodes in the new tied quorum are coming up for the first time, they will try to reset and proceed with a new membership only if the quorum includes the tie-breaker node.

If a tied subset of nodes in the cluster had no previous membership, then the subset of nodes in the cluster with the tie-breaker node attempts to reset nodes in the other subset of nodes in the cluster. If at least one node reset succeeds, a new membership is confirmed.

If a tied subset of nodes in the cluster had previous membership, the nodes in one subset of nodes in the cluster attempt to reset nodes in the other subset of nodes in the cluster. If at least one node reset succeeds, a new membership is confirmed. The subset of nodes in the cluster with the tie-breaker node resets immediately; the other subset of nodes in the cluster attempts to reset after some time.

Resets are done through system controllers connected to tty ports through serial lines. Periodic serial line monitoring never stops. If the estimated serial line monitoring failure interval and the estimated heartbeat loss interval overlap, the cause is likely a power failure at the node being reset.

### No Membership Formed

When no FailSafe membership is formed, you should check the following areas for possible problems:

- Is the `ha_cmsd` FailSafe membership daemon running? Is the `fs2d` database daemon running?

- Can the nodes communicate with each other? Are the control networks configured as heartbeat networks?

- Can the control network addresses be reached by a `ping(1)` command issued from peer nodes?

- Are the quorum majority or tie rules satisfied? Look at the `cmsd` log to determine membership status.

- If a reset is required, are the following conditions met?

  - Is the `crsd` node control daemon up and running?

  - Is the reset serial line in good health?

  You can look at the `crsd` log for the node you are concerned with, or execute an `admin ping` and `admin reset` command on the node to check this.

## Status Monitoring

FailSafe allows you to monitor and check the status of specified clusters, nodes, resources, and resource groups. You can use this feature to isolate the location of system problems.

You can monitor the status of the FailSafe components continuously through their visual representation in the GUI tree view. Using the `cmgr` command, you can display the status of the individual components by using the `show` command.

For information on status monitoring and on the meaning of the states of the FailSafe components, see “System Status” in Chapter 7 of Chapter 7, “IRIS FailSafe System Operation”.

## Dynamic Control of FailSafe Services

FailSafe allows you to perform a variety of administrative tasks that can help you troubleshoot a system with problems without bringing down the entire system. These tasks include the following:

- You can add or delete nodes from a cluster without affecting the FailSafe services and the applications running in the cluster.

- You can add or delete a resource group without affecting other online resource groups.

- You can add or delete resources from a resource group while it is still online.

- You can change FailSafe parameters such as the heartbeat interval and the node timeout and have those values take immediate affect while the services are up and running.

- You can start and stop FailSafe services on specified nodes.

- You can move a resource group online, or take it offline.

- You can stop the monitoring of a resource group by putting the resource group into maintenance mode. This is not an expensive operation, as it does not stop and start the resource group, it just puts the resource group in a state where it is not available to FailSafe.

- You can reset individual nodes.

For information on how to perform these tasks, see Chapter 5, “Configuration”, and Chapter 7, “IRIS FailSafe System Operation”.

## Recovery Procedures

The following sections describe various recovery procedures you can perform when different failsafe components fail. Procedures for the following situations are provided:

- “Single-Node Recovery”

- “Cluster Error Recovery”

- “Resource Group Recovery”

- “Node Error Recovery”

- “Resource Group Maintenance and Error Recovery”

- “Clear Resource Error State”

- “Control Network Failure Recovery”

- “Serial Cable Failure Recovery”

- “Cluster Database Sync Failure”

- “Cluster Database Maintenance and Recovery”

- “GUI and cmgr Inconsistencies”

- “GUI Does Not Report Information”

- “Using the cdbreinit Command”

### Single-Node Recovery

When one of the nodes in a two-node cluster is intended to stay down for maintenance or cannot be brought up, a set of procedures must be followed so that the database on the surviving node knows that that node is down and therefore should not to be considered in the failover domain. Without these procedures, the surviving node will be in what is called the lonely state, meaning that it will never form a cluster by itself.

See the procedure in “Two-Node Clusters: Single-Node Use” in Chapter 7.

### Cluster Error Recovery

Use the following procedure if status of the cluster is `UNKNOWN` in all nodes in the cluster:

1. Check to see if there are control networks that have failed (see “Control Network Failure Recovery”).

2. Determine if there are sufficient nodes in the cluster that can communicate with each other using control networks in order to form a quorum. (At least 50% of the nodes in the cluster must be able to communicate with each other.) If there is an insufficient number of nodes, stop HA services on the nodes that cannot communicate (using the `force` option); this will change the number of nodes used in the quorum calculation.

3. If there are no hardware configuration problems, do the following:

   - Detach all resource groups that are online in the cluster (if any)

   - Stop HA services in the cluster

   - Restart HA services in the cluster

   See “Resource Group Recovery”

For example, the following `cmgr` command detaches the resource group `web-rg` in cluster `web-cluster`:

    cmgr> admin detach resource_group web-rg in cluster web-cluster

To stop HA services in the cluster `web-cluster` and ignore errors (`force` option), use the following command:

    cmgr> stop ha_services for cluster web-cluster force

To start HA services in the cluster `web-cluster`, use the following command:

    cmgr> start ha_services for cluster web-cluster

### Resource Group Recovery

The fact that a resource group is in an error state does not mean that all resources in the resource group have failed. However, to get the resources back into an `online` state, you must first set them to the `offline` state. You can do without actually taking the resources offline by using the following `cmgr` command:

    admin offline_detach_force resource_group [in cluster clustername]

For example:

    cmgr> admin offline_detach_force RG1 in cluster test-cluster

| Caution: You should use the `InPlace_Recovery` failover policy attribute when using this command. This attribute specifies that the resources will stay on the same node where they were running at the time when the `offline_detach_force` command was run.

### Node Error Recovery

When a node is not able to talk to the majority of nodes in the cluster, the `SYSLOG` will display a message that the CMSD is in a lonely state. Another problem you may see is that a node is getting reset or going to an unknown state.

Use the following procedure to resolve node errors:

1. Check to see if the control networks in the node are working (see “Control Network Failure Recovery”).

2. Check to see if the serial reset cables to reset the node are working (see “Serial Cable Failure Recovery”).

3. Verify that the `sgi-cmsd` port is the same in all nodes in the cluster.

4. Check the node configuration; it should be consistent and correct.

5. Check `syslog` and `cmsd` logs for errors. If a node is not joining the cluster, check the logs of the nodes that are part of the cluster.

6. If there are no hardware configuration problems, stop HA services in the node and restart HA services.

   For example, to stop HA services in the node `web-node3` in the cluster `web-cluster`, ignoring errors (`force` option), use the following command:

       cmgr> stop ha_services in node web-node3 for cluster web-cluster force

   For example, to start HA services in the node `web-node3` in the cluster `web-cluster`, use the following command:

       cmgr> start ha_services in node web-node3 for cluster web-cluster

### Resource Group Maintenance and Error Recovery

To do simple maintenance on an application that is part of the resource group, use the following procedure. This procedure stops monitoring the resources in the resource group when maintenance mode is on. You must turn maintenance mode off when performing application maintenance.

| Caution: If there is a node failure on the node where resource group maintenance is being performed, the resource group is moved to another node in the failover policy domain.

For example:

1. To put a resource group `web-rg` in maintenance mode, use the following `cmgr` command:

       cmgr> admin maintenance_on resource_group web-rg in cluster web-cluster

2. The resource group state changes to `ONLINE_MAINTENANCE`. Do whatever application maintenance is required. (Rotating application logs is an example of simple application maintenance).

3. To remove a resource group `web-rg` from maintenance mode, use the following command:

       cmgr> admin maintenance_off resource_group web-rg in cluster web-cluster

   The resource group state changes back to `ONLINE`.

Perform the following procedure when a resource group is in an `ONLINE` state and has an `SRMD EXECUTABLE ERROR`:

1. Look at the SRM logs (default location: `/var/cluster/ha/logs/srmd_`nodename) to determine the cause of failure and the resource that has failed. Search for the `ERROR` string in the SRMD log file:

       Wed Nov 3 04:20:10.135
       <E ha_srmd srm 12127:1 sa_process_tasks.c:627>
       CI_FAILURE, ERROR: Action (start) for resource (192.0.2.45) of type
       (IP_address) failed with status (failed)

2. Check the script logs on that same node for `IP_address` start script errors.

3. Fix the cause of failure. This might require changes to resource configuration or changes to resource type stop/start/failover action timeouts.

4. After fixing the problem, move the resource group offline with the `force` option and then move the resource group online in the cluster.

   For example, the following command moves the resource group `web-rg` in the cluster `web-cluster` offline and ignores any errors:

       cmgr> admin offline resource_group web-rg in cluster web-cluster force

   The following command moves the resource group `web-rg` in the cluster `web-cluster` online:

       cmgr> admin online resource_group web-rg in cluster web-cluster

   The resource group `web-rg` should be in an `ONLINE` state with no error.

Use the following procedure when a resource group is not online but is in an error state. Most of these errors occur as a result of the exclusivity process. This process, run when a resource group is brought online, determines if any resources are already allocated somewhere in the failure domain of a resource group. Note that exclusivity scripts return that a resource is allocated on a node if the script fails in any way. In other words, unless the script can determine that a resource is not present, it returns a value indicating that the resource is allocated.

Some possible error states include: `SPLIT RESOURCE GROUP (EXCLUSIVITY)`, `NODE NOT AVAILABLE (EXCLUSIVITY)`, `NO AVAILABLE NODES` in failure domain. See “Resource Group Status” in Chapter 7, for explanations of resource group error codes.

1. Look at the `failsafe` and SRMD logs (default directory: `/var/cluster/ha/logs`, files: `failsafe_`nodename, `srmd_`nodename) to determine the cause of the failure and the resource that failed.

   For example, suppose that the task of moving a resource group online results in a resource group with error state `SPLIT RESOURCE GROUP (EXCLUSIVITY)`. This means that parts of a resource group are allocated on at least two different nodes. One of the failsafe logs will have the description of which nodes are believed to have the resource group partially allocated:

       [Resource Group:RG_name]:Exclusivity failed -- RUNNING on node1 and node2

       [Resource Group:RG_name]:Exclusivity failed -- PARTIALLY RUNNING on node1 and PARTIALLY RUNNING on node2

   At this point, look at the `srmd` logs on each of these nodes for exclusive script errors to see what resources are believed to be allocated. In some cases, a misconfigured resource will show up as a resource that is allocated. This is especially true for `Netscape_web` resources.

2. Fix the cause of the failure. This might require changes to resource configuration or changes to resource type start/stop/exclusivity timeouts.

3. After fixing the problem, move the resource group offline with the `force` option and then move the resource group online.

Perform the following checks when a resource group shows a `no more nodes in AFD` error:

1. All nodes in the failover domain are not in the membership. Check CMSD logs for errors.

2. Check the SRMC/script logs on all nodes in the failover domain for start/monitor script errors.

There are a few double failures that can occur in the cluster that will cause resource groups to remain in a non-highly-available state. At times a resource group might be stuck in an offline state. A resource group might also stay in an error state on a node even when a new node joins the cluster and the resource group can migrate to that node to clear the error. When these circumstances arise, do the following:

1. If the resource group is offline, try to move it online.

2. If the resource group is stuck on a node, detach the resource group and then bring it back online again. This should clear many errors.

3. If detaching the resource group does not work, force the resource group offline, then bring it back online.

4. If commands appear to be hanging or not working properly, detach all resource groups, then shut down the cluster and bring all resource groups back online.

See “Take a Resource Group Offline” in Chapter 7, for information on detaching resource groups and forcing resource groups offline.

### Clear Resource Error State

Use this procedure when a resource that is not part of a resource group is in an `ONLINE` state with an error. This can happen when the addition or removal of resources from a resource group fails.

Do the following:

1. Look at the SRM logs to determine the cause of failure and the resource that has failed. The default location is:

       /var/cluster/ha/logs/srmd_nodename

2. Fix the problem that caused the failure. This might require changes to resource configuration or changes to resource type stop/start/failover action timeouts.

3. Clear the error state with the GUI or the `cmgr` command:

   - Use the Clear Resource Error State GUI task. Provide the following information:

     - Resource Type: select the type of the resource

     - Resource in Error State: select the name of the resource that should be cleared from the error state

     Click OK to complete the task.

   - Use the `cmgr admin offline_force` command to move the resource offline. For example, to remove the error state of resource `web-srvr` of type `Netscape_Web`, making it available to be added to a resource group, enter the following:

         cmgr> admin offline_force resource web-srvr of resource_type Netscape_Web in cluster web-cluster

### Control Network Failure Recovery

Control network failures are reported in `cmsd` logs. The default location of `cmsd` log is `/var/cluster/ha/logs/cmsd_`node name. Follow this procedure when the control network fails:

1. Use the `ping(1M)` command to check whether the control network IP address is configured in the node.

2. Check node configuration to see whether the control network IP addresses are correctly specified.

   The following `cluster_mgr` command displays node configuration for `web-node3:`

       cmgr> show node web-node3

3. If IP names are specified for control networks instead of IP addresses in XX.XX.XX.XX notation, check to see whether IP names can be resolved using DNS. It is recommended that IP addresses are used instead of IP names.

4. Check whether the heartbeat interval and node timeouts are correctly set for the cluster. These HA parameters can seen using `cluster_mgr show ha_parameters` command.

### Serial Cable Failure Recovery

Serial cables are used for resetting a node when there is a node failure. Serial cable failures are reported in `crsd` logs. The default location for the `crsd` log is `/var/cluster/ha/log/crsd_`nodename.

Check the node configuration to see whether serial cable connection is correctly configured.

The following `cmgr` command displays node configuration for `web-node3`

    cmgr> show node web-node3

Use the `admin ping` command to verify the serial cables. The following command reports serial cables problems in node `web-node3`:

    cmgr> admin ping node web-node3

### Cluster Database Sync Failure

If the cluster database synchronization fails, use the following procedure:

1. Check for the following message in the `/var/adm/SYSLOG` file on the target node:

       Starting to receive CDB sync series from machine <node1_node_ID>
       ...
       Finished receiving CDB sync series from machine <node1_node_ID>

2. Check for control network or `portmapper/rpcbind` problems.

3. Check the node definition in the cluster database.

4. Check the `SYSLOG` and `fs2d` logs on the source node.

### Cluster Database Maintenance and Recovery

When the entire cluster database must be reinitialized, execute the following command:

    # /usr/cluster/bin/cdbreinit /var/cluster/cdb/cdb.db

This command restarts all cluster processes. The contents of the configuration database will be automatically synchronized with other nodes if other nodes in the pool are available.

Otherwise, the cluster database must be restored from backup at this point. For instructions on backing up and restoring the cluster database, see “Backing Up and Restoring Configuration With cmgr” in Chapter 7.

### GUI and `cmgr` Inconsistencies

If the GUI is displaying information that is inconsistent with the FailSafe `cmgr` command, restart `cad` process on the node to which GUI is connected to by executing the following command:

    # killall cad

The cluster administration daemon is restarted automatically by the `cmond` process.

### GUI Does Not Report Information

If the GUI is not reporting configuration information and status, perform the following steps:

1. Check the information using the `cmgr(1M)` command. If `cmgr` is reporting correct information, there is a GUI update problem.

2. If there is a GUI update problem, kill the `cad` daemon on that node. Wait for a couple of minutes to see whether `cad` gets correct information. Check the `cad` logs on that node for errors.

3. Check the CLI logs on that node for errors.

4. If the status information is incorrect, check the `cmsd` or `fsd` logs on that node.

### Using the `cdbreinit` Command

When the configuration databases are not in synchronization on all the nodes in the cluster, you can run the `cdbreinit` command to recover. The `cdbreinit` command should be run on the node which is not in sync.

Perform the following steps.

| Note: Perform each step on all the nodes before proceeding to the next step in the recovery procedure.

1. Stop FailSafe services in the cluster using the GUI or `cmgr`.

2. Stop cluster processes on all nodes in the pool:

       # /etc/init.d/cluster stop
       # killall fs2d

3. Run `cdbreinit` on the node where the cluster database is not in sync.

4. Start cluster processes on all nodes in the pool:

       # /etc/init.d/cluster start

5. Wait a couple of minutes for the cluster database to sync. There will be cluster database sync long messages in the `SYSLOG` on the node.

6. Start FailSafe services in the cluster.

Prev | Table of Contents | Next
Chapter 8. Testing the Configuration |  | Chapter 10. Upgrading and Maintaining Active Clusters