This chapter provides information on FailSafe system recovery, and includes sections on the following topics:
When a FailSafe system experiences problems, you can use some of the FailSafe features and commands to determine where the problem is located.
FailSafe provides the following tools to evaluate and recover from system failure:
Log files
Commands to monitor status of system components
Commands to start, stop, and fail over highly available services
Keep in mind that the FailSafe logs may not detect system problems that do not translate into FailSafe problems. For example, if a CPU goes bad, or hardware maintenance is required, FailSafe may not be able to detect and log these failures.
In general, when evaluating system problems of any nature on a FailSafe configuration, you should determine whether you need to shut down a node to address those problems.
When you shut down a node, perform the following steps:
Stop FailSafe HA services on that node
Shut down the node to perform needed maintenance and repair
Start up the node
Start FailSafe HA services on that node
It is important that you explicitly stop FailSafe services before shutting down a node, where possible, so that FailSafe does not interpret the node shutdown as node failure. If FailSafe interprets the service interruption as node failure, there could be unexpected ramifications, depending on how you have configured your resource groups and your application failover domain.
When you shut down a node to perform maintenance, you may need to change your FailSafe configuration to keep your system running.
If you must disable resources, such as when you want to perform maintenance on a node, use the following procedure:
Offline the resource groups by using the offline_detach option or offline_detach_force option (if the resource group is in error). For more information, see “Resource Group Recovery” , and “Resource Group Maintenance and Error Recovery ”.
Perform the needed maintenance.
Reboot the node.
Online the resource group.
FailSafe maintains system logs for each of the FailSafe daemons. You can customize the system logs according to the level of logging you wish to maintain. Table 9-1 shows the levels of messages.
For information on setting logging for cad, cmond, and fs2d, see “Configure System Files” in Chapter 3. For information on setting up log configurations, see “Set Log Configuration” in Chapter 5 in Chapter 5, “Configuration”.
Examining the log files should enable you to see the nature of the system error. Noting the time of the error and looking at the log files to observe the activity of the various daemons immediately before error occurred, you may be able to determine what situation existed that caused the failure.
 | Note: Many megabytes of disk space can be consumed on the server when debug levels are used in a log configuration. |
In looking over the actions of a FailSafe system on failure to determine what has gone wrong and how processes have transferred, it is important to consider the concept of FailSafe membership. When failover occurs, the runtime failover domain can include only those nodes that are in the FailSafe membership.
Nodes can enter into the FailSafe membership only when they are not disabled and they are in a known state. This ensures that data integrity is maintained because only nodes within the FailSafe membership can access the shared storage. If nodes that are outside the membership and are not controlled by FailSafe were able to access the shared storage, two nodes might try to access the same data at the same time; this situation would result in data corruption. For this reason, disabled nodes do not participate in the membership computation.
| Note: No attempt is made to reset nodes that are configured disabled before confirming the FailSafe membership. |
FailSafe membership in a cluster is based on a quorum majority. For a cluster to be enabled, more than 50% of the nodes in the cluster must be in a known state, able to talk to each other, using heartbeat control networks. This quorum determines which nodes are part of the FailSafe membership that is formed.
If there are an even number of nodes in the cluster, it is possible that there will be no majority quorum; there could be two sets of nodes, each consisting of 50% of the total number of node, unable to communicate with the other set of nodes. In this case, FailSafe uses the node that has been configured as the tie-breaker node when you configured your FailSafe parameters. If no tie-breaker node was configured, FailSafe uses the node with the lowest ID number where HA services have been started.
The nodes in a quorum attempt to reset the nodes that are not in the quorum. Nodes that can be reset are declared DOWN in the membership, nodes that could not be reset are declared UNKNOWN. Nodes in the quorum are UP.
If a new majority quorum is computed, a new membership is declared whether any node could be reset or not.
If at least one node in the current quorum has a current membership, the nodes will proceed to declare a new membership if they can reset at least one node.
If all nodes in the new tied quorum are coming up for the first time, they will try to reset and proceed with a new membership only if the quorum includes the tie-breaker node.
If a tied subset of nodes in the cluster had no previous membership, then the subset of nodes in the cluster with the tie-breaker node attempts to reset nodes in the other subset of nodes in the cluster. If at least one node reset succeeds, a new membership is confirmed.
If a tied subset of nodes in the cluster had previous membership, the nodes in one subset of nodes in the cluster attempt to reset nodes in the other subset of nodes in the cluster. If at least one node reset succeeds, a new membership is confirmed. The subset of nodes in the cluster with the tie-breaker node resets immediately; the other subset of nodes in the cluster attempts to reset after some time.
Resets are done through system controllers connected to tty ports through serial lines. Periodic serial line monitoring never stops. If the estimated serial line monitoring failure interval and the estimated heartbeat loss interval overlap, the cause is likely a power failure at the node being reset.
When no FailSafe membership is formed, you should check the following areas for possible problems:
Is the ha_cmsd FailSafe membership daemon running? Is the fs2d database daemon running?
Can the nodes communicate with each other? Are the control networks configured as heartbeat networks?
Can the control network addresses be reached by a ping(1M) command issued from peer nodes?
Are the quorum majority or tie rules satisfied? Look at the cmsd log to determine membership status.
If a reset is required, are the following conditions met?
Is the crsd node control daemon up and running?
Is the reset serial line in good health?
You can look at the crsd log for the node you are concerned with, or execute an admin ping and admin reset command on the node to check this.
FailSafe allows you to monitor and check the status of specified clusters, nodes, resources, and resource groups. You can use this feature to isolate the location of system problems.
You can monitor the status of the FailSafe components continuously through their visual representation in the GUI tree view. Using the cmgr command, you can display the status of the individual components by using the show command.
For information on status monitoring and on the meaning of the states of the FailSafe components, see “System Status” in Chapter 7 of Chapter 7, “IRIS FailSafe System Operation”.
FailSafe allows you to perform a variety of administrative tasks that can help you troubleshoot a system with problems without bringing down the entire system. These tasks include the following:
You can add or delete nodes from a cluster without affecting the FailSafe services and the applications running in the cluster.
You can add or delete a resource group without affecting other online resource groups.
You can add or delete resources from a resource group while it is still online.
You can change FailSafe parameters such as the heartbeat interval and the node timeout and have those values take immediate affect while the services are up and running.
You can start and stop FailSafe services on specified nodes.
You can move a resource group online, or take it offline.
You can stop the monitoring of a resource group by putting the resource group into maintenance mode. This is not an expensive operation, as it does not stop and start the resource group, it just puts the resource group in a state where it is not available to FailSafe.
You can reset individual nodes.
For information on how to perform these tasks, see Chapter 5, “Configuration”, and Chapter 7, “IRIS FailSafe System Operation”.
The following sections describe various recovery procedures you can perform when different failsafe components fail. Procedures for the following situations are provided:
When one of the nodes in a two-node cluster is intended to stay down for maintenance or cannot be brought up, a set of procedures must be followed so that the database on the surviving node knows that that node is down and therefore should not to be considered in the failover domain. Without these procedures, the surviving node will be in what is called the lonely state, meaning that it will never form a cluster by itself.
See the procedure in “Two-Node Clusters: Single-Node Use” in Chapter 7.
Use the following procedure if status of the cluster is UNKNOWN in all nodes in the cluster:
Check to see if there are control networks that have failed (see “Control Network Failure Recovery”).
Determine if there are sufficient nodes in the cluster that can communicate with each other using control networks in order to form a quorum. (At least 50% of the nodes in the cluster must be able to communicate with each other.) If there is an insufficient number of nodes, stop HA services on the nodes that cannot communicate (using the force option); this will change the number of nodes used in the quorum calculation.
If there are no hardware configuration problems, do the following:
Detach all resource groups that are online in the cluster (if any)
Stop HA services in the cluster
Restart HA services in the cluster
For example, the following cmgr command detaches the resource group web-rg in cluster web-cluster :
cmgr> admin detach resource_group web-rg in cluster web-cluster |
To stop HA services in the cluster web-cluster and ignore errors (force option), use the following command:
cmgr> stop ha_services for cluster web-cluster force |
To start HA services in the cluster web-cluster, use the following command:
cmgr> start ha_services for cluster web-cluster |
The fact that a resource group is in an error state does not mean that all resources in the resource group have failed. However, to get the resources back into an online state, you must first set them to the offline state. You can do without actually taking the resources offline by using the following cmgr command:
admin offline_detach_force resource_group [in cluster clustername] |
cmgr> admin offline_detach_force RG1 in cluster test-cluster |
 | Caution: You should use the InPlace_Recovery failover policy attribute when using this command. This attribute specifies that the resources will stay on the same node where they were running at the time when the offline_detach_force command was run. |
When a node is not able to talk to the majority of nodes in the cluster, the SYSLOG will display a message that the CMSD is in a lonely state. Another problem you may see is that a node is getting reset or going to an unknown state.
Use the following procedure to resolve node errors:
Check to see if the control networks in the node are working (see “Control Network Failure Recovery”).
Check to see if the serial reset cables to reset the node are working (see “Serial Cable Failure Recovery”).
Verify that the sgi-cmsd port is the same in all nodes in the cluster.
Check the node configuration; it should be consistent and correct.
Check syslog and cmsd logs for errors. If a node is not joining the cluster, check the logs of the nodes that are part of the cluster.
If there are no hardware configuration problems, stop HA services in the node and restart HA services.
For example, to stop HA services in the node web-node3 in the cluster web-cluster, ignoring errors (force option), use the following command:
cmgr> stop ha_services in node web-node3 for cluster web-cluster force
For example, to start HA services in the node web-node3 in the cluster web-cluster, use the following command:
cmgr> start ha_services in node web-node3 for cluster web-cluster
To do simple maintenance on an application that is part of the resource group, use the following procedure. This procedure stops monitoring the resources in the resource group when maintenance mode is on. You must turn maintenance mode off when performing application maintenance.
| Caution: If there is a node failure on the node where resource group maintenance is being performed, the resource group is moved to another node in the failover policy domain. |
For example:
To put a resource group web-rg in maintenance mode, use the following cmgr command:
cmgr> admin maintenance_on resource_group web-rg in cluster web-cluster
The resource group state changes to ONLINE_MAINTENANCE. Do whatever application maintenance is required. (Rotating application logs is an example of simple application maintenance).
To remove a resource group web-rg from maintenance mode, use the following command:
cmgr> admin maintenance_off resource_group web-rg in cluster web-cluster
The resource group state changes back to ONLINE.
Perform the following procedure when a resource group is in an ONLINE state and has an SRMD EXECUTABLE ERROR:
Look at the SRM logs (default location: /var/cluster/ha/logs/srmd_nodename) to determine the cause of failure and the resource that has failed. Search for the ERROR string in the SRMD log file:
Wed Nov 3 04:20:10.135 <E ha_srmd srm 12127:1 sa_process_tasks.c:627> CI_FAILURE, ERROR: Action (start) for resource (192.0.2.45) of type (IP_address) failed with status (failed)
Check the script logs on that same node for IP_address start script errors.
Fix the cause of failure. This might require changes to resource configuration or changes to resource type stop/start/failover action timeouts.
After fixing the problem, move the resource group offline with the force option and then move the resource group online in the cluster.
For example, the following command moves the resource group web-rg in the cluster web-cluster offline and ignores any errors:
cmgr> admin offline resource_group web-rg in cluster web-cluster force
The following command moves the resource group web-rg in the cluster web-cluster online:
cmgr> admin online resource_group web-rg in cluster web-cluster
The resource group web-rg should be in an ONLINE state with no error.
Use the following procedure when a resource group is not online but is in an error state. Most of these errors occur as a result of the exclusivity process. This process, run when a resource group is brought online, determines if any resources are already allocated somewhere in the failure domain of a resource group. Note that exclusivity scripts return that a resource is allocated on a node if the script fails in any way. In other words, unless the script can determine that a resource is not present, it returns a value indicating that the resource is allocated.
Some possible error states include: SPLIT RESOURCE GROUP (EXCLUSIVITY), NODE NOT AVAILABLE (EXCLUSIVITY), NO AVAILABLE NODES in failure domain. See “Resource Group Status” in Chapter 7, for explanations of resource group error codes.
Look at the failsafe and SRMD logs (default directory: /var/cluster/ha/logs, files: failsafe_nodename, srmd_nodename) to determine the cause of the failure and the resource that failed.
For example, suppose that the task of moving a resource group online results in a resource group with error state SPLIT RESOURCE GROUP (EXCLUSIVITY). This means that parts of a resource group are allocated on at least two different nodes. One of the failsafe logs will have the description of which nodes are believed to have the resource group partially allocated:
[Resource Group:RG_name]:Exclusivity failed -- RUNNING on node1 and node2
[Resource Group:RG_name]:Exclusivity failed -- PARTIALLY RUNNING on node1 and PARTIALLY RUNNING on node2
At this point, look at the srmd logs on each of these nodes for exclusive script errors to see what resources are believed to be allocated. In some cases, a misconfigured resource will show up as a resource that is allocated. This is especially true for Netscape_web resources.
Fix the cause of the failure. This might require changes to resource configuration or changes to resource type start/stop/exclusivity timeouts.
After fixing the problem, move the resource group offline with the force option and then move the resource group online.
Perform the following checks when a resource group shows a no more nodes in AFD error:
All nodes in the failover domain are not in the membership. Check CMSD logs for errors.
Check the SRMC/script logs on all nodes in the failover domain for start/monitor script errors.
There are a few double failures that can occur in the cluster that will cause resource groups to remain in a non-highly-available state. At times a resource group might be stuck in an offline state. A resource group might also stay in an error state on a node even when a new node joins the cluster and the resource group can migrate to that node to clear the error. When these circumstances arise, do the following:
If the resource group is offline, try to move it online.
If the resource group is stuck on a node, detach the resource group and then bring it back online again. This should clear many errors.
If detaching the resource group does not work, force the resource group offline, then bring it back online.
If commands appear to be hanging or not working properly, detach all resource groups, then shut down the cluster and bring all resource groups back online.
See “Take a Resource Group Offline” in Chapter 7, for information on detaching resource groups and forcing resource groups offline.
Use this procedure when a resource that is not part of a resource group is in an ONLINE state with an error. This can happen when the addition or removal of resources from a resource group fails.
Do the following:
Look at the SRM logs to determine the cause of failure and the resource that has failed. The default location is:
/var/cluster/ha/logs/srmd_nodename
Fix the problem that caused the failure. This might require changes to resource configuration or changes to resource type stop/start/failover action timeouts.
Clear the error state with the GUI or the cmgr command:
Use the Clear Resource Error State GUI task. Provide the following information:
Resource Type: select the type of the resource
Resource in Error State: select the name of the resource that should be cleared from the error state
Click OK to complete the task.
Use the cmgr admin offline_force command to move the resource offline. For example, to remove the error state of resource web-srvr of type Netscape_Web , making it available to be added to a resource group, enter the following:
cmgr> admin offline_force resource web-srvr of resource_type Netscape_Web in cluster web-cluster
Control network failures are reported in cmsd logs. The default location of cmsd log is /var/cluster/ha/logs/cmsd_ node name. Follow this procedure when the control network fails:
Use the ping(1M) command to check whether the control network IP address is configured in the node.
Check node configuration to see whether the control network IP addresses are correctly specified.
The following cluster_mgr command displays node configuration for web-node3:
cmgr> show node web-node3
If IP names are specified for control networks instead of IP addresses in XX.XX.XX.XX notation, check to see whether IP names can be resolved using DNS. It is recommended that IP addresses are used instead of IP names.
Check whether the heartbeat interval and node timeouts are correctly set for the cluster. These HA parameters can seen using cluster_mgr show ha_parameters command.
Serial cables are used for resetting a node when there is a node failure. Serial cable failures are reported in crsd logs. The default location for the crsd log is /var/cluster/ha/log/crsd_nodename.
Check the node configuration to see whether serial cable connection is correctly configured.
The following cmgr command displays node configuration for web-node3
cmgr> show node web-node3 |
Use the admin ping command to verify the serial cables. The following command reports serial cables problems in node web-node3:
cmgr> admin ping node web-node3 |
If the cluster database synchronization fails, use the following procedure:
Check for the following message in the /var/adm/SYSLOG file on the target node:
Starting to receive CDB sync series from machine <node1_node_ID> ... Finished receiving CDB sync series from machine <node1_node_ID>
Check for control network or portmapper/rpcbind problems.
Check the node definition in the cluster database.
Check the SYSLOG and fs2d logs on the source node.
When the entire cluster database must be reinitialized, execute the following command:
# /usr/cluster/bin/cdbreinit /var/cluster/cdb/cdb.db |
This command restarts all cluster processes. The contents of the cluster database will be automatically synchronized with other nodes if other nodes in the pool are available.
Otherwise, the cluster database must be restored from backup at this point. For instructions on backing up and restoring the cluster database, see “Backing Up and Restoring Configuration with cmgr” in Chapter 7.
If the GUI will not run, check the following:
Are the cluster daemons running?
When you first install the software, the following daemons should be running:
To determine which daemons are running, enter the following:
# ps -ef | grep cluster
The following shows an example of the output when just the initial daemons are running; for readability, whitespace has been removed and the daemon names are highlighted:
fs6 # ps -ef | grep cluster root 31431 1 0 12:51:36 ? 0:14 /usr/lib32/cluster/cbe/fs2d /var/cluster/cdb/cdb.db # root 31456 31478 0 12:53:01 ? 0:03 /usr/cluster/bin/crsd -l root 31475 31478 0 12:53:00 ? 0:08 /usr/cluster/bin/cad -l -lf /var/cluster/ha/log/cad_log --append_log root 31478 1 0 12:53:00 ? 0:00 /usr/cluster/bin/cmond -L info -f /var/cluster/ha/log/cmond_log root 31570 31408 0 14:01:52 pts/0 0:00 grep cluster
If you do not see these processes, go to the logs to see what the problem might be. If you must restart the daemons, enter the following:
# /etc/init.d/cluster start
Are the tcpmux and tcpmux/sgi_sysadm services enabled in the /etc/inetd.conf file?
The following line is added to the /etc/inetd.conf file when sysamd_base is installed:
tcpmux/sgi_sysadm stream tcp nowait root ?/usr/sysadm/bin/sysadmd sysadmd
If the tcpmux line is commented out, you must uncomment it and then run the following:
# kill -HUP inetd
Are the inetd or tcp wrappers interfering? This may be indicated by connection refused or login failed messages.
Are you connecting to an IRIX node? The fstask(1M) command can only be executed on an IRIX node. The GUI may be run from a node running an operating system other than IRIX via the Web if you connect the GUI to an IRIX node.
If the GUI is displaying information that is inconsistent with the FailSafe cmgr command, restart cad process on the node to which GUI is connected to by executing the following command:
# killall cad |
The cluster administration daemon is restarted automatically by the cmond process.
If the GUI is not reporting configuration information and status, perform the following steps:
Check the information using the cmgr(1M) command. If cmgr is reporting correct information, there is a GUI update problem.
If there is a GUI update problem, kill the cad daemon on that node. Wait for a couple of minutes to see whether cad gets correct information. Check the cad logs on that node for errors.
Check the CLI logs on that node for errors.
If the status information is incorrect, check the cmsd or fsd logs on that node.
When the cluster databases are not in synchronization on all the nodes in the cluster, you can run the cdbreinit command to recover. The cdbreinit command should be run on the node which is not in sync.
Perform the following steps.
| Note: Perform each step on all the nodes before proceeding to the next step in the recovery procedure. |
Stop FailSafe services in the cluster using the GUI or cmgr.
Stop cluster processes on all nodes in the pool:
# /etc/init.d/cluster stop # killall fs2d
Run cdbreinit on the node where the cluster database is not in sync.
Start cluster processes on all nodes in the pool:
# /etc/init.d/cluster start
Wait a couple of minutes for the cluster database to sync. There will be cluster database sync long messages in the SYSLOG on the node.
Start FailSafe services in the cluster.
FailSafe uses a umount(1M) command with the -k option to move a resource in the case of a CXFS metadata server relocation if the relocate-mds attribute in the CXFS resource definition is set to true. The umount -k command will kill all server process using the CXFS filesystem.