Programmers must write several scripts and add information to the IRIS FailSafe configuration file to turn an application into a highly available service supported by the IRIS FailSafe system. No modifications need to be made to an application to make it a highly available service—all changes are to the IRIS FailSafe system software and to the IRIS FailSafe configuration file.
Not every application can become a highly available service—this chapter explains the characteristics an application must have to become a highly available service. This chapter also describes the procedure for developing the scripts that make an application a highly available service and gives information about IRIS FailSafe system software that is important for programmers developing these scripts.
The major sections in this chapter are as follows:
Applications need not be modified to make them highly available when running in an FailSafe cluster. However, not every application is appropriate for high availability. This section describes the characteristics that an application must have for high availability. When an application has these characteristics, a programmer can develop the scripts required by the IRIS FailSafe system.
The characteristics of an application that can be made highly available are:
The application is crash tolerant.
It should be able to recover from failures. The failure could be a system failure (a hardware failure or an operating system failure) or an application failure.
The application must have a start and stop procedure.
When the application fails over, the instances of the application are stopped on one node using the stop procedure and restarted on the other node using start procedure.
The application can be moved from one node to the other after failures.
If the application has failed, it must still be possible to run the application stop procedure. In addition, the application must recover from the failed state when the application start procedure is executed in the other node.
If the application depends on other applications and resources, they must also be highly available.
IRIS FailSafe and its optional products enable you to configure system resources (interfaces, IP addresses, XLV volumes, and XFS filesystems) and applications (NFS filesystems, Netscape servers, and INFORMIX, Oracle, and Sybase databases) as highly available services. For these highly available services, you must configure them as described in the IRIS FailSafe Administrator's Guide .
If the application you want to make highly available depends on resources and applications not supported by IRIS FailSafe products, you must make these resources and applications highly available, too, using the information in this guide.
To turn an application into a highly available service automatically failed over by the IRIS FailSafe system, follow these steps:
Understand the application and determine:
The system resources on which the application depends.
The type of configuration required for the applications—active/backup or dual-active.
The number of instances of the application.
The commands and arguments required to start, stop, and monitor the application instances.
The relationships between this application and other highly available services; specifically, the order all highly available services need to be started and stopped.
Configure and test the base IRIS FailSafe system as described in the IRIS FailSafe Administrator's Guide .
Add configuration information for the new application to /var/ha/ha.conf. See Chapter 2, “Modifying the Configuration File for a New Highly Available Service.”
Write the local and remote monitoring scripts. See Chapter 3, “Writing a Monitoring Script.”
Write the failover script. This script includes the giveaway(), giveback(), takeback(), and takeover() functions. See Chapter 4, “Writing a Failover Script.”
Install, test, and debug the scripts. See Chapter 5, “Installing and Testing Scripts.”
 | Note: Do not modify the scripts included with the IRIS FailSafe product. New or customized scripts must have different names than the files included with IRIS FailSafe (see Figure 1-14). |
In this section, the architecture of the system software is described: the software components and the messages they send, the states that nodes can be in and how they transition to other states, and the scripts that enable fail over of highly available services. This material supplements the overview IRIS FailSafe system provided in Chapter 1 of the IRIS FailSafe Administrator's Guide .
Table 1-1 describes the IRIS FailSafe system software executables.
Table 1-1. IRIS FailSafe Software
Component | Process | Purpose |
|---|---|---|
Heartbeat daemon | /usr/etc/ha_hbeat | Generates heartbeats, which are messages passed between the two nodes of the cluster over the private Ethernet that connects the nodes. |
Node controller | /usr/etc/ ha_nc | Keeps track of the node state. Determines when a state transition is needed as a result of a heartbeat failure, an application failure, an interface failure, an administrative command, or a node re-entering a cluster. Controls actions such as killing the other node; leaving, rejoining, and reintegrating into a cluster; and starting and stopping monitoring. |
Application monitor | /usr/etc/ ha_appmon | Monitors the node's highly available services. Monitors the remote heartbeat daemon and local node controller. Notifies the node controller of failures. Executes scripts. Sends mail at state transitions and failures. |
Kill daemon | /usr/etc/ ha_killd | Monitors the serial connection to the other node. Has the ability to kill the other node. |
Local monitoring scripts | /var/ha/actions/ha_*_lmon | Each local monitoring script monitors all instances of one highly available service on the local node. |
Database agents | /usr/etc/ha_ifmx, /usr/etc/ha_orcl, /usr/etc/ha_sybs | Each database agent monitors all instances of one type of database. |
Interface agent | /usr/etc/ha_ifa | Monitors the local node's interfaces. |
Failover scripts for highly available services | /var/ha/resources/* | Each script contains four failover functions (takeback(), takeover(), giveaway(), and giveback()) for one highly available service. |
Failover scripts for operations | /var/ha/actions/{giveaway, | Each of these scripts causes the failover scripts in /var/ha/resources to be executed with the name of the calling script as a command-line argument. This causes the function with the same name as the argument (takeback(), takeover(), giveaway(), or giveback()) to be executed. |
Figure 1-1 shows the communication paths between the software components listed in Table 1-1 on the two nodes of an IRIS FailSafe cluster while they are in normal state. All of the processes shown run continuously during normal IRIS FailSafe operation except the monitoring scripts and failover scripts. The monitoring scripts run periodically to check the state of agents and applications. The failover scripts run only during fail over.
Table 1-2 describes the IRIS FailSafe administrative commands.
Table 1-2. IRIS FailSafe Administrative Commands
Command | Purpose |
|---|---|
ha_admin | Enable the system administrator to force IRIS FailSafe failover operations. |
ha_cfgchksum | Compute the checksum of a configuration file. |
ha_cfginfo | Extract information from the configuration file. |
ha_cfgverify | Verify a configuration file. |
ha_exec | Execute a command, possibly repeatedly. |
ha_spng | Verify the serial connections between nodes. |
http_ping | Check if a Netscape™ server is still running. |
macconfig | Display or modify MAC addresses. |
The IRIS FailSafe configuration file /var/ha/ha.conf is a key component of IRIS FailSafe software. It contains cluster-specific configuration information, which is used by IRIS FailSafe monitoring and failover scripts. The configuration files on the nodes in a cluster must be identical.
The condition of each node in a cluster is described by its state. The possible states are as follows:
| standalone | The node is coming up and IRIS FailSafe is starting up. This is a transient state. | |
| joining | The node is coming up and joining the cluster. The node should never remain in this state for more than two or three minutes. | |
| normal | The node is actively providing its own highly available services. It is also monitoring the other node in the cluster. | |
| degraded | The node is providing all highly available services for the cluster; the other node is unavailable. | |
| standby | This node has stopped monitoring the other node in the cluster and is no longer providing highly available services. It is not capable of providing highly available services. | |
| controlled failback |
| |
| error | An unrecoverable failure has occurred. |
Evaluating the current node state and monitoring the state of the other node in the cluster are the responsibilities of the IRIS FailSafe node controller (ha_nc). The state of each node in the cluster is kept by each node controller.
Nodes move from one state to another as a result of system bootup, system halt, failure (system failure or the failure of a highly available service), and ha_admin commands. To move from one state to another, a node controller sends a message to its local application monitor and asks it to perform one or more of these four operations, called failover operations:
| takeback | Start up all services for which this node is listed as the primary node (specified by the server-node parameter for the service) in ha.conf. | |
| takeover | Start up all services for which this node is listed as the backup node (specified by the backup-node parameter for the service) in ha.conf. | |
| giveaway | Stop all services for which this node is listed as the primary node for the service in ha.conf. | |
| giveback | Stop all services for which this node is listed as the backup node for the service in ha.conf. |
The possible state transition scenarios are shown in Figure 1-2 through Figure 1-13 in the following subsections. They show the sequences of states for the two nodes in a cluster, the primary messages passed between the nodes, and the takeback, takeover, giveaway, and giveback operations that occur during the different scenarios. Understanding these scenarios can help you interpret messages written to logs and to debug the scripts you write when you add the failover of a new service to the IRIS FailSafe system.
Figure 1-2 through Figure 1-13 show two nodes over time (time increases from the top of the figures to the bottom). The components of the figures are:
| NORMAL, CONTROLLED FAILBACK, DEGRADED, JOINING, STANDALONE, STANDBY |
| |
| diagonal lines | A block of unlabeled diagonal lines indicates that the node is down or it is up, but IRIS FailSafe is not running. | |
| ovals | Ovals are used to indicate events such as crashes, reboots, and the detection of failures by IRIS FailSafe software. | |
| takeback, takeover, giveaway, giveback |
| |
| ha_admin commands |
| |
| arrows | Arrows are used with message names to indicate messages from one node to the other. The remainder of this list describes these messages. | |
| kill | A kill message from one node to the other causes the recipient to shut down. | |
| rejoin | A message sent when a node is in joining state. It is sent when a node comes up initially. If there is no response from the other node, the node sending the message cannot move to normal state. Additional rejoin messages are sent every 30 seconds until long-timeout seconds have passed, in case the initial messages are lost. The diagrams show only the one rejoin message. | |
| reintegrate | A message similar to rejoin, except that it is sent after the ha_admin -fr command or ha_admin -G command is executed on a node in standby state. It requests permission for the node to move to normal state or degraded state, respectively. | |
| released | A message sent after the giveback script has been executed or as a response to a rejoin or reintegrate message. | |
| standby | A message that tells a node to stay in standby state. | |
| start_rmon | A message from a node in degraded state that tells the other node to begin remote monitoring. Normally, a node in standby state doesn't do heartbeat or serial line monitoring of the node in degraded state. This message, sent when the parameter controlled-failback is true for the other node, tells the node in standby state to begin monitoring. The other node in standby state moves to controlled failback state. | |
| takeover | A message from one node that tells the other node to perform the takeover operation. | |
| timeout | The timeout event indicates that a node does not respond to the previous message. |
In Figure 1-2, two nodes are rebooted or IRIS FailSafe is started on both nodes within the value of long-timeout seconds (specified in ha.conf) of each other. Each node moves to normal state.
In the scenario shown in Figure 1-3, one node (Node B) is down or not running IRIS FailSafe. The other node (Node A) is rebooted or IRIS FailSafe is started. Node A moves to degraded state.
In the scenario shown in Figure 1-4, both nodes begin in normal state. The ha_admin -s command is entered by the system administrator on one node (node B) to cause all highly available services to fail over to the other node (Node A) in preparation for performing system maintenance on Node B.
In the scenario shown in Figure 1-5, one node (Node A) has been providing all highly available services while system maintenance has been performed on the other node (Node B). This system maintenance doesn't include a reboot. The ha_admin -fr command is entered by the system administrator on Node B after system maintenance is completed. Both nodes move automatically to normal state.
In the scenario shown in Figure 1-6, one node (Node A) has been providing all highly available services while system maintenance has been performed on the other node (Node B). (Node B was brought to standby state using the command ha_admin -s.) The system maintenance included a reboot. After the reboot, Node B moves to standby state after attempting to rejoin. The command ha_admin -fr must be used to move both nodes to normal state.
In the scenario shown in Figure 1-7, both nodes are operating normally when one node (Node B) crashes or hangs (shown in Figure 1-7). The other node (Node A) moves to degraded state.
In the scenario shown in Figure 1-8, one node (Node A) is operating in degraded state and the other node (Node B) is down or not running IRIS FailSafe. Node B is in an indeterminate state as a result of the scenario in Figure 1-7. It got to When Node B is rebooted, both nodes are brought automatically to normal state because the configuration parameter controlled failback is set to false for Node B (or not set for Node B).
Figure 1-8. Recovery After Node Failure With the Controlled-Failback Parameter Set to False (or Not Set)
In the scenario shown in Figure 1-9, one node (Node A) is operating in degraded state and the other node (Node B) is down or is not running IRIS FailSafe because it has been killed (by Node A or by an incorrect shutdown of IRIS FailSafe). Node B is rebooted, but because the controlled-failback parameter is set to true for Node B, Node B goes to controlled failback state. After the ha_admin -fr command is entered by the system administrator, both nodes move to normal state.
In the scenario shown in Figure 1-10, both nodes begin in normal state. One node (Node A) detects a local failure and moves to standby state. The other node (Node B) takes over all services and moves to degraded state.
In the scenario shown in Figure 1-11, one node (Node A) is in degraded state and the other node (Node B) is in standby state. Local monitoring on Node A detects that there has been a failure of a highly available service. All highly available services remain on Node A. Neither node changes state. Local monitoring on Node A is stopped.
In the scenario shown in Figure 1-12, one node (Node A) is in degraded state and the other node (Node B) is in controlled failback state. Node A detects the failure of a highly available service that it is providing. Node A moves to standby state and Node B moves to degraded state. Node B provides all highly available services.
Figure 1-12. Node in Degraded State Detects Local Failure While the Other Node Is in Controlled Failback State
In the scenario shown in Figure 1-13, one node (Node A) is in degraded state and the other node (Node B) is in controlled failback state. Heartbeat monitoring on Node B detects a failure on Node A. All highly available services are failed over to node B, which transitions to degraded state. The node that was originally in degraded state is killed and restarted.
Scripts perform monitoring and the failover operations giveaway, giveback, takeback, and takeover. To add the failover of a new class of applications, you must create monitoring and failover scripts (described in Chapter 3, “Writing a Monitoring Script,” and Chapter 4, “Writing a Failover Script”) for the new application class.
IRIS FailSafe scripts and configuration file are stored in the directory /var/ha. Figure 1-14 shows the contents of this directory.
The directory /var/ha contains several categories of files and directories as follows:
| actions/giveaway, actions/giveback, actions/takeback, actions/takeover |
| ||
| actions/common.vars |
T_SERVER is a variable used in the scripts to get the value of the parameter server-node from the configuration file ha.conf. common.vars is sourced by each IRIS FailSafe monitoring and failover script at the beginning of the script. | ||
| actions/mail | This script sends a mail message (using /usr/sbin/Mail) to the address specified by the parameter mail-dest-addr in /var/ha/ha.conf when various types of errors are detected. It is executed by the application monitor (ha_appmon). | ||
| actions/ha_<app>_lmon |
| ||
| logs | This directory contains the logs of all scripts executed by IRIS FailSafe. The outputs and errors from the commands in the scripts are logged. | ||
| resources | This directory contains one script per highly available service, plus the script statd, which is required by NFS. These scripts contain functions for the IRIS FailSafe operations giveaway, giveback, takeback, and takeover. | ||
| resources/<appclass> |
| ||
| actions.d/giveaway |
When the files in actions/giveaway are executed, they are executed with the command-line argument giveaway (the name of the directory). This causes the giveaway() function in the script to be performed. For example, executing
really executes resources/filesystems (because S800 filesystems is a link to resources/filesystems) with the argument giveaway. checksum is the checksum for ha.conf. | ||
| actions.d/giveaway/S<nnn><appclass> |
| ||
| actions.d/giveback, actions.d/takeback, actions.d/takeover |
| ||
| ha.conf | This file is the IRIS FailSafe configuration file. It is not included with the IRIS FailSafe product or any of its options; it must be created by the user. (See Chapter 4, “Creating the IRIS FailSafe Configuration File,” of the IRIS FailSafe Administrator's Guide for more details.) | ||
| templates | This directory contains template files used to create the ha.conf file. (See Chapter 4, “Creating the IRIS FailSafe Configuration File,” of the IRIS FailSafe Administrator's Guide for more details.) |
When the IRIS FailSafe system performs a failover operation (giveaway, giveback, takeback, or takeover), the function with that name in each of the scripts in the /var/ha/resources directory is executed. The order of execution is controlled by the filenames in the /var/ha/actions.d/giveaway, /var/ha/actions.d/giveback, /var/ha/actions.d/takeback, and /var/ha/actions.d/takeover directories as explained in the section “Choosing the Execution Order of Failover Scripts for Each Operation” in Chapter 5. Because all scripts are executed when a failover operation occurs, it is important to understand what tasks are going to be performed by other scripts when writing a new script. For your reference, the tasks performed by each of the standard scripts, the NFS scripts and the Web script for each of the failover operations are described below. The NFS scripts (nfs and statd) and Web script (webserver) are present only if the optional IRIS FailSafe NFS and IRIS FailSafe Web products are installed.
The actions performed by each function in the filesystems script are as follows:
| takeback() | Mounts all filesystems for which this node is the primary node. They are mounted in increasing lexicographic order. | |||
| takeover() | Mounts all filesystems for which this node is the backup node. They are mounted in increasing lexicographic order. | |||
| giveaway() | Unmounts all filesystems for which this node is the primary node in decreasing lexicographic order. Before unmounting, it kills any nfsd processes. After unmounting, it restarts all nfsd processes.
| |||
| giveback() | Unmounts all filesystems for which this node is the backup node in decreasing lexicographic order. Before unmounting, it kills any nfsd processes. After unmounting, it restarts all nfsd processes.
|
The actions performed by each function in the interfaces script are as follows:
| takeback() | Adds all the IP addresses for which this node is the primary node to their network interfaces. Also restores the MAC addresses to the interfaces, if re-mac'ing is used. | |
| takeover() | Adds all the IP addresses for which this node is the backup node to their network interfaces. Also moves the MAC addresses to the interfaces, if re-mac'ing is used. | |
| giveaway() | Removes IP addresses from the network interfaces for which this node is the primary node. | |
| giveback() | Removes IP addresses from the network interfaces for which this node is the backup node. Restores the MAC addresses if re-mac'ing is used. |
The actions performed by each function in the nfs script are as follows:
| takeback() | Exports filesystems for which this node is the primary node. They are exported in a sorted hierarchical order. | |
| takeover() | Exports filesystems for which this node is the backup node. They are exported in a sorted hierarchical order. | |
| giveaway() | Unexports (exportfs -u) any filesystem for which this node is the primary node. | |
| giveback() | Unexports (exportfs -u) any filesystem for which this node is the backup node. |
The actions performed by each function in the statd script are as follows:
| takeback() | Add an NFS locking directory (called the statmon directory) to be used for all NFS filesystems for which this node is the primary node. Also send out a message instructing NFS clients to reacquire NFS locks if that client had an NFS lock to one of the IP addresses for which this node is primary node. | |
| takeover() | Add an NFS locking directory (called the statmon directory) to be used for all NFS filesystems for which this node is the backup node. Also send out a message instructing NFS clients to reacquire NFS locks if that client had an NFS lock to one of the IP addresses for which this node is backup node. | |
| giveaway() | Remove the NFS locking directory added by takeback(). | |
| giveback() | Remove the NFS locking directory added by takeover(). |
The actions performed by each function in the volumes script are as follows:
| takeback() | Changes the node names of all of the XLV volumes that are owned by the backup node to this node's (the primary node) name. Assembles all the local XLV volumes and starts xlv_labd if necessary. | |
| takeover() | Changes the node names of all of the XLV volumes that are owned by the primary node to this node's (the backup node) name. Assembles all the local XLV volumes and starts xlv_labd if necessary. | |
| giveaway() | Shuts down all the XLV volumes for which this node is the primary node. Changes the node name for these XLV volumes to the other node's name. | |
| giveback() | Shuts down all the XLV volumes for which this node is the backup node. Changes the node name for these XLV volumes to the other node's name. |
The actions performed by each function in the webserver script are as follows:
| takeback() | Starts all Netscape servers for which this node is the primary node. Reads all the webserver blocks and puts all directories specified by httpd-dir for which this is the primary node into the configuration file specified by httpd-options-file. Calls the script specified by httpd-script with the start argument. | |
| takeover() | Starts all Netscape servers for which this node is the backup node. Reads all the webserver blocks and puts all directories specified by httpd-dir for which this is the backup node into the configuration file specified by httpd-options-file. Calls the script specified by httpd-script with the start argument. | |
| giveaway() | Stops all Netscape servers for which this is the primary node. Reads all the webserver blocks and puts all the directories specified by httpd-dir for which this is the primary node into the configuration file specified by httpd-options-file. Calls the script specified by httpd-script with the stop argument. | |
| giveback() | Stops all Netscape servers for which this is the backup node. Reads all the webserver blocks and puts all the directories specified by httpd-dir for which this is the backup node into the configuration file specified by httpd-options-file. Calls the script specified by httpd-script with the stop argument. |