Chapter 1. Overview
Prev |  | Next

## Chapter 1. Overview

This chapter provides an overview of the components and operation of the IRIS FailSafe system. It contains the following:

- “High Availability and IRIS FailSafe”

- “Cluster Environment”

- “Additional Features”

- “Administration”

- “Highly Available Resources”

- “Highly Available Applications”

- “Failover and Recovery Processes”

- “Overview of Configuring and Testing a New Cluster”

- “Software Overview”

## High Availability and IRIS FailSafe

In the world of mission-critical computing, the availability of information and computing resources is extremely important. The availability of a system is affected by how long it is unavailable after a failure in any of its components. Different degrees of availability are provided by different types of systems:

- Fault-tolerant systems (continuous availability). These systems use redundant components and specialized logic to ensure continuous operation and to provide complete data integrity. On these systems the degree of availability is extremely high. Some of these systems can also tolerate outages due to hardware or software upgrades. This solution is very expensive and requires specialized hardware and software.

- Highly available systems. These systems survive single points of failure by using redundant off-the-shelf components and specialized software. They provide a lower degree of availability than the fault-tolerant systems, but at much lower cost. Typically these systems provide high availability only for client/server applications, and base their redundancy on cluster architectures with shared resources.

The SGI IRIS FailSafe product provides a general facility for providing highly available services. FailSafe provides highly available services for a cluster that contains multiple nodes (N-node configuration).

If one of the nodes in the cluster or one of the nodes' components fails, a different node in the cluster restarts the highly available services of the failed node. To clients, the services on the replacement node are indistinguishable from the original services before failure occurred. It appears as if the original node has crashed and rebooted quickly. The clients notice only a brief interruption in the highly available service.

In a FailSafe environment, nodes can serve as backup for other nodes. Unlike the backup resources in a fault-tolerant system, which serve purely as redundant hardware for backup in case of failure, the resources of each node in a highly available system can be used during normal operation to run other applications that are not necessarily highly available services. All highly available services are owned by one node in the cluster at a time.

Highly available services are monitored by the FailSafe software. During normal operation, if a failure is detected on any of these components, a failover process is initiated. Using FailSafe, you can define a failover policy to establish which node will take over the services under what conditions. This process consists of resetting the failed node (to ensure data consistency), performing recovery procedures required by the failed over services, and quickly restarting the services on the node that will take them over.

FailSafe supports selective failover in which individual highly available applications can be failed over to a backup node independent of the other highly available applications on that node.

## Cluster Environment

This section discusses the following:

- “Terminology”

- “Hardware Components”

- “Disk Connections”

- “Supported Configurations”

### Terminology

This section defines the terminology necessary to configure and monitor highly available services with FailSafe.

#### Node

A node is an operating system (OS) image, usually an individual computer. The nodes are connected to a storage area network (SAN) that connects the storage systems to the nodes in the cluster. A node can belong to only one cluster.

This use of the term node does not have the same meaning as a node in an SGI Origin 3000 or SGI 2000 system.

#### Cluster Database

The cluster database contains configuration information about all nodes and the cluster. The `fs2d` daemon manages the distribution of the cluster database (CDB) across the nodes in the pool.

#### Pool

The pool is the entire set of nodes that are coupled to each other by networks and are defined as nodes in the cluster database. The nodes are usually close together and should always serve a common purpose. A replicated cluster database is stored on each node in the pool.

All nodes that can be added to a cluster are part of the pool, but not all nodes in the pool must be part of the cluster. There is only one pool. (Other pools may exist, but each is disjoint from the other. They share no node or cluster definitions.)

#### Cluster

The cluster is the set of nodes in the pool that have been defined as a cluster. The cluster is identified by a simple name; this name must be unique within the pool. (For example, you cannot use the same name for the cluster and for a node.)

All nodes in the cluster are also in the pool. However, all nodes in the pool are not necessarily in the cluster; that is, the cluster may consist of a subset of the nodes in the pool. There is only one cluster per pool.

Figure 1-1 shows the concepts of pool and cluster.

Figure 1-1. Pool and Cluster Concepts

#### Membership

There are the following types of membership:

- FailSafe membership is the list of FailSafe nodes in the cluster on which FailSafe can make resource groups online.

- `fs2d` database membership (also known as user-space membership) is the group of nodes in the pool that are accessible to `fs2d`. The `fs2d` daemon manages the distribution of the cluster database across the nodes in the pool; nodes available to `fs2d` are able to receive cluster database updates, and are therefore part of the `fs2d` database membership; this may be a subset of the nodes defined in the pool.

- Process membership is the list of process instances in a cluster that form a cluster process group. There can be multiple process groups per node.

With CXFS coexecution, there is also CXFS membership. For more information about CXFS, see “Coexecution of CXFS and IRIS FailSafe” in Chapter 2, and the CXFS Software Installation and Administration Guide.

#### Quorum

The quorum is the number of nodes required to form a cluster, which differs according to membership:

- For FailSafe membership: >50% (a majority) of the nodes in the cluster where highly available (HA) services were started must be in a known state (successfully reset or talking to each other using heartbeat and control networks) are required to form and maintain a cluster.

- For `fs2d` database membership, 50% (half) of the nodes in the pool must be available to the `fs2d` daemon (and can therefore receive cluster database updates) to form and maintain a cluster.

Figure 1-2 shows an example of FailSafe and `fs2d` memberships. The figure describes the following:

- A pool consisting of six nodes, N1 through N6.

- A cluster that has been defined to have four nodes, N1 to N4.

- HA services have been started on four nodes, N1 to N4. (HA services can only be started on nodes that have been defined as part of the cluster; however, not all nodes within the cluster must have HA services started. )

- Three nodes are up (N1 through N3) and three nodes are down (N4 through N6).

- The `fs2d` membership consists of N1 through N3, 3 of 6 nodes in the pool (50%).

- The FailSafe membership also consists of nodes N1 through N3, 3 of 4 nodes where HA services were started.

Figure 1-2. FailSafe Membership

If a network partition results in a tied membership, in which there are two sets of nodes each consisting of 50% of the cluster, a node from the set containing the tie-breaker node will attempt to reset a node in the other set in order to maintain a quorum. Figure 1-3 shows an example of this, including the following:

- Because nodes N1 and N2 are the set containing the tie breaker, they will both try to reset the nodes in the other set (N3 and N4).

- After the reset is successful and the membership of N1 and N2 is confirmed, resource groups that were in N3 and N4 will be moved to N1 and N2, assuming that is what is defined in their failover policies.

| Note: The figure really shows one sequence of events; the different time designations are for illustration purposes.

Figure 1-3. Tie-breaker and Membership

#### Private Network

A private network is one that is dedicated to cluster communication and is accessible by administrators but not by users:

- The cluster software uses the private network to send the heartbeat/control messages necessary for the cluster configuration to function. If there are delays in receiving heartbeat messages, the cluster software may determine that a node is not responding and remove it from FailSafe membership. The node will be reset.

- Rebooting network equipment can cause the nodes in a cluster to lose communication; the cluster will move into a degraded state if communication between nodes is lost. Using a private network limits the traffic on the network and therefore will help avoid unnecessary resets or disconnects. Also, because the messaging protocol does not prevent snooping (viewing) or spoofing (in which one machine on the network masquerades as another), a network with restricted access is safer than one with user access.

Therefore, because the performance and security characteristics of a public network could cause problems in the cluster and because heartbeat is very timing-dependent (even small variations can cause problems), a private network is required.

In addition, SGI recommends that all nodes be on the same local network segment.

| Note: If there are any network issues on the private network, fix them before trying to use FailSafe.

#### Resource

A resource is a single physical or logical entity that provides a service to clients or other resources. For example, a resource can be a single disk volume, a particular network address, or an application such as a web server. A resource is generally available for use over time on two or more nodes in a cluster, although it can be allocated to only one node at any given time.

Resources are identified by a resource name and a resource type.

#### Resource Type

A resource type is a particular class of resource. All of the resources in a particular resource type can be handled in the same way for the purposes of failover. Every resource is an instance of exactly one resource type.

A resource type is identified by a simple name; this name must be unique within the cluster. A resource type can be defined for a specific node or it can be defined for an entire cluster. A resource type that is defined for a specific node overrides a clusterwide resource type definition with the same name; this allows an individual node to override global settings from a clusterwide resource type definition.

The FailSafe software includes many predefined resource types. If these types fit the application you want to make highly available, you can reuse them. If none fit, you can create additional resource types by using the instructions in the IRIS FailSafe Version 2 Programmer's Guide.

#### Resource Name

A resource name identifies a specific instance of a resource type. A resource name must be unique for a given resource type.

#### Resource Group

A resource group is a collection of interdependent resources. A resource group is identified by a simple name; this name must be unique within a cluster. Table 1-1 shows an example of the resources and their corresponding resource types for a resource group named `Webgroup`.

Table 1-1. Example `Webgroup` Resource Group

Resource | Resource Type
--- | ---
`10.10.48.22` | `IP_address`
`/fs1` | `filesystem`
`vol1` | `volume`
`web1` | `Netscape_web`

If any individual resource in a resource group becomes unavailable for its intended use, then the entire resource group is considered unavailable. Therefore, a resource group is the unit of failover.

Resource groups cannot overlap; that is, two resource groups cannot contain the same resource.

#### Dependency

One resource can be dependent on one or more other resources; if so, it will not be able to start (that is, be made available for use) unless the dependent resources are also started. Dependent resources must be part of the same resource group and are identified in a resource dependency list. Resource dependencies are verified when resources are added to a resource group, not when resources are defined.

Like resources, a resource type can be dependent on one or more other resource types. If such a dependency exists, at least one instance of each of the dependent resource types must be defined. A resource type dependency list details the resource types upon which a resource type depends.

For example, a resource type named `Netscape_web` might have resource type dependencies on resource types named `IP_address` and `volume`. If a resource named `WS1` is defined with the `Netscape_web` resource type, then the resource group containing `WS1` must also contain at least one resource of the type `IP_address` and one resource of the type `volume`. This is shown in Figure 1-4.

Figure 1-4. Resource Type Dependencies

#### Failover

A failover is the process of allocating a resource group (or application) to another node, according to a failover policy. A failover may be triggered by the failure of a resource, a change in the FailSafe membership (such as when a node fails or starts), or a manual request by the administrator.

#### Failover Policy

A failover policy is the method used by FailSafe to determine the destination node of a failover. A failover policy consists of the following:

- Failover domain

- Failover attributes

- Failover script

FailSafe uses the failover domain output from a failover script along with failover attributes to determine on which node a resource group should reside.

The administrator must configure a failover policy for each resource group. A failover policy name must be unique within the pool.

#### Failover Domain

A failover domain is the ordered list of nodes on which a given resource group can be allocated. The nodes listed in the failover domain must be within the same cluster; however, the failover domain does not have to include every node in the cluster.

The administrator defines the initial failover domain when creating a failover policy. This list is transformed into a run-time failover domain by the failover script; FailSafe uses the run-time failover domain along with failover attributes and the FailSafe membership to determine the node on which a resource group should reside. FailSafe stores the run-time failover domain and uses it as input to the next failover script invocation. Depending on the run-time conditions and contents of the failover script, the initial and run-time failover domains may be identical.

In general, FailSafe allocates a given resource group to the first node listed in the run-time failover domain that is also in the FailSafe membership; the point at which this allocation takes place is affected by the failover attributes.

#### Failover Attribute

A failover attribute is a string that affects the allocation of a resource group in a cluster. The administrator must specify system attributes (such as `Auto_Failback` or `Controlled_Failback`), and can optionally supply site-specific attributes.

#### Failover Scripts

A failover script is a shell script that generates a run-time failover domain and returns it to the `ha_fsd` process. The `ha_fsd` process applies the failover attributes and then selects the first node in the returned failover domain that is also in the current FailSafe membership.

The following failover scripts are provided with the FailSafe release:

- `ordered`, which never changes the initial failover domain. When using this script, the initial and run-time failover domains are equivalent.

- `round-robin`, which selects the resource group owner in a round-robin (circular) fashion. This policy can be used for resource groups that can be run in any node in the cluster.

If these scripts do not meet your needs, you can create a new failover script using the information provided in the IRIS FailSafe Version 2 Programmer's Guide.

#### Action Scripts

The action scripts determine how a resource is started, monitored, and stopped. There must be a set of action scripts specified for each resource type.

The following is the complete set of action scripts that can be specified for each resource type:

- `exclusive`, which verifies that a resource is not already running

- `start`, which starts a resource

- `stop`, which stops a resource

- `monitor`, which monitors a resource

- `restart`, which restarts a resource on the same server after a monitoring failure occurs

The release includes action scripts for predefined resource types. If these scripts fit the resource type that you want to make highly available, you can reuse them by copying them and modifying them as needed. If none fits, you can create additional action scripts by using the instructions provided in the IRIS FailSafe Version 2 Programmer's Guide.

#### Cluster Process Group

A cluster process group is a group of application instances in a distributed application that cooperate to provide a service. Each application instance can consist of one or more UNIX processes and spans only one node.

For example, distributed lock manager instances in each node would form a process group. By forming a process group, they can obtain process membership and reliable, ordered, atomic communication services.

| Note: There is no relationship between UNIX process group and cluster process group.

#### Plug-In

A plug-in is the set of software required to make an application highly available, including a resource type and action scripts. There are plug-ins provided with the base FailSafe release, optional plug-ins available for purchase from SGI, and customized plug-ins you can write using the instructions in the IRIS FailSafe Version 2 Programmer's Guide. For a list of provided and optional plug-ins, see “Layers”.

### Hardware Components

Figure 1-5 shows an example of FailSafe hardware components, in this case for a two-node system.

Figure 1-5. Sample System Components

The hardware components are as follows:

- Up to eight SGI 2000, SGI 200, or SGI Origin 3000 nodes.

- More than two interfaces on each node for control networks.

  At least two Ethernet or FDDI interfaces on each node are required for the control network heartbeat connection, by which each node monitors the state of other nodes. The FailSafe software also uses this connection to pass control messages between nodes. These interfaces have distinct IP addresses.

- A serial line from a serial port on each node to a Remote System Control port on another node.

  A node that is taking over services on the failed node uses this line to reboot the failed node during takeover. This procedure ensures that the failed node is not using the shared disks when the replacement node takes them over.

- Disk storage and SCSI bus/Fibre Channel shared by the nodes in the cluster.

  The nodes in the FailSafe system share dual-hosted disk storage over a shared fast and wide SCSI bus or Fibre Channel. The storage connection is shared so that either node can take over the disks in case of failure. The hardware required for the disk storage can be one of the following:

  - Origin SCSI JBOD/RAID

  - Origin FC RAID deskside or rackmount storage systems; each chassis assembly has two storage-control processors (SPs) and at least five disk modules with caching enabled

  - TP9100

  - TP9400

  - TP9900

- An EL-8+ (`FAILSAFE-N_NODE`) hardware component to reset machines in a cluster or, optionally, an ST16XX or EL-16 hardware component.

In addition, FailSafe supports ATM LAN emulation failover when FORE Systems ATM cards are used with a FORE Systems switch.

| Note: The FailSafe system is designed to survive a single point of failure. Therefore, when a system component fails, it must be restarted, repaired, or replaced as soon as possible to avoid the possibility of two or more failed components.

### Disk Connections

A FailSafe system supports the following disk connections:

- RAID support

  - Single or dual controllers

  - Single or dual hubs

  - Single or dual pathing

- JBOD support

  - Single or dual vaults

  - Single or dual hubs

SCSI disks can be connected to two machines only. Fibre Channel disks can be connected to multiple machines.

### Supported Configurations

FailSafe supports the following highly available configurations:

- Basic two-node configuration

- Star configuration of multiple primary and one backup node

- Ring configuration

Figure 1-6. Configuration Types

These configurations provide redundancy of processors and I/O controllers. Redundancy of storage is obtained through the use of multihosted RAID disk devices and plexed (mirrored) disks.

You can use the following reset models when configuring a FailSafe system:

- Server-to-server. Each server is directly connected to another for reset. May be unidirectional.

- Network. Each server can reset any other by sending a signal over the control network to an EL-16 multiplexer.

- IRISconsole. Each server can request that the IRISconsole perform resets.

Figure 1-7. Reset Types

In a basic two-node configuration, the following arrangements are possible:

- All highly available services run on one node. The other node is the backup node. After failover, the services run on the backup node. In this case, the backup node is a hot standby for failover purposes only. The backup node can run other applications that are not highly available services.

- Highly available services run concurrently on both nodes. For each service, the other node serves as a backup node. For example, both nodes can be exporting different NFS filesystems. If a failover occurs, one node then exports all of the NFS filesystems.

## Additional Features

FailSafe provides the following features to increase the flexibility and ease of operation of a highly available system:

- Dynamic management

- Fine-grain failover

- Local restarts

These features are summarized in the following sections.

### Dynamic Management

FailSafe allows you to perform a variety of administrative tasks while the system is running:

- Monitor applications. You can turn monitoring of an application on and off while FailSafe continues to run. This allows you to perform online application upgrades without bringing down the FailSafe system.

- Managed resources. You can add resources while the FailSafe system is online.

- Upgrade FailSafe software. You can upgrade FailSafe software on one node at a time without taking down the entire FailSafe cluster.

### Fine-Grain Failover

The unit of failover is a resource group. This limits the impact of a component failure to the resource group to which that component belongs, and does not affect other resource groups or services on the same node. The process in which a specific resource group is failed over from one node to another node while other resource groups continue to run on the first node is called fine-grain failover.

### Local Restarts

FailSafe allows you to fail over a resource group onto the same node. This feature enables you to configure a single-node system, where backup for a particular application is provided on the same machine, if possible. It also enables you to indicate that a specified number of local restarts be attempted before the resource group fails over to a different node.

## Administration

You can perform all FailSafe administrative tasks by means of the FailSafe Manager graphical user interface (GUI). The GUI provides a guided interface to configure, administer, and monitor a FailSafe-controlled highly available cluster. The GUI also provides screen-by-screen help text.

If you wish, you can perform administrative tasks directly by using the `cmgr(1M)` command, which provides a command-line interface for the administration tasks.

For more information, see Chapter 4, “Administration Tools”, see Chapter 5, “Configuration”, and Chapter 7, “IRIS FailSafe System Operation”.

## Highly Available Resources

This section discusses the highly available resources in a FailSafe system:

- Nodes

- Network interfaces and IP addresses

- Disks

### Nodes

If a node crashes or hangs (for example, due to a parity error or bus error), the FailSafe software detects this. A different node, determined by the failover policy, takes over the failed node's services after resetting the failed node.

If a node fails, the interfaces, access to storage, and services also become unavailable. See the succeeding sections for descriptions of how the FailSafe system handles or eliminates these points of failure.

### Network Interfaces and IP Addresses

Clients access the highly available services provided by the FailSafe cluster using IP addresses. Each highly available service can use multiple IP addresses. The IP addresses are not tied to a particular highly available service; they can be shared by all the highly available services in the cluster.

FailSafe uses the IP aliasing mechanism to support multiple IP addresses on a single network interface. Clients can use a highly available service that uses multiple IP addresses even when there is only one network interface in the server node.

The IP aliasing mechanism allows a FailSafe configuration that has a node with multiple network interfaces to be backed up by a node with a single network interface. IP addresses configured on multiple network interfaces are moved to the single interface on the other node in case of a failure.

| Note: That is, the hostname is bound to a different IP address that never moves.

FailSafe requires that each network interface in a cluster have an IP address that does not fail over. These IP addresses, called fixed IP addresses, are used to monitor network interfaces.

Each fixed IP address must be configured to a network interface at system boot up time. All other IP addresses in the cluster are configured as highly available (HA) IP addresses.

Highly available IP addresses are configured on a network interface. During failover and recovery processes they moved to another network interface in the other node by FailSafe. Highly available IP addresses are specified when you configure the FailSafe system. FailSafe uses the `ifconfig` command to configure an IP address on a network interface and to move IP addresses from one interface to another.

In some networking implementations, IP addresses cannot be moved from one interface to another by using only the `ifconfig` command. FailSafe uses medium access control (MAC) address impersonation (re-MACing) to support these networking implementations. Re-MACing moves the physical MAC address of a network interface to another interface. It is done by using the `macconfig` command. Re-MACing is done in addition to the standard `ifconfig` process that FailSafe uses to move IP addresses. To do re-MACing in FailSafe, a resource of type `MAC_Address` is used.

| Note: Re-MACing can be used only on Ethernet networks. It cannot be used on FDDI networks.

Re-MACing is required when packets called gratuitous ARP packets are not passed through the network. These packets are generated automatically when an IP address is added to an interface (as in a failover process). They announce a new mapping of an IP address to a MAC address. This tells clients on the local subnet that a particular interface now has a particular IP address. Clients then update their internal ARP caches with the new MAC address for the IP address. (The IP address just moved from interface to interface.) When gratuitous ARP packets are not passed through the network, the internal ARP caches of subnet clients cannot be updated. In these cases, re-MACing is used. This moves the MAC address of the original interface to the new interface. Thus, both the IP address and the MAC address are moved to the new interface and the internal ARP caches of clients do not need updating.

Re-MACing is not done by default; you must specify that it be done for each pair of primary and secondary interfaces that requires it. A procedure in the section “Planning Network Interface and HA IP Address Configuration” in Chapter 2, describes how you can determine whether re-MACing is required. In general, routers and PC/NFS clients may require re-MACing interfaces.

A side effect of re-MACing is that the original MAC address of an interface that has received a new MAC address is no longer available for use. Because of this, each network interface has to be backed up by a dedicated backup interface. This backup interface cannot be used by clients as a primary interface. (After a failover to this interface, packets sent to the original MAC address are ignored by every node on the network.) Each backup interface backs up only one network interface.

### Disks

The FailSafe storage system can include shared SCSI or Fibre Channel JBOD or RAID storage systems. It can also include vault peripheral enclosures using SCSI disks with plexing. If highly available applications use filesystems, XFS filesystems or CXFS filesystems must be used. All data for highly available applications must be stored in XLV logical volumes on shared disks or, if CXFS filesystems are used, in XVM logical volumes.

For Fibre Channel RAID storage systems, if a disk or disk controller fails, the RAID storage system is equipped to keep services available through its own capabilities.

With plexed XLV logical volumes on the disks in a FibreVault, the XLV system provides redundancy. No participation of the FailSafe system software is required for a disk failure. If a disk controller fails, the FailSafe system software initiates the failover process.

Figure 1-8 shows disk storage takeover on a two-node system. The surviving node takes over the shared disks and recovers the logical volumes and filesystems on the disks. This process is expedited by the XFS filesystem, which supports fast recovery because it uses journaling technology that does not require the use of the `fsck` command for filesystem consistency checking.

Figure 1-8. Disk Storage Failover on a Two-Node System

## Highly Available Applications

Each application has a primary node and up to seven additional nodes that you can use as a backup node, according to the failover policy you define. The primary node is the node on which the application runs when FailSafe is in normal state. When a failure of any highly available application is detected by FailSafe software, all resources in the affected resource group on the failed node are failed over to a different node and the resources on the failed node are stopped. When these operations are complete, the resources are started on the backup node.

All information about resources, including the primary node, components of the resource group, and failover policy is specified when you configure your FailSafe system with the GUI or with the `cmgr(1M)` command. Information on configuring the system is provided in Chapter 5, “Configuration”. Monitoring scripts detect the failure of a resource.

The FailSafe software provides a framework for making applications highly available services. By writing scripts and configuring the system in accordance with those scripts, you can turn client/server applications into highly available applications. For information, see the IRIS FailSafe Version 2 Programmer's Guide.

## Failover and Recovery Processes

A failure is when the node has crashed, hung, or been shut down, or when a highly available service is no longer operating. The node with the failure is called the failed node. A different node performs a failover of the highly available services that are being provided on the failed node. Failover allows all of the highly available services, including those provided by the failed node, to remain available within the cluster.

Depending on which node detects the failure, the sequence of actions following the failure is different.

If the failure is detected by the FailSafe software running on the same node, the failed node performs the following operations:

- Stops the highly available resource group running on the failed node

- Moves the highly available resource group to a different node, according to the defined failover policy for the resource group

- Asks the new node to start providing all resource group services previously provided by the failed node

When it receives the message, the node that is taking over the resource group performs the following operations:

- Transfers ownership of the resource group from the failed node to itself

- Starts offering the resource group services that were running on the failed node

If the failure is detected by FailSafe software running on a different node, the node detecting the failure performs these operations:

- Power-cycles the failed node (to prevent corruption of data) by using the serial connection between the nodes

- Transfers ownership of the resource group from the failed node to the other nodes in the cluster, based on the resource group failover policy

- Starts offering the resource group services that were running on the failed node

When a failed node comes back up, whether the node automatically starts to provide highly available services again depends on the failover policy you define.

For more information, see “Define a Failover Policy with the GUI” in Chapter 5.

Normally, a node that experiences a failure automatically reboots and resumes providing highly available services. This scenario works well for transient errors (as well as for planned outages for equipment and software upgrades).

For further information on FailSafe execution during startup and failover, see “Execution of FailSafe Action and Failover Scripts”.

## Overview of Configuring and Testing a New Cluster

After the FailSafe cluster hardware has been installed, use the following general procedure to configure and test the FailSafe system:

1. Become familiar with FailSafe terms by reviewing this chapter.

2. Plan the configuration of highly available applications and services on the cluster using Chapter 2, “Configuration Planning”.

3. Perform various administrative tasks, including the installation of prerequisite software, that are required by FailSafe, as described in Chapter 3, “Installation and System Preparation”.

4. Define the configuration as explained in Chapter 5, “Configuration”.

5. Test the system in three phases:

   - Test individual components prior to starting FailSafe software.

   - Test normal operation of the system.

   - Simulate failures to test the operation of the system after a failure occurs.

## Software Overview

This section describes the software layers, communication paths, and cluster database of a FailSafe system.

### Layers

A FailSafe system has the following software layers:

- Plug-ins, which create highly available services. Table 1-2 shows the provided and optional FailSafe plug-ins and their associated resource types.

  Table 1-2. Provided and Optional Plug-Ins

  Provided Plug-In | Resource Type | Optional Plug-In | Resource Type
  --- | --- | --- | ---
  CXFS file system | `CXFS` | FailSafe/DMF | `DMF`
  IP addresses | `IP_address` | FailSafe/NFS | `NFS` and `statd_unlimited`
  MAC addresses | `MAC_address` | FailSafe/Informix | `INFORMIX_DB`
  XFS file systems | `filesystem` | FailSafe/Oracle | `Oracle_DB`
  XLV logical volumes | `volume` | FailSafe/Samba | `Samba`
  |  | FailSafe/TMF | `TMF`
  |  | FailSafe/Web (Netscape) | `Netscape_web`

  See the release notes for information about the specific releases of these products that are supported.

  If the application you want is not available, you can hire the SGI Professional Services group to develop the required software, or you can use the IRIS FailSafe Version 2 Programmer's Guide to write the software yourself.

- FailSafe base, which includes the ability to define resource groups and failover policies.

- Cluster services, which lets you define clusters, resources, and resource types (this consists of the `cluster_services` installation package)

- Cluster software infrastructure, which lets you do the following:

  - Perform node logging

  - Administer the cluster

  - Define nodes

  The cluster software infrastructure consists of the `cluster_admin` and `cluster_control` subsystems.

Figure 1-9 shows a graphic representation of these layers. The cluster services and cluster software infrastructure layers are shared with CXFS. Table 1-3, describes the contents of the `/usr/cluster/bin` directory. For more information about CXFS, see the CXFS Software Installation and Administration Guide.

Figure 1-9. Software Layers

Table 1-3. Contents of `/usr/cluster/bin`

Layer | Subsystem | Process | Description
--- | --- | --- | ---
Plug-ins | `failsafe_informix` `failsafe2_oracle` | `ha_ifmx2` | IRIS FailSafe database agents. Each database agent monitors all instances of one type of database.
IRIS FailSafe Base | `failsafe2` | `ha_fsd` | IRIS FailSafe daemon. Provides basic component of the IRIS FailSafe software.
Cluster services (high-availability processes) | `cluster_services` | `ha_cmsd` | The FailSafe membership daemon. Provides the list of nodes, called FailSafe membership, available to the cluster.
|  | `ha_gcd` | Group membership daemon. Provides group membership and reliable communication services in the presence of failures to IRIS FailSafe processes.
|  | `ha_srmd` | System resource manager daemon. Manages resources, resource groups, and resource types. Executes action scripts for resources.
|  | `ha_ifd` | Interface agent daemon. Monitors the local node's network interfaces. This daemon is described in detail in “Interface Agent Daemon (IFD)”.
Cluster software infrastructure (cluster administrative processes) | `cluster_admin` | `cad` | Cluster administration daemon. Provides administration services.
| `cluster_control` | `crsd` | Node control daemon. Monitors the serial connection to other nodes. Has the ability to reset other nodes.
|  | `cmond` | Daemon that manages all other daemons. This process starts other processes in all nodes in the cluster and restarts them on failures.
|  | `fs2d` | Manages the configuration database and keeps each copy in sync on all nodes in the pool.

### Interface Agent Daemon (IFD)

The IFD is an agent that monitors network interfaces and IP addresses. The IFD monitors all network interfaces and IP addresses configured in the node even when there are no highly available IP addresses in the node.

The IFD checks the number of input packets for each interace. If the number of input packets does not increase for a 10-second period, the IFD contacts the broadcast address of the interface by using the `ping(1)` command. If the input packet count does not increase in the next 10-second period, the network interface and all IP addresses on the interface are marked as bad.

The IFD reads the configuration of IP addresses from the cluster database.

`IP_address` resource type action scripts use the `ha_ifdadmin` command to communicate with the IFD. Action scripts obtain status and configuration IP address from the IFD.

IFD logging can be controlled with the GUI and the `cmgr` command.

### Communication Paths

The following figures show communication paths in FailSafe.

| Note: The following figures do not represent the `cmond` cluster manager daemon. The purpose of this daemon is to keep the other daemons running.

Figure 1-10. Administrative Communication within One Node

Figure 1-11. Daemon Communication within One Node

Figure 1-12. Communication between Nodes in the Pool

Figure 1-13. Communication for a Node Not in the Cluster

### Execution of FailSafe Action and Failover Scripts

The order of execution is as follows:

1. FailSafe starts up by using the `start ha_services` command in `cmgr` or as part of the node bootup procedure. It then reads the resource group information from the cluster database.

2. FailSafe tells the system resource manager (SRM) to run `exclusive` scripts for all resource groups that are in the `Online ready` state.

3. SRM returns one of the following states for each resource group:

   - `running`

   - `partially running`

   - `not running`

4. If a resource group has a state of `not running` in a node where HA services have been started, the following occurs:

   1. FailSafe runs the failover policy script associated with the resource group. The failover policy scripts takes the list of nodes that are capable of running the resource group (the failover domain) as a parameter.

   2. The failover policy script returns an ordered list of nodes in descending order of priority (the run-time failover domain) where the resource group can be placed.

   3. FailSafe sends a request to SRM to move the resource group to the first node in the run-time failover domain.

   4. SRM executes the `start` action script for all resources in the resource group:

      - If the `start` script fails, the resource group is marked `online` on that node with following error:

            srmd executable error

      - If the `start` script is successful, SRM automatically starts monitoring those resources. After the specified start monitoring time passes, SRM executes the `monitor` action script for the resource in the resource group.

5. If the state of the resource group has a status of `running` or `partially running` on only one node in the cluster, FailSafe runs the associated failover policy script:

   - If the highest priority node is the same node where the resource group is `partially running` or `running`, the resource group is made online on the same node. In the `partially running` case, FailSafe tells SRM to execute `start` scripts for all resources in the resource group.

   - If the highest priority node is another node in the cluster, FailSafe tells SRM to execute `stop` action scripts for resources in the resource group on other nodes. FailSafe then makes the resource group online in the highest priority node in the cluster.

6. If the state of the resource group is `running` or `partially running` in multiple nodes in the cluster, the resource group is marked with an `error exclusivity` error. These resource groups will require operator intervention to become online in the cluster.

Figure 1-14 shows the message paths for action scripts and failover policy scripts.

Figure 1-14. Message Paths for Action Scripts and Failover Policy Scripts

### When a `start` Script Fails

When the `start` action script fails, the order of execution is as follows:

1. SRM notifies FailSafe of the `start` action script failure as a resource group failure.

2. FailSafe runs the failover policy script to determine the next node for the resource group.

3. FailSafe sends a request to SRM to release the resource group and allocate the resource group in the next node in the cluster.

### When a `stop` Script Fails

When the `stop` action script fails, the order of execution is as follows:

1. SRM notifies FailSafe of the `stop` action script failure as a resource group failure.

2. FailSafe marks the resource group with the following error:

       srmd executable error

3. The system administrator must use the `offline force` command to clear the error state after stopping the resource group in the node.

### Components

The cluster database is a key component of FailSafe software. It contains all information about the following:

- Resources

- Resource types

- Resource groups

- Failover policies

- Nodes

- Clusters

The cluster database daemon (`fs2d`) maintains identical databases on each node in the cluster.

Table 1-4 shows the contents of the `/var/cluster/ha` directory.

Table 1-4. Contents of the `/var/cluster/ha` directory

Directory or File | Purpose
--- | ---
`comm/` | Directory that contains files that communicate between various daemons. FailSafe processes create temporary files in this directory. FailSafe interprocess communication will fail if there is not sufficient disk space for this directory (approximately 2-3 MB) in the root filesystem on every node in a FailSafe cluster.
`common_scripts/` | Directory that contains the script library (the common functions that may be used in action scripts).
`log/` | Directory that contains the logs of all scripts and daemons executed by IRIS FailSafe. The outputs and errors from the commands within the scripts are logged in the `script_nodename` file.
`policies/` | Directory that contains the failover scripts used for resource groups.
`resource_types/template` | Directory that contains the template action scripts.
`resource_types/rt_name` | Directory that contains the action scripts for the rt_name resource type. For example, `/var/cluster/ha/resource_types/filesystem`.
`resource_types/rt_name/exclusive` | Script that verifies that a resource of this resource type is not already running.
`resource_types/rt_name/monitor` | Script that monitors a resource of this resource type.
`resource_types/rt_name/restart` | Script that restarts a resource of this resource type on the same node after a monitoring failure.
`resource_types/rt_name/start` | Script that starts a resource of this resource type.
`resource_types/rt_name/stop` | Script that stops a resource of this resource type.

Prev | Table of Contents | Next
About This Guide |  | Chapter 2. Configuration Planning