Chapter 10. Upgrading and Maintaining Active Clusters
Prev |  | Next

## Chapter 10. Upgrading and Maintaining Active Clusters

When an IRIS FailSafe system is running, you may need to perform various administration procedures without shutting down the entire cluster. This chapter provides instructions for performing upgrade and maintenance procedures on active clusters. It includes the following procedures:

- “Add a Node to an Active Cluster”

- “Delete a Node from an Active Cluster”

- “Change Control Networks in a Cluster”

- “Upgrade OS Software in an Active Cluster”

- “Upgrade FailSafe Software in an Active Cluster”

- “Add New Resource Groups or Resources in an Active Cluster”

- “Adding a New Hardware Device in an Active Cluster”

## Add a Node to an Active Cluster

Use the following procedure to add a node to an active cluster. This procedure assumes that `cluster_admin`, `cluster_control`, `cluster_ha`, and `failsafe2` products are already installed in this node.

1. Check control network connections from the node to the rest of the cluster using `ping(1M)` command. Note the list of control network IP addresses.

2. Check the serial connections to reset this node. Note the name of the node that can reset this node.

3. Run node diagnostics. For information on FailSafe diagnostic commands, see Chapter 8, “Testing the Configuration”.

4. Make sure that the `sgi-cad`, `sgi-crsd`, `sgi-cmsd`, and `sgi-gcd` entries are present in the `/etc/services` file. The port numbers for these processes should match the port numbers in other nodes in the cluster.

   Example entries:

       sgi-cad         7200/tcp      # SGI cluster admin daemon
       sgi-crsd        7500/udp      # SGI cluster reset services daemon
       sgi-cmsd        7000/udp      # SGI FailSafe membership Daemon
       sgi-gcd         8000/udp      # SGI group communication Daemon

5. Check if the cluster processes (`cad`, `cmond`, `crsd`) are running.

       # ps -ef | grep cad

   If cluster processes are not running, run the `cdbreinit` command.

       # /usr/cluster/bin/cdbreinit /var/cluster/cdb/cdb.db
       	Killing fs2d...
       	Removing database header file /var/cluster/cdb/cdb.db...
       	Preparing to delete database directory /var/cluster/cdb/cdb.db# !!
       	Continue[y/n]y
       	Removing database directory /var/cluster/cdb/cdb.db#...
       	Deleted CDB database at /var/cluster/cdb/cdb.db
       	Recreating new CDB database at /var/cluster/cdb/cdb.db with cdb-exitop...
       	 fs2d
       	 Created standard CDB database in /var/cluster/cdb/cdb.db

       	 Please make sure that "sgi-cad" service is added to /etc/services file
       	 If not, add the entry and restart cluster processes.
       	 Please refer to IRIS FailSafe administration manual for more
       	 information.

       	 Modifying CDB database at /var/cluster/cdb/cdb.db with cluster_ha-exitop...
       	 Modified standard CDB database in /var/cluster/cdb/cdb.db

       	 Please make sure that "sgi-cmsd" and "sgi-gcd" services are added
       	 to /etc/services file before starting HA services.
       	 Please refer to IRIS FailSafe administration manual for more
       	 information.

       	 Starting cluster control processes with cluster_control-exitop...

       	 Please make sure that "sgi-crsd" service is added to /etc/services file
       	 If not, add the entry and restart cluster processes.
       	 Please refer to IRIS FailSafe administration manual for more
       	 information.

       	 Started cluster control processes
       	 Restarting cluster admin processes with failsafe-exitop...

6. Use the GUI, the `cmgr` command, or the `cmgr` template (`/var/cluster/cmgr-templates/cmgr-create-node`), to define the node.

   | Note: This node must be defined from one of nodes that is already in the cluster.

7. Use the `cmgr` command or the GUI to add the node to the cluster.

   For example: the following `cmgr` command adds the node `web-node3` to the cluster `web-cluster`:

       cmgr> modify cluster web-cluster
       Enter commands, when finished enter either "done" or "cancel"

       web-cluster ? add node web-node3
       web-cluster ? done

8. Start highly available (HA) services on this node using the `cmgr` command to the GUI.

   For example, the following `cmgr` command starts HA services on node `web-node3` in cluster `web-cluster`:

       cmgr> start ha_services on node web-node3 in cluster web-cluster

9. Add this node to the failure domain of the relevant failover policy. In order to do this, the entire failover policy must be redefined, including the additional node in the failure domain.

## Delete a Node from an Active Cluster

Use the following procedure to delete a node from an active cluster. This procedure assumes that the node status is `UP`.

1. If resource groups are online on the node, use the `cmgr` command or the GUI to move them to another node in the cluster.

   To move the resource groups to another node in the cluster, there should be another node available in the failover policy domain of the resource group. If you want to leave the resource groups running in the same node, use the `cmgr` command or the GUI to detach the resource group.

   For example, the following command would leave the resource group `web-rg` running in the same node in the cluster `web-cluster`.

       cmgr> admin detach resource_group web-rg in cluster web-cluster

2. Delete the node from the failure domains of any failover policies which use the node. In order to do this, the entire failover policy must be redefined, deleting the affected node from the failure domain.

3. Stop HA services on the node.

   For example, to stop HA services on the node `web-node3`, use the following `cmgr` command. This command will move all the resource groups online on this node to other nodes in the cluster if possible.

       cmgr> stop ha_services on node web-node3 for cluster web-cluster

   If it is not possible to move resource groups that are online on node `web-node3`, the above command will fail. The `force` option is available to stop HA services in a node even in the case of an error. If there are resources that cannot be moved offline or deallocated properly, a side-effect of the offline force command will be to leave these resources allocated on the node.

Perform Steps 4, 5, 6, and 7 if the node must be deleted from the configuration database.

4. Delete the node from the cluster.

   For example, to delete node `web-node3` from `web-cluster` configuration, use the following `cmgr` command:

       cmgr> modify cluster web-cluster
       Enter commands, when finished enter either "done" or "cancel"

       web-cluster ? remove node web-node3
       web-cluster ? done

5. Remove node configuration from the configuration database.

   The following `cmgr` command deletes the `web-node3` node definition from the configuration database:

       cmgr> delete node web-node3

6. Stop all cluster processes and delete the configuration database.

   The following commands stop cluster processes on the node and delete the configuration database:

       # /etc/init.d/cluster stop
       # killall fs2d
       # cdbdelete /var/cluster/cdb/cdb.db

7. Disable cluster and HA processes from starting when the node boots. The following commands perform those tasks:

       # chkconfig cluster off
       # chkconfig failsafe2 off

## Change Control Networks in a Cluster

Use the following procedure to change the control networks in a currently active cluster. This procedure is valid for a two-node cluster consisting of nodes `node1` and `node2`. In this procedure, you must complete each step before proceeding to the next step.

| Note: Do not perform any other administration operations during this procedure.

1. From any node, stop HA services on the cluster. Make sure all HA processes have exited on both nodes.

2. From `node2`, stop the cluster processes on `node2`:

       # /etc/init.d/cluster stop
       # killall fs2d

   Make sure the `fs2d` process have been killed on `node2`.

3. From `node1`, modify the `node1` and `node2` definition. Use the GUI or the following `cmgr` commands:

       cmgr> modify node node1
       Enter commands, when finished enter either "done" or "cancel"
       node1?> remove nic old_nic _address
       node1> add nic new_nic_address
       NIC - new_nic_address set heartbeat to ...
       NIC - new_nic_address set ctrl_msgs to ...
       NIC - new_nic_address set priority to ...
       NIC - new_nic_address done
       node1? done

   Repeat the same procedure to modify `node2`.

4. From `node1`, check if the `node1` and `node2` definitions are correct. Using `cmgr` on `node1`, execute the following commands to view the node definitions:

       cmgr> show node node1
       cmgr> show node node2

5. On both `node1` and `node2`, modify the network interface IP addresses in `/etc/config/netif.options` and execute `ifconfig` to configure the new IP addresses on `node1` and `node2`. Verify that the IP addresses match the node definitions in the cluster database.

6. From `node1`, stop the cluster process on `node1`:

       # /etc/init.d/cluster stop
       # killall fs2d

   Make sure the `fs2d` process have been killed on `node1`.

7. From `node2`, execute the following command to start cluster process on `node2`:

       # /usr/cluster/bin/cdbreinit /var/cluster/cdb/cdb.db

   Answer `y` to the prompt.

8. From `node1`, start cluster processes on `node1`:

       # /etc/init.d/cluster start

   The following messages should appear in the `SYSLOG` on `node2`:

       Starting to receive CDB sync series from machine <node1 node id>
       ...
       Finished receiving CDB sync series from machine <node1 node id>

   Wait for approximately 60 seconds for the synchronization to complete.

9. From any node, start HA services in the cluster.

## Upgrade OS Software in an Active Cluster

When you upgrade your OS software in an active cluster, you perform the upgrade on one node at a time.

If the OS software upgrade does not require reboot or does not impact the FailSafe software, there is no need to use the OS upgrade procedure. If you do not know whether the upgrade will impact FailSafe software or if the OS upgrade requires a machine reboot, follow the upgrade procedure described below.

The following procedure upgrades the OS software on node `web-node3`.

1. If resource groups are online on the node, use a `cmgr` command or the GUI to move them another node in the cluster. To move the resource group to another node in the cluster, there should be another node available in the failover policy domain of the resource group.

   The following `cmgr` command moves resource group `web-rg` to another node in the cluster `web-cluster:`

       cmgr> admin move resource_group web-rg in cluster web-cluster

2. To stop HA services on the node `web-node3`, use the following `cmgr` command or the GUI. This command will move all the resource groups online on this node to other nodes in the cluster if possible.

       cmgr> stop ha_services on node web-node3 for cluster web-cluster

   If it is not possible to move resource groups that are online on node `web-node3`, the above command will fail. You can use the `force` option to stop HA services in a node even in the case of an error.

3. Perform the OS upgrade in the node `web-node3`.

4. After the OS upgrade, make sure cluster processes (`cmond`, `cad`, `crsd`) are running.

5. Restart HA services on the node. For example, the following `cmgr` command restarts HA services on the node:

       cmgr> start ha_services on node web-node3 for cluster web-cluster

   Make sure the resource groups are running on the most appropriate node after restarting HA services.

## Upgrade FailSafe Software in an Active Cluster

When you upgrade FailSafe software in an active cluster, you upgrade one node at a time in the cluster.

The following procedure upgrades FailSafe on node `web-node3`.

1. If resource groups are online on the node, use a `cmgr` command or the GUI to move them another node in the cluster. To move the resource group to another node in the cluster, there should be another node available in the failover policy domain of the resource group.

   For example, the following `cmgr` command moves resource group `web-rg` to another node in the cluster `web-cluster:`

       cmgr> admin move resource_group web-rg in cluster web-cluster

2. To stop HA services on the node `web-node3`, use the following `cmgr` command or the GUI. This command will move all the resource groups online on this node to other nodes in the cluster if possible.

       cmgr> stop ha_services on node web-node3 for cluster web-cluster

   If it is not possible to move resource groups that are online on node `web-node3`, the above command will fail. You can use the `force` option to stop HA services in a node even in the case of an error.

3. Stop all cluster processes running on the node.

       # /etc/init.d/cluster stop

4. Perform the FailSafe upgrade in the node `web-node3`.

5. After the FailSafe upgrade, check whether cluster processes (`cmond`, `cad`, `crsd`) are running. If not, restart cluster processes:

       # chkconfig cluster on; /etc/init.d/cluster start

6. Restart HA services on the node. For example, the following `cmgr` command restarts HA services on the node:

       cmgr> start ha_services on node web-node3 for cluster web-cluster

   Make sure the resource groups are running on the most appropriate node after restarting HA services.

## Add New Resource Groups or Resources in an Active Cluster

The following procedure describes how to add a resource group and resources to an active cluster. To add resources to an existing resource group, perform resource configuration (Step 4), perform resource diagnostics (Step 5), and add resources to the resource group (Step 6).

1. Identify all the resources that have to be moved together. These resources running on a node should be able to provide a service to the client. These resources should be placed in a resource group. For example, Netscape webserver `mfg-web`, its highly available (HA) IP address `192.26.50.40`, and the filesystem `/shared/mfg-web` containing the Web configuration and document pages should be placed in the same resource group (for example, `mfg-web-rg`).

2. Configure the resources in all nodes in the cluster where the resource group is expected to be online. For example, this might involve configuring Netscape Web server `mfg-web` on nodes `web-node1` and `web-node2` in the cluster.

3. Create a failover policy. Determine the type of failover attribute required for the resource group. You can use the following `cmgr` template to create the failover policy:

       /var/cluster/cmgr-templates/cmgr-create-failover_policy

4. Configure the resources in configuration database. There are `cmgr` templates to create resources of various resource types in `/var/cluster/cmgr-templates` directory. For example, the volume resource, the `/shared/mfg-web` filesystem, the `192.26.50.40` `IP_address` resource, and the `mfg-web` `Netscape_web` resource have to be created in the configuration database. Create the resource dependencies for these resources.

5. Run resource diagnostics. For information on the diagnostic commands, see Chapter 8, “Testing the Configuration”.

6. Create resource group and add resources to the resource group. You can use the following `cmgr` template to create resource group and add resources to resource group:

       /var/cluster/cmgr-templates/cmgr-create-resource_group

   All resources that are dependent on each other should be added to the resource group at the same time. If resources are added to an existing resource group that is online in a node in the cluster, the resources are also made online on the same node.

## Adding a New Hardware Device in an Active Cluster

You will add new hardware devices to an active cluster one node at a time.

To add hardware devices to a node in an active cluster, follow the same procedure as when you upgrade OS software in an active cluster, as described in “Upgrade OS Software in an Active Cluster”. In summary:

- You must move the resource groups offline and stop HA services in the node before adding the hardware device.

- After adding the hardware device, make sure cluster processes are running and start HA services on the node.

To include the new hardware device in the configuration database, you must modify your resource configuration and your node configuration, where appropriate.

Prev | Table of Contents | Next
Chapter 9. System Recovery |  | Chapter 11. Performance Co-Pilot for FailSafe