This chapter describes administrative tasks you perform to operate and monitor an FailSafe system. It describes how to perform tasks using the FailSafe Manager GUI and the cmgr command. The major sections in this chapter are as follows:
 | Note: SGI recommends that you perform all FailSafe administration from one node in the pool so that the latest copy of the database will be available, even when there are network partitions. |
On Origin 300, Origin 3200C, Onyx 300, and Onyx 3200C systems, there is only one serial/USB port that provides both L1 and console support for the machine. In a FailSafe configuration, this port (the DB9 connector) is used for system reset. It is connected to a serial port in another node or to the Ethernet multiplexer.
To get access to console input and output, the console must be redirected to another serial port in the machine.
Use the following procedure:
Edit the /etc/inittab file to use an alternate serial port.
Either issue an init q command or reboot.
For example, suppose you had the following in the /etc/inittab file (line breaks added for readability):
# on-board ports or on Challenge/Onyx MP machines, first IO4 board ports t1:23:respawn:/sbin/suattr -C CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip -c "exec /sbin/getty ttyd1 console" # alt console t2:23:off:/sbin/suattr -C CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip -c "exec /sbin/getty -N ttyd2 co_9600" # port 2 |
You could change it to the following:
# on-board ports or on Challenge/Onyx MP machines, first IO4 board ports t1:23:off:/sbin/suattr -C CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip -c "exec /sbin/getty ttyd1 co_9600" # port 1 t2:23:respawn:/sbin/suattr -C CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip -c "exec /sbin/getty -N ttyd2 console" # alt console |
 | Caution: Redirecting the console by using the above method works only when IRIX is running. To access the console when IRIX is not running (miniroot), you must physically reconnect the machine: unplug the serial hardware reset cable from the console/L1 port and then connect the console cable. |
After a FailSafe command is started, it may partially complete even if you interrupt the command by typing Ctrl-c. If you halt the execution of a command this way, you may leave the cluster in an indeterminate state and you may need to use the various status commands to determine the actual state of the cluster and its components.
If you have a two-node cluster, you should create an emergency failover policy (step 1 in “System Status”) for each node in preparation for a time when it may need to run by itself. This situation can occur if the other must stay down for maintenance or if it fails and cannot be brought up.
The following procedure describes the steps required to use just one node in the cluster:
Create an emergency failover policy for each node. Each policy should look like the following example when the cmgr command is issued, where ActiveNode is the name of the node using the policy (in the examples, nodeA) and DownNode is the name of the nonfunctioning node (in the examples, nodeB):
cmgr> show failover_policy emergency-ActiveNode Failover Policy: emergency-ActiveNode Version: 1 Script: ordered Attributes: Controlled_Failback InPlace_Recovery Initial AFD: ActiveNode
For example, suppose you have two nodes, nodeA and nodeB. You would have two emergency failover policies:
cmgr> show failover_policy emergency-nodeA Failover Policy: emergency-nodeA Version: 1 Script: ordered Attributes: Controlled_Failback InPlace_Recovery Initial AFD: nodeA cmgr> show failover_policy emergency-nodeB Failover Policy: emergency-nodeB Version: 1 Script: ordered Attributes: Controlled_Failback InPlace_Recovery Initial AFD: nodeB
For more information, see “Define a Failover Policy” in Chapter 5.
At this point, the procedure assumes that the cluster has one node that has tried to come up but is now in the lonely state. The other node is down. The procedure is for recovering from this point.
Modify each resource group to use the appropriate single-node emergency failover policy, using the following cmgr commands or the GUI:
modify resource_group RGname in cluster Clustername set failover_policy to emergency-ActiveNode
For example, on nodeA:
cmgr> set cluster test-cluster cmgr> modify resource_group group1 Enter commands, when finished enter either "done" or "cancel" resource_group group1 ? set failover_policy to emergency-nodeA resource_group group1 ? done Successfully modified resource group group1
Change the state of all of resource groups to offline. (The last known state of these groups was online before the machines went down; however, the resource groups are not actually online at this point because the cluster was booted with the active node having entered the lonely state because the other node is not functional. This step tells the database to label the state of the resource groups appropriately in preparation for later steps.)
Use the following command:
admin offline_force resource_group RGname in cluster Clustername
For example:
cmgr> set cluster test-cluster cmgr> show resource_groups in test-cluster Resource Groups: group1 group2 cmgr> admin offline_force resource_group group1 cmgr> admin offline_force resource_group group2
Force the stop of HA services for the down node:
stop ha_services on DownNode for cluster Clustername force
Note: This is a long running task that and might take a few minutes to complete. The cmgr command will provide intermediate task status. For example:
cmgr> stop ha_services on nodeB for cluster test-cluster force
Mark the resource groups as online in the database. When HA services are started in future steps, the services will come online using the emergency failover policies.
admin online resource_group RGname in cluster Clustername
For example:
cmgr> set cluster test-cluster cmgr> admin online resource_group group1 FailSafe daemon (ha_fsd) is not running on this local node or it is not ready to accept admin commands. Resource Group (group1) is online-ready. Failed to admin: online admin command failed cmgr> show status of resource_group group1 in cluster test-cluster State: Online Ready Error: No error Check resource group group1 status in an active node if HA services are active in cluster
Start HA services on the active node:
start ha_services on ActiveNode for Clustername
HA services are now active in this single node cluster. From this step through the rest of the recovery, services are active and there should be no downtime experienced.
For example:
cmgr> start ha_services on nodeA for cluster test-cluster
Remove and reinitialize the database on the down node, which is now booted in multiuser mode:
# cd /var/cluster/cdb # rm -rf /var/cluster/cdb/cdb* # /usr/cluster/bin/cdbreinit /var/cluster/cdb/cdb.db
Wait for a few minutes and use the tail command to watch the SYSLOG file for messages indicating that this node knows its identity and has joined the cluster. For example (line breaks added here for readability):
# tail -f /var/adm/SYSLOG ... Dec 14 18:23:16 6D:DownNode cmond[1074]: Notification can not be processed, local machine and cluster name is not known. Dec 14 18:23:16 6D:DownNode cmond[1074]: Local machine belongs to cluster Clustername. Dec 14 18:23:16 6D:DownNode cmond[1074]: Local machine name is DownNode"
To resume using the down node, do the following:
Boot the down node into single user mode.
Within single-user mode, use the chkconfig command to set all cluster services to off:
# chkconfig | grep cluster
Boot the DownNode to multiuser mode.
Start HA services for the down node:
cmgr> start ha_services on node DownNode for cluster Clustername
Use the tail command to watch the SYSLOG file for messages that indicate the formerly down node is now in FailSafe membership and that services are in the UP state. Do not continue until you see messages such as the following in this example:
# tail -f /var/adm/SYSLOG ... blue node ActiveNode [81] : UP incarnation 7 age 2:0 blue node DownNode [82] : UP incarnation 1 age 1:0
Modify the resource groups to restore the original failover policies they were using before the failure:
modify resource_group RGname in cluster Clustername set failover_policy to OriginalFailoverPolicy
Note: This only restores the configuration for the static environment. The runtime environment will still be using the single-node policy at this time. For example, if the normal failover policy was normal-fp:
cmgr> set cluster test-cluster cmgr> modify resource_group group1 Enter commands, when finished enter either "done" or "cancel" resource_group group1 ? set failover_policy to normal-fp resource_group group1 ? done Successfully modified resource group group1 cmgr modify resource_group group2 Enter commands, when finished enter either "done" or "cancel" resource_group group2 ? set failover_policy to normal-fp resource_group group2 ? done Successfully modified resource group group2
Perform an offline_detach command on the resource groups in the cluster. This causes FailSafe to stop monitoring the resource group, but does not physically stop the processes on that group. FailSafe will report the status as offline and will not have any control over the group. The resources will remain in service.
Note: Doing an offline_detach operation on a resource group leaves the resources in the resource group running on the node. FailSafe does not track these resources any longer. You should take care should to prevent resources from running on multiple nodes in the cluster.
Because FailSafe is no longer monitoring the group after the offline_detach (or offline_detach_force) is executed, you should not allow FailSafe to recover on any node other than where it was running at the time the offline_detach was performed.
This also means that no other nodes should be allowed to rejoin the FailSafe membership, especially if Auto_Recovery is set in the resource group's failover policy. This restriction is required because the failover policy scripts are run whenever there is a change in membership; rerunning the scripts could cause your previously offline detached resource group to start on a node other than the node where the offline_detach was performed.
FailSafe policy scripts are run only on nodes where FailSafe is running (that is, nodes where HA services have been started). For example, suppose you have a four-node FailSafe cluster (with nodes A, B, C and D), where nodes A, B, and C are in UP state and node D is DOWN state. If resource group RG is made offline using the offline_detach or offline_detach_force command on node B and HA services are shutdown on node B, node D should not rejoin the cluster before resources in RG are stopped manually on node B. If node D rejoins the cluster, the resource group RG will be made online on nodes A, C or D.Use the following command:
admin offline_detach resource_group RGname [in cluster Clustername]
cmgr> admin offline_detach resource_group group1 in cluster test-cluster
Show the status of the resource groups to be sure that they now show as offline.
Note: The resources are still in service even though this command output shows them as offline. show status of resource_group RGname in cluster Clustername
For example:
cmgr> show status of resource_group group1 in cluster test-cluster
Make the resource groups online in the cluster:
admin online resource_group RGname in cluster Clustername
For example:
cmgr> admin online resource_group group1 in cluster test-cluster cmgr> admin online resource_group group2 in cluster test-cluster
Move the resources back to their original nodes. Because our original policies included the InPlace_Recovery attribute, all of the resources have remained on the node that has been active throughout this process.
admin move resource_group RGname in cluster Clustername to node PrimaryOwner
For example:
cmgr> admin move resource_group group1 in cluster test-cluster to node nodeB
If you run into errors after entering the admin move command, see “Ensuring that Resource Groups are Deallocated” in Chapter 9.
While the FailSafe system is running, you can monitor the status of the FailSafe components to determine the state of the component. FailSafe allows you to view the system status in the following ways:
Keep continuous watch on the state of a cluster using the cluster_status command or the GUI.
Query the status of an individual resource group, node, or cluster using either the GUI or the cmgr command.
Use the haStatus script provided with the cmgr command to see the status of all clusters, nodes, resources, and resource groups in the configuration.
The following sections describe the procedures for performing these tasks.
You can use the cluster_status command to monitor the cluster using a curses interface. For example, the following shows a two-node cluster configured for FailSafe only and cluster_status help text displayed:
# /var/cluster/cmgr-scripts/cluster_status
* Cluster=nfs-cluster FailSafe=ACTIVE CXFS=Not Configured 08:45:12
Nodes = hans2 hans1
FailSafe = UP UP
CXFS =
ResourceGroup Owner State Error
bartest-group Offline No error
footest-group Offline No error
bar_rg2 hans2 Online No error
nfs-group1 hans2 Online No error
foo_rg hans2 Online No error
+-------+ cluster_status Help +--------+
| on s - Toggle Sound on event |
| on r - Toggle Resource Group View |
| on c - Toggle CXFS View |
| j - Scroll up the selection |
| k - Scroll down the selection |
| TAB - Toggle RG or CXFS selection |
| ENTER - View detail on selection |
| h - Toggle help screen |
| q - Quit cluster_status |
+--- Press 'h' to remove help window --+
|
The above shows that a sound will be activated when a node or the cluster changes status. You can override the s setting by invoking cluster_status with the -m (mute) option. You can also use the arrow keys to scroll the selection.
| Note: The cluster_status command can display no more than 128 CXFS filesystems. |
The easiest way to keep a continuous watch on the state of a cluster is to use the GUI view area.
System components that are experiencing problems appear as blinking red icons. Components in transitional states also appear as blinking icons. If there is a problem in a resource group or node, the icon for the cluster turns red and blinks, as well as the resource group or node icon.
The cluster status can be one of the following:
ACTIVE, which means the cluster is up and running and there is a valid FailSafe membership.
INACTIVE, which means the start FailSafe HA services task has not been run and there is no FailSafe membership.
ERROR, which means that some nodes are in a DOWN state; that is, the cluster should be running, but it is not.
UNKNOWN,which means that the state cannot be determined because FailSafe HA services are not running on the node performing the query.
If you minimize the GUI window, the minimized-icon shows the current state of the cluster. Green indicates FailSafe HA services active without an error, gray indicates FailSafe HA services are inactive, and red indicates an error state.
The following tables show keys to the icons and states used in the FailSafe Manager GUI.
The full legend for component states is as follows:
Icon | Entity | |
|---|---|---|
| IRIX Node | |
| Cluster | |
| Resource | |
| Resource group | |
| Resource type | |
| Failover policy | |
| Expanded tree | |
| Collapsed tree | |
| User name | |
| GUI task for which execution privilege may be granted or revoked | |
| Privileged command executed by a given GUI task |
Icon | State | |
|---|---|---|
| Inactive or unknown (HA services may not be active) | |
| Online-ready state for a resource group | |
| Healthy and active or online | |
| (blinking) Transitioning to healthy and active/online or transitioning to offline | |
| Maintenance mode, in which the resource is not monitored by FailSafe | |
| (blinking red) Problems with the component |
To query node and cluster status, use the following command:
show status of cluster Clustername |
You can use cmgr to query the status of a resource or to contact the system controller on a node, as described in the following subsections.
To query a resource status, use the following command:
show status of resource Resourcename of resource_type RTname [in cluster Clustername] |
If you have specified a default cluster, you do not need to specify a cluster when you use this command and it will show the status of the indicated resource in the default cluster.
To query the status of a resource group, you provide the name of the resource group and the cluster which includes the resource group. Resource group status includes the following components:
Resource group state
Resource group error state
Resource owner
These components are described in the following subsections.
If a node that contains a resource group online has a status of UNKNOWN, the status of the resource group will not be available or ONLINE-READY.
A resource group state can be one of the following:
When a resource group is ONLINE, its error status is continually being monitored. A resource group error status can be one of the following:
You can use the view area to monitor the status of the resources in a FailSafe configuration:
Select View: Resources in Groups to see the resources organized by the groups to which they belong.
Select View: Groups owned by Nodes to see where the online groups are running. This view lets you observe failovers as they occur.
To query a resource group status, use the following cmgr command:
show status of resource_group RGname [in cluster Clustername] |
If you have specified a default cluster, you do not need to specify a cluster when you use this command and it will show the status of the indicated resource group in the default cluster.
To query the status of a node, you provide the logical node name of the node. The node status can be one of the following:
When you start HA services, node states transition from INACTIVE to UP. It may happen that a node state may transition from INACTIVE to UNKNOWN to UP.
You can use the cluster_status command to monitor the status of the nodes in the cluster.
You can use the GUI view area to monitor the status of the clusters in a FailSafe configuration. Select View: Groups owned by Nodes to monitor the health of the default cluster, its resource groups, and the group's resources.
To query node status, use the following command:
show status of node nodename |
When FailSafe is running, you can determine whether the system controller on a node is responding with the following command:
admin ping node nodename |
This command uses the FailSafe daemons to test whether the system controller is responding.
You can verify reset connectivity on a node in a cluster even when the FailSafe daemons are not running by using the standalone option of the admin ping command:
admin ping standalone node nodename |
This command does not go through the FailSafe daemons, but calls the ping command directly to test whether the system controller on the indicated node is responding.
The haStatus script provides status and configuration information about clusters, nodes, resources, and resource groups in the configuration. This script is installed in the /var/cluster/cmgr-scripts directory. You can modify this script to suit your needs. See the haStatus(1M) man page for further information about this script.
The following examples show the output of the different options of the haStatus script.
# haStatus -help Usage: haStatus [-a|-i] [-c clustername] where, -a prints detailed cluster configuration information and cluster status. -i prints detailed cluster configuration information only. -c can be used to specify a cluster for which status is to be printed. “clustername” is the name of the cluster for which status is to be printed. # haStatus Tue Nov 30 14:12:09 PST 1999 Cluster test-cluster: Cluster state is ACTIVE. Node hans2: State of machine is UP. Node hans1: State of machine is UP. Resource_group nfs-group1: State: Online Error: No error Owner: hans1 Failover Policy: fp_h1_h2_ord_auto_auto Resources: /hafs1 (type: NFS) /hafs1/nfs/statmon (type: statd_unlimited) 150.166.41.95 (type: IP_address) /hafs1 (type: filesystem) havol1 (type: volume) # haStatus -i Tue Nov 30 14:13:52 PST 1999 Cluster test-cluster: Node hans2: Logical Machine Name: hans2 Hostname: hans2.dept.company.com Is FailSafe: true Is CXFS: false Nodeid: 32418 Reset type: powerCycle System Controller: msc System Controller status: enabled System Controller owner: hans1 System Controller owner device: /dev/ttyd2 System Controller owner type: tty ControlNet Ipaddr: 192.26.50.15 ControlNet HB: true ControlNet Control: true ControlNet Priority: 1 ControlNet Ipaddr: 150.166.41.61 ControlNet HB: true ControlNet Control: false ControlNet Priority: 2 Node hans1: Logical Machine Name: hans1 Hostname: hans1.dept.company.com Is FailSafe: true Is CXFS: false Nodeid: 32645 Reset type: powerCycle System Controller: msc System Controller status: enabled System Controller owner: hans2 System Controller owner device: /dev/ttyd2 System Controller owner type: tty ControlNet Ipaddr: 192.26.50.14 ControlNet HB: true ControlNet Control: true ControlNet Priority: 1 ControlNet Ipaddr: 150.166.41.60 ControlNet HB: true ControlNet Control: false ControlNet Priority: 2 Resource_group nfs-group1: Failover Policy: fp_h1_h2_ord_auto_auto Version: 1 Script: ordered Attributes: Auto_Failback Auto_Recovery Initial AFD: hans1 hans2 Resources: /hafs1 (type: NFS) /hafs1/nfs/statmon (type: statd_unlimited) 150.166.41.95 (type: IP_address) /hafs1 (type: filesystem) havol1 (type: volume) Resource /hafs1 (type NFS): export-info: rw,wsync filesystem: /hafs1 Resource dependencies statd_unlimited /hafs1/nfs/statmon filesystem /hafs1 Resource /hafs1/nfs/statmon (type statd_unlimited): InterfaceAddress: 150.166.41.95 Resource dependencies IP_address 150.166.41.95 filesystem /hafs1 Resource 150.166.41.95 (type IP_address): NetworkMask: 0xffffff00 interfaces: ef1 BroadcastAddress: 150.166.41.255 No resource dependencies Resource /hafs1 (type filesystem): volume-name: havol1 mount-options: rw,noauto monitor-level: 2 Resource dependencies volume havol1 Resource havol1 (type volume): devname-group: sys devname-owner: root devname-mode: 666 No resource dependencies Failover_policy fp_h1_h2_ord_auto_auto: Version: 1 Script: ordered Attributes: Auto_Failback Auto_Recovery Initial AFD: hans1 hans2 # haStatus -a Tue Nov 30 14:45:30 PST 1999 Cluster test-cluster: Cluster state is ACTIVE. Node hans2: State of machine is UP. Logical Machine Name: hans2 Hostname: hans2.dept.company.com Is FailSafe: true Is CXFS: false Nodeid: 32418 Reset type: powerCycle System Controller: msc System Controller status: enabled System Controller owner: hans1 System Controller owner device: /dev/ttyd2 System Controller owner type: tty ControlNet Ipaddr: 192.26.50.15 ControlNet HB: true ControlNet Control: true ControlNet Priority: 1 ControlNet Ipaddr: 150.166.41.61 ControlNet HB: true ControlNet Control: false ControlNet Priority: 2 Node hans1: State of machine is UP. Logical Machine Name: hans1 Hostname: hans1.dept.company.com Is FailSafe: true Is CXFS: false Nodeid: 32645 Reset type: powerCycle System Controller: msc System Controller status: enabled System Controller owner: hans2 System Controller owner device: /dev/ttyd2 System Controller owner type: tty ControlNet Ipaddr: 192.26.50.14 ControlNet HB: true ControlNet Control: true ControlNet Priority: 1 ControlNet Ipaddr: 150.166.41.60 ControlNet HB: true ControlNet Control: false ControlNet Priority: 2 Resource_group nfs-group1: State: Online Error: No error Owner: hans1 Failover Policy: fp_h1_h2_ord_auto_auto Version: 1 Script: ordered Attributes: Auto_Failback Auto_Recovery Initial AFD: hans1 hans2 Resources: /hafs1 (type: NFS) /hafs1/nfs/statmon (type: statd_unlimited) 150.166.41.95 (type: IP_address) /hafs1 (type: filesystem) havol1 (type: volume) Resource /hafs1 (type NFS): State: Online Error: None Owner: hans1 Flags: Resource is monitored locally export-info: rw,wsync filesystem: /hafs1 Resource dependencies statd_unlimited /hafs1/nfs/statmon filesystem /hafs1 Resource /hafs1/nfs/statmon (type statd_unlimited): State: Online Error: None Owner: hans1 Flags: Resource is monitored locally InterfaceAddress: 150.166.41.95 Resource dependencies IP_address 150.166.41.95 filesystem /hafs1 Resource 150.166.41.95 (type IP_address): State: Online Error: None Owner: hans1 Flags: Resource is monitored locally NetworkMask: 0xffffff00 interfaces: ef1 BroadcastAddress: 150.166.41.255 No resource dependencies Resource /hafs1 (type filesystem): State: Online Error: None Owner: hans1 Flags: Resource is monitored locally volume-name: havol1 mount-options: rw,noauto monitor-level: 2 Resource dependencies volume havol1 Resource havol1 (type volume): State: Online Error: None Owner: hans1 Flags: Resource is monitored locally devname-group: sys devname-owner: root devname-mode: 666 No resource dependencies # haStatus -c test-cluster Tue Nov 30 14:42:04 PST 1999 Cluster test-cluster: Cluster state is ACTIVE. Node hans2: State of machine is UP. Node hans1: State of machine is UP. Resource_group nfs-group1: State: Online Error: No error Owner: hans1 Failover Policy: fp_h1_h2_ord_auto_auto Resources: /hafs1 (type: NFS) /hafs1/nfs/statmon (type: statd_unlimited) 150.166.41.95 (type: IP_address) /hafs1 (type: filesystem) havol1 (type: volume) |
The Embedded Support Partner (ESP) consists of a set of daemons that perform various monitoring activities. You can choose to configure ESP so that it will log FailSafe events (the FailSafe ESP event profile is not configured in ESP by default).
FailSafe uses an event class ID of 77 and a description of IRIS FailSafe2.
If you want to use ESP for FailSafe, enter the following command to add the failsafe2 event profile to ESP:
# espconfig -add eventprofile failsafe2 |
FailSafe will then log ESP events for the following:
Daemon configuration error
Failover policy configuration error
Resource group allocation (start) failure
Resource group failures:
Allocation (start) failure
Release (stop) failure
Monitoring failure
Exclusivity failure
Failover policy failure
Resource group status:
online
offline
maintenance_on
maintenance_off
FailSafe shutdown (HA services stopped)
FailSafe started (HA services started)
You can use the espreport or launchESPartner commands to see the logged ESP events. See the esp(5) man page and the Embedded Support Partner User Guide for more information about ESP.
While a FailSafe system is running, you can move a resource group online to a particular node, or you can take a resource group offline. In addition, you can move a resource group from one node in a cluster to another node in a cluster. The following subsections describe these tasks.
This section describes how to bring a resource group online.
Before you bring a resource group online for the first time, you should run the diagnostic tests on that resource group. Diagnostics check system configurations and perform some validations that are not performed when you bring a resource group online.
You cannot bring a resource group online in the following circumstances:
If the resource group has no members
If the resource group is currently running in the cluster
To bring a resource group fully online, HA services must be active. When HA services are active, an attempt is made to allocate the resource group in the cluster. However, you can also execute a command to bring the resource group online when HA services are not active. When HA services are not active, the resource group is marked to be brought online when HA services become active; the resource group is then in an ONLINE-READY state. Failsafe tries to bring a resource group in an ONLINE-READY state online when HA services are started.
You can disable resource groups from coming online when HA services are started by using the GUI or cmgr to take the resource group offline, as described in “Take a Resource Group Offline”.
| Caution: Before bringing a resource group online in the cluster, you must be sure that the resource group is not running on a disabled node (where HA services are not running). Bringing a resource group online while it is running on a disabled node could cause data corruption. For information on detached resource groups, see “Take a Resource Group Offline”. |
Do the following:
Group to Bring Online: select the name of the resource group you want to bring online. The menu displays only resource groups that are not currently online.
Click on OK to complete the task.
To bring a resource group online, use the following command:
admin online resource_group RGname [in cluster Clustername] |
If you have specified a default cluster, you do not need to specify a cluster when you use this command.
cmgr> set cluster test-cluster
cmgr> admin online resource_group group1
FailSafe daemon (ha_fsd) is not running on this local node or it is not ready to accept admin commands.
Resource Group (group1) is online-ready.
Failed to admin:
online
admin command failed
cmgr> show status of resource_group group1 in cluster test-cluster
State: Online Ready
Error: No error
Check resource group group1 status in an active node if HA services are active in cluster |
This section tells you how to take a resource group offline.
When you take a resource group offline, FailSafe takes each resource in the resource group offline in a predefined order. If any single resource gives an error during this process, the process stops, leaving all remaining resources allocated.
You can take a FailSafe resource group offline in any of the following ways:
Take the resource group offline. This physically stops the processes for that resource group and does not reset any error conditions. If this operation fails, the resource group will be left online in an error state.
Force the resource group offline. This physically stops the processes for that resource group but resets any error conditions. This operation cannot fail.
Detach the resource group. This causes FailSafe to stop monitoring the resource group, but does not physically stop the processes on that group. FailSafe will report the status as offline and will not have any control over the group. This operation should rarely fail.
Detach the resource group and force the error state to be cleared. This causes FailSafe to stop monitoring the resource group, but does not physically stop the processes on that group. FailSafe will report the status as offline and will not have any control over the group. In addition, all error conditions of the resource group will be reset. This operation should rarely fail.
If you do not need to stop the resource group and do not want FailSafe to monitor the resource group while you make changes, but you would still like to have administrative control over the resource group (for instance, to move that resource group to another node), you can put the resource group in maintenance mode using the Suspend Monitoring a Resource Group task on the GUI or the admin maintenance_on command of cmgr, as described in “Suspend and Resume Monitoring of a Resource Group”.
If the fsd daemon is not running or is not ready to accept client requests, executing this task disables the resource group in the cluster database only. The resource group remains online and the command fails.
Enter the following:
Detach Only: check this box to stop monitoring the resource group. The resource group will not be stopped, but FailSafe will not have any control over the group.
Detach Force: check this box to stop monitoring the resource group. The resource group will not be stopped, but FailSafe will not have any control over the group. In addition, Failsafe will clear all errors.
Caution: The Detach Only and Detach Force settings leave the resource group's resources running on the node where the group was online. After stopping HA services on that node, do not bring the resource group online on another node in the cluster; doing so can cause data integrity problems. Instead, make sure that no resources are running on a node before stopping HA services on that node. Force Offline: check this box to stop all resources in the group and clear all errors.
Group to Take Offline: select the name of the resource group you want to take offline. The menu displays only resource groups that are currently online.
Click on OK to complete the task.
To take a resource group offline, use the following command:
admin offline resource_group RGname [in cluster Clustername] |
To take a resource group offline with the force option in effect, forcing FailSafe to complete the action even if there are errors, use the following command:
admin offline_force resource_group RGname [in cluster Clustername] |
| Note: Doing an offline_force operation on a resource group can leave resources in the resource group running on the cluster. The offline_force operation will succeed even though all resources in the resource group have not been stopped. FailSafe does not track these resources any longer. You should take care to prevent resources from running on multiple nodes in the cluster. |
To detach a resource group, use the following command:
admin offline_detach resource_group RGname [in cluster Clustername] |
To detach the resource group and force the error state to be cleared:
admin offline_detach_force resource_group RGname [in cluster Clustername] |
This causes FailSafe to stop monitoring the resource group, but does not physically stop the processes on that group. FailSafe will report the status as offline and will not have any control over the group. In addition, all error conditions of the resource group will be reset. This operation should rarely fail.
This section tells you how to move a resource group.
While FailSafe is active, you can move a resource group to another node in the same cluster.
| Note: When you move a resource group in an active system, you may find the unexpected behavior that the command appears to have succeeded, but the resource group remains online on the same node in the cluster. This can occur if the resource group fails to start on the node to which you are moving it. In this case, FailSafe will fail over the resource group to the next node in the application failover domain, which may be the node on which the resource group was originally running. Since FailSafe kept the resource group online, the command succeeds. |
Enter the following:
Group to Move: select the name of the resource group to be moved. Only resource groups that are currently online are displayed in the menu.
Failover Domain Node: (optional) select the name of the node to which you want to move the resource group. If you do not specify a node, FailSafe will move the resource group to the next available node in the failover domain.
Click on OK to complete the task.
To move a resource group to another node, use the following command:
admin move resource_group RGname [in cluster Clustername] [to node Nodename] |
For example, to move resource group nfs-group1 running on node primary to node backup in the cluster nfs-cluster, do the following:
cmgr> admin move resource_group nfs-group1 in cluster nfs-cluster to node backup |
If the user does not specify the node, the resource group's failover policy is used to determine the destination node for the resource group.
If you run into errors after entering the admin move command, see “Ensuring that Resource Groups are Deallocated” in Chapter 9.
This section describes how to stop monitoring of a resource group in order to put it into maintenance mode.
You can temporarily stop FailSafe from monitoring a specific resource group, which puts the resource group in maintenance mode. The resource group remains on the same node in the cluster but is no longer monitored by FailSafe for resource failures.
You can put a resource group into maintenance mode if you do not want FailSafe to monitor the group for a period of time. You may want to do this for upgrade or testing purposes, or if there is any reason that FailSafe should not act on that resource group. When a resource group is in maintenance mode, it is not being monitored and it is not highly available. If the resource group's owner node fails, FailSafe will move the resource group to another node and resume monitoring.
When you put a resource group into maintenance mode, resources in the resource group are in ONLINE-MAINTENANCE state. The ONLINE-MAINTENANCE state for the resource is seen only on the node that has the resource online. All other nodes will show the resource as ONLINE. The resource group, however, should appear as being in ONLINE-MAINTENANCE state in all nodes.
Do the following:
Group to Stop Monitoring: select the name of the group you want to stop monitoring. Only those resource groups that are currently online and monitored are displayed in the menu.
Click OK to complete the task.
This task lets you resume monitoring a resource group.
Once monitoring is resumed and assuming that the restart action is enabled, if the resource group or one of its resources fails, FailSafe will restart each failed component based on the failover policy.
Perform the following steps:
Group to Start Monitoring: select the name of the group you want to start monitoring. Only those resource groups that are currently online and not monitored are displayed in the menu.
Click OK to complete the task.
To put a resource group into maintenance mode, use the following command:
admin maintenance_on resource_group RGname [in cluster Clustername] |
If you have specified a default cluster, you do not need to specify a cluster when you use this command.
You can stop the execution of FailSafe on all the nodes in a cluster or on a specified node only. See “Stop FailSafe HA Services” in Chapter 5.
You can use FailSafe to reset nodes in a cluster. This sends a reset command to the system controller port on the specified node. When the node is reset, other nodes in the cluster will detect this and remove the node from the active cluster, reallocating any resource groups that were allocated on that node onto a backup node. The backup node that is used depends on how you have configured your system.
After the node reboots, it will rejoin the cluster. Some resource groups might move back to the node, depending on how you have configured your system.
You can use the GUI to reset nodes in a cluster. This sends a reset command to the system controller port on the specified node. When the node is reset, other nodes in the cluster will detect the change and remove the node from the active cluster. When the node reboots, it will rejoin the FailSafe membership.
To reset a node, do the following:
Node to Reset: select the node to be reset.
Click on OK to complete the task.
When FailSafe is running, you can reboot a node with the following command:
admin reset node nodename |
This command uses the FailSafe daemons to reset the specified node.
You can reset a node in a cluster even when the FailSafe daemons are not running by using the standalone option of the admin reset command:
admin reset standalone node nodename |
This command does not go through the FailSafe daemons.
You should perform a database backup whenever you want to save the database and be able to restore it to the current state at a later point.
You can use the following methods to restore the database:
If the database is accidentally deleted from a node, use the fs2d daemon to replicate the database from another node in the pool.
If you want to be able to recreate the current configuration, use the build_cmgr_script script. You can then recreate this configuration by running the script generated.
If you want to retain a copy of the database and all node-specific information such as local logging, use the cdbBackup and cdbRestore commands.
If the database has been accidentally deleted from an individual node, you can replace it with a copy from another node. Do not use this method if the cluster database has been corrupted.
Do the following:
Stop the cluster database daemons by running the following command on each node:
# /etc/init.d/cluster stop
Run cdbreinit on nodes that are missing the cluster database.
Restart the cluster database daemons by running the following command on each node:
# /etc/init.d/cluster start
The fs2d daemon will then replicate the cluster database to those nodes from which it is missing
You can use the build_cmgr_script command from one node in the cluster to create a cmgr script that will recreate the node, cluster, switch, and filesystem definitions for all nodes in the cluster database. You can then later run the resulting script to recreate a database with the same contents; this method can be used for missing or corrupted cluster databases.
| Note: The build_cmgr_script script does not contain local logging information, so it cannot be used as a complete backup/restore tool. |
To perform a database backup, use the build_cmgr_script script from one node in the cluster, as described in “Creating a cmgr Script Automatically” in Chapter 4.
| Caution: Do not make configuration changes while you are using the build_cmgr_script command. |
By default, this creates a cmgr script in the following location:
/tmp/cmgr_create_cluster_clustername_processID |
You can specify another filename by using the -o option.
To perform a restore on all nodes in the pool, do the following:
Stop HA services for all nodes in the cluster.
Stop the cluster database daemons on each node.
Remove all copies of the old database by using the cdbreinit command on each node.
Execute the cmgr script (which was generated by the build_cmgr_script script) on the node that is defined first in the script. This will recreate the backed-up database on each node.
Note: If you want to run the generated script on a different node, you must modify the generated script so that the node is the first one listed in the script.
Restart cluster database daemons on each node.
For example, to backup the current database, clear the database, and restore the database to all nodes, do the following:
On one node in the cluster: # /var/cluster/cmgr-scripts/build_cmgr_script -o /tmp/newcdb Building cmgr script for cluster clusterA ... build_cmgr_script: Generated cmgr script is /tmp/newcdb On one node: # stop ha_services for cluster clusterA On each node: # /etc/init.d/cluster stop On each node: # /usr/cluster/bin/cdbreinitOn each node: # /etc/init.d/cluster start On the *first* node listed in the /tmp/newcdb script: # /tmp/newcdb |
The cdbBackup and cdbRestore commands backup and restore the cluster database and node-specific information, such as local logging information. You must run these commands individually for each node.
To perform a backup of the cluster, use the cdbBackup command on each node.
| Caution: Do not make configuration changes while you are using the cdbBackup command. |
To perform a restore, run the cdbRestore command on each node. You can use this method for either a missing or corrupted cluster database. Do the following:
Stop HA services.
Stop cluster services on each node.
Remove the old database by using the cdbreinit command on each node.
Stop cluster services again (these were restarted automatically by cdbreinit in the previous step) on each node.
Use the cdbRestore command on each node.
Start cluster services on each node.
For example, to backup the current database, clear the database, and then restore the database to all nodes, do the following:
On each node: # /usr/cluster/bin/cdbBackup On one node in the cluster: # stop ha_services for cluster clusterA On each node: # /etc/init.d/cluster stop On each node: # /usr/cluster/bin/cdbreinit On each node (again): # /etc/init.d/cluster stop On each node: # /usr/cluster/bin/cdbRestore On each node: # /etc/init.d/cluster start |
For more information, see the cdbBackup(1M) and cdbRestore(1M) man page.
| Note: Do not perform a cdbDump while information is changing in the cluster database. Check the SYSLOG file for information to help determine when cluster database activity is occurring. As a rule of thumb, you should be able to perform a cdbDump if at least 15 minutes have passed since the last node joined the cluster or the last administration command was run. |
You should rotate the log files at least weekly so that your disk will not become full.
For information about log levels, see “Set Log Configuration” in Chapter 5.
You can run the /var/cluster/cmgr-scripts/rotatelogs script to copy all files to a new location. This script saves log files with the day and the month name as a suffix. If you run the script twice in one day, it will append the current log file to the previous saved copy. The root crontab file has an entry to run this script weekly.
The script syntax is as follows:
/var/cluster/cmgr-scripts/rotatelogs [-h] [-d|-u] |
If no option is specified, the log files will be rotated. Options are as follows:
| -h | Prints the help message. The log files are not rotated and other options are ignored. |
| -d | Deletes saved log files that are older than one week before rotating the current log files. You cannot specify this option and -u. |
| -u | Unconditionally deletes all saved log files before rotating the current log files. You cannot specify this option and -d. |
By default, the rotatelogs script will be run by crontab once a week, which is sufficient if you use the default log levels. If you plan to run with a high debug level for several weeks, you should reset the crontab entry so that the rotatelogs script is run more often.
On heavily loaded machines, or for very large log files, you may want to move resource groups and stop HA services before running rotatelogs.
You can use a script such as the following to copy large files to a new location. The files in the new location will be overwritten each time this script is run.
#!/bin/sh
# Argument is maximum size of a log file (in characters) - default: 500000
size=${1:-500000}
find /var/cluster/ha/log -type f ! -name '*.OLD' -size +${size}c -print | while read log_file; do
cp ${log_file} ${log_file}.OLD
echo '*** LOG FILE ROTATION ' `date` '***' > ${log_file}
done |
The GUI lets you grant or revoke access to a specific GUI task for one or more specific users. By default, only root may execute tasks in the GUI. You cannot grant or revoke tasks for users with a user ID of 0.
Access to the task is allowed only on the node to which the GUI is connected; if you want to allow access on another node in the pool, you must connect the GUI to that node and grant access again.
GUI tasks and the cmgr(1M) command operate by executing underlying privileged commands which are normally accessible only to root. When granting access to a task, you are in effect granting access to all of its required underlying commands, which results in also granting access to the other GUI tasks that use the same underlying commands.
To see which tasks a specific user can currently access, select View: Users. Select a specific user to see details about the tasks available to that user.
To see which users can currently access a specific task, select View: Task Privileges. Select a specific task to see details about the users who can access it and the privileged commands it requires.
You can grant access to a specific task to one or more users at a time.
| Note: Access to the task is only allowed on the node to which the GUI is connected; if you want to allow access on another node in the pool, you must connect the GUI to that node and grant access again. |
Do the following:
Select the user or users for whom you want to grant access. You can use the following methods to select users:
Click to select one user at a time
Shift+click to select a block of users
Ctrl+click to toggle the selection of any one user, which allows you to select multiple users that are not contiguous
Click Select All to select all users
Click Next to move to the next page.
Select the task or tasks to grant access to, using the above selection methods. Click Next to move to the next page.
Confirm your choices by clicking OK.
Note: If more tasks than you selected are shown, then the selected tasks run the same underlying privileged commands as other tasks, such that access to the tasks you specified cannot be granted without also granting access to these additional tasks.
To see which tasks a specific user can access, select View: Users. Select a specific user to see details about the tasks available to that user.
To see which users can access a specific task, select View: Task Privileges. Select a specific task to see details about the users who can access it and the privileged commands it requires.
Suppose you wanted to grant user guest permission to define clusterwide and node-specific resources. You would do the following:
Select guest and click Next to move to the next page.
Select the tasks you want sys to be able to execute:
Click Define a Resource
Ctrl+click Redefine a Resource for a Specific Node
Ctrl+click Add or Remove Resources in Resource Group
Click Next to move to the next page.
Confirm your choices by clicking OK.
Figure 7-1 shows the privileged commands that were granted to user guest.
| Note: Modify a Resource Definition is also displayed, even though the administrator did not explicitly select it; the privilege commands for the tasks selected also require this command. |
Figure 7-2 shows the screen that is displayed when you select View: Users and click guest to display information in the details area of the GUI window. The privileged commands listed are the underlying commands executed by the GUI tasks.
Suppose you wanted to give user sys access to all tasks except adding or removing nodes from a cluster. The easiest way to do this is to select all of the tasks and then deselect the one you want to restrict. You would do the following:
Select sys and click Next to move to the next page.
Select the tasks you want sys to be able to execute:
Click Select All to highlight all tasks.
Deselect the task to which you want to restrict access. Ctrl+click Add/Remove Nodes in Cluster.
Click Next to move to the next page.
Confirm your choices by clicking OK.
You can revoke task access from one or more users at a time.
| Note: Access to the task is only revoked on the node to which the GUI is connected; if a user has access to the task on multiple nodes in the pool, you must connect the GUI to those other nodes and revoke access again. |
Do the following:
Select the user or users from whom you want to revoke task access. You can use the following methods to select users:
Click to select one user at a time
Shift+click to select a block of users
Ctrl+click to toggle the selection of any one user, which allows you to select multiple users that are not contiguous
Click Select All to select all users
Click Next to move to the next page.
Select the task or tasks to revoke access to, using the above selection methods. Click Next to move to the next page.
Confirm your choices by clicking OK.
Note: If more tasks than you selected are shown, then the selected tasks run the same underlying privileged commands as other tasks, such that access to the tasks you specified cannot be revoked without also revoking access to these additional tasks.
To see which tasks a specific user can access, select View: Users. Select a specific user to see details about the tasks available to that user.
To see which users can access a specific task, select View: Task Privileges. Select a specific task to see details about the users who can access it.