Currently, the compute nodes run the SUSE Linux Enterprise Server 10 (SLES10) Service Pack 1 (SP1) Linux distribution. To improve the performance of applications running MPI jobs on compute nodes, the following SLES10 services are turned off:

You can add per-host compute node customization to the compute node images. You do this by adding scripts either to the /opt/sgi/share/per-host-customization/global/ directory or the /opt/sgi/share/per-host-customization/ mynewimage/ directory on the system admin controller.

---

Note: When creating custom images for compute blades, make sure you clone the original SGI images. This provides the original images intact that you can fall back to if necessary.

---

Scripts in the global directory apply to all compute nodes images. Scripts under the image name apply only to the image in question. The scripts are cycled through once per host when being installed on the rack leaders. They receive one input argument, which is the full path (on the rack leader controller) to the per-host base directory, for example /var/lib/sgi/mynewimage/i2n11. There is a README file at /opt/sgi/share/per-host-customization/README on the system admin controller, as follows:

This directory contains compute node image customization scripts which are
executed as part of the install-image operations on the leader nodes when
pulling over a new compute node image.

After the image has been pulled over, and the per-host-customization dir has
been rsynced, the per-host /etc and /var directories are populated, then the
scripts in this directory are cycled through once per-host.  This allows the
scripts to source the node specific network and cluster management settings,
and set node specific settings.

Scripts in the global directory are iterated through first, then if a directory
exists that matches the image name, those scripts are iterated through next.

You can use the scripts in the global directory as examples.

An example global script, /opt/sgi/share/per-host-customization/global/sgi-hostname is, as follows:

#!/bin/sh
#
# Set the compute node's hostname to the cluster unique name
#
# This script is excecuted once per-host as part of the install-image operation
# run on the leader nodes.  The full path to the per-host iru+slot directory is
# passed in as $1, e.g. /var/lib/sgi/per-host//i2n11.
#

iruslot=$1

# source cluster management information
. ${iruslot}/etc/opt/sgi/cminfo

# set hostname of blade to cluster unique name
echo ${NAME} > ${iruslot}/etc/HOSTNAME

To customize the compute node operating system image, perform the following steps:

This section describes how to make a copy of a compute node image.

To create a simple compute node image clone from the system admin controller, perform the following steps:

The cimage command allows you to list, modify, and set software images on the compute nodes in your system.

The cimage command accepts the following options:

RACK arguments take the format rX.

NODE arguments take the format rXiYnZ .

X, Y, Z can be single digits, a [start-end] range, or * for all matches.

... indicates more than one RACK or NODE argument can be passed in.

EXAMPLES

# cimage --list-images

image: compute-sles10sp1
        kernel: 2.6.16.46-0.12-carlsbad
        kernel: 2.6.16.46-0.12-smp

# cimage --list-nodes r1
r1i0n0: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n1: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n2: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n3: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n4: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n5: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n6: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n7: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n8: compute-sles10sp1 2.6.16.46-0.7-smp
[...snip...]

# cimage --set compute-sles10sp1 2.6.16.46-0.12-carlsbad r1i0n0

# cimage --list-nodes r1
r1i0n0: compute-sles10sp1 2.6.16.46-0.7-carlsbad
r1i0n1: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n2: compute-sles10sp1 2.6.16.46-0.7-smp
[...snip...]

# cimage --set compute-sles10sp1 2.6.16.46-0.7-carlsbad r*i*n*

# cimage --set compute-sles10sp1 2.6.16.46-0.7-smp r1i[0-2]n[5-6] r1i[2-3]n[0-4] 

# cimage --clone-image compute-sles10sp1 mynewimage
Cloning compute-sles10sp1 to mynewimage ... done

# cp .rpm /var/lib/systemimager/images//tmp
# chroot /var/lib/systemimager/images// bash
# rpm -Uvh /tmp/.rpm

# cimage --del-db mynewimage
# cimage --add-db mynewimage

# cimage --add-rack mynewimage r*
r1lead: install-image: mynewimage
r1lead: install-image: mynewimage done.

# cimage --list-images

image: compute-sles10sp1
        kernel: 2.6.16.46-0.7-carlsbad
        kernel: 2.6.16.46-0.7-smp

image: mynewimage
        kernel: 2.6.16.46-0.7-carlsbad
        kernel: 2.6.16.46-0.7-smp

# cimage --set mynewimage 2.6.16.46-0.7-smp r1i3n*

# cimage --del-image mynewimage
r1lead: delete-image: mynewimage
r1lead: delete-image: mynewimage done.

The cpower command is, as follows:

cpower [<option> ...] [<target_type>] [<action>] <target>

The <option> argument can be one or more of the following:

---

Note: The command will fail if the target contains any wild cards, unless the --all option is specified.

---

The <target> argument is one of the following:

The <action> argument is one of the following:

The target must always be specified except when the --system option is used. Wildcards may be used, but be careful not to accidentally power off or reboot the leader nodes. If wildcard use affects any leader node, the command fails with an error.

IPMI-style Commands

The default operation for the cpower command is to operate on nodes and to provide you the status of these nodes, as follows:

# cpower r1i*

The cpower command also

This example gives you the power status and boot status of all the compute blades in rack 1. This command is equivalent to cpower --node --status r1i*.

This command issues an ipmitool power off command to all of the nodes specified by the wildcard, as follows:

# cpower --off r2i*

The default is to apply to a node.

The following commands behave exactly as you would expect as if you were using ipmitool, and have no special extra logic for ordering:

• # cpower --up r1i*

• # cpower --reset r1i*

• # cpower --cycle r1i*

• # cpower --identify 5 r1i*

---

Note: --up is a synonym for --on and --down is a synonym for --off.

---

The cpower command contains more logic when you go up to higher levels of abstraction, for example, using --iru , --rack, and --system. These higher level domain specifiers tell the command to be smart about how to order various of the actions that you give on the command line.

The --iru option tells the command to use correct ordering with IRU power commands. In this case, it firsts connect to the CMC on each IRU in rack 1 to issue the power on command, which turns on power to the IRU chassis (this is not the equivalent ipmitool command). Then it powers up the compute nodes in the IRU. Powering things down is the opposite, with the power to the IRU being turned off after power to the blades. IRU targets are specified as follows: r3i2 for rack 3, IRU 2.

# cpower --iru --up r1* 

The --rack option ensures power commands to the leader node are down in the correct order relative to compute nodes within a rack. First, it powers up the leader node and waits for it to boot up (if it is not already up). Then it will do the functional equivalent of a cpower --iru --up r4i* on each of the IRUs contained in the rack, including applying power to each IRU chassis. Using the --down option is the opposite, and also turns off the leader node (after doing a shutdown) after all the IRUs are powered down. To avoid including leader nodes in a power command for a rack, use the --noleader option. Rack targets are specified, as follows: r4 for rack 4. Here is an example:

# cpower --rack --up r4

Commands with the --system option ensures that power up commands are applied first to service nodes, then to leader nodes, then to IRUs and compute blades, in just the same way. Likewise, compute blades are powered down before IRUs, leader nodes, and service nodes, in that order. To avoid including service nodes in a system-domain command, use the --noservice option. Note that you must not specify a target with --system option, since it applies to the Altix ICE system.

It useful to be able to shutdown a machine before turning off the power, in most cases. The following cpower options to enable you to do this: --shutdown, --boot , and --reboot. The --shutdown option is self-explanatory, but --reboot will ensure that a system is always rebooted, whereas --boot will only boot up a system if it is not already booted. Thus, --boot is useful for booting up compute blades that have failed to start.

---

Note: The IPMI power commands necessary to enable a system to boot (either with a power reset, or a power on) may be sent to a node, but a node that has been shutdown with the --shutdown option does not have its power automatically turned off.

---

The --shutdown option works on node, IRU, or rack domain levels. It will shut down nodes (in the correct order if you use the --iru or --rack options), and then just leave them as they are, power still applied. Usually you may only specify one action per command, however, with the --shutdown option, you may also specify --off. Using both these actions results in nodes being shutdown, then powered off. This is particularly useful when powering off a rack, since otherwise, the leaders may be shutdown before there is a chance to power off the compute blades. Here is an example:

# cpower --shutdown --rack r1

To boot up systems that have not already been booted, perform the following:

# cpower --boot  r1i2n*

Again, the command boots up nodes in the right orders if you specify the --iru or --rack options and the appropriate target. Otherwise, there is no guarantee that, for example, the command will attempt to power on the leader node before compute nodes in the same rack.

To reboot all of the nodes specified, or boot them if they are already shut down, perform the following:

# cpower --reboot --iru r3i3

The --iru or --rack options ensure proper ordering if you use them. In this case, the command will make sure that power is supplied to the chassis for rack 3, IRU 3, and then the all the compute nodes in that IRU will be rebooted.

EXAMPLES

Example 3-2. cpower Command Examples

To boot compute blade r1i0n8, perform the following:

# cpower --boot r1i0n8

To boot a number of compute blades at the same time, perform the following:

# cpower --boot --rack r1

---

Note: The --boot option will only boot those nodes that have not already booted.

---

To shut down service node 0, perform the following:

# cpower --shutdown service0

To shutdown and switch off everything in rack 3, perform the following:

# cpower --shutdown --off --rack r3

---

Note: Using the --shutdown and the --off options together is the only time you can use more than one command on the cpower command line. This combination will shutdown then power off all of the computer nodes in parallel, then shutdown and power off the leader node. Use the --noleader option if you want the leader node to remain booted up.

---

To shutdown the entire system, including all service nodes and all leader nodes, but not the admin node, and not turn the power off to anything, perform the following:

# cpower --shutdown --system

To shutdown all the compute nodes, but not the service nodes, leader nodes, perform the following:

# cpower --shutdown --system --noleader --noservice

---

Note: The only way to shut down the system admin controller (admin node) is to perform the operation manually.

---

The NTP client on the system admin controller should point to the house network time server. The NTP server provides NTP service to system components so that nodes can consult it when they are booted. The system admin controller sends NTP broadcasts to some networks to keep the nodes in sync after they have booted.

NTP client on the rack leader controller gets time from the system admin controller when it is booted and then stays in sync by watching NTP broadcasts from the system admin controller. The NTP server node provides NTP service to Altix ICE components so that compute nodes can sync their time when they are booted. The rack leader controller sends NTP broadcasts to some networks to keep the compute nodes in sync after they have booted.

The NTP client on managed service nodes ( for a definition of managed, see “discover Command” in Chapter 2) sets its time at initial booting from the system admin controller. It listens to NTP broadcasts from the system admin controller to stay in sync. It does not provide any NTP service.

The NTP Client on the compute node sets its time at initial booting from the rack leader controller. It listens to NTP broadcasts from the rack leader controller to stay in sync.

Sometime, especially during initial deployment of an Altix ICE system when system components are being installed and configured for the first time, NTP is not available to serve time to system components.

A non-modified NTP server, running for the first time, takes quite some time before it offers service. This means the leader and service nodes may fail to get time from the system admin controller as they come on-line. Compute nodes may also fail to get time from the leader when they first come up. This situation usually only happens at first deployment. After the ntp servers have a chance to create their drift files, ntp servers offer time with far less delay on subsequent reboots.

The following work arounds are in place for situations when NTP can not serve the time: