Currently, the compute nodes run the SUSE Linux Enterprise Server 10 (SLES10) Service Pack 1 (SP1) Linux distribution. To improve the performance of applications running MPI jobs on compute nodes, the following SLES10 services are turned off:

This section discusses how to manage various nodes on your SGI Altix ICE system. It describes how to configure the various nodes, including the compute and service nodes. It describe how to augment software packages. Many tasks having to do with package management have multiple valid methods to use.

For information on installing patches and updates, see “Installing SGI Tempo Patches and Updating SGI Altix ICE Systems ” in Chapter 2.

Compute Node Per-Host Customizations

You can add per-host compute node customization to the compute node images. You do this by adding scripts either to the /opt/sgi/share/per-host-customization/global/ directory or the /opt/sgi/share/per-host-customization/mynewimage/ directory on the system admin controller.

---

Note: When creating custom images for compute nodes, make sure you clone the original SGI images. This provides the original images intact that you can fall back to if necessary.

---

Scripts in the global directory apply to all compute nodes images. Scripts under the image name apply only to the image in question. The scripts are cycled through once per host when being installed on the rack leader controllers. They receive one input argument, which is the full path (on the rack leader controller) to the per-host base directory, for example, /var/lib/sgi/mynewimage/i2n11. There is a README file at /opt/sgi/share/per-host-customization/README on the system admin controller, as follows:

This directory contains compute node image customization scripts which are executed as part of the install-image operations on the leader nodes when pulling over a new compute node image. After the image has been pulled over, and the per-host-customization dir has been rsynced, the per-host /etc and /var directories are populated, then the scripts in this directory are cycled through once per-host. This allows the scripts to source the node specific network and cluster management settings, and set node specific settings. Scripts in the global directory are iterated through first, then if a directory exists that matches the image name, those scripts are iterated through next. You can use the scripts in the global directory as examples.

An example global script, /opt/sgi/share/per-host-customization/global/sgi-hostname is, as follows:

#!/bin/sh # # Copyright (c) 2007 Silicon Graphics, Inc. # All rights reserved. # # Set the compute node's hostname to the cluster unique name # # This script is excecuted once per-host as part of the install-image operation # run on the leader nodes, which is called from cimage on the admin node. # The full path to the per-host iru+slot directory is passed in as $1, # e.g. /var/lib/sgi/per-host//i2n11. # # sanity checks . /opt/sgi/share/per-host-customization/global/sanity.sh iruslot=$1 # source cluster management information . ${iruslot}/etc/opt/sgi/cminfo # set hostname of blade to cluster unique name echo ${NAME} > ${iruslot}/etc/HOSTNAME

Customizing Software Images

---

Note: Procedures in this section describe how to work with service node and compute node images. Always use a cloned image. If you are adjusting an RPM list, use your own copy of the RPM list.

---

The service and compute node images are created during the configure-cluster operation (or during your upgrade to SGI ProPack 5 SP3 if you were running SGI ProPack 5 SP2 previously). This process uses an RPM list to generate a root on the fly.

You can clone a compute node image, or create a new one based on an RPM list. For service nodes, SGI does not support a clone operation. For compute images, you can either clone the image and work on a copy or you can always make a new compute node image from the SGI supplied default RPM list.

Procedure 3-1. Clone a Compute Node Image

To clone a compute node image, perform the following steps:

1. From the system admin controller, create a clone of the compute node image, as follows:

   # cimage --clone-image compute-sles10sp2 new

   
   

   After that command is complete, you will have a new image located in /var/lib/systemimager/images/new on the system admin controller.

2. To see that the image is now available, perform the following command:

   # cimage --list-images image: compute-sles10sp1 kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp image: new kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp

   
   

For RPM lists, the default RPM lists are located in /etc/opt/sgi/rpmlists on the system admin controller. SGI suggests you never change these files, but rather, create your own versions using the ones supplied by SGI as a base.

Please note, it is important that certain packages be in the rpmlist for a given node. For example, an rpmlist used for compute nodes should have packages sgi-compute-node and sgi-cluster. Service nodes must have sgi-service-node and sgi-cluster.

Procedure 3-2. Manually adding a Package to a Compute Node Image

To manually add a package to a compute node image, perform the steps:

1. Make a clone of the compute node image, as described in “Customizing Software Images”.

2. Determine what images and kernels you have available now, as follows:

   # cimage --list-images image: compute-sles10sp1 kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp image: compute-sles10sp1-new kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp

   
   

3. From the system admin controller, change directory to the images directory, as follows:

   # cd /var/lib/systemimager/images/

   
   

4. From the system admin controller, copy the RPMs your wish to add, as follows:

   # cp /newrpm.rpm new/tmp

   
   

5. The new RPMs now reside in /tmp direcory in the image named new. To install them into your new compute node image, perform the following commands:

   # chroot new bash

   
   

   And then perform the following:

   # rpm -Uvh /tmp/newrpm.rpm

   
   

6. The image on the system admin controller is updated. However, you still need to push the changes out. Ensure there are no nodes currently using the image and then run this command:

   # cimage --push-rack new r\*

   
   

   This will push the updates to the rack lead controllers and the changes will be seen by the compute nodes the next time they start up. For information on how to ensure the image is associated with a given node, see the cimage --set command and the example in Procedure 3-3.

Procedure 3-3. Creating a Simple Compute Node Image Clone

---

Note: Always work from a clone image, see “Customizing Software Images”.

---

To create a simple compute node image clone from the system admin controller, perform the following steps:

1. To clone the compute node image, perform the following:

   # cimage --clone-image compute-sles10sp1 compute-sles10sp1-new

   
   

2. To see the images and kernels in the list, perform the following:

   # cimage --list-images image: compute-sles10sp1 kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp image: compute-sles10sp1-new kernel: 2.6.16.46-0.12-carlsbad kernel: 2.6.16.46-0.12-smp

   
   

3. To change the compute nodes to use the cloned image/kernel pair, perform the following:

   # cimage --set compute-sles10sp1-new 2.6.16.46-0.12-smp "r*i*n*"

   
   

Procedure 3-4. Manually Adding a Package to the Service Node Image

To manually add a package to the service node image, perform the following steps:

1. Use the mksiimage command to create your own version of the service node image. See “Creating Compute and Service Images Using the mksiimage Command”.

2. Change directory to the images directory, as follows:

   # cd /var/lib/systemimager/images/

   
   

3. From the system admin controller, copy the RPMs your wish to add, as follows, where my-service-image is your own service node image:

   # cp /newrpm.rpm my-service-image/tmp

   
   

4. The new RPMs now reside in /tmp direcory in the image named my-service-image. To install them into your new compute node image, perform the following commands:

   # chroot new bash

   
   And then perform the following:

   # rpm -Uvh /tmp/newrpm.rpm

   
   

   At this point, the image has been updated with the rpm . Please note, that unlike compute node images, changes made to a service node image will not be seen by service nodes until they are re-installed with the image. If you wish to install the package on running systems, you can copy the rpm to the running system and use rpm from there.

The cimage command allows you to list, modify, and set software images on the compute nodes in your system.

The cimage command accepts the following options:

RACK arguments take the format rX.

NODE arguments take the format rXiYnZ .

X, Y, Z can be single digits, a [start-end] range, or * for all matches.

... indicates more than one RACK or NODE argument can be passed in.

EXAMPLES

# cimage --list-images

image: compute-sles10sp1
        kernel: 2.6.16.46-0.12-carlsbad
        kernel: 2.6.16.46-0.12-smp

# cimage --list-nodes r1
r1i0n0: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n1: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n2: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n3: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n4: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n5: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n6: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n7: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n8: compute-sles10sp1 2.6.16.46-0.7-smp
[...snip...]

# cimage --set compute-sles10sp1 2.6.16.46-0.12-carlsbad r1i0n0

# cimage --list-nodes r1
r1i0n0: compute-sles10sp1 2.6.16.46-0.7-carlsbad
r1i0n1: compute-sles10sp1 2.6.16.46-0.7-smp
r1i0n2: compute-sles10sp1 2.6.16.46-0.7-smp
[...snip...]

# cimage --set compute-sles10sp1 2.6.16.46-0.7-carlsbad r*i*n*

# cimage --set compute-sles10sp1 2.6.16.46-0.7-smp r1i[0-2]n[5-6] r1i[2-3]n[0-4] 

# cimage --clone-image compute-sles10sp1 mynewimage
Cloning compute-sles10sp1 to mynewimage ... done

# cp *.rpm /var/lib/systemimager/images//tmp
# chroot /var/lib/systemimager/images/mynewimage/ bash
# rpm -Uvh /tmp/*.rpm

# cimage --del-db mynewimage
# cimage --add-db mynewimage

# cimage --push-rack mynewimage r*
r1lead: install-image: mynewimage
r1lead: install-image: mynewimage done.

# cimage --list-images

image: compute-sles10sp1
        kernel: 2.6.16.46-0.7-carlsbad
        kernel: 2.6.16.46-0.7-smp

image: mynewimage
        kernel: 2.6.16.46-0.7-carlsbad
        kernel: 2.6.16.46-0.7-smp

# cimage --set mynewimage 2.6.16.46-0.7-smp r1i3n*

# cimage --del-image mynewimage
r1lead: delete-image: mynewimage
r1lead: delete-image: mynewimage done.

The packages that make up SGI Tempo and SLES are available in repositories at which yum is configured to look.

---

Note: Always work with copies of software images.

---

SGI provides a wrapper around yum that makes it simple to install a package in to an image.

However, yum only looks at packages that are part of a yum repository. So if you are installing your own rpm, you will need to configure yum to look at your own repositories in addition to the others. See the appropriate yum docomentation.

---

Note: The yum software maintains a cache of the repository metdata and it will not update its cache of the metadata until a certain number of minutes have passed. This time limit is defined by the metadata_expire option of the yum.conf file. See the yum.conf(5) man page. If you happen to synchronize your repository to SUSE or Novell shortly after doing a yum operation, and you notice that yum now says there are no updates when there should be, you can run this command to clear the caches and force yum to try again: yum clean all.

---

This example shows how to install the zlib-devel package in to the service node image so that the next time you image or install a service node it has this new package.

You can run the following command:

# yum-image-wrapper /var/lib/systemimager/images/my-service-sles10sp1 install zlib-devel

These instructions only apply to service nodes.

You can use the yum command to install a package on a service node. From the system admin controller, you can issue a command like the following. Please note that SGI suggests using the -y option. This prevents yum from asking for input.

# ssh service0 yum install zlib-devel

You can create service node and compute node images using the mksiimage(1) command. This will generate a root on the fly.

Fresh installations of SGI ProPack 5 SP3 create these images during the configure-cluster installation step.

The rpm lists that drive which packages get installed in the images are listed in files located in /etc/opt/sgi/rpmlists . For example, /etc/opt/sgi/rpmlists/compute-sles10sp1.rpmlist . You should not edit the default SGI RPM lists but instead make copies and work on the copy.

To use the mksiimage command to create a service node image, perform the following:

To use the mksiimage(1) command to create a compute node image, perform the following:

This section describes how to copy a software image from an existing service node so you can use the image when adding a new service node.

To copy a software image from an existing service node for use on a new service node, perform the following steps:

After you have updated or created a service node image, you can install that image on to a managed service node, such as a login node.

---

Note: Re-installing the service node using the discover process will destroy everything previously on the root drive.

---

By default, discover uses the SGI default service-sles10sp1 image. For example:

# discover --service 2,image=my-service-node-image

The image above directs the installation of the described operating system image.

For more information on the discover command, see “discover Command” in Chapter 2.

This section describes how to maintain packages specific to your site and have them available to yum and mksiimage .

SGI suggests putting site-specific packages in a separate location. They should not reside in the same location as SGI or Novell supplied packages.

To set up a custom repository for your site packages, perform the following steps:

All node types that are part of an SGI Altix ICE system can have configuration settings adjusted by the configuration framework. There is some overlap between the per-host customization instructions and the configuration framework instructions. Each approach plays a role in configuring your system. The major differences between the two methods are, as follows:

This framework exists to make it easy to adjust configuration items. There are SGI-supplied scripts already present. You can add more scripts as you wish. You can also exclude scripts from running without purging the script if you decide a certain script should not be run. The following set of questions in bold and bulleted answers describes how to use the system configuration framework.

How does the system configuration framwork operate?

These files could be added, for example, to a running service node, or to an already created service or compute image. Remember that images destined for compute nodes need to be pushed with the cimage command after being altered. For more information, see “cimage Command”.

From where is the framework called?

How do I adjust my system configuration?

Why do scripts need to tolerate files that do not exist but should?

How does the yum plugin work?

How does yume work?

The SGI Tempo 1.3 release includes a new cluster configuration repository/update framework. This framework generates and distributes configuration updates to admin, service, and leader nodes in the cluster. Some of the configuration files managed by this framework include C3 conserver, DNS, Ganglia, hosts files, and NTP.

When an event occurs that requires these files to be updated, the framework executes on the admin node. The admin node stores the updated configuration framework in a special cached location and updates the appropriate nodes with their new configuration files.

In addition to the updates happening as required, the configuration file repository is consulted when a admin, service, or leader node boots. This happens shortly after networking is started. Any configuration files that are new or updated are transferred at this early stage so that the node is fully configured by the time the node is fully operational.

There are no hooks for customer configuration in the configuration repository at this time.

This update framework is tied in with the /etc/opt/sgi/conf.d configuration framework to provide a full configuration solution. As mentioned earlier, customers are encouraged to create /etc/opt/sgi.conf.d scripts to do cluster configuration.

The cpower command is, as follows:

cpower [<option> ...] [<target_type>] [<action>] <target>

The <option> argument can be one or more of the following:

---

Note: The command will fail if the target contains any wild cards, unless the --all option is specified.

---

The <target> argument is one of the following:

The <action> argument is one of the following:

The target must always be specified except when the --system option is used. Wildcards may be used, but be careful not to accidentally power off or reboot the leader nodes. If wildcard use affects any leader node, the command fails with an error.

IPMI-style Commands

The default operation for the cpower command is to operate on nodes and to provide you the status of these nodes, as follows:

# cpower r1i*

The cpower command also

This example gives you the power status and boot status of all the compute blades in rack 1. This command is equivalent to cpower --node --status r1i*.

This command issues an ipmitool power off command to all of the nodes specified by the wildcard, as follows:

# cpower --off r2i*

The default is to apply to a node.

The following commands behave exactly as you would expect as if you were using ipmitool, and have no special extra logic for ordering:

• # cpower --up r1i*

• # cpower --reset r1i*

• # cpower --cycle r1i*

• # cpower --identify 5 r1i*

---

Note: --up is a synonym for --on and --down is a synonym for --off.

---

The cpower command contains more logic when you go up to higher levels of abstraction, for example, using --iru, --rack, and --system. These higher level domain specifiers tell the command to be smart about how to order various of the actions that you give on the command line.

The --iru option tells the command to use correct ordering with IRU power commands. In this case, it firsts connect to the CMC on each IRU in rack 1 to issue the power on command, which turns on power to the IRU chassis (this is not the equivalent ipmitool command). Then it powers up the compute nodes in the IRU. Powering things down is the opposite, with the power to the IRU being turned off after power to the blades. IRU targets are specified as follows: r3i2 for rack 3, IRU 2.

# cpower --iru --up r1* 

The --rack option ensures power commands to the leader node are down in the correct order relative to compute nodes within a rack. First, it powers up the leader node and waits for it to boot up (if it is not already up). Then it will do the functional equivalent of a cpower --iru --up r4i* on each of the IRUs contained in the rack, including applying power to each IRU chassis. Using the --down option is the opposite, and also turns off the leader node (after doing a shutdown) after all the IRUs are powered down. To avoid including leader nodes in a power command for a rack, use the --noleader option. Rack targets are specified, as follows: r4 for rack 4. Here is an example:

# cpower --rack --up r4

Commands with the --system option ensures that power up commands are applied first to service nodes, then to leader nodes, then to IRUs and compute blades, in just the same way. Likewise, compute blades are powered down before IRUs, leader nodes, and service nodes, in that order. To avoid including service nodes in a system-domain command, use the --noservice option. Note that you must not specify a target with --system option, since it applies to the Altix ICE system.

---

Note: The --shutdown --off combination of actions were deprecated in the SGI Tempo v1.2 release. Use the --halt option in it's place.

---

You need to configure the order in which service nodes are booted up and shut down as part of the overall system power management process. This is done by setting a boot_order for each service node. Use the cadmin command to set the boot order for a service node, for example:

# cadmin --set-boot-order --node service0 2

The cpower --system --boot command boots up service nodes with a lower boot order, first. It then boots up service nodes with a higher boot order. The reverse is true when shutting down the system with cpower. For example, if service1 has a boot order of 3 and service2 has a boot order of 5, service1 is booted completely, and then service2 is booted, afterwards. During shutdown, service2 is shut down completely before service1 is shutdown.

There is a special meaning to a service node having a boot order of zero. This value causes the cpower --system command to skip that service node completely for both start up and shutdown (although not for status queries). Negative values for the service node boot order setting are not permitted.

---

Note: The IPMI power commands necessary to enable a system to boot (either with a power reset, or a power on) may be sent to a node. The --halt option, halts the target node and then powers it off.

---

The --halt options works on node, IRU, or rack domain levels. It will shut down nodes (in the correct order if you use the --iru or --rack options), and then just leave them as they are, power still applied. Using both these actions results in nodes being halted, then powered off. This is particularly useful when powering off a rack, since otherwise, the leaders may be shutdown before there is a chance to power off the compute blades. Here is an example:

# cpower --halt --rack r1

To boot up systems that have not already been booted, perform the following:

# cpower --boot  r1i2n*

Again, the command boots up nodes in the right orders if you specify the --iru or --rack options and the appropriate target. Otherwise, there is no guarantee that, for example, the command will attempt to power on the leader node before compute nodes in the same rack.

To reboot all of the nodes specified, or boot them if they are already shut down, perform the following:

# cpower --reboot --iru r3i3

The --iru or --rack options ensure proper ordering if you use them. In this case, the command will make sure that power is supplied to the chassis for rack 3, IRU 3, and then the all the compute nodes in that IRU will be rebooted.

EXAMPLES

Example 3-2. cpower Command Examples

To boot compute blade r1i0n8, perform the following:

# cpower --boot r1i0n8

To boot a number of compute blades at the same time, perform the following:

# cpower --boot --rack r1

---

Note: The --boot option will only boot those nodes that have not already booted.

---

To shut down service node 0, perform the following:

# cpower --halt service0

To shutdown and switch off everything in rack 3, perform the following:

# cpower --halt --rack r3

---

Note: This command will shutdown and then power off all of the computer nodes in parallel, then shutdown and power off the leader node. Use the --noleader option if you want the leader node to remain booted up.

---

To shutdown the entire system, including all service nodes and all leader nodes, but not the admin node, and not turn the power off to anything, perform the following:

# cpower --halt --system

To shutdown all the compute nodes, but not the service nodes, leader nodes, perform the following:

# cpower --halt --system --noleader --noservice

---

Note: The only way to shut down the system admin controller (admin node) is to perform the operation manually.

---

When you used the configure-cluster command, it guided you through setting up NTP on the admin node. The NTP client on the system admin controller should point to the house network time server. The NTP server provides NTP service to system components so that nodes can consult it when they are booted. The system admin controller sends NTP broadcasts to some networks to keep the nodes in sync after they have booted.

NTP client on the rack leader controller gets time from the system admin controller when it is booted and then stays in sync by connecting to the admin node for time. The NTP server on the leader node provides NTP service to Altix ICE components so that compute nodes can sync their time when they are booted. The rack leader controller sends NTP broadcasts to some networks to keep the compute nodes in sync after they have booted.

The BMC controllers on managed service nodes, compute nodes, and leader nodes are also kept in sync with NTP. Note that you may need the latest BMC firmware for the BMCs to sync with NTP properly. The NTP server information for BMCs is provided by special options stored in the DHCP server configuration file.

The NTP client on managed service nodes ( for a definition of managed, see “discover Command” in Chapter 2) sets its time at initial booting from the system admin controller. It listens to NTP broadcasts from the system admin controller to stay in sync. It does not provide any NTP service.

The NTP Client on the compute node sets its time at initial booting from the rack leader controller. It listens to NTP broadcasts from the rack leader controller to stay in sync.

Sometime, especially during initial deployment of an Altix ICE system when system components are being installed and configured for the first time, NTP is not available to serve time to system components.

A non-modified NTP server, running for the first time, takes quite some time before it offers service. This means the leader and service nodes may fail to get time from the system admin controller as they come on-line. Compute nodes may also fail to get time from the leader when they first come up. This situation usually only happens at first deployment. After the ntp servers have a chance to create their drift files, ntp servers offer time with far less delay on subsequent reboots.

The following work arounds are in place for situations when NTP can not serve the time: