This chapter describes how to use the SGI Tempo systems management software to discovery, install, and configure your Altix ICE system and covers the following topics:
 | Note: If you are upgrading from a prior release or installing SGI Tempo software patches, see “Installing SGI Tempo Patches and Updating SGI Altix ICE Systems ” and “Upgrading from SGI ProPack 5 SP4 to SGI ProPack 5 SP5”. |
The configure-cluster command launches a cluster configuration tool. It allows you to perform the following:
Creates the root images for the service nodes, leader nodes, and compute blades
Prompts for the SLES10 SP1 media and directs creation of image repositories which you can use to customize your software image
Runs a set of commands that allows you to setup the cluster
Change the subnet numbers for the various cluster networks
Configure the subdomain of the cluster (which is likely different than the domain of eth0 on the system admin controller itself)
Information on using this tool is described in the procedure in the following section, see “Installing Software on the System Admin Controller”.
This section describes what you should do if you wish to use the pre-installed software on the system admin controller (admin node).
To configure the pre-installed software that comes on the admin node, perform the following steps:
Use YaST to configure the first interface of the admin node for your house network. Settings to adjust may include the following:
Network settings including IP, default route, and so on
Root password
Time zone
If you need to adjust SGI Altix ICE settings such as the Altix ICE cluster domain or any internal network ranges, you will need to reset the database and rediscover the leader nodes and service nodes, as follows:
Start the configure-cluster command (see “configure-cluster Command”).
Choose the Reset Database operation. Read the on-screen instructions.
After the database has been reset, choose Initial Setup Menu.
Start the options in this menu in order starting at Configure Time Client/Server (NTP).
Note: You will get a message about the systemimager images already existing. You may choose to use the existing images instead of re-creating them. This will save about 30 minutes. Either choice is OK. Do not choose use existing images if you changed the root password or time zone as these settings are stored in the image when the image is created.
At this point, you can begin to discover leader and service nodes and continue cluster installation. See “discover Command”.
This section describes how to install software on the system admin controller (admin node). The system admin controller contains software for provisioning, administering, and operating the SGI Altix ICE 8200 system. The SGI Admin Node Autoinstallation DVD contains a software image for the system admin controller (admin node) and contains SGI Tempo and SGI ProPack for Linux packages, used in conjunction with the packages from the SLES10 SP1 DVD, to create leader, service, and compute images.
The root image for the admin node appliance is created by SGI and installed on to the admin node using the admin install DVD.
| Note: If you are reinstalling the admin node, you may want to make a backup of the cluster configuration snapshot that comes with your system so that you can recover it later. You can find it in the /opt/sgi/var/ivt directory on the admin node; it is the earliest snapshot taken. You can use this information with the interconnect verification tool (IVT) to verify that the current system shows the same hardware configuration as when it was shipped. For more information on IVT, see “Inventory Verification Tool” in Chapter 5. |
To install software images on the system admin controller, perform the following steps:
Turn on, reset, or reboot the system admin controller. The power on button is on the right of the system admin controller, as shown in Figure 2-1.
Prior to the SGI Tempo 1.2 release, the serial console was always used even if the admin node install itself went to the vga screen.
The new method configures the default serial console used by the system to match the console used for installation.
If the you type "serial" at the Admin dvd install prompt, the system is also configured for serial console operations after installation and the yast2-firstboot questions appear on the serial console.
If the you hit Enter at the prompt or type vga, the VGA screen is used for installation, as previously, but also, the system is configured to use VGA as the default console, thereafter.
If a you want to install to the VGA screen, but also want the serial console to be used for operations after initial installation, you should add a console= parameter to /boot/grub/menu.lst for each kernel line. This is done when the admin node boots for the first time after installation is completed. An example of this is, as follows:
kernel /boot/vmlinuz-2.6.16.46-0.12-smp root=/dev/disk/by-label/sgiroot console=ttyS1,38400n8 splash=silent showopts
The appropriate entries were added to the inittab and /etc/security. The change, above, is the only one needed to switch the default console from VGA to serial. Likewise, to move from serial to VGA, simply remove the console= parameter, altogether.
Insert the SGI Admin Node Autoinstallation DVD in the DVD drive on the left of the system admin controller as shown in Figure 2-1.
An autoinstall message appears on your console, as follows:
SGI Admin Node Autoinstallation DVD This is the SGI Admin Node autoinstall DVD. If you proceed, the entire system will be erased and re-installed. You may install from the vga screen or from the serial console. The default system console will match the console you used for installation. Hit ENTER for the vga screen or type "serial" for serial. The first time you boot after installation, you will be prompted for system setup questions eary in the startup process. These questions will appear on the same console you use to install the system. Experts: You may choose to use the "auto" label (auto reboot and skip firstboot questions). You may also append the "netinst" option with an nfs path (hostname:/mntpoint/file.iso) to nfs mount the ISO. Press ENTER to send autoinstallation output to the vga screen. Type "serial" at the boot prompt to send autoinstallation output to the serial console.
Note: If you want to use the serial console, enter serial at the boot: prompt, otherwise, output for the install procedure goes to VGA screen.
You can hit the ENTER button at the boot prompt. The boot initrd.image executes, the hard drive is patitioned creating a swap area and a root file system, the Linux operating system and the cluster manager software is installed and a repository is set up for the rack leader controller, service node, and compute node software RPMs.
Note: When you boot with the admin install DVD, any previous data on the disks is destroyed. This step takes several minutes. When the installation is complete, the system admin controller DVD drive automatically ejects the DVD.
Once installation of software on the system admin controller is complete, remove the DVD from the DVD drive.
Once the system has been installed, enter the reboot command to reboot your system.
Note: The output will go to the VGA screen unless you used serial for the admin install DVD earlier.
You will see messages about the system admin controller booting the kernel. You can ignore any messages about a few services that may fail to start.
Note: If you used the serial console for installation ( serial is not the default), the console output and configuration questions from yast2 firstboot will go to the serial port. Pressing Ctrl -l will re-draw the yast2 firstboot screen when you are using the serial console. After the reboot completes, the YaST first boot installation tool starts and a Welcome screen appears, as shown in Figure 2-2. Click on the Next button to proceed.
Note: The YaST Installation Tool has a main menu with sub-menus. You will be redirected back to the main menu, at various times, as you follow the steps in this procedure.
You will be prompted by YaST firstboot installer to enter your system details including the root password, network configuration, time zone, and so on.
From the Hostname and Name Server Configuration screen, as shown in Figure 2-3, enter the hostname and domain name of your system in the appropriate fields. Make sure that Change Hostname via DHCP is unselected (no x should appear in the box). Click on the Next button to continue.
Note: You can use Ctrl L to refresh the YaST screen as necessary.
From the Network Card Configuration Interfaces screen, shows the suggested configuration as shown in Figure 2-4. Click Next to continue.
From the Network Card Configuration Overview screen, configure the first card under Name to establish the public network (sometimes called the house network) connection to your SGI Altix ICE 8200 system.
Note: Do NOT configure the second interface at this time. A script will do this for you in a later step.
Click on the Next button to continue.From the Network Address Setup screen, choose dynamic address setup via DHCP or enter the IP address for the system admin controller. This is your public/house network information. Click on the Next button to continue.
From the Hostname and Name Server Configuration screen, enter the name and DNS domain name as shown in Figure 2-7. Note that the hostname was entered in step 7.
From the Routing Configuration screen, enter the appropriate gateway address and netmask. Click on the OK button to continue.
From the Clock and Time Zone screen, select the appropriate region and time zone. Click on the Next button to continue.
From the Password for the System Administrator “root” screen, set the root password. Click on the Next button to continue.
From the User Authentication Method screen, select the authentication method to use for the users on your system. Click on the Accept button to continue.
Enter the user's full name, username, and user password in the New Local User screen. Click on the Next button to continue.
From the Hardware Configuration screen, select Use Following Configuration. Click on the Next button to continue.
An Installation Competed screen appears, as show in Figure 2-8. Click on the Finish button.
After you have completed the YaST first boot installation instructions, login into the system admin controller. You can use YaST to confirm or correct any configuration settings.
Note: It is important that you make sure that you network settings are correct before proceeding with cluster configuration.
To start cluster configuration, enter the following command:
% /opt/sgi/sbin/configure-cluster
The Cluster Configuration Tool: Initial Configuration Check screen appears, as shown in Figure 2-9. This tool provides instructions on the steps you need to take to configure your cluster. Click OK to continue.
The Cluster Configuration Tool: Initial Cluster Setup screen appears, as shown in Figure 2-10. Read the notice and then click OK to continue.
Note: The Cluster Configuration Tool has a main menu with sub-menus. You will be redirected back to the main menu, at various times, as you follow the steps in this procedure.
Copy the RPMs from your local SLES media.
You need to copy the SLES10 SP1 media to construct a repository for your system.
Insert the SLES10 SP1 DVD into the system admin controller DVD drive. You can either select Mount inserted DVD or select Use custom mount point. If you choose a custom mount point, use a command similar to the following command:
% mount /dev/dvd /mnt
The first of two Copy RPMS screens appears, as shown in Figure 2-12.
The Copy of RPMS from media complete message appears, as shown in Figure 2-13. Click OK to continue.
After choosing the Network Settings option, the Cluster Network Setup screen appears, as shown in Figure 2-14.
The subnet addresses allows you to change the cluster internal network addresses. SGI recommends that you do NOT change these. Click OK to continue to adjust subnets. Otherwise, select Domain Name: Configure Cluster Domain Name and then skip to step 30. A warning screen appears, as shown in Figure 2-15.
Once you deploy your Altix ICE system, to change the network IP values or change domain names, you must reset the system data base and then rediscover the system. You do not need to reinstall the admin node, however. Click OK to continue.The Update Subnet Addresses screen appears, as shown in Figure 2-16.
The default IP address of the system admin controller which is the Head Network for the Altix ICE system is shown. SGI recommends that you do NOT change the IP address of the system admin controller (admin node) or rack leader controllers (leader nodes) if at all possible. You can adjust the IP addresses of the InfiniBand network (ib0 and ib1) to match the IP requirements of the house network. Click OK to continue.
Enter the domain name for your Altix ICE system, as shown in Figure 2-17. Click OK to continue (this will be a subdomain to your house network, by default).
If this is not your first time configuring the cluster, you may see the following screen:
The next operation in the Initial Cluster Setup menu is Configure Time Client/Server (NTP). This procedure changes your NTP configuration file. Click on OK to continue. This sets the system admin controller to serve time to the Altix ICE system and allows you to add time servers on your house networks, which you may optionally use.
Configure NTP time service as shown in Figure 2-20. Click Next to continue.
The YaST tool completes. Click OK to continue.
The next step in the Initial Cluster Setup menu directs you to select Perform Initial Admin Node Infrastructure Setup. This step runs a series of scripts that will configure the system admin controller of the Altix ICE system.
The script installs and configures your system and you should see an install-cluster completed line in the output.
The root images for the service, rack leader controller, and compute nodes are then created. The output of the mksiimage commands are stored in a location similar to the following:
/tmp/mksiimage-cmds-out.12285
You can review the output if you so choose.
The final output of the script reads, as follows:
/opt/sgi/sbin/create-default-sgi-images Done!
Note: As it notes on the Admin Infrastructure One Time Setup screen, this step takes about 30 minutes.
Click OK to continue.
The next step in the Initial Cluster Setup menu is to configure the house DNS resolvers. It is OK to set these resolvers to the same name servers used on the system admin controller itself. Configuring these servers is what allows service nodes to resolve host names on your network. For a description of how to set up service nodes, see “Service Node Installation and Configuration”. This menu has default values printed that match your current admin node resolver setup. If this is ok, just select OK to continue. Otherwise, make any changes you wish to the resolver listing and select OK. If you do not wish to have any house resolvers, select Disable House DNS.
After entering the IPs, click OK to enable, click Disable House DNS to stop using house DNS resolution, click Back to leave house DNS resolution as it was when you started (disabled at installation).
The setting DNS forwarding screen appears. Click Yes to continue.
Proceed to “Installing Software on the Rack Leader Controllers and Service Nodes”. It describes the discovery process for the rack leader controllers in your system and how to install software on the rack leader controllers.
Note: The main menu contains a reset the database function that allows you to start software installation over without having to reinstall the system admin controller.
The discover command is used to discover rack leader controllers (leader nodes), service nodes, including the their associated BMC controllers, in an entire system or in a set of one or more racks that you select. Rack numbers generally start at one. Service nodes generally start at zero. When you use the discover command to perform the discovery operation on your Altix ICE system, you will be prompted with instructions on how to proceed (see “Installing Software on the Rack Leader Controllers and Service Nodes”).
The discover command is, as follows:
/opt/sgi/sbin/discover --rack <#>[,<hw-type>] /opt/sgi/sbin/discover --rackset <start-number>,<count>[,<options>] /opt/sgi/sbin/discover --service <#>[,<options>] |
The discover command accepts the following options:
| Option | Description | |
| --rack | Discovers a specific rack or set of racks | |
| --rackset | Discovers count racks starting at start-number | |
| --service | Discovers the specified service node | |
| --force | Use --force to avoid sanity checks that require input | |
| --ignoremac | Ignores one or more MAC addresses | |
| --delrack | Deletes racks and associated leaders and blades | |
| --delservice | Deletes a service node | |
| --help | Usage and help text |
The options parameter is is a list of comma separated options that modify how discover proceeds for the associated node and sets it up for installation. Hardware types (see below) have no variable style naming with equal signs. All other option types take the form "name=value".
The options paramater include the following:
hw-type
The hw-type parameter is a hardware model that affects how the discover command proceeds. If hw-type is not specified, a default value is used. Use the other hardware type for a service node you supply and manage. This mode allocates IP addresses for you and print them to the screen. This other type of service node is not managed by the Tempo systems management software.
Valid hardware type specifiers are, as follows:
ice-csn (default type)
xe210
xe240
xe310
altix450 (NAS cube)
altix4000
altix4700
other
image type
For service nodes only, you can specify an alternate image to install on to the target system. See the examples for how to specify this.
If you wish to re-discover an existing service node or rack, simply run the discover command in the same manner you normally would. If you wish to purge a rack or service node entirely, (never to be seen again), use --delservice and --delrack options.
EXAMPLES
Example 2-1. discover Command Examples
The following examples walk you through some typical discover command operations.
To discover rack 1 and service node 0, perform the following:
# /opt/sgi/sbin/discover --rack 1 --service 0,xe210 |
In this example, service node 0 is an Altix XE210 system.
To discover racks 1-5, and service node 0-2, perform the following:
# /opt/sgi/sbin/discover --rackset 1,5 --service 0,xe240 --service 1,altix450 --service 2,other |
In this example, service node 1 is an Altix 450 system. Service node 2 is other hardware type.
To discover service 0, but use service-myimage instead of service-sles10sp1 (default), perform the following:
# /opt/sgi/sbin/discover --service 0,image=service-myimage |
To discover racks 1 and 4, service node 1, and ignore MAC address 00:04:23:d6:03:1c , perform the following:
# /opt/sgi/sbin/discover --ignoremac 00:04:23:d6:03:1c --rack 1 --rack 4 --service |
The discover command, described in “discover Command”, sets up the leader and managed service nodes for installation and discovery. This section describes the discovery process you use to determine the Media Access Control (MAC) address, that is, the unique hardware address, of each rack leader controller (leader nodes) and then how to install software on the rack leader controllers.
To install software on the rack leader controllers, perform the following steps:
Use the discover command from the command line, as follows:
# /opt/sgi/sbin/discover --rack 1
Note: You can discover multiple racks at a time using the --rackset option. Service nodes can be discovered with the --service option.
The discover script executes. When prompted, turn the power on to the node being discovered and only that node.
Note: Make sure you only power on the node being discovered and nothing else in the system. Make sure not to power the system up itself.
When the node has electrical power, the BMC starts up even though the system is not powered on. The BMC does a network DHCP request that the discover script intercepts and then configures the cluster database and DHCP with the MAC address for the BMC. The BMC then retrieves its IP address. Next, this script instructs the BMC to power up the node. The node performs a DHCP request that the script intercepts and then configures the cluster database and DHCP with the MAC address for the node. The rack leader controller installs itself using the systemimager software and then boots itself.
The discover script will turn on the chassis identify light for 2 minutes. Output similar to the following appears on the console:
Discover of rack1 / leader node r1lead complete r1lead has been set up to install itself using systemimager The chassis identify light has been turned on for 2 minutes
The blue chassis identify light is your cue to power on the next rack leader controller and start the process all over.
You may watch install progress by using the console command. For example, console r1lead connects you to the console of the r1lead so that you can watch installation progress. The sessions are also logged. For more information on the console command, see “Console Management” in Chapter 3.
Using the identify light, you can configure all the rack leader controllers and service nodes in the cluster without having to go back and fourth to and from your workstation between each discovery operation. Just use the identify light on the node that was just discovered as your cue to move to the next node to plug in.
Shortly after the discover command reports that discovery is complete for a given node, that node installs itself. If you supplied multiple nodes on the discover command line, it is possible multiple nodes could be in different stages of the imaging/installation process at the same time. For rack leaders, when the leader boots up for the first time, one process it starts is the blademond process. This process discovers the IRUs and attached blades and sets them up for use. The blademond process is described in “discover-rack Command”, including which files to watch for progress.
If your discover process does not find the appropriate BMC after a few minutes, the following message appears:
============================================================================== Warning: Trouble discovering the BMC! ============================================================================== 3 minutes have passed and we still can't find the BMC we're looking for. We're going to keep looking until/if you hit ctrl-c. Here are some ideas for what might cause this: - Ensure the system is really plugged in and is connected to the network. - This can happen if you start discover AFTER plugging in the system. Discover works by watching for the DHCP request that the BMC on the system makes when power is applied. Only nodes that have already been discovered should be plugged in. You should only plug in service and leader nodes when instructed. - Ensure the CMC is operational and passing network traffic. - Ensure the CMC firwmare up to date and that it's configured to do VLANs. - Ensure the BMC is properly configured to use dhcp when plugged in to power. - Ensure the BMC, frusdr, and bios firmware up to date on the node. - Ensure the node is connected to the correct CMC port. Still Waiting. Hit ctrl-c to abort this process. That will abort discovery at this problem point -- previously discovered components will not be affected. ==============================================================================
If your discover process finds the appropriate BMC, but cannot find the leader or service node that is powered up after a few minutes, the following message appears:
============================================================================== Warning: Trouble discovering the NODE! ============================================================================== 4 minutes have passed and we still can't find the node. We're going to keep looking until/if you hit ctrl-c. If you got this far, it means we did detect the BMC earlier, but we never saw the node itself perform a DHCP request. Here are some ideas for what might cause this: - Ensure the BIOS boot order is configured to boot from the network first - Ensure the BIOS / frusdr / bmc firmware are up to date. - Is the node failing to power up properly? (possible hardware problem?) Consider manually pressing the front-panel power button on this node just in case the ipmitool command this script issued failed. - Try connecting a vga screen/keyboard to the node to see where it's at. - Is there a fault on the node? Record the error state of the 4 LEDs on the back and contact SGI support. Consider moving to the next rack in the mean time, skippnig this rack (hit ctrl-c and re-run discover for the other racks and service nodes). Still Waiting. Hit ctrl-c to abort this process. That will abort discovery at this problem point -- previously discovered components will not be affected. ==============================================================================
You are now ready to discover and install software on the compute blades in the rack. For instructions, see “Discovering Compute Nodes”.
With the Tempo v1.3 release, you no longer need to explicitly call the discover-rack command to discover a rack and integrate new blades. This is done automatically by a new blademond daemon that runs on the leader nodes.
The blademond daemon is started up when the leader node boots after imaging and begins to poll the chassis management control (CMC) blade in each IRU to determine if any new blades are present. It polls the CMCs every two minutes to see if anything has changed. If something has changed (a new blade, a blade removed, or a blade swapped), it sends the new slot map to the admin node and calls the discover-rack command to integrate the changes. It then boots new nodes on the default compute image.
The blademond daemon maintains its log file at /var/log/blademond on the leader nodes.
You can turn on debug mode in the blademond daemon by sending it a SIGUSR1 signal from the leader node, as follows:
# kill -USR1 pid |
To turn debug mode off, send it another SIGUSR1 signal. You should see a message in the blademond log about debug mode being enabled or disabled.
The blademond daemon maintains the slot map at /var/opt/sgi/lib/blademond/slot_map on the leader nodes. This appears as /var/opt/sgi/lib/blademond/slot_map. rack_number on the admin node.
This section describes how to discover compute nodes in your Altix ICE system.
| Note: With the Tempo v1.3 release, you no longer need to explicitly call the discover-rack command to discover a rack and integrate new compute nodes (blades). This is done automatically by a new blademond daemon that runs on the leader nodes (see “discover-rack Command”). |
To discover compute nodes (blades) in your Altix ICE system, perform the following:
Complete the steps in “Installing Software on the Rack Leader Controllers and Service Nodes”.
For instructions on how to configure, start, verify, or stop the InfiniBand Fabric management software on your Altix ICE system, see Chapter 4, “System Fabric Management”.
| Note: The InfiniBand fabric does not automatically configure itself. For information on how to configure and start up the InfiniBand fabric, see Chapter 4, “System Fabric Management”. |
Service nodes are discovered and deployed similar to rack leader controllers (leader nodes). The discover command, with the --service related commands, allow you to discover service nodes in the same discover operation that discovered the leader nodes.
Like rack leader controllers, the service node is automatically installed. The service-image is used to install the service node.
Unlike system admin controllers (admin nodes), eth0 on the service node connects to the Altix ICE network (like rack leader controllers). If you wish to have the service node on your house network, you need to configure the second Ethernet interface (eth1).
The yast2-firstboot software does not start automatically on the system console after the first boot after installation (unlike the admin node). This is because the service node installation is a somewhat automated process. A configuration script called /opt/sgi/sbin/configure-service-node is available. This script is very simple and simply pops up a couple of dialog windows and then forces yast2-firstboot to start up in the current shell session.
The pop-up dialog windows contain information about system management operations on the service node, as follows:
eth1 is the house network that you should configure in firstboot.
If you change the default host name, you need to make sure that the cluster service name is still resolvable as tools depend on that.
Name service configuration is handled by the admin and leader nodes. Therefore, service node resolv.conf files need to always point to the admin and leader nodes in order to resolve cluster names. If you wish to resolve host names on your "house" network, use the configure-cluster command to configure the house name servers. The admin and leader nodes will then be able to resolve your house network addresses, in addition to the internal cluster hostnames. Besides, the cluster configuration update framework may replace your resolv.conf file anyway when cluster configuration adjustments are made.
Do not change resolv.conf and do not configure different name servers in yast.
This section describes how to configure a service node and covers the following topics:
You may want to reach network services outside of your SGI Altix ICE 8200 system. For this type of access, SGI recommends using Network Address Translation (NAT), also known as IP Masquerading or Network Masquerading. Depending on the amount of network traffic and your site needs, you may want to have multiple service nodes providing NAT services.
To enable NAT on your service node, perform the following steps:
Use the configuration tools provided on your service node to turn on IP forwarding and enable NAT/IP MASQUERADE.
Specific instructions should be available in the third-party documentation provided for your storage node system. For service node running SUSE Linux Enterprise Server (SLES), there is documentation at /opt/sgi/docs/setting-up-NAT/README . This document describes how to get NAT working for both IB interfaces.
Note: This file is only on the service node. You need to # ssh service0 and then from service 0 # cd /opt/sgi/docs/setting-up-NAT.
Update the all of the compute node images with default route configured for NAT.
SGI recommends a script on the system admin controller at /opt/sgi/share/per_host_customization/global/sgi-static-routes that can customize the routes based upon rack, IRU, and slot of the compute blade. Some examples are available in that script.
Use the use the cimage --push-rack command to propagate the changes to the proper location for compute nodes to boot. For more information on using the cimage command, see “cimage Command” in Chapter 3 and “Customizing Software On Your SGI Altix ICE System” in Chapter 3.
Use the cimage --set command to select the image
Reboot/reset the compute nodes using that desired image.
Once the service node(s) has NAT enabled, is attached to an operational house network, and the compute nodes are booted from an image which sets their routing to point at the service node, test the NAT operation by using the ping(8) command to ping known IP addresses on the house network from an interactive session on the compute blade.
See the troubleshooting discussion that follows.
Troubleshooting can become very complex. The first steps are to determine that the service node(s) are correctly configured for the house network and can ping the house IP addresses. Good choices are house name servers possibly found in the /etc/resolv.conf or /etc/name.d.conf files on the admin node. Additionally, the default gateway addresses for the service node may be a good choice. You can use the netstat -rn command for this information, as follows:
system-1:/ # netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 128.162.244.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 172.16.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 128.162.244.1 0.0.0.0 UG 0 0 0 eth0 |
If the ping command executed from the service node to the selected IP address gets responses, network monitoring tools such as tcpdump(1) should be used. On the service node, monitor the eth1 interface and simultaneously in a separate session monitor the ib[01] interface. You should specify monitoring specific-enough to not have additional noise then attempt execute a ping command from the compute node.
Example 2-2. tcpdump Command Examples
tcpdump -i eth1 ip proto ICMP # Dump ping packets on the public side of service node. tcpdump -i ib1 ip proto ICMP # Dump ping packets on the IB fabric side of service node. tcpdump -i eth1 port nfs # Dump NFS traffic on the eth1 side of service node. tcpdump -i ib1 port nfs # Dump NFS traffic on the eth1 side of service node. |
If packets do not reach the service nodes respective IB interface, perform the following:
Check the system admin controller's compute image configuration of the default route
Verify that this image has been pushed to the compute nodes
Verify that the compute nodes have booted with this image
If the packets reach the service nodes IB interface, but do not exit the eth1 interface, verify the NAT configuration on the service node.
If the packets exit the eth1 interface, but replies do not return, verify the house network configuration and that IP masquerading is properly configured so that the packets exiting the interface appear to be originating from the service node and not the compute node.
You may chose to connect your compute nodes using resolvable addresses on the house network. This requires planning before the installation by reserving a large block of resolvable IP addresses on the house network and the correct steps early in installation.
| Note: Placing a fabric on the house network does make it more susceptible to bandwidth and latency fluctuations due to undesired or unexpected network traffic. |
To connect your compute nodes using resolvable addresses on the house network, perform the following steps:
Enter IP values into the configure-cluster script while you make sure to assign IP addresses in the resolvable range to the IB fabric(s) you desire.
You can make either ib0, ib1, or both resolvable on the house network. Careful planning is required.
After house network addresses are assigned, you need to use the service node(s) operating system tools to enable IP forwarding and configure the house routers or network infrastructure to route addresses for the desired fabrics through the desired service nodes.
All of these steps are extremely site specific, therefore, you need to rely on your network administrators to set up this type of configuration.
For information on setting up DNS, see Figure 2-23.
Assuming the installation has either NAT or Gateway operations configured on one or more service nodes, the compute nodes can directly mount the house NFS server's exports (see the exports(5) man page).
To allow the compute nodes to directly mount the house NFS server's exports, perform the following steps:
Edit the system admin controller's /opt/sgi/share/per_host_customization/global/sgi-fstab file or alternatively an image-specific script. An example of the sgi-fstab file is, as follows:
system-1-admin:/opt/sgi/share/per-host-customization/global # cat sgi-fstab #!/bin/sh # # Set up the compute node's /etc/fstab file. # # Modify per your sites requirements. # # This script is excecuted once per-host as part of the install-image operation # run on the leader nodes. The full path to the per-host iru+slot directory is # passed in as $1, e.g. /var/lib/sgi/per-host//i2n11. # iruslot=$1 cat <${iruslot}/etc/fstab # tmpfs /tmp tmpfs defaults 0 0 EOF
Add the mount point, push the image, and reset the node.
The server's export should get mounted. If it is not, use the technique for troubleshooting outlined in “Troubleshooting Service Node Configuration for NAT”.
This section describes two different ways to configure a service node for NIS, as follows:
NIS with the compute nodes directly accessing the house NIS infrastructure
NIS with a service node as a NIS slave server to the house NIS master
Assuming the installation has either Network Address Translation (NAT) or Gateway operations configured on one or more service nodes, the compute nodes can directly access the house NIS servers. Broadcast operations for discovering NIS servers do not typically work. Therefore, you need to configure the compute images with the IP address of the NIS server to which you want them to connect.
Procedure 2-8. Service Node Configuration for NIS with the Compute Nodes Directly Accessing the House NIS Infrastructure
To configure NIS on a compute node, perform the following steps:
Clone a compute image which you would like to extend to use NIS (see “cimage Command” in Chapter 3 and “Customizing Software On Your SGI Altix ICE System” in Chapter 3).
Note: The default installation does not contain the ypbind package. You need to install it for use in your cloned image.
Install the ypbind package using the operating system package manager.
Use the operating system configuration tools to configure the ypbind software. See your operating system documentation for instructions on configuring ypbind for NIS operations and the ypbind(8) man page.
Push this new image out to the compute nodes and reboot the system to test the configuration.
If the compute blades fail to connect to the NIS server, use the technique for troubleshooting outlined in “Troubleshooting Service Node Configuration for NAT”.
To configure NIS with a service node as a NIS slave server to the house NIS master, perform the following steps:
Make sure your network administrator has authorized the service node to act as a slave server.
Use the service node operating system tools to configure the NIS slave server on the service node.
Use the ypwhich(1) command to verify that it shows localhost as the current server and ypcat(1) passwd looks consistent with what you expect.
Note: You may have some issues with configuration tools, such as, removing parts of the host name or IP for the server. This can be solved by creating a /etc/hosts record.
Install the ypbind package using the operating system package manager.
Use the operating system configuration tools to configure the ypbind software. See your operating system documentation for instructions on configuring ypbind for NIS operations and the ypbind(8) man page.
Push this new image out to the compute nodes and reboot the system to test the configuration.
If the compute blades fail to connect to the NIS server, use the technique for troubleshooting outlined in “Troubleshooting Service Node Configuration for NAT”.
| Note: Multiple service nodes can be used as NIS slave servers. |
These section describes how to make a service node an NFS home directory server for the compute nodes.
| Note: Having a single, small server provide filesystems to the whole Altix ICE system could create network bottlenecks that the hierarchical design of Altix ICE is meant to avoid, especially if large files are stored there. Consider putting your home filesystems on an NAS file server. For instructions on how to do this, see “Service Node Configuration for NFS ”. |
The instructions in this section assume you are using the service node image provided with the Tempo software. If you are using your own installation procedures or a different operating system, the instructions will not be exact but the approach is still appropriate.
| Note: The example below specifically avoids using /dev/sdX style device names. This is because /dev/sdX device names are not persistent and may change as you adjust disks and RAID volumes in your system. In some situations, you may assume /dev/sda is the system disk and that /dev/sdb is a data disk; this is not always the case. To avoid accidental destruction of your root disk, follow the instructions given below. |
When you are choosing a disk, please consider the following:
To pick a disk device, first find the device that is being currently used as root. Avoid re-partitioning the installation disk by accident. To find which device is being used for root, use this command:
# ls -l /dev/disk/by-label/sgiroot lrwxrwxrwx 1 root root 10 2008-03-18 04:27 /dev/disk/by-label/sgiroot -> ../../sda2 |
At this point, you know the sd name for your root device is sda.
SGI suggests you use by-id device names for your data disk. Therefore, you need to find the by-id name that is NOT your root disk. To do that, use ls command to list the contents of /dev/disk/by-id, as follows:
# ls -l /dev/disk/by-id total 0 lrwxrwxrwx 1 root root 9 2008-03-20 04:57 ata-MATSHITADVD-RAM_UJ-850S_HB08_020520 -> ../../hdb lrwxrwxrwx 1 root root 9 2008-03-20 04:57 scsi-3600508e000000000307921086e156100 -> ../../sda lrwxrwxrwx 1 root root 10 2008-03-20 04:57 scsi-3600508e000000000307921086e156100-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 2008-03-20 04:57 scsi-3600508e000000000307921086e156100-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 2008-03-20 04:57 scsi-3600508e000000000307921086e156100-part5 -> ../../sda5 lrwxrwxrwx 1 root root 10 2008-03-20 04:57 scsi-3600508e000000000307921086e156100-part6 -> ../../sda6 lrwxrwxrwx 1 root root 9 2008-03-20 04:57 scsi-3600508e0000000008dced2cfc3c1930a -> ../../sdb lrwxrwxrwx 1 root root 10 2008-03-20 04:57 scsi-3600508e0000000008dced2cfc3c1930a-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 9 2008-03-20 09:57 usb-PepperC_Virtual_Disc_1_0e159d01a04567ab14E72156DB3AC4FA -> ../../sr0 |
In the output, above, you can see that ID scsi-3600508e000000000307921086e156100 is in use by your system disk because it has a symbolic link pointing back to ../../sda. So do not consider that device.nThe other disk in the listing has ID scsi-3600508e0000000008dced2cfc3c1930a and happens to be linked to /dev/sdb.
Therefore, you know the by-id name you should use for your data is /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a because it is not connected with sda, which we found in the first ls example happened to be the root disk.
The following example uses /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a ID as the empty disk on which you will put your data. It is very important that you know this for sure. In “Setting Up an NFS Home Server on a Service Node for Your Altix ICE System”, an example is provided that allows you to determine where your root disk is located so you can avoid accidently destroying it. Remember, in some cases, /dev/sdb will be the root drive and /dev/sda or /dev/sdc may be the data drive. Please confirm that you have selected the right device, and use the persistent device name to help prevent accidental overwriting of the root disk.
| Note: Steps 1 through 7 of this procedure are performed on the service node. Steps 8 and 9 are performed from the system admin controller (admin node). |
To partition and create filesystems for an NFS home server on a service node, perform the following steps:
Use the parted(8) utility, or some other partition tool, to create a partition on /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a . The following example makes one filesystem out of the disk. You can use the parted utility interactively or in a command-line driven manner.
Make a new msdos label, as follows:
# parted /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a mklabel msdos
Find the size of the disk, as follows:
# # parted /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a print Disk geometry for /dev/sdb: 0kB - 249GB Disk label type: msdos Number Start End Size Type File system Flags Information: Don't forget to update /etc/fstab, if necessary.
Create a partition that spans the disk, as follows:
# # parted /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a mkpart primary ext2 0 249GB Information: Don't forget to update /etc/fstab, if necessary.
Issue the following command to cause the /dev/disk/by-id partition device file is in place and available for use with the mkfs command that follows:
# udevtrigger
Create a filesystem on the disk. You can choose the filesystem type.
Note: The mkfs.ext3 command takes more than 10 minutes to create a single 500GB filesystem using default mkfs.ext3 options. If you do not need the number of inodes created by default, use the -N option to mkfs.ext3 or other options that reduce the number of inodes. The following example creates 20 million inodes. XFS filesystems can be created in much shorter time.
An ext3 example is, as follows:# mkfs.ext3 -N 20000000 /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a-part1
An xfs example is, as follows:# mkfs.xfs /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a-part1
Add the newly created filesystem to the server's fstab file and mount it. Ensure that the new filesystem is exported and that the NFS service is running, as follows:
Append the following line to your /etc/fstab file.
/dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a-part1 /home ext3 defaults 1 2
Note: If you are using XFS, replace ext3 with xfs. This example uses the /dev/disk/by-id path for the device and not a /dev/sd device.
Mount the new filesystem (the fstab entry, above, enables it to mount automatically the next time the system is rebooted), as follows:
# mount -a
Make sure the NFS server service is enabled, as follows:
# chkconfig nfsserver on # /etc/init.d/nfsserver restart
Note: Steps 8 and 9 are performed from the system admin controller (admin node).
The following steps describe how to mount the home filesystem on the compute nodes, as follows:
Note: SGI recommends that you always work on clones of the SGI-supplied compute image so that you always have a base to copy to fall back to if necessary. For information on cloning a compute node image, see “Customizing Software Images” in Chapter 3. Make a mount point in the blade image. In the following example, /home already is a mount point. If you used a different mount point, you need to do something similar to the following on the system admin controller. Note that the rest of the examples will resume using /home.
# mkdir /var/lib/systemimager/images/compute-sles10sp1-clone/my-mount-point
Add the /home filesystem to the compute nodes. SGI supplies an example script for managing this. You just need to add your new mount point to the sgi-fstab post-host-customization script.
Use a text editor to edit the following file:
/opt/sgi/share/per-host-customization/global/sgi-fstab
Insert the following line just before the "EOF" line in sgi-fstab file:
service0-ib1:/home /home nfs hard 0 0
Note: In order to maximize performance, SGI advises that the ib0 fabric be used for all MPI traffic. The ib1 fabric is reserved for storage related traffic.
Use the cimage command to push the update to the rack leader controllers serving each compute node, as follows:
# cimage --push-rack compute-sles10sp1-clone "r*"
Using --push-rack on an image that is already on the rack leader controllers has the simple affect of updating them with the change you made above. For more information on using the cimage, see “cimage Command” in Chapter 3.
When you reboot the compute nodes, they will mount your new home filesystem.
For information on centrally managed user accounts, see “Setting Up a NIS Server for Your Altix ICE System”. It describes NIS master set up. In this design, the master server residing on the service node provides the filesystem and the NIS slaves reside on the rack leader controllers. If you have more than one home server, you need to export all home filesystems on all home servers to the server acting as the NIS master. You also need to export the filesystems to the NIS master using the no_root_squash exports flag.
If you want to use NAS server for scratch storage or make home filesystems available on NAS, you can follow the instructions in “Setting Up an NFS Home Server on a Service Node for Your Altix ICE System”. In this example, you need to replace service0-ib1 with the ib1 InfiniBand host name for the NAS server and you need to know where on the NAS server the home filesystem is mounted to craft the sgi-fstab script properly.
All operations are from the service node acting as the NFS proxy except where noted.
This procedure described in this section does not require the NAT/gateway operations and may be more efficient. This method does require that an unsupported package be installed. It is available from the SGI support page as described below.
To set up a service node for re-exporting house NFS servers, perform the following steps:
Download the unsupported nfs-server RPM from the SGI support server:
Login to Supportfolio (https://support.sgi.com/ )
Click on Browse Collections.
Click on Download Cool Software .
Find the nfs-server package.
Remove nfs-utils on the service node, as follows:
# rpm -e nfs-utils
Install the newly downloaded nfs-server RPM, as follows:
# rpm -Uvh /usr/src/packages/RPMS/x86_64/nfs-server-2.2beta51-246*.x86_64.rpm
Edit the /etc/sysconfig/nfs file and change the REEXPORT_NFS option to "yes"
Enable the NFS server at start-up, as follows:
# chkconfig nfsserver on
Start it on the service node, as follows:
# rcnfsserver start
Add the mount to the "house nfs server" on to the service node acting as the proxy for NFS. An example fstab line is, as follows:
house-server:/mirror /mirror nfs defaults 0 0
Ensure the filesystem is mounted, as follows:
# mount -a
Export the filesystem by adding a line to /etc/exports similar to the example. You also need to change the subdomain to match your site's.
/mirror *.ice.americas.sgi.com(ro,sync)
Now configure the compute blades to mount this directory from the service node acting as a proxy. In this example, it is assumed that service0 is the node from which the blades will mount /mirror. To do this, add a line similar to this to the following before 'EOF' in /opt/sgi/share/per-host-customization/global/sgi-fstab file. This file is located on the system admin controller (admin node).
service0-ib1:/mirror /mirror nfs hard 0 0
Recall that the mount point for the compute blades needs to exist. Therefore, you might need to create a directory within the systemimager image on the admin node, for example, mkdir /var/lib/systemimager/images/compute-sles10sp1/mirror.
Tell NFS about the exports change, as follows:
# rcnfsserver reload
Earlier, in this procedure, you changed the sgi-fstab per-host customization script and created a mount point within one or more compute blade systemimager images. From the admin node, you need to push the images so they are available on the leader nodes serving your racks. The compute blades in the rack in question should be shut down prior to running this command. You should do this for all compute images you may have and for all racks.
# cimage --push-rack compute-sles10sp1 r1
Now you may boot up your compute blades. The filesystem will now be mounted on each one. When you access /mirror on a compute blade, the service node proxy NFS server then accesses its /mirror, which contacts the actual NFS server on the house network.
This section describes how to set up a network information service (NIS) server running SLES10 for your Altix ICE system. If you would like to use an existing house network NIS server, see “Service Node Configuration for NIS for the House Network”. This section covers the following topics:
In the procedures that follow in this section, here are some of the tasks you need to perform and system features you need to consider:
Make a service node the NIS master
Make the rack leader controllers (leader nodes) the NIS slave servers
Not make the system admin controller as the NIS master because it may not be able to mount all of the storage types. Having the storage mounted on the NIS master server makes it far less complicated to add new accounts using NIS.
If multiple service nodes provide home filesystems, the NIS master should mount all remote home filesystems. They should be exported to the NIS master service node with the no_root_squash export option. The example in the following section assumes a single service node with storage and that same node is the NIS master.
No NIS traffic goes over the InfiniBand network.
Compute node NIS traffic goes over Ethernet, not InfiniBand, by way of using a the lead-eth server name in the yp.conf file. This design feature prevents NIS traffic from affecting the InfiniBand traffic between the compute nodes.
This section describes how to set up a service node as a NIS master. This section only applies to service nodes running SLES10.
To set up a service node as a NIS master, from the service node, perform the following steps:
| Note: These instructions use the text-based version of YaST. The graphical version of YaST may be slightly different. |
Start up YaST, as follows:
# yast nis_server
Choose Create NIS Master Server and click on Next to continue.
Choose an NIS domain name and place it in the NIS Domain Name window. This example, uses ice.
Select This host is also a NIS client.
Select Active Slave NIS server exists .
Select Fast Map distribution.
Select Allow changes to passwords .
Click on Next to continue.
Set up the NIS master server slaves.
Note: You are now in the NIS Master Server Slaves Setup. Just now, you can enter the already defined rack leader controllers (leader nodes) here. If you add more leader nodes or re-discover leader nodes, you will need to change this list. For more information, see “Tasks You Should Perform After Changing a Rack Leader Controller ”.
Select Add and enter r1lead in the Edit Slave window. Enter any other rack leader controllers you may have just like above. Click on Next to continue.
You are now in NIS Server Maps Setup . The default selected maps are okay. Avoid using the hosts map (not selected by default) because can interfere with Altix ICE system operations. Click on Next to continue.
You are now in NIS Server Query Hosts Setup. Use the default settings here. However, you may want to adjust settings for security purposes. Click on Finish to continue.
At this point, the NIS master is configured. Assuming you checked the This host is also a NIS client box, the service node will be configured as a NIS client to itself and start yp ypbind for you.
This section describes how to use YaST to set up your other service nodes to be broadcast binding NIS clients. This section only applies to service nodes running SLES10.
| Note: You do not do this on the NIS Master service node that you already configured as a client in “Setting Up a Service Node as a NIS Master”. |
To set up a service node as a NIS client, perform the following steps:
Enable ypbind, perform the following:
# chkconfig ypbind on
Set the default domain (already set on NIS master). Change ice (or whatever domain name you choose above) to be the NIS domain for your Altix ICE system, as follows:
# echo "ice" > /etc/defaultdomain
In order to ensure that no NIS traffic goes over the IB network, SGI does not recommend using NIS broadcast binding on service nodes. You can list a few leader nodes the in /etc/yp.conf file on non-NIS-master service nodes. The following is an example /etc/yp.conf file. Add or remove rack leader nodes as appropriate. Having more entries in the list allows for some redundancy. If r1lead is hit by excessive traffic or goes down, ypbind can use the next server in the list as its NIS server. SGI does not suggest listing other service nodes in yp.conf file because all resolvable names for service nodes on service nodes use IP addresses that go over the InfiniBand network. For performance reasons, it is better to keep NIS traffic off of the InfiniBand network.
ypserver r1lead ypserver r2lead
Start the ypbind service, as follows:
# rcypbind start
The service node is now bound.
Add the NIS include statement to the end of the password and group files, as follows:
# echo "+:::" >> /etc/group # echo "+::::::" >> /etc/passwd # echo "+" >> /etc/shadow
This section provides two sets of instructions for setting up rack leader controllers (leader nodes) as NIS slave servers. It is possible to make all these adjustments to the leader image in /var/lib/systemimager/images . Currently, SGI does not recommend using this approach.
| Note: Be sure the InfiniBand interfaces are up and running before proceeding because the rack leader controller gets its updates from the NIS Master over the InfiniBand network. If you get a "can't enumerate maps from service0" error, check to be sure the InfiniBand network is operational. |
Use the following set of commands from the system admin controller (admin node) to set up a rack leader controller (leader node) as a NIS slave server and client.
| Note: Replace ice with your NIS domain name and service0 with the service node you set up as the master server. |
# cexec --head --all chkconfig ypserv on # cexec --head --all chkconfig ypbind on # cexec --head --all chkconfig portmap on # cexec --head --all chkconfig nscd on # cexec --head --all rcportmap start # cexec --head --all "echo ice > /etc/defaultdomain" # cexec --head --all "ypdomainname ice" # cexec --head --all "echo ypserver 127.0.0.1 > /etc/yp.conf" # cexec --head --all /usr/lib/yp/ypinit -s service0 # cexec --head --all rcportmap start # cexec --head --all rcypserv start # cexec --head --all rcypbind start # cexec --head --all rcnscd start |
This section describes how to set up the compute nodes to be NIS clients. You an configure NIS on the clients to use a server list that only contains the their rack leader controller (leader node). All operations are performed from the system admin controller (admin node).
To set up the compute nodes to be NIS clients, perform the following steps:
Create a compute node image clone. SGI recommends that you always work with a clone of the compute node images. For information on how to clone the compute node image, see “Customizing Software Images” in Chapter 3.
Change the compute nodes to use the cloned image/kernel pair, as follows:
# cimage --set compute-sles10sp1-clone 2.6.16.46-0.12-smp "r*i*n*"
Set up the NIS domain, as follows ( ice in this example):
# echo "ice" > /var/lib/systemimager/images/compute-sles10sp1-clone/etc/defaultdomain
Set up compute nodes to get their NIS service from their rack leader controller (fix the domain name as appropriate), as follows:
# echo "ypserver lead-eth" > /var/lib/systemimager/images/compute-sles10sp1-clone/etc/yp.conf
Enable the ypbind service, using the chroot command, as follows:
# chroot /var/lib/systemimager/images/compute-sles10sp1-clone chkconfig ypbind on
Set up the password, shadow, and group files with NIS includes, as follows:
# echo "+:::" >> /var/lib/systemimager/images/compute-sles10sp1-clone/etc/group # echo "+::::::" >> /var/lib/systemimager/images/compute-sles10sp1-clone/etc/passwd # echo "+" >> /var/lib/systemimager/images/compute-sles10sp1-clone/etc/shadow
Push out the updates using the cimage command, as follows:
# cimage --push-rack compute-sles10sp1-clone "r*"
The NAS cube needs to get configured with each InfiniBand fabric interface in a separate subnet. These fabrics will be separated from each other logically, but attached to the same physical network. For simplicity, this guide assumes that the -ib1 fabric for the compute nodes has addresses assigned in the 10.149.0.0/16 network. This guide also assumes the lowest address the cluster management software has used is 10.149.0.1 and the highest is 10.149.1.3 (already assigned to the NAS cube).
For the NAS cube, you need to configure the large physical network into four, smaller subnets, each of which would be capable of containing all the nodes and service nodes. It will have subnets 10.149.0.0/18 , 10.149.64.0/18, 10.149.128.0/18 , and 10.149.192.0/18.
After the discovery of the storage node has happened, SGI personnel will need to log onto the NAS box and change the network settings to use the smaller subnets, and then define the other three adapters with the same offset within the subnet; for example: Initial configuration of the storage node had set ib0 fabric's IP to 10.149.1.3 netmask 255.255.0.0. After the addresses are changed, ib0=10.149.1.3:255.255.192.0, ib1=10.149.65.3:255.255.192.0 , ib2=10.149.129.3:255.255.192.0, ib3=10.149.193.3:255.255.192.0 . The NAS cube should now have all four adapter connections connected to the fabric with IP addresses which can be pinged from the service node.
| Note: The service nodes and the rack leads will remain in the 10.149.0.0/16 subnet. |
For the compute blades, log into the admin node and modify /opt/sgi/share/per-host-customization/global/sgi-setup-ib-configs file. Following the line iruslot=$1, insert:
# Compute NAS interface to use
IRU_NODE=`basename ${iruslot}`
RACK=`cminfo --rack`
RACK=$(( ${RACK} - 1 ))
IRU=`echo ${IRU_NODE} | sed -e s/i// -e s/n.*//`
NODE=`echo ${IRU_NODE} | sed -e s/.*n//`
POSITION=$(( ${IRU} * 16 + ${NODE} ))
POSITION=$(( ${RACK} * 64 + ${POSITION} ))
NAS_IF=$(( ${POSITION} % 4 ))
NAS_IPS[0]="10.149.1.3"
NAS_IPS[1]="10.149.65.3"
NAS_IPS[2]="10.149.129.3"
NAS_IPS[3]="10.149.193.3" |
Then following the line $iruslot/etc/opt/sgi/cminfo add:
IB_1_OCT12=`echo ${IB_1_IP} | awk -F "." '{ print $1 "." $2 }'`
IB_1_OCT3=`echo ${IB_1_IP} | awk -F "." '{ print $3 }'`
IB_1_OCT4=`echo ${IB_1_IP} | awk -F "." '{ print $4 }'`
IB_1_OCT3=$(( ${IB_1_OCT3} + ${NAS_IF} * 64 ))
IB_1_NAS_IP="${IB_1_OCT12}.${IB_1_OCT3}.${IB_1_OCT4}" |
Then change the IPADDR='${IB_1_IP}' and NETMASK='${IB_1_NETMASK}' lines to the following:
IPADDR='${IB_1_NAS_IP}'
NETMASK='255.255.192.0' |
Then add the following to the end of the file:
# ib-1-vlan config
cat << EOF >$iruslot/etc/sysconfig/network/ifcfg-vlan1
# ifcfg config file for vlan ib1
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='${IB_1_IP}'
MTU=''
NETMASK='255.255.192.0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ETHERDEVICE='ib1'
EOF
if [ $NAS_IF -eq 0 ]; then
rm $iruslot/etc/sysconfig/network/ifcfg-vlan1
fi |
To update the fstab for the compute blades, edit /opt/sgi/share/per-host-customization/global/sgi-fstab file. Perform the equivalent steps as above to add the # Compute NAS interface to use section into this file. Then to specify mount points, add lines similar to the following example:
# SGI NAS Server Mounts
${NAS_IPS[${NAS_IF}]}:/mnt/data/scratch /scratch nfs defaults 0 0 |
The example used in this section assumes that the home directory is mounted on the NIS Master service and that the NIS master is able to create directories and files on it as root. The following example use command line commands. You could also create accounts using YaST.
To create user accounts on the NIS server, perform the following steps:
Log in to the NIS Master service node as root.
Issue a useradd command similar to the following:
# useradd -c "Joe User" -m -d /home/juser juser
Provide the user a password, as follows:
# passwd juser
Push the new account to the NIS servers, as follows:
# cd /var/yp && make
If you add or remove a rack leader controller (leader node), for example, if you use discover command to discover a new rack of equipment, you will need to configure the new rack leader controller to be an NIS slave server as described in “Setting Up a Service Node as a NIS Client”.
In addition, you need to add or remove the leader from the /var/yp/ypservers file on NIS Master service node. Remember to use the -ib1 name for the leader, as service nodes cannot resolve r2lead style names. For example, use r2lead-ib1.
# cd /var/yp && make |
This section describes how to update the software on an SGI Altix ICE system.
SGI supplies updates to SGI Tempo software via the SGI update server at https://update.sgi.com/. Access to this server requires a Supportfolio login and password. Access to SUSE Linux Enterprise Server updates requires a Novell login account and registration.
The initial installation process for the SGI Altix ICE system set up a number of package repositories in the /tftpboot directory on the admin node. The SGI Tempo related packages are in the /tftpboot/oscar directory and SUSE Linux Enterprise Linux (SLES) packages are in the /tftpboot/distro/sles-10-x86_64 directory.
When SGI releases updates, you may run sync-repo-updates (described later) to download the updated packages that are part of a patch. The sync-repo-updates command automatically positions the files properly under /tftpboot.
Once the local repositories contain the updated packages, it is possible to update the various SGI Altix ICE admin, leader, and managed service node images using the yum(8) command. The yum update and yum install commands are used for all updates, including updates to existing images.
For additional information on updating your system, see “Upgrading from SGI ProPack 5 SP4 to SGI ProPack 5 SP5”.
There is a small amount of preparation required, in order to setup an SGI Altix ICE system, so that updated packages can be downloaded from the SGI update server and then installed with the yum command.
This following sections describe these steps:
This section explains how to update the local product package repositories needed to share updates on all of the various nodes on an SGI Altix ICE system.
SGI provides an example script called sync-repo-updates to help keep your local package repositories on the admin node synchronized with available updates for the SGI Tempo, SGI ProPack and SLES products. You can use this script directly or use it as a template to develop more customized solutions. The script is located in /opt/sgi/sbin/sync-repo-updates on the admin node.
The sync-repo-updates script requires two parameters; your Supportfolio user name and password. Using that information, the script contacts SGI's update server and downloads the updated packages into the appropriate local package repositories. If you installed and configured the yup tool as described in “Update the SLES Package Repository”, the sync-repo-updates script will also download any updates to SLES from the Novell update server. When all package downloads are complete, the script runs the yume command to update the repository metadata.
Once the script completes, the local package repositories on the admin node should contain the latest available package updates and be ready to use with the yum(8) command to update node images.
| Note: If you manually add updates to any of the local package repositories on the admin node, remember to run the yume --prepare --repo command to update the package repository metadata. Failure to do so, will cause the yum command to report checksum failures |
“Update the SLES Package Repository” contains further details on using the sync-repo-updates script to get SLES package updates. The sync-repo-updates script will only try to get updates if certain conditions are met, so it's safe to use the script to update the local SGI Tempo and SGI ProPack repositories before completing the tasks listed in that section.
As described in “Update the SGI Package Repositories on the Admin Node”, it is possible to download updates for SUSE Linux Enterprise Server to the local SLES package repository on the admin node. Tools like YaST Online Update and rug(1) are designed to update a running system, but not well suited to managing a repository of packages for use within a clustered environment.
You can use the yup(1) tool to mirror the update packages from Novell's update servers. This tool is not provided as part of the SLES10 operating system release, but is provided as part of the SLE10 SDK. SGI recommends that you use the latest version of yup before attempting to mirror SLES10 SP1 updates. SGI tested with version 222-2.4, which can be downloaded directly from Novell and installed on the admin node with the following steps:
Point your web browser to support.novell.com and login using your Novell account username and password.
Select Download - Patches in the Self Support section.
Search for yup in the Linux Updates and Patches section.
Click the Download button for the entry labeled as Recommended update for yup x86_64 07/10/07; the file name is yup-2220-2.4.noarch.rpm
Click the proceed to download button and then the download button.
As root user on the admin node, run the following command:
# rpm -ivh yup-2220-2.4.noarch.rpm
The yup tool stores packages downloaded from the Novell update server in a directory structure that is not compatible with the local SGI Tempo, SGI ProPack and SLES package repositories located on the admin node. After yup is run, the packages need to be copied to the appropriate SLES repository.
If you plan to use the SGI-supplied /opt/sgi/sbin/sync-repo-updates script to keep repositories up to date, you will see that it copies the packages retrieved via yup in to the appropriate SLES package repository and then updates the repository metadata. If you plan to use your own customized scripts, please use the sync-repo-updates script, as an example.
Before you can download packages via yup, you must register the admin node with Novell. The following steps explain how to register the admin node with Novell using YaST, although the general steps are the same for graphical yast2:
Run the /sbin/yast command and select Software -> Novell Customer Center Configuration
You will be prompted to enter your email address and registration code, if you have it.
Note: If you do not have your registration code(s) handy, you may proceed and Novell will provide you with temporary access; however, you will need to register your SLES purchase in the Novell Customer Center (http://www.novell.com/center ) in order to continue to get access to updates. If you already have a Novell account, use the email address associated with your Novell account here.
Sometimes the registration process will error out with a "failed to contact the server" message. This is due to an issue with unknown/untrusted key error caused by the manner in which the initial admin node images are created. Repeat step 3, you will not have to fill in the field forms again, and then yast will prompt you to import the keys. Once the keys are imported, you should be able to complete this step.
As part of the registration process, yast will add an install source for updates. DO NOT TRY TO USE YaST ONLINE UPDATE. At the end of the Novell Customer Center Configuration module, yast will display information on how to create a Novell account. If you do not already have a Novell account, go create one per the instructions.
Now that you have created a Novell account and registered your admin node with Novell, you can configure yup to mirror the SLES update packages. The following instructions cover only the essential steps required to mirror the SLES updates; the yup man pages have more detailed information on available options:
Edit /etc/sysconfig/yup file and modify the following variables:
YUP_DEST_DIR
Set this variable to /var/spool/yup-updates. This value is referenced by the /opt/sgi/sbin/sync-repo-updates script.
YUP_ID
Paste the contents of the /etc/zmd/deviceid file into this variable.
YUP_PASS
Paste the contents of the /etc/zmd/secret file into this variable.
YUP_ARCH
Set this variable to x86_64.
YUP_SUBVERSIONS
Set this variable to SP1.
Create the yup directory referenced above with the following command:
# mkdir /var/spool/yup-updates
The yup command downloads packages to the /var/spool/yup-updates directory. The sync-repo-updates script moves the SLES packages from that directory into the /tftpboot/distro/sles-10-x86_64 directory so that the local SLES package repository on the admin node contains the updated packages.
Execute the /opt/sgi/sbin/sync-repo-updates script or run yup manuall.
The sync-repo-updates script should now be able to download SLES updates directly from the update server of Novell and update the local SLES package repository on the admin node so that you can use the yum command to install or update SLES packages throughout the node.
Note: The variables in /etc/sysconfig/yup are important because by default you will only have access to the SLES updates for x86_64 by default. Setting YUP_ARCH or YUP_SUBVERSIONS variables incorrectly can lead to 403/permission denied errors. The yup may also error out when accessing SLES10-SP1-Online files; it is safe to ignore that error.
To install updates on the admin node, run the following command as root on the admin node:
# yum update |
Because SGI pre-installs certain package updates on systems before they leave the factory, there may not be any package updates updates for the admin node.
To install the SGI Altix ICE System Administrator's Guide and other SGI ProPack related documentation, run the following command:
# yum install sgi-propackdocs |
This sections explain how to configure existing node images in the /var/lib/systemimager image directories and node images in use on live nodes so that you can install update packages for the SGI Tempo, SGI ProPack and SLES products.
The following sections explain how to update leader, service and compute node images in the /var/lib/systemimager image directories with newer packages, how to update leader and service node images in use on live nodes with new packages and the additional steps required when updating the kernel package on compute node images.
If you want to quickly install the latest updates to the node images already running on a live system, start with “Update Packages on Running Leader and Managed Service Nodes”. If you would rather install the latest update packages to node images in the image directories first and then redeploy those images to the system nodes, go to “Update Packages Inside Images”.
It is possible to update live images running on leader and managed service nodes using the yum command.
| Note: Changes made to live node images are not reflected in the node images in the /var/lib/systemimager/images/{image}-sles10sp1 image directories. This means that changes made to live images may not get carried forward the next time you add or re-discover leader and/or service nodes. |
The following examples show how to update a live leader node and/or managed service node using the yum command:
# ssh r1lead yum update (update live leader node) # ssh service0 yum update (update live service node) |
SGI provides the yum-image-wrapper script, which makes using the yum command inside an image directory fairly straight forward. You can use the script directly, or use the script as a template for a more customized solution.
| Note: Changes to the kernel package inside the compute image directory require some additional steps before the new kernel can be used on compute nodes (see section "5.3 Additional Steps for Compute Image Kernel Updates" for more details). This note does not apply to leader or managed service nodes. |
The following examples show how to upgrade the packages inside three SGI- supplied node images :
# yum-image-wrapper /var/lib/systemimager/images/lead-sles10sp1 update # yum-image-wrapper /var/lib/systemimager/images/service-sles10sp1 update # yum-image-wrapper /var/lib/systemimager/images/compute-sles10sp1 update |
| Note: Changes to the compute image on the admin node are not seen by the compute nodes until the updates have been pushed to the leader nodes with the cimage command. Updating leader and managed service node images ensure that the next time you add or re-discover a leader or service node, it will already contain the updated packages. |
While not recommended, it is possible to manually perform all the steps the yum-image-wrapper does for you. The following series of steps will walk you through the manual process:
Use the chroot command to start a shell within the image.
Perform the following steps:
Add an entry for the admin node to /etc/hosts in the compute image; without this, the yum command will fail inside the chroot; this issue only affects compute images, not leader or service node images.
Mount /proc in the chroot; the yum command requires this.
Mount /sys in the chroot; certain packages require this before they can be installed/updated.
Create a dummy /etc/fstab file; this is required when updating kernel packages; mkinitrd , which runs whenever a kernel package is updated, requires /etc/fstab in order to complete.
Use the yum command to update the packages in the chroot.
| Note: Before pushing the compute image to the leaders using the cimage command, it is good idea to clean the yum cache. This cache can grow and is in the writable portion of the compute blade image. This means it is replicated 64 times per compute blade image per rack and the space that may be used by compute blades is limited by design to minimize network and load issues on rack leaders. |
To clean the yum cache, from the system admin controller (admin node), perform the following:
# yum-image-wrapper /var/lib/systemimager/images/compute-sles10sp1 clean all |
It is also possible to use standard rpm (8) commands to update images. This can be quite cumbersome depending on the number of images you need to update and the number of packages you intend to update. The steps are as follows:
Copy the update packages you want to use in to the image directory in /var/lib/systemimager/images/{image}-sles10sp1.
Use the chroot command to start a shell within the root tree.
Install/update packages using standard rpm commands; certain packages require /sys to be mounted in the chroot, so you may have to mount /sys in order to work around any failures.
In general, SGI recommends using the yum-image-wrapper script to update the node images in the image directory.
Any time a compute image is updated with a new kernel, you will need to run some additional steps in order to make the new kernel available. The following example assumes that the compute node image name is compute-sles10sp1 and that you have already updated the compute node image in the image directory per the instructions in “Update Packages Inside Images”. If you have named your compute node image something other than compute-sles10sp1, replace this in the example that follows:
Shut down any compute nodes that are running the compute-sles10sp1 image (see “Power Management Commands” in Chapter 3).
Push out the changes with the cimage --push-rack command, as follows:
# cimage --push-rack compute-sles10sp1 r\*
Update the database to reflect the new kernel in the compute-sles10sp1, as follows:
# cimage --update-db compute-sles10sp1
Verify the available kernel versions and select one to associate with the compute-sles10sp1 image, as follows:
# cimage --list-images
Associate the compute nodes with the new kernel/image pairing, as follows:
# cimage --set compute-sles10sp1 2.6.16.46-0.12-smp "r*i*n*"
Note: Replace 2.6.16.46-0.12-smp with the actual kernel version.
Reboot the compute nodes with the new kernel/image.
For information on upgrading your system from SGI ProPack 5 Service Pack 4 to SGI ProPack 5 Service Pack 5, see the release notes. The SGI ProPack 5 SP5 release notes can be found in a file named README.TXT that is available in /docs directory on the SGI ProPack 5 for Linux Service Pack 5 CD.
The SGI ProPack 5 for Linux release notes get installed to the following location on a system running SGI ProPack 5 SP5: /usr/share/doc/packages/sgi-propack-5/README.txt