The SGI Altix ICE 8200 system is a blade-based, scalable, high density compute system. The basic building block is the individual rack unit (IRU). The IRU provides power, cooling, system control, and the network fabric for 16 compute blades, as shown in Figure 1-1. Each compute blade supports two either dual-core or quad-core Xeon processor sockets and eight fully-buffered, double-data-rate two (DDR2) memory dual in-line memory module (DIMMs). Four IRUs can reside in a custom designed 42U high rack.

One rack supports a maximum of 512 processor cores and 2TB of memory.

Figure 1-1. Basic System Building Blocks

InfiniBand Fabric

The SGI Altix ICE 8200 system topology is based on an InfiniBand interconnect. Internal InfiniBand switch ASICs of the IRU eliminate the need for external InfiniBand switches. The dual high-speed, low-latency double data rate (DDR) InfiniBand backplanes built into the IRUs provide for fast communication between nodes and racks.

An InfiniBand switch blade provides the interface between compute blades within the same chassis and also between compute blades in separate IRUs. Fabric management software monitors and controls the InfiniBand fabric. SGI Altix ICE 8200 systems are configured with two InfiniBand fabrics, designated as ib0 and ib1. In order to maximize performance, SGI advises that the ib0 fabric be used for all MPI traffic, in this case, MVAPICH MPI. The ib1 fabric is reserved for storage related traffic. The default configuration for MVAPICH MPI is to use only the ib0 fabric. For more information on the InfiniBand fabric, see Chapter 4, “System Fabric Management”. For more information on MVAPICH MPI, see Chapter 6, “MVAPICH MPI”.

---

Note: The “ib0 fabric" is a convenient shorthand for "the fabric which is connected to the ib0 interface on most of the nodes”. In the case of the storage service node, there are four interfaces called ib0 through ib3, all of which are connected to the ib1 fabric (see “Storage Service Node ” and “NAS Configuration for Multiple IB Interfaces” in Chapter 2).

---

Four-tier, Hierarchical Framework

The SGI Altix ICE 8200 system has a unique four-tier, hierarchical management framework as follows:

• System admin controller (admin node) - one per system

• Rack leader controller (leader node) - one per rack

• Chassis management controller (CMC) - one per IRU

• Baseboard Management Controller (BMC) - one per compute node, admin node, leader node, and managed service node

Unlike traditional, flat clusters, the SGI Altix ICE 8200 system does not have a head node. The head node is replaced by a hierarchy of nodes that enables system resources to scale as your add processors. This hierarchy is, as follows:

• System admin controller (admin node)

• Rack leader controller (leader node)

• Service Nodes

  • Login

  • Batch

  • Gateway

  • Storage

The one system admin controller can provision and control multiple leader nodes in the cluster. It receives aggregated cluster management data from the rack leader controllers (leader nodes).

Each system rack has its own leader node. The leader node holds the boot images for the compute blades and aggregates cluster management data for the rack.

Ethernet traffic for managing the nodes in a rack is constrained within the rack by the leader node. Communication and control is distributed across the entire cluster avoiding the admin node becoming a communication bottleneck. Administrative tasks, such as booting the cluster, can be done in parallel rack-by-rack in a matter of seconds. For very large configurations, the access infrastructure can also be scaled by adding additional login and batch service nodes. It is the VLAN logical networks that help prevent network traffic bottlenecks.

---

Note: Understanding the VLAN logical networks is critical to administering an SGI Altix ICE system. For more detailed information, see “VLANs” and “Network Interface Naming Conventions”.

---

The rack leader controller (leader node) and system admin controller (admin node) are described in the section that follows (“System Nodes”).

Chassis Manager

Figure 1-2 shows chassis manager cabling.

---

Note: All nodes reside in the Altix ICE custom designed rack. Figure 1-2 and Figure 1-3 show how systems are cabled up prior to shipment. These figures are meant to give you a functional view of the Altix ICE hierarchical design. They are not meant as cabling diagrams.

---

The chassis manager in each rack connects to the leader node in its own rack and also the chassis manager in the adjacent rack. The system admin controller (admin node) connects to one leader node in the rack. The system admin controller accesses the BMC on each compute node in the rack via VLAN running over a Gigabit Ethernet (GigE) connection (see Figure 1-7).

Figure 1-2. Chassis Manager Cabling

Figure 1-3 shows cabling for a service node and storage service node (NAS cube).

This section describes the system nodes that are part of SGI Altix ICE 8200 system and covers the following topics:

Chassis Management Control (CMC) Blade

---

Note: The following CMC description is the same as the information presented in “Basic System Building Blocks”.

---

Each IRU has a one chassis management control (CMC) blade located directly below compute blade slot 0 as shown in Figure 1-1. This is the chassis manager that performs environmental control and monitoring of the IRU. The CMC controls master power to the compute blades under direction of the rack leader controller RLC (leader node). The RLC can also query the CMC for monitored environmental data (temperatures, fan speeds, and so on) for the IRU. Power control for each blade is handled by the Baseboard Management Controller (BMC) also under direction of the rack leader controller. Once the RLC has asked the CMC to enable master power, the RLC can then command each BMC to power up its associated blade. The RLC can also query each BMC to obtain some environmental and error log information about each blade.

Compute Node

Figure 1-1 shows an IRU with 16 compute nodes. Users submit MPI jobs to run in parallel on the Altix ICE system compute nodes using a public network connection via the service node. The service node provides login services and a batch scheduling service, such as PBS Professional or Torque (OpenPBS), as shown in Figure 1-4. The compute nodes are controlled and monitored by the leader node to their rack as shown in Figure 1-2. Compute nodes are booted and mount the shared, read-only portion of the root file system from the rack leader controller (leader node). The leader node provides the network connections to the compute nodes in the same rack via the InfiniBand fabric and to compute nodes in other racks via the InfiniBand fabric connections to other leaders nodes. The system admin controller does not communicate directly with the CMC or compute blades. Actions for the CMC and compute blades are sent to the appropriate rack leader controller, which communicates to the appropriate CMC and compute blades. The compute nodes do not communicate directly to the CMC or admin nodes, or RLCs outside their rack.

Generally, the CMC controller is not meant to be accessed directly by system administrators, however, in some situations you may need to access it to change a configuration using the LCD control panel. For example, if you added a NAS cube to your system you need to reconfigure the CMC.

---

Note: The LCD control panel is not operational for the first release.

---

Storage Service Node

The storage service node is a network-attached storage (NAS) appliance bundle that provides InfiniBand attached storage for the Altix ICE system. There can be multiple storage service nodes for larger Altix ICE system configurations. Figure 1-3 shows a service node and storage service node (NAS cube).

---

Note: All nodes reside in the Altix ICE custom designed rack. Figure 1-2 and Figure 1-3 how systems are cabled up prior to shipment. These figures are meant to give you a functional view of the Altix ICE hierarchical design. They are not meant as cabling diagrams.

---

Figure 1-3. Service Nodes

This section describes the various network connections in the SGI Altix ICE 8200 system. Users access the system via a public network through services nodes such as the login node and the batch service node, as shown in Figure 1-4. A single service node can provide both login and batch services.

System administrators provision (install software) and manage the Altix ICE system via the logical VLAN network running over the GigE connection (see Figure 1-6, Figure 1-7, and Figure 1-8. The system admin controller (admin node) is on the house network (public network) and you access it directly.

The rack leader controller (leader node) provides boot and root filesystem images for the compute nodes in the same rack. The RLC is connected to blades in its rack via the GigE VLAN. It is connected to all blades and service nodes via InfiniBand fabric.

The gateway service node is the gateway from the InfiniBand fabric to services such as storage, lightweight directory access protocol (LDAP) services, file transfer protocol (FTP), and so on, on the public network. Typically, it is combined with the login/batch service node.

The system admin controller (admin node) and service nodes communicate with the leader node over a GigE fabric that has logically separate, virtual local area networks (VLANs). This GigE fabric is embedded in the backplane of each IRU. This GigE fabric electrically connects much of the Altix ICE system (see Figure 1-4).

Users access compute nodes strictly from the service nodes. Jobs are started on compute nodes using commands on the service node, such as, the OpenSSH client remote login program ssh (1), the submit a script to create a batch job qsub(1) command, or the Cluster Command Control (C3) tool cexec(1) utility that enables the execution of any standard command on all Altix ICE system nodes.

Figure 1-4. Network Connections In a System With Two IRUs

You can use the interconnect verification tool (IVT) to verify that all the various 10/100 Ethernet, Gigabit Ethernet (GigE), and InfiniBand (IB) network links between the various system admin controllers (admin nodes), such as the admin or login node, the leader node, the compute nodes, the CMC and the BMC nodes are correctly connected and working properly after a system is installed or for maintenance purposes. For more information on IVT, see “Inventory Verification Tool” in Chapter 5.

The SGI Altix ICE 8200 system has several Ethernet networks that facilitate booting and managing the system. These networks are built onto the backplane of each IRU for connection to the compute blades and transverse cables between IRUs and between racks. Each compute blade has a Gigabit Ethernet (GigE) and 10/100 Ethernet connection to the backplane.

The GigE connection is an interface that is accessible to the operating system and the basic input/output (BIOS) running on the blade. It is the interface over which the BIOS uses the preboot execution environment (PXE) to PXE boot and it is eth0 to the Linux kernel.

The 10/100 Ethernet interface is accessible to the management interface (BMC) built onto each compute blade. The operating system running on the blade cannot directly access this 10/100 interface. It belongs to the processor on the BMC. Likewise, the BMC cannot access the GigE interface.

Figure 1-5 shows a more detailed view of the Chassis manager.

Figure 1-5. Chassis Manager

The chassis management control (CMC) blade has two embedded Ethernet switches . One is a 24-port GigE switch and the other a 24-port 10/100 switch. The 10/100 switch is a sub-switch (hanging off one port of) the GigE switch.

The primary GigE interface from each of sixteen blades connects to the GigE switch and the sixteen blade BMCs connect to the 10/100 switch. The GigE connections also connect the service nodes and service storage nodes.

The GigE switches in each IRU is "stacked" using a special stacking connection between each IRU in a rack. This connection runs a special intra-switch protocol. All switches in a rack are ganged together to form one large 96 port switch. The connections from each CMC to another are labeled UP and DN as shown in Figure 1-5. The switches are stacked in a ring so failure of one link still allows traffic to flow in the opposite direction on the ring.

The processor on the CMC manages these switches effectively forming a large, intelligent Ethernet switch. A VLAN mechanism runs on top of this network to allow management control software to query port statistics and other port metrics including the attached peer's MAC address.

The CMC has five additional RJ45 connections on its front panel as shown in Figure 1-5. The function of these jacks is, as follows:

A NAS cube storage service node uses both the LL and RL jacks to connect to the Altix ICE system as shown in Figure 1-3.

For small, one IRU configurations, the L58 and R58 ports (see Figure 1-5) can be used to connect service nodes.

The following virtual local area networks (VLANs) are established to optimize the control traffic flow across the entire Ethernet infrastructure:

The rack leader controllers (leader nodes) must run 802.1Q VLAN protocol over their downstream GigE connection to the CMC and the CMC LL port must also run 802.1Q. This is done for you when the rack leader controllers are installed from the system admin controller. For more information, see “Installing Software on the System Admin Controller” in Chapter 2. Each VLAN should present itself as a separate, pseudo interface to the operating system kernel running on that leader node. VLAN _HEAD, VLAN_BMC, and VLAN_GBE must all transition the single Ethernet segment which connects the leader to the CMC in the rack below it.

Figure 1-6. VLAN_GBE and VLAN_BMC Network Connections - IRU View

The VLAN_GBE and VLAN_BMC networks connect the leader node in a given rack with the compute nodes (blades). In the case of VLAN_BMC, the network also connects the CMC with the compute blades and rack leader controller (leader node).

Figure 1-7. VLAN_GBE and VLAN_BMC Network Connections - Rack View

Figure 1-8. VLAN_HEAD Network Connections

In an SGI Altix ICE system with just one IRU, the CMC's R58 and L58 ports are assigned to VLAN_HEAD by a field configurable setting. This provides two additional Ethernet ports that can be use to connect service nodes to your system.

The InfiniBand fabric connects the service nodes, leader nodes, and the compute blades. It does not connect to the admin node or the CMCs. The InfiniBand network has two separate network fabrics, ib0 and ib1. The host channel adapter (HCA) in the leader node has two ports that connect separately to the bottom IRU in the rack.

Figure 1-9. Two InfiniBand Fabrics in a System with Two IRUs

Even though you may be communicating on different VLANs, you may in fact be communicating with the same physical network interface on the system. Naming the logical connections by function allows flexibility to change the number or type of the underlying physical networks. At the topmost level, the admin and service node nodes can communicate with the leader nodes over the VLAN_HEAD virtual network. The system component terms used in this section are described, as follows:

System-wide unique names are node names and non-routable names.

X, Y, and Z in the following tables in this section are all integers.

Table 1-1 shows the VLAN_Head network connection names. See Figure 1-8.

There is one admin node per system. You can have multiple service nodes labelled service0, service1, and so on. The BMC controllers for the service nodes are normally accessible inside the network, however, you can make them accessible on your own external network instead.

Table 1-2 shows the VLAN_GBE network connections.

The GBE VLAN is entirely internal to each rack (see Figure 1-6). The naming scheme is replicated between each rack, so the name i2n4-eth (identifying the VLAN_GBE interface on IRU 2, node 4) may match several different nodes, but only ever one in each rack. To identify a node uniquely, use the rXiYnZ syntax. When more than one GigE interface is present, the names lead-eth1, iYnZ-eth1, and so on, may be used.

Table 1-3 shows the VLAN_BMC network connections.

The BMC VLAN is also local to each rack, in the same way as the GBE VLAN (see Figure 1-6).

Note that the interface lead-bmc on the leader node is not an interface to the BMC on the leader, but rather is an interface on the leader to the VLAN_BMC network in that leaders rack. Software running on other nodes in an Altix ICE system, outside of a given rack, cannot directly address the BMC's, or CMC, within said rack. Rather such requests much go through suitable application level software running on that rack's leader, when can in turn access the BMCs and CMC in its rack, via this lead-bmc interface to the racks VLAN_BMC network.

Connecting to the leader node's BMC is only possible from an admin node, service, or other leader node, when you should use rXlead-bmc .

The CMC does not have a BMC connection, but instead the VLAN_BMC connection is to the CMC's console interface.

Table 1-4 shows the VLAN_1588 network connections.

The 1588 VLAN carries the time synchronization traffic and connects CMCs in all the racks in the Altix ICE system. For this reason, the full rack-qualified name is needed to uniquely identify the target CMC.

Sometimes a rack, an IRU, a blade (node), or a CMC needs to be uniquely identified within the Altix ICE system. Table 1-5 shows the names that may be used for this, but there is no IP address associated with them. Therefore, DNS lookup will not succeed for these names. The names are used by certain Altix ICE management tools and are parsed internally to indicate which leader node to use in order to connect to the destination system.

Hostnames are distinct from the non-routable names and are shown in Table 1-6. In general, this is the name that you get by typing hostname at the command prompt on the system, and is used as a way of identifying the system to the user. Often, the command prompt is set up to contain the hostname. This is a benefit since with multiple windows open to different systems, it allows the user to avoid executing commands in the wrong window.

---

Note: For the first release, hostnames should not be changed. Service hostnames need to match the node name, that is, serviceX.

---

The Infiniband fabric is connected to service nodes, system admin controllers (leader nodes), and compute nodes, but not to the system admin controller (admin node) or CMCs. Table 1-7 shows InfiniBand names. There are two IB connections to each of the nodes that use it. Since IB is not local to each rack, you must use the fully-qualified, system-unique node name when specifying a destination interface. It may be neceassary to alias the rXiYnZ names (currently non-resolvable) to rXiYnZ-ib0 if this is needed by MPI.