This chapter provides an overview of the physical and architectural aspects of your Altix 350 system. System configurations and components are described and illustrated. This chapter includes the following sections:
The Altix 350 system is the latest advancement in the SGI NUMAflex approach to mid-range modular computing. It is designed to deliver maximum sustained performance in a compact system footprint. Independent scaling of computational power, I/O bandwidth, and in-rack storage lets you configure a system to meet your unique computational needs. The small footprint and highly modular design of the Altix 350 system makes it ideal for computational throughput, media streaming, or complex data management.
The Altix 350 system can be expanded from a standalone single-module system with 2GB of memory and 4 PCI/PCI-X slots to a high-performance system that contains 32 processors, one or two routers, up to 192 GB of memory, and 64 PCI/PCI–X slots. For most configurations, the Altix 350 system is housed in one 17U rack or one 39U rack as shown in Figure 2-1; however, for small system configurations, the Altix 350 system can be placed on a table top.
Systems that are housed in 17U racks have a maximum weight of approximately 610 lb (277 kg). The maximum weight of systems that are housed in 39U racks is approximately 1,366 lb (620 kg). The racks have casters that enable you to remove the system from the shipping container and roll it to its placement at your site.
See Chapter 1, “Installation and Operation” for more information about installing your system. Check with your SGI service representative for additional physical planning documentation that may be available.
For more information about the technical specifications of your system, see Appendix A, “Technical Specifications” in this manual.
The Altix 350 system is based on the SGI NUMAflex architecture, which is a shared-memory system architecture that is the basis of SGI HPC servers and supercomputers. The NUMAflex architecture is specifically engineered to provide technical professionals with superior performance and scalability in a design that is easy to deploy, program, and manage. It has the following features:
Shared access of processors, memory, and I/O. The Super Hub (SHub) ASICs and the NUMAlink-4 interconnect functions of the NUMAflex architecture enable applications to share processors, memory, and I/O devices.
Each SHub ASIC in the system acts as a memory controller between processors and memory for both local and remote memory references.
The NUMAlink interconnect channels information between all the modules in the system to create a single contiguous memory in the system of up to 384 GB and enables every processor in a system direct access to every I/O slot in the system.
Together, the SHub ASICs and the NUMAlink interconnect enable efficient access to processors, local and remote memory, and I/O devices without the bottlenecks associated with switches, backplanes, and other commodity interconnect technologies.
System scalability. The NUMAflex architecture incorporates a low-latency, high-bandwidth interconnect that is designed to maintain performance as you scale system computing, I/O, and storage functions. For example, the computing dimension in some system configurations can range from 1 to 32 processors in a single system image (SSI).
Efficient resource management. The NUMAflex architecture is designed to run complex models and, because the entire memory space is shared, large models can fit into memory with no programming restrictions. Rather than waiting for all of the processors to complete their assigned tasks, the system dynamically reallocates memory, resulting in faster time to solution.
The Altix 350 system can be configured using a ring topology, as described in Figure 2-2. In a ring topology, the components that make up the system are connected in a closed-loop fashion; the components have a direct connection with two neighboring components and indirect connection with the other components in the loop. For example, component 1 in Figure 2-2 can communicate with component 2 and 8 directly; however, to communicate with component 4, the message passes through components 2 and 3. Note that the diagram is for explanatory purposes and does not represent any actual configuration. For more detailed information about this type of configuration, see the section “System Configurations”.
The Altix 350 system can be optionally configured using one or two NUMAlink-4 routers. This is also referred to as single-plane or dual-plane router configuration. Using routers, eight to sixteen modules can be interconnected for a system with up to 32 processors in a single system image.
In a single-plane router configuration a single NUMAlink-4 connection is made to the port 0 connector on the back of (up to) eight Altix 350 modules. Additional modules (9 thru 16) are connected to the open NUMAlink-4 port (port 1) on the modules already connected to the router. See Figure 2-3 for an example diagram of single-plane router connection.
In a dual-plane configuration a NUMAlink-4 connection is made to the NUMAlink ports (port 1) on the back of two modules in the configured system. These connections are made from port A on the back of each of two NUMAlink-4 routers. The two Altix 350 modules are then joined together by routing a NUMAlink cable between the two open ports (port 0). This creates a dual-communication pathway between the modules, the routers and other modules in the configuration. See Figure 2-4 for an example diagram of a dual-plane router system.
The components shown in Figure 2-2 are 2U modules that can be configured as one system using the following devices:
Base compute module. All Altix 350 systems contain at least one base compute module that contains the following components:
One or two Intel Itanium 2 processors; each processor has integrated L1, L2, and L3 caches
Up to 24 GB of local memory
Four PCI/PCI–X slots
One base I/O PCI card that comes factory-installed in the lowermost PCI/PCI–X slot
Note: Each system or partition requires one base I/O PCI card. This card provides the base I/O functionality for the system. Additional I/O cards are required if you want additional hard disk drives and/or DVD-ROM drives in additional modules. These cards must reside in additional base compute modules (one card per module). The system base I/O card has a real time interrupt input port and output port, an Ethernet port, and a either a serial port or SCSI connector (the IO10 has serial, the IO9 has SCSI. ). The base I/O card is also needed to support a base module's system disk drive(s), optional DVD–ROM, and a console serial port.
Note: The RT interrupt input and RT interrupt output functionality of either base I/O PCI card is not supported under SGI Linux + ProPack. One SHub ASIC (the crossbar between the processors, local memory, the network interface, and the I/O interface).
For more information about the base compute module, see Chapter 3, “Base Compute Module”.
CPU Expansion module. The only difference between the base compute module and the CPU expansion module is that the CPU expansion module does not contain any usable PCI/PCI-X slots or disk drives. For more information about the CPU expansion module, see Chapter 4, “CPU Expansion Module”.
CMPX module. The only difference between the base compute module and the CMPX module is that the CMPX module does not contain a base I/O PCI card or disk drives. For more information about the CMPX module, see Chapter 5, “CMPX Module”.
When the system consists of a base compute module only, the maximum number of processors is 2 and the maximum amount of memory is 24 GB. To increase the number of processors and/or memory in the system, the base compute module can connect to additional CPU expansion and/or CMPX modules.
Table 2-1 lists the minimum and maximum ranges of the configurable items for the Altix 350 system.
Table 2-1. Altix 350 Configuration Ranges
| Configuration | Configuration |
|---|---|---|
Compute modules | 1 | 16 |
Processors | 1 | 32 |
Peak performance of processors: 1.5 GHz |
~ 6.0 GFLOPS | Peak performance will vary over time based on new technology available; check with your SGI sales or service representative. |
Memory capacity | 2 GB | 384 GB |
Internal disk storage | One or two disks | 16 to 32 disk drives |
Internal PCI/PCI–X slots[a] | 4 | 64 |
External storage devices | None | Customer-configurable |
Compute rack | None | 2 |
L2 controller | None | 1 |
[a] Each base compute module has four internal PCI/PCI–X slots; however, one slot is required for the base IO PCI card. Therefore, the number of available slots in the base compute module is reduced by one. | ||
As described in “Functional Architecture” some Altix 350 systems use a type of network configuration that is referred to as a ring topology. As the name implies, the network connection between the base compute modules, CPU expansion modules, and the CMPX modules forms a ring. A message is passed around the ring until it reaches its destination.
The data flow of this ring topology flows in both directions, enabling the modules to have direct connection to two other modules and providing an alternative path when a connection fails between two modules.
The bisection bandwidth of the ring depends on the number of modules on the ring; the bisection bandwidth is greatest when there are only two modules on the ring. See Table 2-2.
Table 2-2. Bisection Bandwidth of Ring Topology
Module Count | Bisection Bandwidth |
|---|---|
2 | 3.2 GB/s per processor |
3 | 2.13 GB/s per processor |
4 | 1.6 GB/s per processor |
5 | 1.28 GB/s per processor |
6 | 1.06 GB/s per processor |
7 | 914 MB/s per processor |
8 | 800 MB/s per processor |
Figure 2-5 through Figure 2-11 provide some examples of the ring topology.
Altix 350 systems with eight or more modules can optionally be connected using one (single-plane) or two (dual-plane) router modules.
The router module or R-brick is an eight-port 2U high module that functions as a high-speed switch to route network packets between Altix 350 modules within a system. This creates a NUMAlink interconnect fabric (as opposed to a ring topology which is normally used in smaller system configurations). The ring topology interconnect method is described in the previous section (“Ring Topology Configurations”). For more information on router module features, see Chapter 6, “Optional Router”.
Figure 2-3 and Figure 2-4 show the single and dual router configuration concepts from a block diagram perspective.
Table 2-3 describes the cable connections made in a single-router configuration with a maximum of 16 modules. An example of a single-plane (single router) system interconnect topology is shown in Figure 2-12.
Table 2-3. Single Plane Router Cable Connection Descriptions
1st NUMAlink Connection point | 2nd NUMAlink Connection point | Description |
|---|---|---|
From Router connector (A) | To (A) module 5 NUMAlink 1 | First NUMAlink-4 cable connection (A to A) |
From Router connector (B) | To (B) module 6 NUMAlink 1 | Second NUMAlink-4 cable connection (B to B) |
From Router connector (C) | To (C) module 7 NUMAlink 1 | Third NUMAlink-4 cable connection (C to C) |
From Router connector (D) | To (D) module 8 NUMAlink 1 | Fourth NUMAlink-4 cable connection (D to D) |
From Router connector (E) | To (E) module 9 NUMAlink 1 | Fifth NUMAlink-4 cable connection (E to E) |
From Router connector (F) | To (F) module 10 NUMAlink 1 | Sixth NUMAlink-4 cable connection (F to F) |
From Router connector (G) | To (G) module 11 NUMAlink 1 | Seventh NUMAlink-4 cable connection (G to G) |
From Router connector (H) | To (H) module 12 NUMAlink 1 | Eighth NUMAlink-4 cable connection (H to H) |
From module 9 NUMAlink 0 (I) | To module 13 NUMAlink 0 (I) | Ninth NUMAlink-4 cable connection (I to I) |
From module 10 NUMAlink 0 (J) | To module 14 NUMAlink 0 (J) | Tenth NUMAlink-4 cable connection (J to J) |
From module 11 NUMAlink 0 (K) | To module 15 NUMAlink 0 (K) | Eleventh NUMAlink-4 cable connection (K to K) |
From module 12 NUMAlink 0 (L) | To module 16 NUMAlink 0 (L) | Twelfth NUMAlink-4 cable connection (L to L) |
From module 1 NUMAlink 0 (M) | To module 5 NUMAlink 0 (M) | Thirteenth NUMAlink-4 cable connection (M to M) |
From module 2 NUMAlink 0 (N) | To module 6 NUMAlink 0 (N) | Fourteenth NUMAlink-4 cable connection (N to N) |
From module 3 NUMAlink 0 (O) | To module 7 NUMAlink 0 (O) | Fifteenth NUMAlink-4 cable connection (O to O) |
From module 4 NUMAlink 0 (P) | To module 8 NUMAlink 0 (P) | Sixteenth NUMAlink-4 cable connection (P to P) |
In a dual-plane (two router) system every module is connected directly to one of the two routers. Each module is then interconnected to another module to create a high-speed NUMAlink fabric interconnection. Table 2-4 lists the cable interconnects as they relate to a 32 processor (16 module) maximum configuration system. An example of a dual-plane (two router) system interconnect topology is shown in Figure 2-13.
Table 2-4. Dual Plane Cable Interconnect Points
1st NUMAlink cable | 2nd NUMAlink cable | 3rd NUMAlink cable |
|---|---|---|
From 1st router port 1(A) | From module 5 port 0 (M) | From module 1 port 1 (A1) |
From 1st router port 2 (B) | From module 6 port 0 (N) | From module 2 port 1 (B1) |
From 1st router port 3 (C) | From module 7 port 0 (O) | From module 3 port 1 (C1) |
From 1st router port 4 (D) | From module 8 port 0 (P) | From module 4 port 1 (D1) |
From 1st router port 5 (E) | From module 9 port 0 (I) | From module 13 port 1 (E1) |
From 1st router port 6 (F) | From module 10 port 0 (J) To module 14 port 0 (J) | From module 14 port 1 (F1) |
From 1st router port 7 (G) | From module 11 port 0 (K) | From module 15 port 1 (G1) |
From 1st router port 8 (H) | From module 12 port 0 (L) | From module 16 port 1 (H1) |
In order to be able to efficiently monitor and direct all the L1 controllers in a dual-router Altix 350 system, an optional Ethernet switch is available. This switch allows connection of a laptop or workstation using SGI's optional L3 controller software.
The connection is made by way of an Ethernet to USB adapter that plugs into the rear of each of the router modules. The adapters plug into the router's USB connections and then each adapter is connected to the Ethernet switch mounted at the top of the rack.
The switch connects to a public or local area Ethernet. You can then connect a system console with optional L3 controller software to that Ethernet to monitor and control the dual-plane Altix 350 system remotely.
For information on using the controller software, see the SGI L1 and L2 Controller Software User's Guide (P/N 007-3938-xxx). This guide describes how to use the L1 and L2 controller commands at your system console to monitor and manage your SGI system.
Figure 2-14 shows an example diagram of a system with two routers connected to an Ethernet switch.
This section briefly describes the major system components of an Altix 350 system, in the following subsections:
The base compute module is a 2U AC-powered device that consists of the following:
One or two Intel Itanium 2 processors; each processor has integrated L1, L2, and L3 caches
Up to 24 GB of memory.
One to four PCI/PCI–X cards.
Note: At least one base compute module comes factory-installed with a base I/O card in the bottom PCI/PCI-X slot. One or two sled-mounted disk drives (at least one disk drive is required in the system).
DVD-ROM (optional).
The system disk drives and the DVD-ROM require a base I/O card to function.
Each base compute module also contains an L1 controller that provides the following services:
Controls and sequences power.
Controls and monitors the environment.
Initiates a reset.
Stores identification and configuration information.
Figure 2-15 shows the front and rear views of a base compute module. See Chapter 3, “Base Compute Module” for more information about this module.
The CPU expansion module is a 2U AC-powered device that consists of the following:
One or two Intel Itanium 2 processors; each has integrated L1, L2, and L3 caches
Up to 24 GB of memory.
One L1 controller that provides the following services:
Controls and sequences power.
Controls and monitors the environment.
Initiates a reset.
Stores identification and configuration information.
Figure 2-16 shows the front and rear views of a CPU expansion module.
The CMPX module is a 2U AC-powered module that offers the following options:
Zero, one or two optional processors
Up to 24 GB of optional memory
Four PCI/PCI-X slots (see Figure 2-17)
One L1 controller that provides the following services:
Controls and sequences power.
Controls and monitors the environment.
Initiates a reset.
Stores identification and configuration information.
See Chapter 5, “CMPX Module” for more information about the CMPX module.
Each base compute module contains a base I/O card and two disk-drive bays. You can add additional storage to the system as follows:
For a SCSI (small computer system interface) JBOD (just a bunch of disks) solution, SGI offers the TP900 storage module, that can be added to base compute modules or CMPX expansion modules (with optional IO9 or other optional SCSI PCI cards).
For a Fibre Channel solution that supports both JBOD and RAID configurations, SGI offers the 2Gb SGI TP9100 storage system.
For Fibre Channel RAID solutions, SGI offers the SGI TP9400 storage system and the SGI TP9500 or TP9500S storage system family. Check with your SGI sales or service representative for additional RAID storage systems available.
The server system also supports a number of tape devices; check with your SGI sales or support representative for available options.
The various storage devices are discussed in the subsections that follow.
The TP900 storage module, shown in Figure 2-18, is a 2U-high 8-drive storage system that provides compact, high-capacity, high-availability JBOD storage. The enclosure backplane connects the 8 drives on one SCSI bus. As an option, the storage module can also be configured on two SCSI buses (2 strings of 4 drives).
This storage module has the following features:
It mounts in a standard 19-inch rack; it is available in factory-installed configurations.
It uses SCSI Parallel Interface 3 (SPI-3) capable Low Profile (1-inch high) 3.5-inch disk drives.
Its drive carriers accept SGI-qualified 10,000- or 15,000-RPM SCSI disk drives.
For more information about the TP900 storage module, see SGI Total Performance 900 Storage System User's Guide (007-4428-00x).
The 2Gb SGI TP9100, shown in Figure 2-19, is an affordable, entry-level RAID storage array that is easily expandable and comes in either a deskside tower or a rackmounted configuration. You can start with a basic JBOD configuration and later add RAID controllers, or you can start with a RAID configuration.
The 2Gb SGI TP9100 storage system connects to base compute and/or CMPX modules via a Fibre Channel PCI card. For more information about the SGI TP9100 storage system, see SGI Total Performance 9100 (2 Gb TP9100) Storage System User's Guide (007-4522-00x).
The SGI TP9400, shown in Figure 2-20, and the SGI TP9500 and TP9500S are highly scalable RAID storage subsystems. These storage systems have vast storage capacities and can grow to whatever size you require without disruption to normal processing activities. This continuous availability enables all active components to be configured redundantly and installed “hot” as customer-replaceable or expansion units.
The TP9400 and TP9500 family of storage systems connect to base compute modules and/or CMPX modules via Fibre Channel PCI cards.
For more information about the TP9400 and TP9500 family of storage systems, see the SGI InfiniteStorage TP9400 and SGI InfiniteStorage TP9500 and TP9500S RAID User's Guide (007-4304-00x).
The L2 controller (see Figure 2-21), which is an optional component, is a rack-level controller that performs the following functions:
Controls resource sharing.
Controls L1 controllers.
Maintains system configuration and topology information.
Enables remote maintenance.
Routes data between upstream and downstream devices, as follows:
Upstream devices (for example, the system console) provide control for the system, initiate commands for the downstream devices, and act on the messages that they receive from downstream devices.
Downstream devices (for example, L1 controllers) perform the actions specified by the L2 controller commands, send responses to the L2 controller that indicate the status of the commands, and send error messages to the L2 controller.
All components within a rack that have an L1 controller can connect to the L2 controller (see Figure 2-21). For example, base compute, CPU expansion, and CMPX, modules can connect to the L2 controller directly.
The Altix 350 system can consist of the following power components:
One or two power distribution units (PDUs) (see Figure 2-22). The second PDU is added to the system when more than 10 AC power receptacles are needed within the rack.
The PDU inputs AC voltage from an external power receptacle and it can output AC voltage to the base compute modules, CPU expansion modules, CMPX modules and TP900 storage modules.
Figure 2-22 shows the power connections for a sample Altix 350 system.
The Altix 350 system supports two rack types: a short rack and a tall rack. The racks are measured in standard units (U); one U is equal to 1.75 in. (4.45 cm). The short rack is a 17U rack and the tall rack is a 39U rack (see Figure 2-23).
The components within the rack are identified by the lowest U number that they occupy. For example, the top CMPX module shown in Figure 2-23 is identified as U7 in the short rack and U15 in the tall rack.
Both rack types are industry-standard 19-inch racks, and they support two types of mounting rails (optional slide rails and shelf rails) that hold the modules within the rack. For example, the base compute, CPU expansion, and CMPX modules can use shelf rails or optional slide-mounting rails (see Figure 2-24). The TP900 storage modules always use shelf rails, which are two parallel L-shaped mounting rails within the rack (see Figure 2-25).
Both rack types, as shown in Figure 2-26, have front and rear doors that have keylocks to prevent unauthorized access of the system. The racks also have cable entry/exit areas at the bottom of the racks. The 39U racks have cable management hardware in the rear.
Both rack types are mounted on four casters, two of which are swivel casters. The casters enable the rack to be rolled out of a shipping crate and to its placement at your site.
The base of the racks have seismic tie-down attachment points. The base of the tall rack also has leveling pads.