The InfiniBand network on SGI Altix ICE 8000 series systems uses Open Fabrics Enterprise Distribution (OFED) 1.2 software. This section describes the InfiniBand fabric and how to manage it. For background information on OFED, see http://www.openfabrics.org.
Fabric management on SGI Altix ICE 8000 series systems uses the OFED 1.2 OpenSM software package. The InfiniBand fabric connects the service nodes, rack leader controllers (leader nodes), and the compute nodes. It does not connect to the system admin controller (admin node) or the chassis management control (CMC) blades. The InfiniBand network has two separate network fabrics, ib0 and ib1 (see “InfiniBand Fabric” in Chapter 1) with the following characteristics:
Each network fabric has its own subnet manager (SM).
For a system with two racks or more, one rack leader controller (leader node) runs an instance of SM to manage the ib0 fabric and a second leader node runs an instance of SM to manage the ib1 fabric.
On a system with a single rack, both instances of opensm run on the same rack leader node.
Each instance of SM on the rack leader controller is controlled by the /etc/opensm-ib0.conf or /etc/opensm-ib1.conf configuration file.
Rack leader controllers run the opensm daemon for each fabric over separate HCA ports (see Figure 1-9).
Note: After a system reboot, you need to manually restart the opensm daemons running on the InfiniBand fabric. If the opensm daemons are allowed to start automatically, as the leader nodes boot, you will not know which leader is the Master and it is highly likely that the fabric will be routed incorrectly. To start the InfiniBand fabric, you can use the following command: scalimanage-cli restartaltixiceopensm
Each fabric is addressed by a global unique identifier (GUID) and unique HCA port.
The GUID and HCA port is set in the configuration file.
Coherency of the fabric database is handled by sldd-ib[01].sh. You must make sure OSM_HOSTS is configured correctly in the /etc/opensm-ib0.conf or /etc/opensm-ib1.conf configuration files.
| Note: Currently, the InfiniBand fabric ib0 is reserved for MPI or interprocess communication traffic and the InfiniBand fabric ib1 is reserved for storage. |
For more information on the InfiniBand fabric, see Appendix A, “InfiniBand Fabric Details” and Appendix B, “InfiniBand Fabric Troubleshooting”.