Chapter 3. System Fabric Management
Prev |  | Next

## Chapter 3. System Fabric Management

The InfiniBand network on SGI Altix ICE 8000 series systems uses Open Fabrics Enterprise Distribution (OFED) 1.2 software. This section describes the InfiniBand fabric and how to manage it. For background information on OFED, see http://www.openfabrics.org.

Fabric management on SGI Altix ICE 8000 series systems uses the OFED 1.2 OpenSM software package. The InfiniBand fabric connects the service nodes, rack leader controllers (leader nodes), and the compute nodes. It does not connect to the system admin controller (admin node) or the chassis management control (CMC) blades. The InfiniBand network has two separate network fabrics, `ib0` and `ib1` (see “InfiniBand Fabric” in Chapter 1) with the following characteristics:

- Each network fabric has its own subnet manager (SM).

- For a system with two racks or more, one rack leader controller (leader node) runs an instance of SM to manage the `ib0` fabric and a second leader node runs an instance of SM to manage the `ib1` fabric.

- On a system with a single rack, both instances of `opensm` run on the same rack leader node.

- Each instance of SM on the rack leader controller is controlled by the `/etc/opensm-ib0.conf` or `/etc/opensm-ib1.conf` configuration file.

- Rack leader controllers run the `opensm` daemon for each fabric over separate HCA ports (see Figure 1-9).

  | Note: After a system reboot, you need to manually restart the `opensm` daemons running on the InfiniBand fabric. If the `opensm` daemons are allowed to start automatically, as the leader nodes boot, you will not know which leader is the `Master` and it is highly likely that the fabric will be routed incorrectly. To start the InfiniBand fabric, you can use the following command:
      scalimanage-cli restartaltixiceopensm

- Each fabric is addressed by a global unique identifier (GUID) and unique HCA port.

  The GUID and HCA port is set in the configuration file.

- Coherency of the fabric database is handled by `sldd-ib[01].sh`. You must make sure `OSM_HOSTS` is configured correctly in the `/etc/opensm-ib0.conf` or `/etc/opensm-ib1.conf` configuration files.

| Note: Currently, the InfiniBand fabric `ib0` is reserved for MPI or interprocess communication traffic and the InfiniBand fabric `ib1` is reserved for storage.

For more information on the InfiniBand fabric, see Appendix A, “InfiniBand Fabric Details” and Appendix B, “InfiniBand Fabric Troubleshooting”.

Prev | Table of Contents | Next
Chapter 2. Getting Started with Scali Manage |  | Appendix A. InfiniBand Fabric Details