Appendix B. InfiniBand Fabric Troubleshooting
Prev |  | Next

## Appendix B. InfiniBand Fabric Troubleshooting

This appendix describes some useful utilities and diagnostics for trouble shooting the InfiniBand fabric.

## Useful Utilities and Diagnostics

The `openib-diags` package contains useful tools and diagnostic software for Open Fabrics Enterprise Distribution (OFED). This section describes some of these tools. These tools reside on the rack leader controller (leader node) in the `/usr/bin` directory, as follows:

    r01lead:~ # cd /usr/bin
    r01lead:/usr/bin # ls ib*
    ibaddr            ibcheckstate     ibdiscover.pl        ibnetdiscover     ib_rdma_bw   ibstatus        ...
    ibcheckerrors     ibcheckwidth     ibdmchk              ibnlparse         ib_rdma_lat  ibswitches      ...
    ibcheckerrs       ibclearcounters  ibdmsh               ibnodes           ib_read_bw   ibsysstat       ...
    ibchecknet        ibclearerrors    ibdmtr               ibping            ib_read_lat  ibtopodiff      ...
    ibchecknode       ib_clock_test    ibfindnodesusing.pl  ibportstate       ibroute      ibtracert       ...
    ibcheckport       ibdiagnet        ibhosts              ibprintca.pl      ib_send_bw   ibv_asyncwatch  ...
    ibcheckportstate  ibdiagpath       ibis                 ibprintswitch.pl  ib_send_lat  ibv_devices     ...
    ibcheckportwidth  ibdiagui         iblinkinfo.pl        ibqueryerrors.pl  ibstat       ibv_devinfo

This section covers the following topics:

- “ibstat Command”

- “perfquery Command”

- “ibnetdiscover Command”

- “ibdiagnet Command”

### `ibstat` Command

You can use the `ibstat` command to see the current status of the host channel adapaters (HCA) in your InfiniBand fabric incluing the HCAs on rack leader controllers. The following view is prior to starting the fabric management:

    r01lead:/usr/bin # ibstat
    CA 'mthca0'
            CA type: MT25208 (MT23108 compat mode)
            Number of ports: 2
            Firmware version: 4.7.600
            Hardware version: a0
            Node GUID: 0x0008f104039881a8
            System image GUID: 0x0008f104039881ab
            Port 1:
                    State: Initializing
                    Physical state: LinkUp
                    Rate: 20
                    Base lid: 0
                    LMC: 0
                    SM lid: 0
                    Capability mask: 0x02510a68
                    Port GUID: 0x0008f104039881a9
            Port 2:
                    State: Initializing
                    Physical state: LinkUp
                    Rate: 20
                    Base lid: 0
                    LMC: 0
                    SM lid: 0
                    Capability mask: 0x02510a68
                    Port GUID: 0x0008f104039881aa

The following shows output from the `ibstat` command after the fabric management software has been started:

    r01lead:/opt/sgi/sbin # ibstat
    CA 'mthca0'
            CA type: MT25208 (MT23108 compat mode)
            Number of ports: 2
            Firmware version: 4.7.600
            Hardware version: a0
            Node GUID: 0x0008f104039881a8
            System image GUID: 0x0008f104039881ab
            Port 1:
                    State: Active
                    Physical state: LinkUp
                    Rate: 20
                    Base lid: 1
                    LMC: 0
                    SM lid: 1
                    Capability mask: 0x02510a6a
                    Port GUID: 0x0008f104039881a9
            Port 2:
                    State: Active
                    Physical state: LinkUp
                    Rate: 20
                    Base lid: 1
                    LMC: 0
                    SM lid: 1
                    Capability mask: 0x02510a6a
                    Port GUID: 0x0008f104039881aa

### `ibstatus` Command

You can use the `ibstatus` (less verbose that `ibstat`) command to show the link rate, as follows:

    r01lead:/opt/sgi/sbin # ibstatus
    Infiniband device 'mthca0' port 1 status:
            default gid:     fe80:0000:0000:0000:0008:f104:0398:81a9
            base lid:        0x1
            sm lid:          0x1
            state:           4: ACTIVE
            phys state:      5: LinkUp
            rate:            20 Gb/sec (4X DDR)

    Infiniband device 'mthca0' port 2 status:
            default gid:     fe80:0000:0000:0000:0008:f104:0398:81aa
            base lid:        0x1
            sm lid:          0x1
            state:           4: ACTIVE
            phys state:      5: LinkUp
            rate:            20 Gb/sec (4X DDR)

| Note: If link rate is not 20 Gb/sec 4xDDR, there is a physical link problem with your system.

## `perfquery` Command

The `perfquery` command is useful for find errors on a particular or number of HCA's and switch ports. You can also use `perfquery` to reset HCA and switch port counters.

To see a usage statement for the `perfquery` command, perform the following:

    r01lead:/opt/sgi/sbin # perfquery --help
    Usage: perfquery [-d(ebug) -G(uid) -a(ll_ports) -r(eset_after_read) -C ca_name -P ca_port -R(eset_only)
     -t(imeout) timeout_ms -V(ersion) -h(elp)] [<lid|guid> [[port] [reset_mask]]]
            Examples:
                    perfquery               # read local port's performance counters
                    perfquery 32 1          # read performance counters from lid 32, port 1
                    perfquery -e 32 1       # read extended performance counters from lid 32, port 1
                    perfquery -a 32         # read performance counters from lid 32, all ports
                    perfquery -r 32 1       # read performance counters and reset
                    perfquery -e -r 32 1    # read extended performance counters and reset
                    perfquery -R 0x20 1     # reset performance counters of port 1 only
                    perfquery -e -R 0x20 1  # reset extended performance counters of port 1 only
                    perfquery -R -a 32      # reset performance counters of all ports
                    perfquery -R 32 2 0x0fff        # reset only error counters of port 2
                    perfquery -R 32 2 0xf000        # reset only non-error counters of port 2

Some sample output from the `perfquery` command is, as follows:

    r01lead:/opt/sgi/sbin # perfquery
    # Port counters: Lid 1 port 1
    PortSelect:......................1
    CounterSelect:...................0x0000
    SymbolErrors:....................0
    LinkRecovers:....................0
    LinkDowned:......................0
    RcvErrors:.......................0
    RcvRemotePhysErrors:.............0
    RcvSwRelayErrors:................0
    XmtDiscards:.....................0
    XmtConstraintErrors:.............0
    RcvConstraintErrors:.............0
    LinkIntegrityErrors:.............0
    ExcBufOverrunErrors:.............0
    VL15Dropped:.....................0
    XmtData:.........................0
    RcvData:.........................0
    XmtPkts:.........................0
    RcvPkts:.........................0

## `ibnetdiscover` Command

The `ibnetdiscover` command allows you discover the IB fabric.

To see a usage statement for the `ibnetdiscover` command, perform the following:

    r01lead:/opt/sgi/sbin # ibnetdiscover --help
    Usage: ibnetdiscover [-d(ebug)] -e(rr_show) -v(erbose) -s(how) -l(ist)
    -g(rouping) -H(ca_list) -S(witch_list)
    -V(ersion) -C ca_name -P ca_port -t(imeout) timeout_ms
    --switch-map switch-map] [<topology-file>]
    --switch-map <switch-map>  specify a switch-map file

| Note: Only abbreviated output is shown in the this example.

Some sample output from the `ibnetdiscover` command is, as follows:

    r01lead:/opt/sgi/sbin # ibnetdiscover
    #
    # Topology file: generated on Tue Jul 17 14:05:20 2007
    #
    # Max of 3 hops discovered
    # Initiated from node 0008f104039881a8 port 0008f104039881a9

    vendid=0x2c9
    devid=0xb924
    sysimgguid=0x8006900000000dd

    ...

    Switch   : 0x08006900000000dc ports 24 devid 0xb924 vendid 0x2c9
    "MT47396 Infiniscale-III Mellanox Technologies"
    Switch   : 0x08006900000000a4 ports 24 devid 0xb924 vendid 0x2c9
    "MT47396 Infiniscale-III Mellanox Technologies"

    r01lead:/opt/sgi/sbin # ibnetdiscover -H (HCA's)
    Ca       : 0x0030487aa7940000 ports 1 devid 0x6274 vendid 0x2c9 "MT25204 InfiniHostLx Mellanox Technologies"
    Ca       : 0x0030487aa78c0000 ports 1 devid 0x6274 vendid 0x2c9 "r1i0n8-ib0 HCA-1"
    Ca       : 0x0008f10403988198 ports 2 devid 0x6278 vendid 0x8f1 " HCA-1"
    Ca       : 0x0030487aa7840000 ports 1 devid 0x6274 vendid 0x2c9 "r1i0n1-ib0 HCA-1"
    Ca       : 0x0030487aa79c0000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n0-ib0 HCA-1"
    Ca       : 0x0030487aa7900000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n8-ib0 HCA-1"
    Ca       : 0x0030487aa7980000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n1-ib0 HCA-1"
    Ca       : 0x0008f104039881a8 ports 2 devid 0x6278 vendid 0x8f1 " HCA-1"

    ======================================================================================================

## `ibdiagnet` Command

The `ibdiagnet` command is a useful diagnostic tool.

To see a usage statement for the `ibdiagnet` command, perform the following:

    r01lead:/opt/sgi/sbin # ibdiagnet --help
    Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
    NAME
      ibdiagnet
    SYNOPSYS
      ibdiagnet [-c ] [-v] [-r] [-o ]
         [-t ] [-s ] [-i ] [-p ]
         [-pm] [-pc] [-P <>]
         [-lw <1x|4x|12x>] [-ls <2.5|5|10>]

    DESCRIPTION
      ibdiagnet scans the fabric using directed route packets and extracts all the
      available information regarding its connectivity and devices.
      It then produces the following files in the output directory defined by the
      -o option (see below):
        ibdiagnet.lst    - List of all the nodes, ports and links in the fabric
        ibdiagnet.fdbs   - A dump of the unicast forwarding tables of the fabric
                           switches
        ibdiagnet.mcfdbs - A dump of the multicast forwarding tables of the fabric
                           switches
        ibdiagnet.masks  - In case of duplicate port/node Guids, these file include
                           the map between masked Guid and real Guids
        ibdiagnet.sm     - A dump of all the SM (state and priority) in the fabric
        ibdiagnet.pm     - In case -pm option was provided, this file contain a dump
                           of all the nodes PM counters
      In addition to generating the files above, the discovery phase also checks for
      duplicate node/port GUIDs in the IB fabric. If such an error is detected, it
      is displayed on the standard output.
      After the discovery phase is completed, directed route packets are sent
      multiple times (according to the -c option) to detect possible problematic
      paths on which packets may be lost. Such paths are explored, and a report of
      the suspected bad links is displayed on the standard output.
      After scanning the fabric, if the -r option is provided, a full report of the
      fabric qualities is displayed.
      This report includes:
        SM report
        Number of nodes and systems
        Hop-count information:
             maximal hop-count, an example path, and a hop-count histogram
        All CA-to-CA paths traced
        Credit loop report
        mgid-mlid-HCAs matching table
      Note: In case the IB fabric includes only one CA, then CA-to-CA paths are not
      reported.
      Furthermore, if a topology file is provided, ibdiagnet uses the names defined
      in it for the output reports.

    OPTIONS
      -c                      : The minimal number of packets to be sent
                                       across each link (default = 10)
      -v                             : Instructs the tool to run in verbose mode
      -r                             : Provides a report of the fabric qualities
      -o                    : Specifies the directory where the output
                                       files will be placed (default = /tmp)
      -t                  : Specifies the topology file name
      -s                   : Specifies the local system name. Meaningful
                                       only if a topology file is specified
      -i                  : Specifies the index of the device of the port
                                       used to connect to the IB fabric (in case of
                                       multiple devices on the local system)
      -p                   : Specifies the local device's port number used
                                       to connect to the IB fabric
      -pm                            : Dumps all pmCounters values into ibdiagnet.pm
      -pc                            : reset all the fabric links pmCounters
      -P <>: If any of the provided pm is greater then its
                                       provided value, print it to screen
      -lw <1x|4x|12x>                : Specifies the expected link width
      -ls <2.5|5|10>                 : Specifies the expected link speed

      -h|--help                      : Prints this help information
      -V|--version                   : Prints the version of the tool
         --vars                      : Prints the tool's environment variables and
                                       their values

    ERROR CODES
      1 - Failed to fully discover the fabric
      2 - Failed to parse command line options
      3 - Failed to interact with IB fabric
      4 - Failed to use local device or local port
      5 - Failed to use Topology File
      6 - Failed to load required Package

Output which shows no errors means the system is operating correctly:

    r01lead:/opt/sgi/sbin # ibdiagnet
    Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
    Loading IBDM from: /usr/lib64/ibdm1.2
    -W- Topology file is not specified.
        Reports regarding cluster links will use direct routes.
    -W- A few ports of local device are up.
        Since port-num was not specified (-p option), port 1 of device 1 will be
        used as the local port.
    -I- Discovering the subnet ... 10 nodes (2 Switches & 8 CA-s) discovered.

    -I---------------------------------------------------
    -I- Bad Guids Info
    -I---------------------------------------------------
    -I- No bad Guids were found

    -I---------------------------------------------------
    -I- Links With Logical State = INIT
    -I---------------------------------------------------
    -I- No bad Links (with logical state = INIT) were found

    -I---------------------------------------------------
    -I- PM Counters Info
    -I---------------------------------------------------
    -I- No illegal PM counters values were found

    -I---------------------------------------------------
    -I- Bad Links Info
    -I---------------------------------------------------
    -I- No bad link were found

    -I- Done. Run time was 0 seconds.

    You can use ibdiagnet to load the fabric to test it.
    like this  :-

    r01lead:/opt/sgi/sbin # ibdiagnet -c 5000
    Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
    Loading IBDM from: /usr/lib64/ibdm1.2
    -W- Topology file is not specified.
        Reports regarding cluster links will use direct routes.
    -W- A few ports of local device are up.
        Since port-num was not specified (-p option), port 1 of device 1 will be
        used as the local port.
    -I- Discovering the subnet ... 10 nodes (2 Switches & 8 CA-s) discovered.

    -I---------------------------------------------------
    -I- Bad Guids Info
    -I---------------------------------------------------
    -I- No bad Guids were found

    -I---------------------------------------------------
    -I- Links With Logical State = INIT
    -I---------------------------------------------------
    -I- No bad Links (with logical state = INIT) were found

    -I---------------------------------------------------
    -I- PM Counters Info
    -I---------------------------------------------------
    -I- No illegal PM counters values were found

    -I---------------------------------------------------
    -I- Bad Links Info
    -I---------------------------------------------------
    -I- No bad link were found

    -I- Done. Run time was 8 seconds.

Prev | Table of Contents | Next
Appendix A. InfiniBand Fabric Details |  | Index