This appendix describes some useful utilities and diagnostics for trouble shooting the InfiniBand fabric.
The openib-diags package contains useful tools and diagnostic software for Open Fabrics Enterprise Distribution (OFED). This section describes some of these tools. These tools reside on the rack leader controller (leader node) in the /usr/bin directory, as follows:
r01lead:~ # cd /usr/bin r01lead:/usr/bin # ls ib* ibaddr ibcheckstate ibdiscover.pl ibnetdiscover ib_rdma_bw ibstatus ... ibcheckerrors ibcheckwidth ibdmchk ibnlparse ib_rdma_lat ibswitches ... ibcheckerrs ibclearcounters ibdmsh ibnodes ib_read_bw ibsysstat ... ibchecknet ibclearerrors ibdmtr ibping ib_read_lat ibtopodiff ... ibchecknode ib_clock_test ibfindnodesusing.pl ibportstate ibroute ibtracert ... ibcheckport ibdiagnet ibhosts ibprintca.pl ib_send_bw ibv_asyncwatch ... ibcheckportstate ibdiagpath ibis ibprintswitch.pl ib_send_lat ibv_devices ... ibcheckportwidth ibdiagui iblinkinfo.pl ibqueryerrors.pl ibstat ibv_devinfo |
This section covers the following topics:
You can use the ibstat command to see the current status of the host channel adapaters (HCA) in your InfiniBand fabric incluing the HCAs on rack leader controllers. The following view is prior to starting the fabric management:
r01lead:/usr/bin # ibstat
CA 'mthca0'
CA type: MT25208 (MT23108 compat mode)
Number of ports: 2
Firmware version: 4.7.600
Hardware version: a0
Node GUID: 0x0008f104039881a8
System image GUID: 0x0008f104039881ab
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 20
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510a68
Port GUID: 0x0008f104039881a9
Port 2:
State: Initializing
Physical state: LinkUp
Rate: 20
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510a68
Port GUID: 0x0008f104039881aa |
The following shows output from the ibstat command after the fabric management software has been started:
r01lead:/opt/sgi/sbin # ibstat
CA 'mthca0'
CA type: MT25208 (MT23108 compat mode)
Number of ports: 2
Firmware version: 4.7.600
Hardware version: a0
Node GUID: 0x0008f104039881a8
System image GUID: 0x0008f104039881ab
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x02510a6a
Port GUID: 0x0008f104039881a9
Port 2:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x02510a6a
Port GUID: 0x0008f104039881aa |
You can use the ibstatus (less verbose that ibstat) command to show the link rate, as follows:
r01lead:/opt/sgi/sbin # ibstatus
Infiniband device 'mthca0' port 1 status:
default gid: fe80:0000:0000:0000:0008:f104:0398:81a9
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
Infiniband device 'mthca0' port 2 status:
default gid: fe80:0000:0000:0000:0008:f104:0398:81aa
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR) |
 | Note: If link rate is not 20 Gb/sec 4xDDR, there is a physical link problem with your system. |
The perfquery command is useful for find errors on a particular or number of HCA's and switch ports. You can also use perfquery to reset HCA and switch port counters.
To see a usage statement for the perfquery command, perform the following:
r01lead:/opt/sgi/sbin # perfquery --help
Usage: perfquery [-d(ebug) -G(uid) -a(ll_ports) -r(eset_after_read) -C ca_name -P ca_port -R(eset_only)
-t(imeout) timeout_ms -V(ersion) -h(elp)] [<lid|guid> [[port] [reset_mask]]]
Examples:
perfquery # read local port's performance counters
perfquery 32 1 # read performance counters from lid 32, port 1
perfquery -e 32 1 # read extended performance counters from lid 32, port 1
perfquery -a 32 # read performance counters from lid 32, all ports
perfquery -r 32 1 # read performance counters and reset
perfquery -e -r 32 1 # read extended performance counters and reset
perfquery -R 0x20 1 # reset performance counters of port 1 only
perfquery -e -R 0x20 1 # reset extended performance counters of port 1 only
perfquery -R -a 32 # reset performance counters of all ports
perfquery -R 32 2 0x0fff # reset only error counters of port 2
perfquery -R 32 2 0xf000 # reset only non-error counters of port 2 |
Some sample output from the perfquery command is, as follows:
r01lead:/opt/sgi/sbin # perfquery # Port counters: Lid 1 port 1 PortSelect:......................1 CounterSelect:...................0x0000 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtData:.........................0 RcvData:.........................0 XmtPkts:.........................0 RcvPkts:.........................0 |
The ibnetdiscover command allows you discover the IB fabric.
To see a usage statement for the ibnetdiscover command, perform the following:
r01lead:/opt/sgi/sbin # ibnetdiscover --help Usage: ibnetdiscover [-d(ebug)] -e(rr_show) -v(erbose) -s(how) -l(ist) -g(rouping) -H(ca_list) -S(witch_list) -V(ersion) -C ca_name -P ca_port -t(imeout) timeout_ms --switch-map switch-map] [<topology-file>] --switch-map <switch-map> specify a switch-map file |
| Note: Only abbreviated output is shown in the this example. |
Some sample output from the ibnetdiscover command is, as follows:
r01lead:/opt/sgi/sbin # ibnetdiscover # # Topology file: generated on Tue Jul 17 14:05:20 2007 # # Max of 3 hops discovered # Initiated from node 0008f104039881a8 port 0008f104039881a9 vendid=0x2c9 devid=0xb924 sysimgguid=0x8006900000000dd ... Switch : 0x08006900000000dc ports 24 devid 0xb924 vendid 0x2c9 "MT47396 Infiniscale-III Mellanox Technologies" Switch : 0x08006900000000a4 ports 24 devid 0xb924 vendid 0x2c9 "MT47396 Infiniscale-III Mellanox Technologies" r01lead:/opt/sgi/sbin # ibnetdiscover -H (HCA's) Ca : 0x0030487aa7940000 ports 1 devid 0x6274 vendid 0x2c9 "MT25204 InfiniHostLx Mellanox Technologies" Ca : 0x0030487aa78c0000 ports 1 devid 0x6274 vendid 0x2c9 "r1i0n8-ib0 HCA-1" Ca : 0x0008f10403988198 ports 2 devid 0x6278 vendid 0x8f1 " HCA-1" Ca : 0x0030487aa7840000 ports 1 devid 0x6274 vendid 0x2c9 "r1i0n1-ib0 HCA-1" Ca : 0x0030487aa79c0000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n0-ib0 HCA-1" Ca : 0x0030487aa7900000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n8-ib0 HCA-1" Ca : 0x0030487aa7980000 ports 1 devid 0x6274 vendid 0x2c9 "r1i1n1-ib0 HCA-1" Ca : 0x0008f104039881a8 ports 2 devid 0x6278 vendid 0x8f1 " HCA-1" ====================================================================================================== |
The ibdiagnet command is a useful diagnostic tool.
To see a usage statement for the ibdiagnet command, perform the following:
r01lead:/opt/sgi/sbin # ibdiagnet --help
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
NAME
ibdiagnet
SYNOPSYS
ibdiagnet [-c ] [-v] [-r] [-o ]
[-t ] [-s ] [-i ] [-p ]
[-pm] [-pc] [-P <>]
[-lw <1x|4x|12x>] [-ls <2.5|5|10>]
DESCRIPTION
ibdiagnet scans the fabric using directed route packets and extracts all the
available information regarding its connectivity and devices.
It then produces the following files in the output directory defined by the
-o option (see below):
ibdiagnet.lst - List of all the nodes, ports and links in the fabric
ibdiagnet.fdbs - A dump of the unicast forwarding tables of the fabric
switches
ibdiagnet.mcfdbs - A dump of the multicast forwarding tables of the fabric
switches
ibdiagnet.masks - In case of duplicate port/node Guids, these file include
the map between masked Guid and real Guids
ibdiagnet.sm - A dump of all the SM (state and priority) in the fabric
ibdiagnet.pm - In case -pm option was provided, this file contain a dump
of all the nodes PM counters
In addition to generating the files above, the discovery phase also checks for
duplicate node/port GUIDs in the IB fabric. If such an error is detected, it
is displayed on the standard output.
After the discovery phase is completed, directed route packets are sent
multiple times (according to the -c option) to detect possible problematic
paths on which packets may be lost. Such paths are explored, and a report of
the suspected bad links is displayed on the standard output.
After scanning the fabric, if the -r option is provided, a full report of the
fabric qualities is displayed.
This report includes:
SM report
Number of nodes and systems
Hop-count information:
maximal hop-count, an example path, and a hop-count histogram
All CA-to-CA paths traced
Credit loop report
mgid-mlid-HCAs matching table
Note: In case the IB fabric includes only one CA, then CA-to-CA paths are not
reported.
Furthermore, if a topology file is provided, ibdiagnet uses the names defined
in it for the output reports.
OPTIONS
-c : The minimal number of packets to be sent
across each link (default = 10)
-v : Instructs the tool to run in verbose mode
-r : Provides a report of the fabric qualities
-o : Specifies the directory where the output
files will be placed (default = /tmp)
-t : Specifies the topology file name
-s : Specifies the local system name. Meaningful
only if a topology file is specified
-i : Specifies the index of the device of the port
used to connect to the IB fabric (in case of
multiple devices on the local system)
-p : Specifies the local device's port number used
to connect to the IB fabric
-pm : Dumps all pmCounters values into ibdiagnet.pm
-pc : reset all the fabric links pmCounters
-P <>: If any of the provided pm is greater then its
provided value, print it to screen
-lw <1x|4x|12x> : Specifies the expected link width
-ls <2.5|5|10> : Specifies the expected link speed
-h|--help : Prints this help information
-V|--version : Prints the version of the tool
--vars : Prints the tool's environment variables and
their values
ERROR CODES
1 - Failed to fully discover the fabric
2 - Failed to parse command line options
3 - Failed to interact with IB fabric
4 - Failed to use local device or local port
5 - Failed to use Topology File
6 - Failed to load required Package
|
Output which shows no errors means the system is operating correctly:
r01lead:/opt/sgi/sbin # ibdiagnet
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
Loading IBDM from: /usr/lib64/ibdm1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering the subnet ... 10 nodes (2 Switches & 8 CA-s) discovered.
-I---------------------------------------------------
-I- Bad Guids Info
-I---------------------------------------------------
-I- No bad Guids were found
-I---------------------------------------------------
-I- Links With Logical State = INIT
-I---------------------------------------------------
-I- No bad Links (with logical state = INIT) were found
-I---------------------------------------------------
-I- PM Counters Info
-I---------------------------------------------------
-I- No illegal PM counters values were found
-I---------------------------------------------------
-I- Bad Links Info
-I---------------------------------------------------
-I- No bad link were found
-I- Done. Run time was 0 seconds.
You can use ibdiagnet to load the fabric to test it.
like this :-
r01lead:/opt/sgi/sbin # ibdiagnet -c 5000
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
Loading IBDM from: /usr/lib64/ibdm1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering the subnet ... 10 nodes (2 Switches & 8 CA-s) discovered.
-I---------------------------------------------------
-I- Bad Guids Info
-I---------------------------------------------------
-I- No bad Guids were found
-I---------------------------------------------------
-I- Links With Logical State = INIT
-I---------------------------------------------------
-I- No bad Links (with logical state = INIT) were found
-I---------------------------------------------------
-I- PM Counters Info
-I---------------------------------------------------
-I- No illegal PM counters values were found
-I---------------------------------------------------
-I- Bad Links Info
-I---------------------------------------------------
-I- No bad link were found
-I- Done. Run time was 8 seconds. |