The SGI Altix system can support many more PCI-X I/O devices than other traditional Linux servers. Each PX-brick has 6 PCI-X buses and 12 slots..
When you have more than one PX-brick on your SGI Altix system, you must be able to physically and programmatically locate your devices. This chapter provides the information about physical and logical Linux addresses that you need to find your PCI-X device.
When your PCI device is slotted into a PCI-X slot, it has a physical address. The components of the physical address are as follows:
Function number (0 to 7)
Physical device/slot number on the PCI-X bus (1 or 2)
PCI-X bus number on the host bridge adapter (1 or 2)
Host bridge adapter number (also known as widgets) (12, 13, or 15)
Compute brick slab number (0 or 1)
Compute brick module identification number
PX-brick/IX-brick module identification number
Figure 5-1, shows physical address components.
When the Linux PCI subsystem encounters your PCI device, it is given a logical address. The components of the logical address are as follows:
Function number
Device number
Logical PCI bus number
On SGI Altix systems, because of the vast number of PCI-X buses that the system can support, SGI has provided a hardware graph topology that maps all PCI devices in the system from their Linux logical address to the actual hardware address. This facility allows you to correctly identify a particular device on the system for any purpose, one of which may be to find the physical failing component that has been reported by the system.
During SGI Altix platform initialization, as the system hardware probes outwards from the compute bricks to initialize the various components attached to the system, the system software uses the Linux device filesystem (devfs) to create a hierachical topology of all the components discovered and initialized.
An example:
/dev/hw/module/001c11/slab/0/Pbrick/xtalk/12/pci-x/0/1 |
This topology describes the actual hardware location of a specific PCI-X device. The following section describes how to use this topology information to physically find your device.
The /proc/pci file provides a user level display of all PCI devices seen and initialized by the Linux kernel. Consider the following example:
[root@pumpkin root]# cat /proc/pci
....
Bus 3, device 1, function 0:
Fibre Channel: QLogic Corp. QLA2300 64-bit FC-AL Adapter (#5) (rev 1).
IRQ 567.
Master Capable. Latency=64. Min Gnt=64.
I/O at 0xc00000080fa00000 [0xc00000080fa000ff].
Non-prefetchable 64 bit memory at 0xc00000080fe00000 [0xc00000080fe00fff].
.... |
 | Note: The bus number (3) in the preceding example is in decimal notation and the bus numbers in /dev/hw/linux/bus/pci-x/* are in hexadecimal notation. |
You can locate your device physically from the logical Linux PCI bus number and device number by using the /proc/pci output. The PCI device in the previous example is on logical PCI bus 3, device 1. The device number is the same as the slot number on the PCI bus. Therefore, you can determine that the device is on slot 1 of that PCI bus.
SGI Altix systems provide topology information (known as the hwgraph) for all system components. The hwgraph depicts the connections among these components. The hwgraph topology information is rooted at /dev/hw. The following example shows how to find the physical location of your device. It shows the physical location of Linux bus numbers as provided by the /proc/pci display.
[root@pumpkin root]# ls -al /dev/hw/linux/bus/pci-x/*
lr-xr-xr-x 1 root root 56 Dec 31 1969 /dev/hw/linux/bus/pci-x/1-> ../../
../hw/module/001c27/slab/0/Pbrick/xtalk/12/pci-x/0
lr-xr-xr-x 1 root root 56 Dec 31 1969 /dev/hw/linux/bus/pci-x/2-> ../../
../hw/module/001c27/slab/0/Pbrick/xtalk/12/pci-x/1
lr-xr-xr-x 1 root root 56 Dec 31 1969 /dev/hw/linux/bus/pci-x/3-> ../../
../hw/module/001c27/slab/0/Pbrick/xtalk/15/pci-x/0
lr-xr-xr-x 1 root root 56 Dec 31 1969 /dev/hw/linux/bus/pci-x/4-> ../../
../hw/module/001c27/slab/0/Pbrick/xtalk/15/pci-x/1
|
| Note: The bus numbers in the preceding example are in hexadecimal notation. |
The components are defined as follows:
| module/001c27 | Indicates the module identification number. Each module on the system has a module identification number. This number is displayed on the LED on the front of the module. | |
| /slab/0 | Indicates the slab number. Each compute brick has two slabs (also known as nodes) -- slab 0 and slab 1. | |
| Pbrick | Indicates the I/O brick type. All IX-bricks and PX-bricks are identified as Pbrick in the topology. | |
| xtalk/15 | Indicates the PCI-X host bridge adapter number. There are three PCI-X host bridge adapters in each I/O brick. They are numbered 12, 13, and 15. Each PCI-X host bridge adapter number is "stamped" at the back of the I/O brick. | |
| pci-x/0 | Indicates the logical PCI-X bus number on the host bridge adapter. Each PCI-X host bridge adapter has two PCI-X buses -- bus 1 and bus 2. This number is also "stamped" on the back panel of the I/O brick. Unfortunately, the hwgraph assigns the PCI bus numbers as 0 and 1 instead of 1 and 2. Therefore, pci-x/0 is actually physical bus 1 and pci-x/1 is physical bus 2. |
To physically locate your device, use the hwgraph topology information, as follows: If, from the /proc/pci display, your Linux logical bus number is 3, your device number is 1, and /dev/hw/linux/busnum points at ../../../hw/module/0012c27/slab/0/Pbrick/xtalk/15/pci-x/1 , you know that your device (device 1) is in slot 1, on physical bus number 2 (pcix-x/1), on PCI-X host bridge adapter number 15, that is connected to slab 0 (node 0) of the compute brick with module identification number 0012c27.
PCI services provided by the Linux kernel driver kernel interface (DKI) require a handle that maps to your PCI device. This handle is actually a pointer to your device pci_dev structure. Therefore, before the device driver can call the Linux kernel for any PCI services, it must first locate its PCI device handle.
You can locate the PCI device pci_dev structure by using the following methods:
struct pci_dev *
pci_find_device(unsigned int vendor, unsigned int device,
const struct pci_dev *from) |
This routine scans from *from to locate the pci_dev structure that matches the given vendor and device identification. If *from is NULL, it starts from the beginning of the pci_dev list. If a match is found, the address of pci_dev is returned in *from .
Other search routines of interest are:
struct pci_dev *pci_find_subsys (unsigned int vendor,
unsigned int device,
unsigned int ss_vendor, unsigned int ss_device,
const struct pci_dev *from);
|
struct pci_dev *pci_find_class (unsigned int class,
const struct pci_dev *from);
|
struct pci_dev *pci_find_slot (unsigned int bus, unsigned int devfn); |
The device driver can also scan through the PCI device list by searching the list in reverse order (see Example 5-1) and scanning the list in order (see Example 5-2). The advantage of these scanning mechanisms is that they allow the device driver to match any attributes required for locating its PCI device.
Example 5-1. Scanning the list in reverse order
#define pci_for_each_dev_reverse(dev)\
for(dev = pci_dev_g(pci_devices.prev); \
dev != pci_dev_g(&pci_devices); \
dev = pci_dev_g(dev->global_list.prev)) |
Example 5-2. Scanning the list in order
#define pci_for_each_dev(dev) \
for(dev = pci_dev_g(pci_devices.next); \
dev != pci_dev_g(&pci_devices); \
dev = pci_dev_g(dev->global_list.next))
|