The peripheral component interconnect (PCI) bus, initially designed at Intel, is standardized by the PCI special interest group, a nonprofit consortium of vendors. PCI-X is the successor to PCI. Both PCI and PCI-X devices can be used on a PCI-X bus. SGI Altix system architecture supports the PCI-X bus. All peripheral devices on SGI Altix systems are connected via PCI-X buses. The PCI-X bus is designed as a high-performance local bus to connect peripherals to memory and a microprocessor.
This chapter describes PCI-X implementation in larger architectures such as the SGI Altix systems.
For more information about PCI-X system architecture, see PCI-X System Architecture.
The PX-brick with BaseIO is a Crosstalk-to-PCI-X based I/O subsystem. It has two 1200-MB/s Xtown2 connectors and one or two of them can be used to connect to SC-bricks. There are 12 PCI slots that are configured on 6 buses and 2 hard disk bays that support SCSI disk drives and a DVD-ROM. The DVD-ROM is not SCSI; it is connected through parallel ATA. For peak bandwidth values, see Table 3-1. Figure 3-1, depicts the PX-brick with BaseIO.
Table 3-1. Bandwidth Characteristics of the IX-brick
Description | Peak Bandwidth |
|---|---|
Xtown2 ports -- A and B | 1200 MB/s |
PCI-X bus frequency: 33 MHz 66 MHz 100 MHz 133 MHz | 64-bit mode: 256 MB/s 512 MB/s 800 MB/s 1024 MB/s |
Note 1: PCI-X mode achieves a higher percentage of theoretical peak versus PCI mode.
Note 2: To run the bus at 133 MHz requires only one card on that bus and it must be a 133-MHz capable card.
Note3: The IO9 is a 66-MHz PCI card, so bus 1 runs at 66 MHz PCI in the IX-brick.
The PX-brick has two 1200-MB/s Xtown2 ports that connect SC-bricks. There are 12 PCI slots that are configured on 6 buses. For peak and sustained bandwidth values, see Table 3-2. Figure 3-2, depicts a PX-brick PCI-X expansion.
Table 3-2. Bandwidth Characteristics of the PX-brick
Description | Peak Bandwidth |
|---|---|
Xtown2 ports -- A and B | 1200 MB/s |
PCI-X bus frequency: 33 MHz 66 MHz 100 MHz 133 MHz | 64-bit mode: 256 MB/s 512 MB/s 800 MB/s 1024 MB/s |
Note 1: PCI-X mode achieves a higher percentage of theoretical peak versus PCI mode.
Note 2: To run the bus at 133 MHz requires only one card on that bus and it must be a 133-MHz capable card.
In SGI Altix systems, the PCI-X adapter connects to the high-speed XIO bus. This bridge joins the PCI-X bus into the connection fabric, so that any PCI-X bus can be addressed from any module, and any PCI-X bus can access memory that is physically located in any module, as shown in Figure 3-3.
|
In SGI Altix systems, the multimedia features have substantial local resources, so contention with multimedia for the use of main memory is lower. However, these systems also have multiple CPUs and multiple layers of address translation, and these factors can introduce latencies in PCI-X transactions.
It is important to understand that there is no guaranteed order of execution between separate PCI-X transactions in these systems . There can be multiple hardware layers between the CPU, memory, and the device. One or more data transactions can be “in flight” for durations that are significant. For example, suppose that a PCI-X bus-master device completes the last transfer of a DMA write of data to memory, and then executes a DMA write to update a status flag elsewhere in memory.
Under circumstances that are unusual but not impossible, the status in memory can be updated and acted upon by software, while the data transaction is still “in flight” and has not completely arrived in memory. The same can be true of a programmable I/O (PIO) read that polls the device. It can return “complete” status from the device while data sent by DMA has yet to reach memory.
Ordering is guaranteed when interrupts are used. An interrupt handler is not executed until all writes initiated by the interrupting device have completed.
When the Linux kernel scans the PCI-X buses on an SGI Altix system and finds an active device, it initializes the device configuration registers as follows:
| Command register | The enabling bits for I/O access, memory access, and master are set to 1. Other bits, such as memory write, invalidate, and fast back-to-back are left at 0. | ||||||||||||||||
| Cache line size | Set at 0x20 (32, 32-bit words, or 128 bytes). | ||||||||||||||||
| Latency timer | Setting depends on bus speed and the device's Min_Gnt (minimum grant) register. If the device's Min_Gnt value is 0, the latency timer is set to 1 microsecond. Otherwise, it is set to (min_gnt_mult * Min_Gnt). Values for min_gnt_multdepend on bus speed, as follows:
| ||||||||||||||||
| Base address registers | Each register that requests PCI memory or PCI I/O address space is programmed with a starting address.
|
When attaching a device, the device driver can set any other configuration parameters.
 | Caution: If the driver changes the contents of a base address register, the results are unpredictable. Do not do this. |
The following optional signal lines are not supported:
The LOCK# signal is ignored; atomic access to memory is not supported.
The cache-snoop signals SBO# and SDONE are ignored. Cache coherency is ensured by the PCI-X adapter and the memory architecture, with assistance by the driver.
In SGI Altix systems, addresses are translated not once but at least twice and sometimes more often between the CPU and the device, or between the device and memory. Also, some of the logic for features, such as prefetching and byte-swapping, is controlled by the use of high-order address bits. There is no simple function on a physical memory address that yields a PCI-X bus address (nor vice-versa). The device driver must use the PIO addresses presented in the pci-dev structure. For more information, see Chapter 7, “PCI-X I/O and Memory Resources”.
It is also necessary for the device driver to call the relevant DMA mapping routines for DMA addresses (PCI-X bus addresses). For more information, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.
SGI Altix systems support 64-bit data transactions. Use of 64-bit data transactions results in best performance. The PCI-X adapter accepts 64-bit addresses produced by a bus-master device. The PCI-X adapter does not generate 64-bit addresses itself. (This is because the PCI-X adapter generates addresses only to implement PIO transactions, and PIO targets are always located in 32-bit addresses).
In SGI Altix systems, the PCI-X adapter implements a standard, 64-bit PCI-X bus operating as follows:
One 133-MHz PCI-X card in one slot of the bus with the other slot empty and configured down.
Two 100-MHz PCI-X cards or two 66-MHz PCI-X cards on one bus.
Two 66-MHz PCI cards or two 33-MHz PCI cards on one bus.
For mixed-speed cards, the PCI-X bus will downgrade to the lowest card speed.
For PIO purposes, memory space defined by each PCI-X device in its configuration registers is allocated in the lowest gigabyte of PCI-X address space, below 0x400 0000. These addresses are allocated dynamically, based on the contents of the configuration registers of active devices. The I/O address space requested by each PCI-X device in its configuration registers is also allocated dynamically as the system comes up. For further information on PIO address use, see Chapter 7, “PCI-X I/O and Memory Resources”.
Any part of physical address space can be mapped into PCI-X bus address space for purposes of DMA access from a PCI-X bus-master device. The SGI Altix system architecture uses a 50-bit physical address, of which some bits designate a node board. The PCI-X adapter sets up a translation between an address in PCI-X memory space and a physical address, which can refer to a different node from the one to which the PCI-X bus is attached.
The device driver ensures correct mapping through the use of PCI DMA map routines.
If the PCI-X device supports only 32-bit addresses, DMA addresses can be established in 32-bit PCI-X space. When this is requested, extra mapping hardware is used to map a window of 32-bit space into the 50-bit memory space.
| Caution: The number of mapping registers is limited, so it is possible that a request for DMA translation could fail. |
Because of the possibility of the failure of a DMA translation request, it is preferable to use 64-bit DMA mapping when the device supports it. When the device supports 64-bit PCI-X bus addresses for DMA, the PCI-X adapter can use a simpler mapping method from a 64-bit address into the target 50-bit address, and there is no contention for mapping hardware. The device driver must request a 64-bit DMA map, and must program the device with 64-bit values. For further information on DMA mapping, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.
The PCI-X adapter maintains two priority groups, the real-time group and the low-priority group. Both groups are arbitrated in round-robin style. Devices in the real-time group always have priority for use of the bus. There is no kernel interface for changing the priority of a device.
Each PCI-X bus contains two unique interrupt signals. The INTA# and INTC# signals are wired together, and the INTB# and INTD# signals are wired together. A PCI-X device that uses two distinct signals must use INTA# and INTB#, or INTC# and INTD#. A device that needs more than two signals can use the additional signal lines, but such a device must also provide a register from which the device driver can learn the cause of the interrupt.
The PCI-X bus adapter chip that is used on all SGI Altix systems has eight input interrupts. PCI-X cards, however, can implement up to four different interrupts (A, B, C, and D), which might create a shared condition. Table 3-3, shows how interrupts can be shared on an SGI Altix system.
PCI-X slots | PCI-X Interrupt line A | PCI-X Interrupt line B | PCI-X Interrupt line C | PCI-X Interrupt line D |
|---|---|---|---|---|
Slot 0 | 0 | 4 | 0 | 4 |
Slot 1 | 1 | 5 | 1 | 5 |
Slot 2 | 2 | 6 | 2 | 6 |
Slot 3 | 3 | 7 | 3 | 7 |
Slot 4 | 4 | 0 | 4 | 0 |
Slot 5 | 5 | 1 | 5 | 1 |
Slot 6 | 6 | 2 | 6 | 2 |
Slot 7 | 7 | 3 | 7 | 3 |
For example, if a card in slot 0 uses INTA# and a card in slot 4 uses INTB#, there will be a conflict. In this case, the interrupt service routines (ISRs) of both cards will be called when the bridge interrupt pin 0 changes to active. If you try to connect to all four interrupt lines from the card, you will create a shared condition. This is called interrupt overloading.
Because SGI Altix systems support two slots and line A and line C are wired together and line B and line D are wired together, devices on the same bus will never share the same interrupt line.