This chapter includes the following main sections:
The two most common peripheral protocols for device communication in the computer industry are networks and channels.
Networks
Involve I/O interfaces that usually support many small transactions with relatively high overhead due to software involvement in the flow of information
Allow a host or device in the network to communicate with any other device
Operate in an open, unstructured, and unpredictable environment
Channels
Supply peripheral I/O interface to a host and transport large amounts of data between the host and peripherals
Keep data processing overhead to a minimum by handling data transfer in hardware, with little or no software involvement once an I/O operation begin
Operate in a closed, structured, and predictable environment where all devices that can communicate with a host are known in advance and any change requires host software or configuration table changes
Fibre channel attempts to combine the best of these two methods into an I/O interface that meets the needs of both channel users and network users. Fibre channel communications can be conducted over copper coax, twisted pair, or optical fiber. Note that Silicon Graphics currently supports only copper coax, with optical cable and a media interface adapter (MIA) as an option.
Fibre channel technology provides three different interconnect topologies to serve the combined needs of channel and network usages. These three topologies are:
Fabric
Point-to-point
Arbitrated loop
Currently, Silicon Graphics supports only arbitrated loop technology, (see “Silicon Graphics Fibre Channel Ports and Topologies”).
A fabric topology permits dynamic interconnections between nodes through ports connected to the fabric. This is a standard network configuration. In the future, it may be possible to carry both network traffic and disk I/O over the same wire.
In point-to-point host applications, two ports are used, connected to a link. The transmitter of each port is connected directly to the receiver of the opposite port. This topology limits the number of connections that can be made across the wire.
The arbitrated loop topology used by Silicon Graphics is called fibre channel arbitrated loop (FC-AL). In this topology, each port arbitrates for access to the loop. Ports that “lose” the arbitration act as repeaters of all traffic on the loop. The loop contains a dedicated transmit channel and a dedicated receive channel that are clad together into one cable to form a loop out and back. This protocol allows up to 127 ports connected in a serial loop (one FL_Port and 126 NL_Ports). Silicon Graphics supports a maximum of 110 ports in a single rack.
Ports are called N_Ports (Node_Ports), NL_Ports (Node_Loop Ports), F_Ports (Fabric_Ports), or FL_Ports (Fabric_Loop Ports).
FC-AL ports represent a superset of functions and must work correctly with all three topologies (fabric, point-to-point, and arbitrated loop).
An NL_Port represents each disk in a disk array. Each NL_Port sees all messages and passes messages not addressed to that port. Ports passing messages are said to be in “repeat mode”.
An FL_Port provides the connection from the fibre channel loop to the I/O board on the host, (in this case XIO or PCI). In the future, FL_Ports may provide a link from an arbitrated loop to a fabric.
Silicon Graphics fibre channel topology uses an arbitrated loop (FC-AL), as diagrammed in Figure 2-1.[2]
FC-AL is a shared-bandwidth distributed topology with arbitration fairness. The Silicon Graphics FC-AL option potentially supports 126 active L_Ports. Current support is limited to a maximum of 110 disks within a single rack. Communication on a loop is between two devices at a time. All nonactive FC-AL ports act as repeaters; all devices on the loop run at the same interface speed.
The arbitrated loop topology is a distributed topology in which each port includes the minimum necessary functions to establish a circuit. The ports are all L_Ports.
Disk L_Ports use the point-to-point protocol to create a point-to-point circuit between two L_Ports.
Port arbitration begins with one port replacing fill words between frames with an arbitration primitive called “ARBx.” If it goes completely around the loop and returns to the sending port, then that port has won arbitration.
A port that wins arbitration stops retransmitting received data and opens the loop between its receiver and transmitter by sending out the open primitive, OPNyx, where “y” represents the physical address of the destination port (AL_PD=Arbitrated Loop Physical Destination) and “x” represents the sending port. See Figure 2-2 for an example of loop functions.
Closing a loop is done by transmitting the close primitive (CLS). The partner port finishes its work and retransmits the CLS, and then the loop is available for arbitration again.
The FC-AL loop interface is an implementation of SCSI-3 architecture that provides high-performance, greater connectivity, and high-availability features. FC-AL was designed as a low-cost method of connecting multiple hosts and storage devices without requiring the use of switches and fabrics. Note that multi-host configurations require special optional software and hardware; ask your Silicon Graphics sales or service representative for information.
Data transfers in fibre channel use the following parameters
DC-balanced 8b/10b signals with odd or even disparity
variable-length data frames (maximum 2 KB)
32-bit CRC on frames
Combining channel and LAN characteristics, fibre channel uses buffers, one at the source node (port) and one at the destination. Each buffer can be any size.
FC layers 0, 1, and 2 move data from one buffer to another without regard to the format or meaning of data, thus avoiding the overhead required for handling different network communications protocols. FC layers 0, 1, and 2 provide only control of the transfer and simple error detection, relying on the content of the message header to trigger actions such as routing data to the proper buffer. Except for how FC formats serial data for transmission, protocols are irrelevant; communication protocols between devices can be whatever those devices require. The Silicon Graphics fibre implementation uses SCSI Framing Protocol (FCP) for FC-AL communications between devices and the host system.
Information can flow between two ports—actually between their buffers—in both directions simultaneously. The mechanism coordinating the flow of information between two N_Ports is an exchange.
Data transfer takes place in units called frames, each a maximum of 2048 bytes of data. The frame contains the information to be transmitted, the address of the source and destination ports, and link control information. Frames are of two types, Data frames and Link_Control frames. Data frames can be used as Link_Data frames and Device_Data frames. A set of related frames for one operation is called a sequence.
There are no limits on the size of a transfer. Frame sizes are transparent to upper-level software because the unit of transfer is a sequence. A specific layer of fibre channel (FC-2, described in “FC-2” later in this chapter) is responsible for breaking a sequence into the frame size that was negotiated between ports. This layer of FC also detects transmission errors using 32-bit CRC.
Fibre channel is a layered architecture with five layers: FC-0, FC-1, FC-2, FC-3, and FC-4. Figure 2-3 diagrams the relationship between FC layers and OSI layers.
FC-PH is the FC physical and signaling interface, revision 4.1, defined in the FC-PH standard (ANSI X4.3). FC-PH is made up of FC-0, FC-1, and FC-2. Each FC layer is described and explained in the following separate sections:
FC-0 defines the physical interface (media), or link, encompassing a wide variety of media, speed, and distance combinations. It defines the physical link in the system, including the fibre, connectors, and optical and electrical parameters for a variety of data rates. Table 2-1 summarizes the Silicon Graphics implementation for FC-0.
 | Note: In FC, there is no change in protocol between copper and fiber optic. |
Table 2-1. Silicon Graphics Fibre Channel Cable Characteristics
Parameter | Copper Cable | Fiber Optic Cable Option |
|---|---|---|
Value | 100-TW-EL-S full-duplex (quad conductor) | 100-M5-SL/N-I |
Connector | Male DB9 | SC duplex connector |
Media type | 150 ohm copper | 62.5-micron multimode fiber |
Speed | (1.0626 Gbaud) physical interfaces | 1.0626 Gbaud |
Transmitter type | PECL | 780 nm optical laser |
Distance | Maximum 25 meters from controller to enclosure | Maximum 300 m from controller to enclosure with optical cable |
 | Caution: To maintain acceptable quality and signal integrity, use only Silicon Graphics supplied copper and optical cables with your fibre enclosures. |
FC-1 defines the transmission protocol, including serial encoding and decoding rules, special characters, and error control. The information transmitted over a fiber is encoded 8 bits at a time into a 10-bit transmission character. The transmission code, which must be DC balanced to support the electrical requirements of the receiving units, improves the transmission characteristic of information. The transmission characters ensure that short-run lengths and enough transitions are present in the serial bitstream to make clock recovery possible.
FC-2, the signaling protocol, is the transport mechanism of fibre channel. It defines the framing rules of the data to be transferred between ports, the different mechanisms for controlling service classes, and the means of managing the sequence of a data transfer.
The standard defines these building blocks to aid in the transport of data across the link:
An ordered set is a four-byte transmission word containing data and special characters that have a special meaning. Ordered sets provide the availability to obtain bit and word synchronization, which also establishes word boundary alignment. An ordered set always begins with the special character K28.5. Three major types of ordered sets are defined by the signaling protocol:
Frame delimiters: start-of-frame (SOF) and end-of-frame (EOF), which immediately precede or follow the contents of a frame.
Two primitive signals (primitives), idle and receiver ready (R_RDY):
Idle indicates an operational port facility ready for frame transmission and reception.
R_RDY indicates that the interface buffer is available for receiving further frames.
The primitives are transmitted and repeated continuously to indicate specific conditions within a port or conditions encountered by the receiver logic of a port.
Four primitive sequences:
Offline (OLS)
Not Operational (NOS)
Link Reset (LR)
Link Reset Response (LRR)
When a primitive sequence is received and recognized, a corresponding primitive sequence or idle is transmitted in response. Recognition of a primitive sequence requires consecutive detection of three instances of the same ordered set.
The basic building block of an FC connection is the frame, which contains the information to be transmitted (payload), the address of the source and destination ports, and link control information. Frames are broadly categorized as data frames and Link_control frames. Data frames can be used as Link_Data frames and Device_Data frames. Link_control frames are classified as acknowledge (ACK) and Link_Response (busy and reject) frames.
Each frame begins and ends with a frame delimiter. The frame header immediately follows the SOF delimiter. The frame header controls link applications and device protocol transfers and detects missing or out-of-order frames. An optional header can contain additional link control information. A field up to 2112 bytes (payload) contains the information to be transferred from a source N_Port to a destination N_Port. The four-byte cyclic redundancy check (CRC), which precedes the EOF delimiter, detects transmission errors.
A sequence is formed by a set of one or more related frames transmitted unidirectionally from one port to another. Each frame in a sequence is uniquely numbered with a sequence count. Error recovery, controlled by an upper protocol layer is usually performed at sequence boundaries. For more detail, see “Data Transfer in Fibre Channel” in this chapter.
An exchange is composed of one or more nonconcurrent sequences for a single operation. The exchanges can be unidirectional or bidirectional between two ports. Within a single exchange, only one sequence can be active at any one time, but sequences of different exchanges can be active concurrently.
The protocols are related to the services offered by
Primitive sequence protocols for link failure
N_Port login protocol: before performing data transfer, an N_Port interchanges its service parameters with another N_Port
N_Port logout protocol: performed when an N_Port requests removal of its service parameters from the other N_Port (used to free resources at the connected N_Port)
Data transfer protocol: methods of transferring upper-layer protocol (ULP) data using FC flow control management
Flow control paces the flow of frames between N_Ports and between an N_Port and the fabric to prevent overrun at the receiver. Class 3 uses only buffer-to-buffer flow control. Flow control is managed by the sequence initiator (source) and sequence recipient (destination) ports using Credit and Credit_CNT:
Credit is the number of buffers allocated to a transmitting port.
Credit_CNT represents the number of data frames that have not been acknowledged by the sequence recipient.
Buffer-to-buffer flow control is managed between an N_Port and an F_Port or between N_Ports in point-to-point topology. Each port is responsible for managing BB_Credit_CNT. BB_Credit is established during the fabric login. The sequence recipient (destination) port signals by sending a Receiver_Ready primitive signal to the transmitting port when it has free receive buffers for the incoming frames. Frame delivery is not confirmed.
Silicon Graphics fibre channel provides for service class 3. A connectionless service (“hopeful” or “datagram”), Class 3 is the most practical for connections requiring a lot of time. Class 3 has no end-to-end flow control or delivery acknowledgments. A datagram is the transfer of data without first establishing that data will be delivered. Timeouts are used to decide whether information is lost and retransmission required.
FC-3 defines the common services provided by FC-PH to the ULPs. These services, which include striping, hunt groups, and multicast, are not yet implemented.