Fibre Channel arbitrated loop (FC-AL)

Overview
When Fibre Channel was first introduced, it was a new technology and everything was expensive. Switches, hubs, and node transceivers proved to be costly. Arbitrated Loop topology lies between point-to-point and switched fabric in that it provides more connectivity than point-to-point with up to 126 NL_Ports in a loop, but less than switched fabric which has the ability in theory to support up to 16 million ports. It was a cost-effective way of connecting a limited number of ports in a loop single network.
Fibre Channel arbitrated loop (FC-AL) is a daisy-chain connecting up to 126 devices in a loop configuration over attachment points called L_Ports (loop ports). FC-AL is a low-cost connectivity solution because it does not require switches. FC-AL is a good choice for small to medium-sized configurations, and provides a growth path by allowing connection of a loop to a switched fabric.



Efficiency and connectivity is enhanced by incorporating one or more hubs into the loop. Routing traffic through a hub on each leg of a loop eliminates the loss of the entire loop, as happens in a hubless loop.
The arbitrated loop topology promoted the introduction of Fibre Channel by removing the cost of a fabric switch and, depending on the number of nodes in the loop, the amount of transceivers could also be reduced.
Arbitrated loop provides more connectivity than point-to-point in that it can support 126 NL_Ports and 1 FL_Port on a single loop and is a middle ground between point-to-point and switched fabric. In arbitrated loop, the transmit output of one port is connected to the receive input of the next and these connections are made between all the nodes until a closed loop is formed. This type of configuration is usually made using a Fibre Channel hub which eliminates the need to form the logical loop using cabling.

In arbitrated loop each port sees all messages on the loop and ignores or passes those messages which are not addressed to that particular port.

 
Picture.2 Arbitrated loop


Loop construction
An arbitrated loop is constructed by connecting nodes together in a single loop. Loops can be constructed by physically connecting each node in the loop or through the use of a Fibre Channel hub. The transmit of one port is attached to the receive of the next. This is continued until the loop has been formed. A loop can contain 126 N_L Ports and one F_L Port which is used for connection to a Fabric environment. Information that is passed around the loop is repeated by each port and passed on if it is not the required destination.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Picture.3 Arbitrated loop with 4 nodes

Picture 3 shows an example of how an arbitrated loop might be constructed. In this loop, there are four nodes and four transceivers, keeping the Fibre Channel circuitry to a minimum. In this configuration, the bandwidth is shared between all nodes on the loop and if any new nodes were added it would be further reduced. In this configuration if any of the ports failed then the complete loop would be lost. With the absence of a hard failure, this loop could be thrown into turmoil if any intermittent type failure should arise. Any blip whatsoever would force the whole loop into a re-initialization state and halt all I/O operations until the loop is back to a stable state. This is not desirable in a highly-available storage subsystem which is why today most fibre connectivity to a Symmetrix is through a fabric switch. However, with the emergence of dual port fibre devices, the arbitrated loop is a viable option, especially as a back-end solution within a storage array.
Arbitrated loop performance is dependant on a number of factors.
Some of these are obvious, such as the number of nodes on the loop.
Clearly, in a shared bandwidth medium, the population fighting for that medium will have a direct impact on performance. However, this is not the only way the number of nodes can affect performance.

With the introduction of more nodes, the roundtrip time of the loop is also increased. This is because each frame or sequence may have additional ports to pass through to reach the final destination, and with each additional port adding some latency, overall throughput can be affected. Likewise, the time it takes to win arbitration can increase.


Loops with hubs
Another way of constructing a loop is to use a hub. As the cost of transceivers and hubs have dropped dramatically from when they were first introduced, the benefits obtained with this configuration can outweigh the extra expense. As shown on picture 4, the loop is constructed within the hub.
 

Picture.4 Arbitrated loop with a hub

The first improvement is the ease of cabling. On picture.4, each transmitter had to be routed to the receiver of the next node and so on. As shown on picture.4, all that is needed to construct the loop is to plug each cable directly to the hub. This example shows that for the same number of nodes in the loop, the number of transceivers will double, adding to the cost.

The hub brings another useful feature to the table. A failure would cause the whole loop to collapse. However, the hub can bypass a faulty port and allow the remaining ports to operate as normal, as illustrated on picture 5.

 
Picture.5 Hub port bypass


Arbitration
The process through which a fibre port gains sole control of the loop is called arbitration. Once a port has control of the loop, and it has opened another port, it is free to transmit frames to that port. Picture.6 shows a loop which is available; in other words devices are free to arbitrate to gain access to this loop.

 
Picture.6 Arbitrated loop (FC-AL)

Arbitration prevents multiple ports from sending frames at the same time on the loop. After a device has won arbitration and opened another device then, in effect, a point-to-point connection has been established. All remaining ports cannot participate in any way other than passing along whatever comes their way until the loop has been closed and is once again available for arbitration to all.

 
Picture.7 Loop after arbitration won and ports opened

Picture.7 illustrates the state of the loop after a successful arbitration by port one or four. The arbitration winner opened the other port and now ports two and three have been logically removed from the loop forming a virtual point-to-point connection between port one and four.

EMC first introduced arbitrated loop in a direct connect from HBA to Symmetrix FA configuration (2 Node Arbitrated Loop). (Refer to Picture.8). This was the beginning of Fibre Channel connectivity to the Symmetrix and the configuration was limited to a minimum until all of the teething problems associated with a new technology had been identified and corrected.

 
Picture.8 First implementation on Symmetrix

Very quickly, hubs were introduced to the configuration and the connectivity was expanded (Picture.9). The hubs provided resiliency in that they provided a bypass circuit on each port which is used to bypass a particular port if there is not a valid signal present, allowing the rest of the loop to continue to operate.
Basically, they ensure that a bad port or a broken fibre will not take down the entire loop.

 
Picture.9 Expanded connectivity


Dual port devices
In dual loop drive systems each disk drive has two ports and thus the drive can be connected on two separate loops. As shown on picture 10, if a failure occurs on one loop, for example Initiator 1 fails or a port fails on one of the disks, the disks will remain accessible through the second initiator. This could be compared to PowerPath at the disk level.

 
Picture.10 Dual loop disk drives

This is not the only solution available with dual port disks. Hubs could also be included which would provide further connectivity and resiliency options (Picture 11).

 
Picture.11 Highly available dual port disk solution


Arbitrated loop addressing
Fibre Channel specifies a three-byte field for the address used in routing frames. In arbitrated loop, only one of these three bytes (least significant 8 bits) is used for the address which is known as the Arbitrated Loop Physical Address (AL_PA). This address is used in the Source ID (S_ID) and Destination ID (D_ID) of frames transmitted in the loop.

 
Picture.12 Loop addressing

Picture.12 shows the full 24-bit address defined by the Fibre Channel standard. Eight (8) bits used by the AL_PA. Bits 8 to 23 are used for the FL_port identifier and the full 24 bits are used by an N_Port in a fabric switch environment.
The AL_PA values used are limited to characters that result in neutral disparity after encoding. AL_PA 00 is reserved for FL_Port and the remaining 126 AL_PA are distributed irregularly between 01 and EF.
If two ports arbitrate to get access to the loop at the same time then only one port can be given access. This is decided by assigning priorities to the AL_PA addresses with 00 having the highest and 01 through to EF having decreasing priority.

 
Picture.13 AL_PA priority


Loop ID
The usable AL_PA values are made up of the neutral disparity characters in the range 0x00 to 0xFF and are not distributed regularly.
The used AL_PA values range from 0x00 to 0xEF as is shown on Picture.14. When you configure a Symmetrix FA to have a certain AL_PA you do not set an AL_PA value directly but instead specify a Loop ID value in the bin file and this Loop ID then corresponds to a certain AL_PA value. The Loop ID values are sequential between 0x00 and 0x7E and thus can be easier to manage.
The correlation of Loop ID to AL_PA is shown on Picture.14.

 
Picture.14 AL_PA to loop ID chart

Arbitrated loop has several ordered sets used in loop arbitration and opening and closing of loop circuits. An ordered set is a group of four transmission characters, the first being the K28. Five special characters and the remaining three (data) characters define the meaning of the ordered set. These ordered sets can exist either at the start or end of the frame (in the case of frame delimiters), or can exist on their own (in the case of primitive signals and sequences). For more information on ordered sets.
Primitive signals are normally used to indicate events or actions at the sending port. A table of these primitive signals is shown on Picture.14. For example, the ARB(x) primitive signal is used by a port in the arbitrating state to indicate that it is arbitrating for access to the loop. The x indicates the AL_PA assigned to that specific port.

 
Picture.15 Primitive signals

Primitive sequences are used to indicate states or conditions and are normally transmitted continuously until something causes the state to change. A minimum of three consecutive occurrences of the same ordered set is required before a primitive sequence is recognized and action taken. Picture.16 shows Arbitrated Loop primitive sequences used in link initialization and loop port bypass. Switched fabric uses a different method of link initialization than arbitrated loop, but it does still use primitive sequences.

 
Picture.16 Primitive sequences

An AL_PA identifies either a source or destination port in arbitrated loop. In some cases it is necessary to identify whether it is the source or destination AL_PA that is being referred to. When it is necessary to identify a destination port, the term AL_PD is used. In the case of the source port, the term AL_PS is used. Picture.16 shows different types of LIP (Loop Initialization Primitive) sequence used in loop initialization. Each is discussed below.
Loop Initialization LIP(F7, F7). A port transmitting LIP(F7, F7) indicates that the port in the initializing state is requesting loop initialization but does not have a valid AL_PA. This mainly occurs when a device is hot-plugged into a Loop or when a port that was non-participating wants to become participating and requires an AL_PA to do so.
Loop Failure LIP(F8, F7). A port transmitting LIP(F8, F7) indicates that the port in the initializing state is requesting a loop initialization due to a loop failure. The port does not have an AL_PA and uses F7.
This could occur if a non-participating loop port without an AL_PA detects a loop failure or a node in the process of getting an AL_PA detects a loop failure.
Loop Initialization LIP(F7, AL_PS). This LIP indicates that the loop port identified in the AL_PS value is requesting loop initialization.
This can be used if the port detects a performance degradation, arbitration wait time-out, or for another unspecified reason.

Loop Failure LIP(F8, AL_PS). This LIP indicates that the loop port identified in the AL_PS value has detected a loop failure. This may occur when a loop interconnection has failed, a loop port has failed, a loop port has been powered off or removed from the loop when no bypass circuit is present, or if the bypass circuit fails.
Selective Reset LIP(AL_PD, AL_PS). The selective reset LIP is used to perform a vendor specific reset at the loop port specified in the AL_PD value. The AL_PS value indicates the port that originated the request. This LIP could be used during error recovery to reset a port that is in a hung state.
Along with the loop initialization primitive sequences, there are a number of sequences to set and reset the LP_Bypass variable in the Loop Port State Machine (LPSM). With this variable set, the LPSM retransmits frames and does not attempt to arbitrate or participate in the loop. The state of this variable is also used to control an optional port bypass circuit to electrically bypass the loop port.
Loop Port Enable LPE(yx), LPE(fx). These primitive sequences cause either a designated port (yx) or all ports (fx) to reset the LP_Bypass variable and deactivate a control line to an external port bypass circuit if present.
Loop Port Bypass LPB(yx), LPB(fx). These primitive sequences, when received, cause the designated port (yx) or all the ports (fx) to set the LP_Bypass variable and optionally activate a control line to an external port bypass circuit if present.


Loop Port State Machine (LPSM)
A port on an arbitrated loop is required to behave in a certain manner in order to operate correctly on the Loop. The LPSM defines this behavior (Picture.17). The particular states that can occur on a loop include: initialization, arbitration, open circuit, close circuit, and implement fairness. These various states are implemented using specific ordered sets which are processed by the LPSM of the ports on the loop.

 
Picture.18 Loop Port State Machine (LPSM)

The basic operations of a Loop Port State Machine are:
◆    When a port is first attached to the loop it enters the initializing state and starts loop initialization where each port transmits a LIP continuously and monitors for a LIP returning. On receipt of a LIP the ports go into a Open-Init state where most of the Loop initialization steps take place. If the loop is not operational the port may go to into the Old_Port state disabling Arbitrated Loop functionality and begin operating as an N_Port.
◆    If the ports have no work to do they are in the monitoring state transmitting and receiving Idles.
◆    When the loop port requires access to other ports in the loop it enters the arbitrating state and begins arbitrating for access. When the port wins arbitration it enters the ARB-WON state.
◆    When the port needs to access another port on the loop it sends an OPN to select the destination port. The source port goes into the OPEN state and the destination port goes into the OPENED state.
◆    Once both ports are in these states transfer of frame can begin.
◆    When either of the ports has completed its transfer and wants to close the connection it sends a close (CLS). The port that sends the CLS enters the transmitted close state and the when the other port receives the CLS it enters the received close state. It transmits its remaining frames, if necessary, as long as it has available credit and then transmits a CLS back to the originator. The two ports are now logically disconnected from the loop and enter the monitoring state again.
◆    If a loop port is in the monitoring state or arbitrating for the loop and receives an OPEN from another port it enters the OPENED state.
◆    There is another state that a port can enter called the transfer state.
◆    If a port has frames to send to multiple ports and is finished communicating with the first of these ports it can send a CLS, move to the transfer state, and once it receives the CLS from the first port it can open the second port without going through an arbitration cycle.



Loop initialization
Before discussing all the steps in detail that take place during a loop initialization, it is important to understand why this step must be done and exactly what is accomplished during this step.

Overview
The loop initialization step performs a number of functions in arbitrated loop including the assignment of addresses (AL_PAs) to loop ports, notification that the configuration may have changed, and notification of a loop failure. Events that can cause a loop initialization include: a) if a port was powered on it may need to acquire an AL_PA and notify other ports on the loop that the configuration has changed; or b) a port detects a physical connection problem and begins the loop initialization process to notify other ports and check if the loop is still operational. The main steps involved in loop initialization are listed on Picture.19.

 
Picture.19 Initialization procedure

Each node on a loop is capable of initiating loop initialization by transmitting one of the LIP sequences shown on Picture.17. When a node transmits this LIP sequence all the other ports on the loop recognize this and enter the OPEN_ INIT state and retransmit the LIP along the loop. This occurs until the LIP arrives
back at the port that is initiating the initialization and it too enters the ‘OPEN_ INIT’ state.
With all ports on the loop now in the OPEN_ INIT state, the next step is a process to select a port to become the temporary loop master. This step is initiated by each node on the loop that has entered ‘OPEN_INIT’ state continuously sending out LISM (Loop Initialization Select Master) frames. LISM frames have a certain format, detailed on Picture.20, and serve the purpose of allowing each node on the loop a chance at becoming the temporary loop master.

 
Picture.20 LISM frame format

The structure of the LISM frame is shown on Picture.20 with the S_ID and D_ID fields in the frame header showing a value of ‘0000xx’. The least significant byte is a value of 00 if the port is an FL_Port, or EF if the port is an NL_Port which identifies the port type. The payload of the LISM frame contains the 8 byte World Wide Name of the port.
If there is a single FL_Port on the loop it becomes the loop master. If there are more than one FL_Port then the one with the lowest port worldwide number becomes the master. The reason the FL_Port becomes the master is based on the fact that the FL_Port is part of a fabric and therefore has more knowledge of the configuration. If there are no FL_Ports present then the NL_Port with the lowest port worldwide number is selected as temporary loop master. Normally, it is an HBA that becomes the loop master as its worldwide number is lower than that of a Symmetrix FA.
The process involved at the LISM stage involves each port transmitting LISM frames and each port checking the LISM frames it is receiving for the port type field which is denoted in the least significant byte of the S_ID and D_ID fields. If an NL_Port receives a LISM frame from an FL_Port it stops transmitting its own LISM frames and begins retransmitting the LISM of the FL_Port. If an FL_Port receives a LISM frame from an NL_Port it discards the received frame and begins transmitting its own frame. If the port type in the S_ID and D_ID fields of the frame header is the same as that of the receiving port, the port worldwide number in the payload of the LISM frame is compared with that of the receiving port. If the port worldwide number in the received LISM frame is higher than that of the receiving port, the port discards the frame and the port continues to transmit its own LISM. If the port worldwide number in the received LISM frame is not higher, it stops transmitting its own LISM frame and begins transmitting the received frame. Eventually one of the loop ports will receive back around the loop its own LISM frame and when this happens this port becomes the temporary loop master.
This port then begins transmitting the ARB(F0) primitive signal to inform the other ports that the LISM procedure has been completedand a loop master selected.
The next step in the process is AL_PA assignment. This process involves the assignment of Arbitrated Loop Physical Addresses (AL_PA) to each port on the loop. A port’s AL_PA is its Fibre Channel address on the loop and is used to identify it on the loop and is put in the S_ID of the frame header of all data frames that are transmitted by the port and is in the D_ID of all data frames that are to be received by the port. This is a multistep process where addresses are assigned using four distinct steps depending on the particular ports operation.
These four steps are discussed next and all use the concept of populating an AL_PA bitmap with a value depending on whether that corresponding AL_PA is assigned or not.
In arbitrated loop there are 127 possible addresses on a loop (126 for NL_Ports and one for the FL_Port). To identify which AL_PA values have been taken in a loop a 128 bit (four word) map is used where each bit corresponds to a certain AL_PA. If a bit is set to 1 then the corresponding AL_PA is assigned and that address is in use. If the bit is 0 then the corresponding AL_PA is available to be acquired. Word 0 bit 31 of the AL_PA bit map is the login-required bit (L-bit) which is set by an FL_Port to indicate that the configuration has changed and that all ports are logged out.

 
Picture.21 LIFA / LIPA / LIHA / LISA frames

Step 1. The first step in the AL_PA assignment is the LIFA (Loop Initialization Fabric Address) which basically allows public ports that had previously been logged in with the fabric (FL_Ports) to reclaim the AL_PA they had been using prior to loop initialization. The temporary loop master first initializes the AL_PA bitmap to all ‘0’s and builds the LIFA frame with this bitmap in payload words two through five and the loop initialization identifier in word 1, as is shown on Picture.21.
If the loop master had a fabric assigned address prior to the loop initialization it sets the bit for that AL_PA in the bitmap and then transmits this frame. If the next port on the loop had a fabric assigned
AL_PA it also sets the corresponding bit in the AL_PA bitmap (if not already set) and retransmits the frame. If the next loop port did not previously have a fabric assigned AL_PA then it would leave the AL_PA map unchanged and retransmits the frame. This continues until every port on the loop processes the LIFA frame and is received back by the temporary loop master.

Step 2. The next step of the process is LIPA (Loop Initialization Previous Address) where private ports (NL_Ports) that had an AL_PA prior to loop initialization can reclaim the same AL_PA. This step is initiated by the temporary loop master by changing the loop initialization identifier in word 1 of the LIFA frame it received to LIPA identifier and then transmitting this around the loop. Each port starting with the loop master checks to see if it had a nonfabric assigned AL-PA prior to loop initialization. If it had, it first checks to see if that bit is set in the AL_PA bit map and if it is not set then sets it to 1 and reclaims the AL_PA. If the bit is set then the port would have to wait for a soft-assigned address. This continues until every port on the loop has processed the frame and it is received back by the loop master.

Step 3. The next step of the initialization process is for ports that did not have an AL-PA prior to loop initialization but do have a preferred AL_PA that is set by either jumper settings or some other configuration method. This step is known as the LIHA (Loop Initialization Hard Address)/ The Symmetrix FA would be an example where the AL_PA can be configured in the bin file by setting a corresponding Loop ID value. The temporary loop master changes the received LIPA frame loop initialization identifier in word 1 of the payload to LIHA which indicates this is now a LIHA frame which will be transmitted around the loop. Each port that did not have an AL_PA prior to link initialization but does have hard assigned AL_PA checks the bit corresponding to that AL_PA in the LIHA frame payload words two to five and then sets the bit and claims that AL_PA. If the bit is already set then the port would again need to wait for a soft-assigned address. This continues until every port has processed the LIHA frame and it is received back around the loop by the temporary loop master.

Step 4. The final step in acquiring an AL_PA is the LISA (Loop Initialization Soft Address) where the port may select the first available AL_PA in the bitmap. Once the loop master has received back the LIHA frame it changes the identifier in word 1 from LIHA to LISA and leaves the AL_PA map unchanged. It then transmits the frame and every port that does not have an AL_PA scans the bitmap in the LISA frame to find the first available AL_PA and then claims that soft-assigned address by setting that bit in the AL_PA bitmap.
The AL_PA values are normally assigned starting at the most significant bit of the AL_PA map and proceeding to the least significant bit. This provides the most efficient ordering of AL_PA values around the loop. For the best performance the AL_PA values should be arranged in descending priority in the direction of information flow around the loop. AL_PA 00 is the highest priority.
If after these steps a port has not acquired an address, for example if all the available AL_PA s have been taken or if a ports hard address is already in use, then the port will enter the non- participating mode. If the port does not support loop positional mapping steps which follow AL_PA assignment it sets the third byte of the loop initialization identifier (Word 1) to 00.
A method assigning addresses during the LISA step is to have initiators acquire AL_PA values in the higher priority end of the AL_PA bitmap starting at the most significant bit and proceeding to the least significant bit and targets do the opposite. However, this does not lead to the most efficient ordering of the AL_PAs on a loop for target devices as they would get AL_PAs assigned starting with the lower priority.
A preferred method of assigning AL_PA values during the LISA process is to have a range of AL_PA values at the higher order end of the AL_PA bit map reserved for initiators and have targets assign soft addresses beginning at the end of this range. This would avoid the AL_PA ordering problem in that the initiators would get the higher priority AL_PA values in descending order and the targets would get the lower priority AL_PA values also in descending order around the loop.
There are two additional steps in the initialization stage that are needed to provide information on the positioning of AL_PAs in the loop for managing the configuration and problem analysis. These two steps are Loop Initialization Report Position (LIRP) and Loop Initialization Loop Position (LILP):
The LIRP step involves building a map of the AL_PA values according to their position on the loop relative to the temporary loop master. The temporary loop master begins this process by building a LIRP frame with the structure shown on Picture.22.

 
Picture.22 LIRP/LILP frame format

The basic structure of the frame payload is a one word loop initialization identifier, a one byte offset and up to 127 AL_PA entries. The temporary loop master first of all initializes the structure by setting the offset to 01 and storing its AL_PA at offset 01 in the AL_PA map and puts FF in all the other remaining positions. This frame is then passed to the next port on the loop, the offset is incremented by 1 and the next port stores its AL_PA at that location in the map. This process continues until the LILP frame comes back around the loop to the temporary loop master.

When the loop master receives the LIRP frame back it changes the identifier in the payload of the frame to a LILP and retransmits the entire frame so each port can have a copy of the AL_PA positional map. Once this frame comes back around the loop the loop initialization is complete.


Login process
Once the loop is initialized each port has acquired an address but each initiator (HBA in server) does not know what target devices are on the loop. For each HBA to discover what targets are on the loop it needs to perform some extra steps. These steps provide ports with a means of exchanging information about each other that is used to control any communication that is initiated by these ports. Fibre Channel provides three different types of login that can occur between ports by the use of the following extended link services:
◆ Fabric Login (FLOGI)
◆ N_Port Login (PLOGI)
◆ Process Login (PRLI)
The Fabric login process is used in a switched fabric environment to allow an N_Port establish a session with the fabric. During this step both the N_Port and the fabric exchange parameters with each other and identify themselves to each other. This step also assigns an address to the attached N_Port.

 
Picture.23 FLOGI and Accept

Picture.23 shows the exchange of information that takes place with the N_Port sending the initial FLOGI frame (Command code 04) with all its information and the Fabric returning the Accept (Command code 02) with its associated parameters.
In arbitrated loop the Fabric login does not take place and the ports exchange service parameters with N_Port login, which is also known as PLOGI. Service parameters are basically information regarding the FC-2 capabilities of a port such as maximum frame size that can be received or end-to-end credit values. Again the N_Port login is performed by the initiator sending a PLOGI request frame and the target returning an accept.

 
Picture.24 PLOGI and Accept

Picture.24 shows the PLOGI and the Accept back from the target with their respective command codes. The payload of both the request and the accept PLOGI frames contain the service parameters of the initiator and the target.
Process Login (PRLI) is the process to allow two ports exchange service parameters relating to the FC-4 type they are using. Specifically what you would see here is SCSI type information being exchanged by both ports in a request and accept fashion as is shown on Picture.25.

 
Picture.25 PRLI and Accept

Some of the parameters exchanged during process login are whether the port is an initiator or target, transfer ready being used for reads or writes, and the FC4 type being used (0x08 for SCSI in these implementations).


Arbitration process

Overview
Arbitration is the process that allows a loop port to gain access to the loop and ensure that only one port at a time is sending information. If this was not the case then ports could send data at the same time and interfere with each other. If two ports arbitrate at the same time to get on to the loop then there has to be a mechanism to handle multiple simultaneous requests. This is done by allowing the port with the lowest value AL_PA to have priority over ports with higher value AL_PAs. This could cause a situation where higher priority loop ports monopolize the loop, but to handle this a fairness mechanism is incorporated into the arbitration protocol.
This section discusses the following topics:
◆ “Arbitration process steps ”
◆ “Access fairness ”
◆ “Opening and Closing the loop ”

Before going into the arbitration process the concept of Fill Words needs to be understood. In Fibre Channel, even if there are no frames being transmitted, idle words are continuously transferred around the loop. When frames are being sent around the loop a certain amount of idles are also transmitted between these frames. These idles are a form of fill word and must exist between frames in a Fibre Channel environment. During arbitration it is necessary for a port to remove a fill word it has received and transmit a different fill word in its place. This is known as fill word substitution. When a port needs to transmit a fill word it will use the value contained in the current fill word. For example, if a port wants to start arbitrating on a loop which is not being used it needs to change the current fill word from ARB(x) to IDLE and once this is done the port can transmit ARB(x) instead of the received IDLEs.
Once the loop initialization phase has completed the loop is filled with IDLEs as each port is in the monitoring state. Once a port needs access to the loop it has to arbitrate to get this access. The six steps involved are discussed next, using an example of a loop containing four ports with one port arbitrating as shown on Picture.26.

 
Picture.26 Arbitration Step 1

Step 1 of the Arbitration process, shown on Picture.26, shows a loop which is filled with idles and all ports in the monitoring status. The current fill word on all the ports is IDLE and thus received IDLEs are substituted with the current fill word which is IDLE on all the ports.

 
Picture.27 Arbitration Step 2

Step 2 (Picture.27) shows Port 1 arbitrating for access to the loop by changing its current fill word to ARB(1) and substituting ARB(1) for every received IDLE. If Port 1 receives any ARBs, it needs to check the priority of these ARBs and then either discard if they are a lower priority than ARB(1) or substitute if they are a higher priority.

 
Picture.28 Arbitration Step 3
In Step 3 (Picture.28), once the ARB(1) is received by port 2 it changes its current fill word to ARB(1) and the ARB(1) is transmitted to the next port on the loop. The current fill word on the remaining ports is also changed in this fashion allowing the ARB(1) to propagate around the loop.

 
Picture.29 Arbitration Step 4

In Step 4 (Picture.29), once the ARB(1) is received by Port 1 it has won arbitration and immediately changes its current fill word to ARB(F0). Port 1 now will discard any received ARB(x) and thus this prevents any other loop port from winning arbitration as its ARB(x) could not make it around the loop.

 
Picture.30 Arbitration Step 5

In Step 5 (Picture.30) each port that receives the ARB(F0) changes its current fill word to ARB(F0) and transmits the ARB(F0) whenever required. This is transmitted by the winner of the arbitration to determine if any other ports are arbitrating. If there are other ports arbitrating then they substitute their ARB(x) for the ARB(F0).
As long as no other port attempts to start arbitrating, the ARB(F0) makes it around the loop to port 1 and the current fill word on port 1 is changed to IDLE.

 
Picture.31 Arbitration Step 6

In Step 6 (Picture.31) each port that receives the IDLE changes its current fill word to IDLE and as long as no other port is arbitrating the IDLE’s travel around the loop. As long as Port 1 owns the loop it discards any received ARB(x) or IDLE and continues to send IDLE as its fill word. As it is discarding any received ARB(x) it prevents any other loop port from winning arbitration.
Picture.30 details how a single port arbitrating win ownership of the loop but one must also consider what happens if multiple ports begin arbitrating at the same time. This is handled by the fact that AL_Pas have different priorities and in this case the higher priority AL_PA (lower numerical value) will win the arbitration.


Access fairness
Arbitrated loop assigns different priorities to different AL_PA values and thus the situation could occur on a loop where the higher priority AL_PA ports could dominate the loop and prevent access from the lower AL_PA ports. To prevent this, access fairness is used to counteract AL_PA priority and ensure every loop port has an equal chance to access the loop. Access fairness does not mean how long a port that has won arbitration can use the loop, but rather ensures that once a loop port has won arbitration and given up control of the loop it will not arbitrate again until all other loop ports that are arbitrating have had an opportunity to access the loop. Ports that are not arbitrating do not affect the access fairness protocol. A loop port that follows the access fairness protocol is called a fair port and a port that does not is called an unfair port.
Access fairness applies in a certain time interval from when the first port on a loop wins arbitration to when no other port is arbitrating for ownership of the loop. This is controlled by each port on the loop setting a control variable called the Access variable. When a port wins arbitration it sets the access variable to 0 to ensure it cannot arbitrate again in the current fairness window until all other ports arbitrating have had a chance to win arbitration. The winning port knows what other ports are arbitrating as it is sending ARB(F0) and the ports that are trying to arbitrate are substituting their ARB(x) for this.

 
Picture.32 Access fairness window

If the winning port receives around an ARB(F0) it knows that no other port was arbitrating during the time that ARB(F0) was being sent around the loop which means the end of the current access fairness window. At the end of a fairness window all fair ports that have their access variable set to 0 to prevent them arbitrating in that fairness window have this bit set again allowing them to arbitrate once more.
If the winning port does not receive back an ARB(F0), but instead an ARB(x) from a port trying to arbitrate, then it knows that another port is also arbitrating for access to the loop. Once the current winner relinquishes ownership of the loop it cannot arbitrate in the same fairness window so the other port arbitrating has a chance to win arbitration. This is shown on Picture.32 where Port 1 has won arbitration but Port 3 is also arbitrating for ownership of the loop.
Access fairness will ensure that Port 3 will win arbitration once Port 1 has relinquished ownership.


Opening and Closing the loop
After a port has won arbitration it needs to select a destination port before it can send frames to it. It does this by opening a connection with the destination port by sending an open (OPN) ordered set with the AL_PA of the destination port. Once the destination port has received the OPN ordered set the loop circuit between both ports is open. At this stage all other ports on the loop behave as repeaters and allow frames to pass through on the way to the destination port.

 
Picture.33 Opening a loop circuit

Picture.33 shows Port 1 on a loop opening a circuit with Port 3 by sending an OPN ordered set with the destination port of 3 and once this is received Port 3 responds saying it was ready to receive a frame by sending back the R_RDY.

As can be seen on Picture.34, there are different open ordered sets to open a circuit between ports.

 
Picture.34 Open primitive signals

OPN(yx) is used to open a connection between two ports where the frame transmission will be in both directions. This operation allows both ports transmit at the same time and gives maximum utilization of the bandwidth since frame flow can be in both directions simultaneously.
OPN(yy) is used when the originator of the open only wants to send frames to the destination port and wants to prevent the destination from sending frames to it other than link control frames such as Acknowledge, Busy, or Reject.
Open Replicate is a mode ordered set used to implement multicast and broadcast capabilities in an arbitrated loop environment. The open ordered sets set the mode on the destination port to replicate which basically causes it to retransmit all received information and process the frames that have its address in the frame header.
• OPN(yr) is used to selectively set replicate mode on a port specified by the AL_PD field of the ordered set.
• OPN(fr) causes all ports on the loop to set replicate mode except the port transmitting the OPN(fr) ordered set.


Closing the loop
When all the frames are transmitted from one port to another then theports may close the loop by sending a Close ordered set from one port to the other. When a port is finished transmitting frames it sends a CLS to the destination port and it enters the transmitted close state and waits for the other port to finish. When a port receives a CLS ordered set it enters the Received Close state finishes its frame transmission and sends a CLS. Once the port that initiated the close receives back a CLS from the other port it either enters the monitoring state to allow another port win arbitration or it may send an open to establish a connection with another port.

The following three steps explain how to close a loop.

 
Picture.35 Close loop: Step 1

Step 1. Picture.35 shows an existing open loop between Port 1 and Port 3 Port 3 has received the OPN, responded by sending the R_RDY, Port 1 has received the R_RDY and is transmitting a frame to Port 3.
 

CFW = Current Fill Word

Picture.36 Close loop: Step 2

Step 2. Picture.36 shows Port 1 sending a Close to Port 3 and Port 1 enters the transmitted close state. Port 3 can continue to transfer frames as long as it has credits to do so.

 
Picture.37 Close loop: Step 3

Step 3. When Port 3 receives the Close it enters the received close state, finishes its frame transmission and sends a CLS of its own (Picture.37). Port 3 also enters the monitoring state. When Port 1 receives the close it can enter the monitoring state or open a loop circuit with another port.


Alternate Buffer-to-Buffer Credit (BB_Credit)
In arbitrated loop the flow of frames between two ports is handled using a different credit model than the ones mentioned previously.
Buffer-to-buffer credit and end-to-end credit flow control are both credit models that are negotiated between ports during the login process
Alternate BB_Credit is used in arbitrated loop environments instead of login BB_Credit as each port on a loop may open up to 126 other ports on the loop and thus would have to keep track of the credit it has with each of the other ports on the loop from N_Port login time.
This would be a significant complication. The Alternate BB_Credit model is an enhancement of the login BB_Credit model in that it begins with an initial login credit value which can be zero and can then be increased dynamically once a loop circuit between two ports is opened.
Before the ports log in, the available credit is assumed to be zero and even the login BB_Credit granted to another port may be zero. This login BB_Credit is used to set the available BB_Credit value when the loop circuit is opened, although ports may or may not use this login BB_ credit value.

 
Picture.38 Alternate credit

BB_Credit is then signalled dynamically by a port by sending one R_RDY for each available receive buffer. This allows ports that were opened with an available BB_Credit value of zero to begin frame transmission as soon as they receive an R_RDY from the receiving port. A loop port must indicate whether it supports login BB_Credit or Alternate Credit model. This is done during the N_Port login where the BB_Credit management bit (BBM bit) is either on for support of Alternate credit or off for support of the login BB_Credit model.
Picture.38 shows the situation where Port 1 and Port 3 during N_Port login indicated Alternate credit support and BB_Credit of zero. Thus when port 1 opens a loop circuit with port 3 by sending the OPN(3, 1) then port 1 has to wait until the R_RDY is received back from port 3 before a receive it can send a frame. The number of R_RDYs received by port 1 indicates the number of receive buffers available on port 3. Thus, when the loop is opened with an initial available BB_Credit value of zero there is a time delay in waiting for the first R_RDY to come back around where no data transfer can take place.
When the loop is closed the available BB_Credits are set to zero and any outstanding R_RDYs are not sent. The available BB_Credit value is then re-established when a loop circuit is opened between two ports.