Buffer-to-Buffer Credits and Their Effect on FICON Performance
March, 2005
by Steve Guendert

About the Author

Dr. Steve Guendert, Global Solutions Architect, Brocade Communications Steve is a PMI certified Project Management Professional (PMP), and he has an MBA, and a M.S. in Management Information Systems. He is a well regarded FICON consultant and is one of the industry’s experts on the subject. He lectures and consults regularly on FICON technology as well as business case justifications for migrating from ESCON to FICON. He is currently a Ph D Candidate and is writing his doctoral dissertation on Enterprise I/O Subsystems.

[Hide]

This article will introduce the reader to the concept of Buffer-to-Buffer Credits in a FICON environment and the impact they play on performance in a mainframe storage environment. The discussion is intended to apply primarily to an environment utilizing FICON directors for connectivity; however, the concepts herein apply equally well to an environment with FICON channels directly attached from host to storage devices. For readers without a rudimentary understanding of FICON and/or Fibre Channel architecture and theory, there is a list of references at the end of the article that are outstanding sources of information for all levels of understanding of Fibre Channel and FICON architecture.

Basics of Fibre Channel Flow Control and Buffer Credit Theory

The basic information carrier in Fibre Channel (and, hence, FICON) is the frame. Other than ordered sets, which are used for communicating low-level link conditions, all information is contained within frames. I like to use the analogy of an envelope when discussing the concept of frames: When you send a letter via the United States Postal Service, your letter is encapsulated within an envelope. When you want to send data via a FICON network, your data is encapsulated within a frame. Fortunately, FICON networks have far better service times than the USPS.

To prevent a target device (either host or storage) from being overwhelmed with frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frame(s). If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire fibre channel sequences needing to be retransmitted across the link.

Upon arrival at a receiver, a frame goes through several steps. It is received, deserialized, decoded, and stored in a receive buffer where it is processed by the receiving port. If another frame arrives while the receiver is processing the first frame, a second receive buffer is needed to hold this new frame. Unless the receiver is capable of processing frames as fast as the transmitter is capable of sending them, it is possible for all of the receive buffers to fill up with received frames. At this point, if the transmitter should send another frame, the receiver will not have a receive buffer available and the frame will be lost. To prevent this type of error condition, the Fibre Channel architecture provides a two level flow control mechanism that allows the receiver to control when the transmitter may send frames. The receiving port controls the frame transmission by giving the sending port permission to send one or more frames to the receiving port in question. This permission is called a credit. The actual credit(s) are granted during the login process between two ports. The credit value is decremented when a frame is sent and replenished when a response is received. If the available credits for a given port reaches zero, the supply of credits is said to be exhausted. Further transmission of frames with that port is then suspended until the amount of credits can be replenished to a non-zero value. A good analogy would be a pre-paid calling card: you get a certain amount of minutes to use, and you can talk until there is no more time (minutes) on the card.

One of the goals of Fibre Channel (and hence FICON) is to provide reliable delivery of information from sender to receiver. Providing a data link with a low bit-error rate is a good start, but simply minimizing the quantity of bit-level transmission errors is not enough. We need to guarantee/ensure consistent and reliable frame delivery. Flow control is one of the primary mechanisms for providing this reliability. There are two types of flow control mechanisms in FICON/Fibre Channel; that are used. The first is End-to-End Flow Control. The second, which will be the focus for the remainder of this paper, is called Buffer-to-Buffer Flow Control.

End-to-End Flow Control.

Transmission credit is initially established when two communicating nodes log in and exchange their respective communication parameters. End-to-End Flow Control, also known as EE Credit, is used by Class 1 and Class 2 service between two end nodes. The nodes monitor end-to-end flow control themselves. Intervening switches or directors do not participate in EE Credit. As data is sent from the sending port to the destination port, the sender subtracts a credit from its pool of end-to-end credits. Next, when the destination port receives the transmission, it sends an acknowledgement control word (ACK) back to the sender indicating that the frame was received. When this acknowledgement is received back at the sending port, it then adds a credit back to its own credit pool. Therefore, end-to-end credits used by the sending port are replenished when it receives the acknowledgement from the destination port. What we need to keep in mind is that End-to-End Flow Control is always managed between a specific pair of node ports. Therefore, an individual node port may have many different end-to-end credit values, each corresponding to a different destination node port.

Buffer to Buffer Flow Control

In contrast, Buffer-to-Buffer Flow Control is flow control between adjacent ports in the I/O path, i.e., transmission control over individual network links. A separate, independent pool of credits is used to manage Buffer-to-Buffer Flow Control. Similar to what was earlier discussed for End-to-End Flow Control, Buffer-to-Buffer Flow Control works by a sending port using its available credit supply and waiting to have the credits replenished by the port on the opposite end of the link. These Buffer-to-Buffer Credits (BB Credits) are used by Class 2 and Class 3 service and rely on the fibre channel receiver-ready (R_RDY) control word to be sent by the receiving link port to the sender. An end node attached to a FICON director will establish its BB Credit during login to the fabric. A communicating partner attached elsewhere on the FICON director will establish its own and most likely different BB Credit value to the director during its login process. Hence, BB Credit has no end to end component. The sender then decrements the BB Credit by 1 for each R_RDY received. The initial value of BB Credit must be non-zero. The rate of frame transmission is regulated by the receiving port based on the availability of buffers to hold received frames.

At first glance, it is readily apparent that this system may leave something to be desired in terms of overall performance and efficiency. This is due to the time required for frames to travel from the sending port to the receiving port and responses to return from the receiving port back to the sending port. Now, consider that it takes light approximately 5nsec to propagate through 1 meter of optical fiber, or 50 microseconds to travel 10km. This behavior becomes even less efficient and more of a performance drag on faster links, longer distance links, or when traveling through complex topologies that contribute significant delivery latencies. So, to achieve the higher performance while preventing the overrun of receive buffers, we need to use BB Credit values>1 and frame streaming. If a sending port is allowed to send more than one frame without having to wait for a response to each, performance can be improved. This is referred to as frame streaming. As more credits (beyond 1) are made available, link utilization (and performance) will increase until link utilization reaches 100%. When the link is thus fully utilized, frames can be sent as rapidly as allowed but additional credits will not help matters. So, the key questions become how many frames can be transmitted by the sending port prior to a response to the first frame being received from the receiving port and, most importantly, what is the optimal amount of credits?

What is the optimal number of BB Credits?

Kembel states that the optimal amount of credits is determined by the distance (frame delivery time), the processing time at the receiving port; link signaling rate, and the size of the frames being transmitted. He developed a formula to determine the optimal amount of credits:

Credit=(Round_trip_time+Receiving _port_processing_time)/Frame Transmission _time.

In other words, the optimal number of BB credits depends on 3 key parameters:

1) round trip time, i.e., the distance
2) frame processing time
3) frame transmission time*

*As the link speed increases, the frame transmission time is reduced; therefore, as we get fast iterations of FICON such as FICON Express and FICON Express 2, the amount of credits need to be increased to obtain full link utilization, even in a short distance environment!

Let’s consider this in another way. Assuming the speed of light in a fiber optic cable (in a vacuum) is 200,000,000 meters/second, it requires 500 microseconds to send a bit of information 100 km. Thus, a 100 km link can contain 62,500 bytes by the time the first bit is read at the other end (or 53,125 bytes when accounting for the FC 8-bit/10-bit encoding at the FC-1 layer). In addition, for each frame that is transferred, the hardware at the other end must acknowledge that the frame has been received before a successful transmission occurs. This requires enough capacity in the hardware to allow continuous transmission of frames on the link, while waiting for the acknowledgement to be sent by the receiver at the other end. Therefore, in order to maintain 100% utilization of a 1 Gb link for 100 km, the sending director hardware must have enough resources (BB credits) to keep 106.250 bytes on the link and the receiving switch hardware must have enough resources to allow the sender to transmit continuously. Recall our earlier discussions of the BB Credit mechanism/BB Flow Control. To theoretically achieve 100% utilization of a 1 Gbps link for 100 km, the required BB Credits range from 49 to 1155 depending on the average frame size. When the link speed is increased to 2 Gbps, the required BB Credits range from 98 to 2310. This assumption is that the smallest frame size is 68 bytes (36 bytes of Fibre Channel Header plus 32 bytes of command information for a FICON FC-SB2 command IU) and the largest frame size is 2148 bytes (36 bytes of Fibre Channel Header plus 2112 bytes of data).

Typical environments generally do not sustain one pattern for the transfer of data. Normally, reads and writes are mixed and the data block sizes vary. To get an idea of the resources necessary to support long distances in a FICON environment, the data transfers can be expressed in terms of average block sizes. Below are two tables that characterize the BB Credit requirements for 1 Gbps and 2 Gbps links at 100 km based on different average block sizes for data transfers.

Table 1-Theoretical 1 Gbps utilization at 100 km (these values are close approximations and are intended to convey the approximate Buffer-to-Buffer Credit requirements).

Table 2-Theoretical 2 Gbps utilization at 100Km (these values are close approximations and are intended to convey the approximate buffer to buffer credit requirements.)

Figure 1. Data Rates Based on Buffer Credits.

The graph pictured above represents data from an IBM White paper on Cascaded FICON performance considerations written by Cronin and Basener. This graph shows that at 100 km distance between 2 nodes, having double the number of buffer credits yields double the effective data regardless of block size. Figure 2 below shows a similar type of result specifically for cascaded FICON.

Figure 2 Effect of Buffer Credit Starvation over Distance.

Conclusion.

In this article we touched on several things:

• the two methods /types of flow control in FICON storage networks.
• Buffer-to-Buffer Flow Control and Buffer-to-Buffer Credits in more detail.
• method to determine the correct amount of BB credits of a FICON environment (without doing too much math!).
• examples of what kind of performance drop off can occur if you do not have enough BB-credits.

As companies bring DR back in house, distance becomes more important and, therefore, the amount of BB Credits present has become much more important in the past two years. We showed some illustrations/examples of why that should be so.

It is the author’s opinion that, much like friends, you can never have enough BB Credits in your FICON environment. Performance makes the world go round and those BB Credits are what enable good performance in high workload FICON environments.

Key References

1. Alan F. Bener. Fibre Channel for SANs, McGraw-Hill, March 2001
2. Tom Clark. Designing Storage Area Networks, 2nd Ed.
3. Catherine Cronin, Performance Considerations for Cascaded FICON Directors, http://www-1.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130237.pdf, referenced 3/6/2005
4. Mark Farley, Building Storage Networks 2nd Ed.,
5. Robert W. Kembel, Fibre Channel: A Comprehensive Introduction.