Distance extension technologies overview
To comprehend the distance extension solutions for Storage Area Networks it is important to understand and recall the challenges when implementing SAN connectivity over remote distances. The following information is provided in this overview section:
◆ “Early implementations of SAN environments ,” next
◆ “DWDM ”
◆ “CWDM ”
◆ “SONET ”
◆ “GbE ”
◆ “TCP/IP ”

Early implementations of SAN environments
To increase a single port between two Fibre Channel switches separated by a large geographical distance, every two strands (transmit, receive) of optical fiber cable were required to be physically added by the distance provider. The customer would generally incur expensive construction, service, and maintenance costs when adding a bulk of fiber cables intended to satisfy current E_Port connectivity requirements while allowing future growth potential and redundancy against accidental fiber breaks. Existing fibers that were used for Ethernet implementations could not be shared and required separate dedicated channels per protocol. The challenges involved with this process would stem anywhere from mandatory to extraneous costs associated with fiber cable maintenance. In addition to costs, there were physical hardware limitations to achieving connectivity between (at least) two geographically separated sites. Fibre Channel optics installed on the Fibre Channel switch were at the mercy of the limited optical output transmission power. Even with repeater technology, distortion of the optical wavelength transmitted by the optics can occur over several hops.
The Fibre Channel switches provided limitations as well. Link initialization and flow control were solely controlled by the Fibre Channel switches. The Fibre Channel standard would actually dictate the thresholds in regards to supporting large distances through optical connectivity and the obtainable bandwidth between two Fibre Channel ports. To finalize the list of challenges that SAN environments had to overcome, each Fibre Channel switch provider had its own non-standard and standard ways of implementing their native environments. This may deviate from the mass interpretation of the Fibre Channel standards.

DWDM
Dense Wavelength Division Multiplexing (DWDM) is a process in which different channels of data are carried at different wavelengths over one pair of fiber-optic links. This is in contrast with a conventional fiber-optic system in which just one channel is carried over a single wavelength traveling through a single fiber.
Using DWDM, several separate wavelengths (or channels) of data can be multiplexed into a multicolored light stream transmitted on a single optical fiber (dark fiber). This technique to transmit several independent data streams over a single fiber link is an approach to opening up the conventional optical fiber bandwidth by breaking it up into many channels, each at a different optical wavelength (a different color of light). Each wavelength can carry a signal at any bit rate less than an upper limit defined by the electronics, typically up to several gigabits per second.
Different data formats being transmitted at different data rates can be transmitted together. Specifically, IP data, ESCON SRDF, Fibre Channel SRDF, SONET data, and ATM data can all be traveling at the same time within the optical fiber.
DWDM systems are independent of protocol or format, and no performance impacts are introduced by the system itself.

Figure 1 illustrates the DWDM technology concept:
 
Figure 1 DWDM example

For EMC customers it means that multiple SRDF® channels and Fibre Channel ISL (Inter Switch Links) can be transferred over one pair of fiber links along with traditional network traffic. This is especially important where fiber links are at a premium. For example, a customer may be leasing fiber, so the more traffic they can run over a single link, the more cost effective the solution.
With today's technology, the capacity of a single pair of fiber strands is virtually unlimited. The limitation comes from the DWDM itself. Optical-to-electrical transfers for switching and channel protection are required and limit the input traffic per channel.
Available DWDM topologies include point-to-point and ring configurations with protected and unprotected schemas. DWDM technology can also be used to tie two or more metro area data centers together as one virtual data center.
DWDM systems can multiplex and de-multiplex a large amount of channel quantities. Each channel is allocated its own specific wavelength (lambda) band assignment. Each wavelength band is generally separated by 10 nm spacing(s). As optical technologies improve, separations between each channel may be further reduced enabling more channels to be packed (tighter) onto a single duplex dark fiber.
DWDM has a higher cost associated due to greater channel consolidation, flexibility, utilization of higher quality hardware precision-cooling components (to prevent low frequency signal drift) and the capabilities of regenerating, re-amplifying and reshaping (3R) wavelengths assigned to channels to ensure optical connectivity over vast distances.
Varying circuits pack capabilities are also offered in a DWDM environment. DWDM circuit packs / blades can provide the following protocol conversions:
◆ Fibre Channel to SONET
◆ Fibre Channel to Gigabit Ethernet
◆ Fibre Channel to IP
In addition, some circuit packs can enable features such as write acceleration and buffer-to-buffer credit spoofing. To verify the latest supported distance systems and features, refer to the EMC Support Matrix.
Figure 2 shows a general concept of Fibre Channel link extension using DWDM.
 
d1 = DWDM signal over dark fiber medium.
d2 and d3 = Local ISL connections between switches and DWDM input.
Can be SM or MM depending on DWDM and switch interfaces or local distance requirements.
d4 and d5 = Local storage or server connections into the fabric.

 

 

Figure 2 Fibre Channel link extension

Note: All components are randomly selected and do not reflect a specific setup or configuration.
Note: Distance limitation may also be affected by application response time-out values and should consider signal propagation delay over site distance.
The following list provides general envelope guidelines for using DWDM systems:
◆ May be used for ESCON RDF distance extension, with direct connection between Symmetrix ESCON director ports and DWDM input ports.
◆ May be used for ISL extension of Fibre Channel switched fabrics. (E-Lab Navigator describes switch compatibility.)
◆ Fabric topology guidelines are provided per Fibre Channel switch topology documentation.
◆ Direct connections between host HBA or Symmetrix Fibre Channel director to a DWDM port are not supported. E-Lab Navigator contains specific DWDM distance and topology guidelines.
◆ As a general approach, two distances need to be measured. The shorter of the two is the maximum distance to be supported in the site.

CWDM
Coarse Wave Division Multiplexing (CWDM), like DWDM, uses similar processes of multiplexing and de-multiplexing different channels by assigning different wavelengths to each channel. CWDM is intended to consolidate environments containing a low number of channels at a reduced cost.
CWDM contains 20 nm separations between each assigned channel wavelength. CWDM technology generally uses cost-effective hardware components that require a reduced amount of precision-cooling components usually dominant in DWDM solutions due to the wider separations. With CWDM technology the number of channel wavelengths to be packed onto a single fiber is greatly reduced.
CWDM implementations, like DWDM, utilize an optical-to-electrical-to-optical technology where all the channels are multiplexed into a single CWDM device performing the optical-to-electrical-to-optical conversion.
A CWDM connectivity solution can use optics generating a higher wavelength with increased output optical power. Each channel is designated its own specific wavelength by the specific hot-pluggable CWDM GBIC/SFP optic installed on the Fibre Channel Switches.
With clean fibers, minimal patch panel connections, and ample optical power, CWDM optics alone can provide connectivity distances of up to 100 km per channel. To complete this solution a passive MUX/DEMUX is required to consolidate multiple channel-wavelengths into a single duplex 9-micron dark fiber.

Differences between DWDM and CWDM
The following are differences between DWDM and CWDM:
◆ Number of channels that are supported per solution.
DWDM systems can support channels ranging from 16 channels or above while CWDM supports 16 channels or below.
◆ CWDM GBIC/SFP optics can be used to increase the wavelength output of a channel (i.e., FC-switch optics).
The CWDM GBIC/SFP optics is usually installed in the Fibre Channel switch or client device. The wavelength and optical power enhanced links are then multiplexed and de-multiplexed to and from a single-mode 9-micron dark fiber.
◆ Costs.
Hardware components included with DWDM units are higher in cost due to precision-cooling techniques required to prevent signal drift. DWDM offers greater channel flexibility and capacity.
◆ Configurations can be complex with CWDM.
CWDM requires specific optics for each specific wavelength. Growth for a CWDM environment is limited and difficult to manage when supporting environments growing to larger channel support. More cabling would be required, thereby increasing complexity.
◆ DWDM devices offer circuit packs with numerous features (i.e., protocol conversions, buffer-to-buffer credit spoofing, write acceleration).

SONET
Synchronous Optical NETwork, (SONET), is a standard for optical telecommunications transport, developed by the Exchange Carriers Standards Association for ANSI. SONET defines a technology for carrying different capacity signals through a synchronous optical network. The standard defines a byte-interleaved multiplexed transport occupying the physical layer of the OSI model.
Synchronization is provided by one principal network element with a very stable clock (Stratum 3), which is sourced on its outgoing OC-N signal. This clock is then used by other network elements for their clocks (loop timing).
SONET is useful in a SAN for consolidating multiple low-frequency channels (Client ESCON and 1, 2 Gb Fibre Channel) into a single higher-speed connection. This can reduce DWDM wavelength requirements in an existing SAN infrastructure. It can also allow a distance solution to be provided from any SONET service carrier, saving the expense of running private optical cable over long distances.
The basic SONET building block is an STS-1 (Synchronous Transport Signal), composed of the transport overhead plus a Synchronous Payload Envelope (SPE), totaling 810 bytes. The 27-byte transport overhead is used for operations, administration, maintenance, and provisioning. The remaining bytes make up the SPE, of which an additional nine bytes are path overhead. It is arranged as depicted in Figure 3. Columns 1, 2, and 3 are the transport overhead.

 
Figure 3 STS-1 organization

An STS-1 operates at 51.84 Mb/s, so multiple STS-1s are required to provide the necessary bandwidth for ESCON, Fibre Channel, and Ethernet, as shown in Table 1.
Multiply the rate by 95% to obtain the usable bandwidth in an STS-1 (reduction due to overhead bytes).

Table 1 SONET/Synchronous Digital Hierarchy (SDH)

One OC-48 can carry approximately 2.5 channels of 1 Gb/s traffic, as shown in Table 1. To achieve higher data rates for client connections, multiple STS-1s are byte-interleaved to create an STS-N. SONET defines this as byte-interleaving three STS-1s into an STS-3, and subsequently interleaving STS-3s.
By definition, each STS is still visible and available for ADD/DROP multiplexing in SONET, although most SAN requirements can be met with less complex point-to-point connections. The addition of DWDM can even further consolidate multiple SONET connections (OC-48), while also providing distance extension.

GbE
Gigabit Ethernet (GbE) is a terminology describing an array of technologies involved in the transmission of Ethernet packets at the rate of 1024 megabits (Mb/s) or 1 gigabit per second. Gigabit Ethernet is specifically designed to surpass the traditional 10/100 Mb/s link speeds. GbE is defined by the IEEE publication 802.3z, which was standardized in June, 1998. This is a physical layer standard following elements of the ANSI Fibre Channel’s physical layer. This standard is one of many additions to the original Ethernet standard (802.3 - Ethernet Frame) published in 1985 by the IEEE organization. The following are nomenclature and characteristics of GbE.
◆ 1000Base-SX is defined as a fiber-optic Gigabit Ethernet standard encompassing the use of multi-mode (50 or 62.5 micron) fiber with 850 nanometer wavelengths. Distances of over 500 meters can be achieved.
◆ 1000Base-Lx is defined as a fiber-optic Gigabit Ethernet standard encompassing the use of single-mode (9 micron) fiber with 1310 nanometer wavelengths. Distances of 10 km or more can be achieved.
◆ Copper coaxial cabling, multi-mode fiber-optic cabling (50 and 62.5 micron) and single-mode (9 micron) cabling are available choices for the 802.3z standard.
◆ GbE is mainly used in distance extension products as the transport layer for protocol such as TCP/IP. However, in some cases the product is based on a vendor-unique protocol.
◆ Distance products using GbE may offer features such as compression, write acceleration, and buffer credit spoofing

TCP/IP
As discussed in, ”TCP/IP technology overview” the Transmission Control Protocol (TCP) is a connection-oriented transport protocol that guarantees reliable in-order delivery of a stream of bytes between the endpoints of a connection. TCP achieves this by assigning each byte of data a unique sequence number, maintaining timers, acknowledging received data through the use of acknowledgements (ACKs), and retransmission of data if necessary. Once a connection is established between the endpoints data can be transferred. The data stream that passes across the connection is considered a single sequence of eight-bit bytes, each of which is given a sequence number.
For information on the following, refer to, ”TCP/IP technology overview”:
◆ “TCP terminology”
◆ “TCP error recovery”
◆ “Network congestion”
◆ “Internet Protocol security (IPsec)”
◆ “Tunneling and IPsec”