Challenges for Connecting Remote Storage Devices over Distance · 10 Gb/s Maximum throughput from NYC to Chicago is 29Mb/s with 0.1% packet loss rate and standard IP packet sizes

Presented at the THIC Meeting at the Sony Auditorium, 3300 Zanker Rd, San Jose CA 95134-

1940March 4-5, 2003

Challenges for Connecting Remote Storage Devices over Distance

Andy HellandDirector, Product Management

LightSand, 279 Sinclair Frontage Rd, Milpitas CA 95035-5415Phone: +1-408-404-3119, FAX: +1-408-941-2166, e-mail: [email protected]

Challenges for ConnectingRemote Storage Devices over Distance

Andy HellandDirector, Product Management

[email protected](408) 404-3119

March 4, 2003

Storage Applications over the MAN/WAN

Moving large amounts of data between sitesFile sharing – supercomputing applicationsRemote mirroring / disaster recovery

GatewayGateway

PrimaryStorage

RemoteStorage

MAN/WAN

Tape backupVirtualizationHigh availability / clusteringVideo streamingAsynchronous mirroringSynchronous mirroring

Moving Terabytes Between Sites

Fibre channel is the workhorse for moving data inside the data centerOutside the data center there are two ways to move Terabytes between sites

Non real time – i.e. FedExReal time – direct connection

Fundamentally, four things are necessary for a direct connection to efficiently move large amounts of data over distance

Highest possible bandwidth interconnectionFlow control - ability to fill and sustain bandwidth flow over timeLowest possible latencyMinimal packet loss

Why Fibre Channel?

Storage to storage transfers demands a high QoS

Needs “deterministic” communications

Generally very sensitive to latency

Requires in-order block delivery

Fibre Channel is deterministic

Provides low latency and high bandwidth interconnect

High performance SAN extension must maintain same QoS features over distance

High bandwidth

Low latency

How Much Bandwidth is Enough…?

Moving 10 TB requires:2.25 hours using OC-192 (10Gb/s)9 Hours using OC-48 (2.5Gb/s)14 hours using “2G” FC (1600 Mb/s)28 hours using “1G” FC (800Mb/s)35.7 hours using OC-12 (622 Mb/s)6 days using OC-3 (155 Mb/s)20 days using T3 (45 Mb/s)1.6 years using T1 (1.5 Mb/s)

……If the pipe is fully utilized!

Data Movement Today –A Recent Case Study

Sandia National

Laboratory (SNL)

Los AlamosNational

Laboratory(LANL)

IP NetworkServer Server

Fibre ChannelStorage AreaNetwork

Fibre ChannelStorage AreaNetwork

Scientists at LANL currently dump 100GB of supercomputing data to tape and FedEx it to SNL because it is faster than trying to use the existing 155Mb/s IP WAN connection

Actual measured throughput of 16Mb/s! (10% bandwidth utilization)http://www-unix.mcs.anl.gov/discovery/wufeng.htm

Scientists at LANL currently dump 100GB of supercomputing data to tape and FedEx it to SNL because it is faster than trying to use the existing 155Mb/s IP WAN connection

Actual measured throughput of 16Mb/s! (10% bandwidth utilization)http://www-unix.mcs.anl.gov/discovery/wufeng.htm

The Better Way –Directly Between Storage Systems

Utilizing FC/SONET, the data could have been moved in just over 22 minutes at OC-12!

(less than 6 minutes at OC-48)

RemoteData Center

LocalData Center

IPNetwork

FC SAN

Server Server

FC SAN

SAN GatewayHigh QoS

Link SAN Gateway

Extending a FC SAN – the Inverted Pyramid

Physical layer

Transportlayer

Networklayer

File System

SCSI

CustomerApplication

SONETDWDM

DF

Data integrity

RoutingFlow control

“Determinism”

MirroringRemote backup

Virtualizationetc

Carriers

DatabaseFTP

Transport Layer Extension

FC over Dark FiberFC over DWDMFC over ATMFC over SONETFC over IP

How do I move the signal?How do I move the signal?

DWDM Muxwith FC Blade

ATMNetwork

SONETFC/SONETGateway

FCIPGateway

IP

DWDM

Dark Fiber

FC/ATMGateway

Transport Layer Extension through the MAN and WAN

FC Data Encapsulation Efficiency

80%98%90%Mapping Efficiency

Mapping

Storage over ATM

Storage over SONETStorage over IP

What’s Wrong with Routed IP in the WAN?

Routed IP is designed for scalability and connectivityRouted IP is NOT designed for performanceTCP algorithms are designed to make the end user back-off quickly at the first sign of congestionThroughput is controlled by latency (geographical distance) and packet loss rateIP Carriers will always have packet loss

Sell service by peak capacityAllocate equipment based on average capacity

Multiple Sources for Latency

FCGateway

FCGateway

FCSAN

FCSAN

EquipmentLatency

EquipmentLatency

DistanceLatency

ProtocolLatency

(TCP over lossy network)

1

2

3

4

“Fat Pipes” Don’t Guarantee High Throughput with TCP/IP

Sender ReceiverInfinite BandwidthPipe

With TCP/IP over the WAN,BER and Latency affect throughput

more than the bandwidth of the pipes

With TCP/IP over the WAN,BER and Latency affect throughput

more than the bandwidth of the pipes

Latency

“Lossy Pipe”Typical bit error rate (BER) is 0.1%

1 out of every 1000 packets is dropped

1.00E+05

1.00E+06

1.00E+07

1.00E+08

1.00E+09

1.00E+10

0.00 5.00 10.00 15.00 20.00

BER = 0.01%BER = 0.1%BER = 1%BER = 10%

TCP/IP Performance vs. BER and Latency

1 Mb/s

10 Mb/s

100 Mb/s

1 Gb/sTypical Carrier

Operating Region*

* = Network statisticsobtained Fromwww.matrix.net

* = Network statisticsobtained Fromwww.matrix.net

10 Gb/s Maximum throughput from NYC to Chicago is 29Mb/s with 0.1% packet loss rate and standard IP packet sizesMaximum throughput from NYC to Chicago is 29Mb/s

with 0.1% packet loss rate and standard IP packet sizes

10 ms 20 ms5 ms 15 ms0 ms

New York Boston Chicago Denver

Fibre Channel over SONET / WDMFibre Channel over SONET / WDM

0100200300400500600700

0 1000 2000 3000 4000

OC-12 IPOC-12 SONET

Data re-transmission due to IP packet losslimits actual IP throughput over distance

OC-12 Routed IPOC-12 FC/SONET

Throughput(Mb/s)

Distance(Kilometers)

We have measured 67 MB/s (536 Mb/s)

sequential SCSI Read to a disk at

6000 Km

Real data throughputReal data throughput

Fibre Channel over SONETFibre Channel over SONETThe High Efficiency, Long Distance AlternativeThe High Efficiency, Long Distance Alternative

Hours to Send 1 TeraByte

0

50

100

150

200

250

0 1000 2000 3000 4000

OC-12 IPOC-12 SONET

Data re-transmission due to IP packet losslimits actual IP throughput over distance

Distance(kilometers)

NewYork

Boston Chicago Denver

Hours

Fibre Channel over SONET:The Cost Effective Alternative

$47$109,0002320 Mb/sOC-48 FC/SONET

$67$39,000580 Mb/sOC-12 FC/SONET

$338$196,000580 Mb/sOC-12 Internet Access

New Yorkto

Boston

$119$277,0002320 Mb/sOC-48 FC/SONET

$150$ 87,000580 Mb/sOC-12 FC/SONET

$338$ 196,000580 Mb/sOC-12 Internet AccessNew York

toChicago

$245$568,0002320 Mb/sOC-48 FC/SONET

$290$ 168,000580 Mb/sOC-12 FC/SONET

$338$ 196,000580 Mb/sOC-12 Internet AccessNew York

toDenver

$ per Mb/sMonthly CostData RateConnection TypeDistance

Comparison of Transport Options

FC over DWDM

FCIP

FC over ATM

FC over SONET

FC over Dark Fiber

Encapsulation


Dark FiberFC over DWDM

IP Transport (DWDM)

GbE

ISP

FCIP

100BT

SONETFC over ATM

GbE

100BT

T1

SONETFC over SONET

Dark FiberFC over Dark Fiber

TransportEncapsulation


LowDark FiberFC over DWDM

IP Transport (DWDM)

GbE

ISP

FCIP

Low100BT

Variable

LowSONETFC over ATM

GbE

100BT

T1

LowSONETFC over SONET

LowDark FiberFC over Dark Fiber

Latency TransportEncapsulation


150 KmLowDark FiberFC over DWDM

IP Transport (DWDM)

None

GbE

ISP

FCIP

MetroLow100BT

Variable

NoneLowSONETFC over ATM

GbE

100BT

T1

NoneLowSONETFC over SONET

100 KmLowDark FiberFC over Dark Fiber

Distance LimitationsLatency TransportEncapsulation


No150 KmLowDark FiberFC over DWDM

IP Transport (DWDM)

YesNone

GbE

ISP

FCIP

MetroLow100BT

Variable

No

Yes

No

No

Droppable Data?

NoneLowSONETFC over ATM

GbE

100BT

T1

NoneLowSONETFC over SONET

100 KmLowDark FiberFC over Dark Fiber

Distance LimitationsLatency TransportEncapsulation


Available but limited to MANNo150 KmLowDark FiberFC over DWDM

Emerging

Emerging slowly

IP Transport (DWDM)

Readily Available

YesNone

GbE

ISP

FCIP

EmergingMetroLow

100BT

Variable

No

Yes

No

No

Droppable Data?

Readily AvailableNoneLowSONETFC over ATM

GbE

100BT

Emerging

T1

Readily AvailableNoneLowSONETFC over SONET

Available but limited to MAN100 KmLowDark FiberFC over Dark Fiber

Infrastructure AvailabilityDistance LimitationsLatency TransportEncapsulation

LightSand’s Gateway Products

Up to 3 ports of Fibre Channel

or Gigabit Ethernet

SONET OC-48 / 12SDH STM-4

Dedicated FiberDWDM

WAN / MAN ConnectionLocal Storage Connection

Large data movement over great distances (up to 8000 Km)Highly predictable inter-site latency (Synchronous Mirror)

Hardware-based ‘ping’ for precise latency measurement

Buffer credits with fully coupled flow control sized for wide bandwidth and long distanceSmall footprint 1U package

Doesn’t drop data

400 Mbytes/Sec over 55 miles

S-2500

TP9500StorageArray

3800

Fibre Channel (data)Ethernet (meta-data)OC-48 FC/SONET

Onyx 3800

NRL

InPInternetworking

Processor

InPInternetworking

ProcessorS-2500

3800

OC-48 ATMOC-192 ATM

BXR-48000 BXR-48000

Deskside Onyx

Fuel

S-2500S-2500

12000

Summary

High performance distributed storage systems are being deployed today using FC/SONETPacket loss and latency cripple performanceLatency must be managed at all layers of the storage process

NetworkSCSIFile SystemApplication

Disk arrays and tape libraries can now be hundreds or even thousands of kilometers away from their serversGeographic diversity means safety for stored data

Questions?

Andy [email protected]

(408) 404-3119

Documents

Challenges for Connecting Remote Storage Devices over Distance · 10 Gb/s Maximum throughput from NYC to Chicago is 29Mb/s with 0.1% packet loss rate and standard IP packet sizes