All clouds-are-made-of-metal-banv-april-2432014-for-distribution

Ultimately All Clouds Are

Made of [email protected]

4/23/2014HV Quackenboss 2014 All Rights Reserved [email protected] 1

Why the Title?


Dave Roberts Tweet

“But all clouds are

ultimately made of

metal.”

Dave Roberts

Senior Vice President, ServiceMesh

Tweet, August 2, 2012


Most Everybody Thinks SDN

Is the Solution


SDN

Mobile NetworkDatacenter

Opinion (I)

The more you understand the metal,

the more you will understand SDN, NFV

and Network Virtualization


Opinion (2)

WAN SDN ≠ LAN SDN

WAN Virtualization ≠ LAN Virtualization

Differences come from:

Physics

Economics4/23/2014HV Quackenboss 2014 All Rights Reserved [email protected] 6

Opinion (3)

Mobile WAN ≠ Wired WAN


Physics & Econ Differences

WAN LAN

LATENCY ∞ 0

BANDWIDTH 0 ∞

COST ∞ 04/23/2014HV Quackenboss 2014 All Rights Reserved [email protected] 8

Do the same SDN

solutions apply?


Campus Wi-Fi

Datacenter

LAN

Mobile

Networks

Broadband and

Long haul Networks

WAN Providers’ Challenge“A common refrain among the network executives we’ve talked to is that, if IT services can be provided in secondswith cloud computing technologies, it should not take days or weeks to deliver the network services that support them.”

Practical Implementation of SDN & NFV in the WAN

Heavy Reading

October 2013


London Mobile Infrastructure


600 Base Stations

See the Cell Tower?


Mobile Operators’ Finance Problem


3G Mobile Network Metal


Big device population

Complex

Heterogeneous

10 year depreciation

3G May Be the Future of IoT


Mobile SDN/NFV Perspective


Been “SDN” since before SDN

Salivating over IoT

NFV Mandatory Requirements

Existing OSS/BSS integration

Existing equipment interoperability

• Including legacy 3G

“Wired” WAN


Sprint Long Haul Network


Latency and Light Speed300,000 km/sec (in vacuum)

~200,000 km/sec (in fiber)

Slower because of refraction

Ask your neighborhood physicist


http://en.wikipedia.org/wiki/Optical_fiber

Cross-Country Latency


Source: Comcast

(4,100/200,000) =~.020 seconds

Packets go through routers, which buffer

adding delay

WAN Latency Is Speed-Independent


Source: Comcast.com 40G/sec network

BANDWIDTH LATENCY

40Gbps 20 milliseconds

400Gbps 20 milliseconds

1Mbps 20 milliseconds

Ping (Half Moon Bay to Palo Alto)


Half Moon Bay to New York


Half Moon Bay to Beijing


Half Moon Bay to New Delhi


Latency Vs. BandwidthThought experiment:

Transfer 1 Petabyte of data on 1TB SATA drives from SFO to New Delhi• Note: SATA III throughput: 6 Gbits/sec

• SATA drive: ~100 Mbytes/sec (~1.1Gbps)

Alternatives

A. Lease dedicated 40G circuit

B. Airplane 1000 drives (~2000 lbs)


Latency Vs. BandwidthA: 40G pipe

1-way latency: ~100ms (best case)

• (first bit in to first bit out)

~70 hours

• Hypothetical ideal throughput

Option B: Airplane

~18 hours (assuming non-stop)

1-way latency: ~18 hours


WAN Cable CostsResidential broadband installation

CAPEX: ~$1,000/home passed

OPEX: ~$5/month/homehttp://fastnetnews.com/fiber-news/175-d/4835-fiber-economics-quick-and-dirtynth

Long distance fiber installation

$30K/mile for conduit in open

trench during road construction

(< 1% of cost of road)


http://fastnetnews.com/fiber-news/175-d/4835-fiber-economics-quick-and-dirtynth

Transportation Industry LessonsUtilization is key to low cost

Synchronize

Schedule in advance

Take time to pack (doesn’t hurt overall time much)

Load big stuff first

Make sure there is a full buffer at on ramp

Make sure there isn’t a bottleneck unloading


Datacenter LANs


Samsung Backup

Datacenter Fire

April 20,2014

Mice + Elephants


Source: Facebook OIC 2013 presentation

East-West Traffic Example


Source: Facebook OIC 2013 presentation

Typical Cloud Data Center20 to 30 1U(1.75”) servers/rack

2x1U Top of Rack switches

Switches are 5% to 15% of rack

space, CAPEX and electricity

BUT: about 3 switch ports per server

(15% to 45% of server CAPEX)


Datacenter Topology


(HVQ Note: For each TOR Switch server port, there will be ~1 to 2

aggregation ports)

Source: Nathan Farrington Facebook OIC13 Presentation

TOR (Top of Rack) Switch48x10GE + 4x40GE

Vol. price (est.)

~$2500-$4000

Per port $50+


Source: Opencomute.org Intel draft

spec

Example OCP

Bare Metal Switch

Cabling Cost > TOR CostCopper Twinax: ~$75/link

Intra-rack (server to TOR)

Fiber optics: ~$150 - $400

(2Xoptics + cable)

Rack to spine switch


Aggregation CablingMTP/MPO multifibre cables

12,24…up to72 fibers

Intel/Corning Silicon Photonics (future)

Up to 64 fibers - 25Gbps each


http://www.intel.com/content/www/us/en/research/intel-labs-silicon-photonics-mxc-connector.html

Data Center Metal Economics


Datacenter Metal Key PointsServer networking switches CAPEX is

20% to 30% of server cost

TOR switch hardware is pretty much same for everybody

Weakening perception that brand name SW is better is reducing margins

In-rack connectivity with copper cables is cheap, but still as much or more than the switch $/port

Limited opportunities to reduce hardware CAPEX


Bandwidth Vs CPU1 instruction needs 1 byte of

memory and generates 1 bit/sec of bandwidth

(1994) SPARC10: 60Mhz+100Mbps

(2000) 1Ghz CPU+1GbE

(2014) (3Ghz*15 cores) * 2 sockets

90Gbps (!)

Do datacenters have

enough NIC bandwidth?


Performance in Datacenter LANs

Latency and Bandwidth


Server Latency (One Way)


TCP/IP

Stack

GBE NIC

Server With GbE

~15µsTCP/IP

Stack

10GE NIC

Server With GbE

~5µs

Note: numbers may vary based

on specific benchmarks

Switch Latency Is Less Than

Host Stack Latency

First bit in to first bit out

Measurement favors “cut-through” (versus “store and forward” switches

Typical 10GE (per-chip) .3µs to 1.5µs


IP

PHY

Data Link Data Link

PHY

Switch Latency With Buffering

Switch buffers if output port busy

Dramatic increase in latency

60µs observed in benchmark tests

Dropped packets/retransmits: ∞ latency


IP

PHY

Data Link Data Link

PHY

BUFFER


Source: HP Mellanox Low Latency Benchmark Report 2012

Round-trip user space to

user space

Who Cares About LAN Latency?Not if traversing the WAN

But high latency and dropped hurt

server throughput

Applications spin or get swapped

out

HFT cares!


HFT: Milliseconds = $$$ While You Were Blinking

High-frequency-trading activity is not constant; it occurs in microbursts. The line at the bottom of this graphic is the stock-market activity involving General Electric shares over 100 milliseconds (one-tenth of a second) at 12:44 p.m. on Dec. 19, 2013. The gray box magnifies a five-millisecond window, during which GE experienced heavy bid and offer activity and a total of 44 trades.

Graphic: CLEVERºFRANKE.

Data source: IEX.


44 trades in 5 milliseconds

http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0

Leveling the HFT Playing FieldTo allow customers to

pick and choose

different pricing tiers

based on different

latencies, HFT systems

add latency using fiber

38 miles of fiber cable =

~260 microseconds


http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0

Latency Key PointsFrom viewpoint of end-end across WAN, LAN

latency is near 0

10GE Host NIC and IP Stacks have improved dramatically

1GE is > 10X higher than 10GE

Few 1GE switches support cut through

Higher chance of busy ports

Switch latency in cut through is great

If switches have to buffer because of busy output port, latency goes way up

Latency spikes are short duration so hard to see, let alone measure


OpenFlow For Datacenter LANs


Note: The control plane

network switches are

not shown in this

diagram

OpenFlow Reaction Time in LAN


OpenFlow

Application

Transport Layer

Internetwork Layer

NIC

OpenFlow

Agent

Transport Layer

Internetwork Layer

NIC

1000µs

30µs

20-100µs

30µs

1000µs

OpenFlow Controller

App + IP Stack

OpenFlow Switch

Agent + IP Stack

OpenFlow MicroControllers?


http://www.theregister.co.uk/2012/12/11/pica8_sdn_bundles/

OpenFlow

MicroControllerOpenFlow

MicroController

Master

OpenFlow

Controller

Lessons From Transportation


Multi-pathingHOV lanes

Priority based on

policy

Inefficient utilization

What if nobody

changed lanes?

(802.11ag and ECMP

are static)


Dynamic Load Balancing Can Increase

Throughput



“It's tough to make predictions, especially about the future.”

― Yogi Berra

http://www.goodreads.com/author/show/79014.Yogi_Berra



http://www.pcper.com/news/General-Tech/Atom-aint-dead-yet-New-ultra-low-power-Avoton-chips-servers

Lower Performance =

Higher Efficiency


http://www.pcper.com/news/General-Tech/Atom-aint-dead-yet-New-ultra-low-power-Avoton-chips-servers

New Rack DesignsSince the 1U server was invented by

Cobalt Networks (acquired by Sun)

the processors are > 100X faster

Network switching has

(mostly)lagged Moore’s Law

Will ratio of servers to network

switches decreases over time?

Time for new form factors?


TOR Switch Obselescence


Source: Intel IDF 14 Shenzhen presentation

Data Center Server Future



TOR Switch Evolves to Tray Switch



Conclusions


WAN Issues and OpportunitiesMobile: physical infrastructure

complexity has led to management

complexity

NFV promise is to reducing

physical devices, simplify mgt.

Long haul: utilization of capacity is

still opportunity, SDN offers ways to

simplify mgt. and improve utilization


Datacenter LANsLow cost of switching has limited

opportunities for independent SDN controller sales

Distributed SDN MicroController hierarchy should broaden appeal

Controller needs to be bundled with switch or switch software

Effective dynamic load balancing will improve throughput and latency


Important DifferencesWAN LAN

COST ∞ 0

BANDWIDTH 0 ∞LATENCY ∞ 0


Ultimately All Clouds Are

Made of [email protected]


Technology

All clouds-are-made-of-metal-banv-april-2432014-for-distribution