August 11, 2012

OpenStackAPAC Conference

Building Applications with OpenStack Swift

Joe Arnold, CEOSwiftStack Inc

@joearnold

乔 .阿诺德行政总裁斯威夫特斯达克

2

August 11, 2012

4

Compute | Storage |

Networking“Nova” “Swift” “Quantum” Global community of cloud software developers & users

Apache 2 Open Source License

+180 Participating Companies - 1,000’s of developers

3

August 11, 2012

2009 2010 2011

Developed in large-scale production environments

Currently +70 developers - has doubled every 6 month

OpenStack Swift - Born in Production

4

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

Racks

pace

Racks

pace

USAUSA

5

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

HP Clo

ud

HP Clo

ud

USAUSA

6

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

Inte

rnap

Inte

rnap

USAUSA

7

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

KT Ucl

oud

KT Ucl

oud

Korea

Korea

8

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

Softla

yer

Softla

yer

USAUSA

9

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

Haylix

Haylix

Austra

lia

Austra

lia

10

August 11, 2012

Swift

Dep

loym

ent

Swift

Dep

loym

ent

Exam

ple:

Exam

ple:

eNov

ance

eNov

ance

Fran

ce

Fran

ce

11

August 11, 2012

About SwiftStack

Cloud Storage System based on OpenStack Swift

Cloud storage technical leadershipSwift Core teamProject lead

ExperienceBuilding large-scale cloud storage at

Rackspace, Engine Yard, Internap, Korea Telecom

12

August 11, 2012

Field Observations

13

August 11, 2012

Infrastructure as a Service

Web/Mobile Applications

Massive Number of Users

Web/Mobile Applications

Field Observations

14

August 11, 2012

Field Observation: Web/Mobile Applications

15

August 11, 2012

Shrinking Storage on Mobile Devices

Field Observation: Web/Mobile Applications

16

August 11, 2012

30

Speaks: HTTP

Storage

Standards-Based OpenStack API

HTTP API

OpenStack Swift enables Web/Mobile applications

17

August 11, 2012

30

Upload direct from device

Storage

HTTP Form Post Mechanism

OpenStack Swift enables Web/Mobile applications

HTTP API

18

August 11, 2012

30

Cache with HTTP

StorageCache

HTTP API

OpenStack Swift enables Web/Mobile applications

19

August 11, 2012

30

Native Content-Delivery Integration

OpenStack Swift enables Web/Mobile applications

20

August 11, 2012

OpenStack Swift enables Web/Mobile applications

21

August 11, 2012

Infrastructure as a Service

Web/Mobile Applications

Massive Number of Users

Field Observations

22

August 11, 2012

Field Observation: Massive # of users

Enterprise Application

23

August 11, 2012

Field Observation: Massive # of users

24

August 11, 2012

Multi-Tenant Application

Field Observation: Massive # of users

25

August 11, 2012

$/GB

FREQ

UEN

CY

$/IO

Long-Tail Data

Field Observation: Massive # of users

26

August 11, 2012

Field Observation: Massive # of users

27

August 11, 2012

Field Observation: Massive # of users

28

August 11, 2012

Access NodeAccess Node Access NodeAccess Node Access NodeAccess Node Access NodeAccess Node

Add Storage Nodes capacity can be added by growing existing availability zones, or adding new availability zones.

Add Proxy Nodes A hash ring is shared amongst each node in the cluster.

Swift Scales to Massive # of users

29

August 11, 2012

1. account data2. object data3. All data distributed via hash ring

Swift uses shared-nothing architecture

Simple mechanisms proven at scale1. Whole files on disk2. Routed networking (Layer 3 networks)3. Proven techniques with HTTP for transport4. Proven techniques with rsync for replication

Swift Scales to Massive # of users

30

August 11, 2012

Sharding

Swift Scales to Massive # of users

31

August 11, 2012

HTTP + Proven Cloud Architecture

Swift Scales to Massive # of users

32

August 11, 2012

Infrastructure as a Service

Web/Mobile Applications

Massive Number of Users

Field Observations

33

August 11, 2012

Improve agility

Improve Efficiencies

Sell Services

Field Observation: Infrastructure as a service

Our Our CustomeCustomer Wantsr Wants

34

August 11, 2012

Storage deployments becoming larger

Multi-tenancy important

Field Observation: Infrastructure as a service

35

August 11, 2012

Swift enables Infrastructure as a Service

Scales to large capacity

High concurrency

Multi-tenancy from ground-up

36

August 11, 2012

Swift enables Infrastructure as a Service

OperationalOperationalEfficienciesEfficienciesOperationalOperationalEfficienciesEfficiencies

37

August 11, 2012

SwiftStack Nodes

38

August 11, 2012

SwiftStack Controller

39

August 11, 2012C

ap

aci

ty

time

Text

40

August 11, 2012

Time-Series BackendGanglia / Graphite

Time-Series BackendGanglia / Graphite

StatsDStatsD

LightweightUDP StatsDSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift Process

Watch Everything

41

August 11, 2012

60

Be NotifiedSee Everything

42

August 11, 2012

SwiftStack Plug-Ins

User Dashboard

Active Directory/LDAPIntegration

Utilization API for Billing

On-disk Encryption

Metadata Search

Make each look like a mini-product. Box around each.

Make each look like a mini-product. Box around each.

43

August 11, 2012

Infrastructure as a Service

Web/Mobile Applications

Massive Number of Users

Field Observations

August 11, 2012

OpenStackAPAC Conference

Thank you!Thank you!

Joe ArnoldCEO, SwiftStackjoe@swiftstack.com@joearnold

乔 .阿诺德行政总裁斯威夫特斯达克