Extreme Storage Scale and Efficiency: Lessons from a

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1

Extreme Storage Scale and Efficiency: Lessons from a 100,000-Person Organization

YoungTae Kim

System Consultant , Oracle Korea


Program Agenda

Oracle IT and Storage Strategy

NAS Storage Configurations and Performance

Oracle IT Disk Use Cases

Cloud IT Backup and Recovery

Oracle on Oracle vs. Legacy Solutions

Backup and Recovery Platform Overview

Backup and Recovery Best Practices

Next Steps


Oracle IT

Improve Service

Overall cost reductions

– Eliminate/Reduce purchase and license costs

– Reduce space and power footprint

– Reduce operations costs

Run Oracle on Oracle

– Showcase technology

– Improvements and opportunities in performance, usability, continuity

스토리지 교체 효과


Oracle IT – 고객 위주 TRADITIONAL

ORACLE ON DEMAND

DEVELOPMENT

ORACLE UNIVERSITY

Supporting 100K+ internal users & 1.2M+ external users in

145 countries

Supporting over 5.5M users over 4,000 environments

Supporting 21K developers building over 3K products

globally

Supporting 352K students resulting in 41,000 classes

annually in 24 languages

• Global voice & network infrastructure

• Global data center operations

• Oracle employee support

• Computing device management & support

• Operation, administration & management of Production

business systems

• Operation of Development-managed compute spaces

• Management & support of specialized development

computing devices

• Operation, administration & management of customer

facing computing resources

• Administration & management of computing architecture

& standards

• Business continuity operation & planning

• Operation, administration & management of computing

resources supporting delivery of curriculum & classroom

content


Over 19 Petabytes Total Storage

Over 9 Petabytes Migrated To Sun ZFS Storage Arrays

0

2000

4000

6000

8000

10000

12000

Traditional Development Cloud Services Oracle University

Total Size in TB's Migrated So Far in TB's

스토리지 용량은?


스토리지 전환 현황 Transition to Oracle Storage

Source: Horison Information Strategies, 2009

EMC

7%

Others

4%

NetApp

25%

Sun

Oracle

64%

Others

3%

EMC

7%

Sun

Oracle

77%

NetApp

13%

Sun

0%

Others

14% EMC

17%

NetApp

69%

Current Status After Refresh Before Refresh


Storage Strategy

• Large variety of customer needs and workloads

– From low latency file access to highest speed read/write operations

– Mixed mode solutions within the Cloud

• Transition tooling

– Ensuring compatibility to automation infrastructure

• Increased inspection into performance and behavior during transition

• Matching transition to dependent efforts

– Server Refresh, Virtualization Strategies, Network Upgrade, Backup Solution

도전 과제


IT 예산을 초과하는 데이터 증가폭 스토리지 계층화를 필요로 하는 데이터 센터

Source: Horison Information Strategies, 2009

Single

Tier of Disk

Storage

Modern

Multi-tiered

Storage

Disk

Multi-tiered

Storage

Performance Disk

$7 - $22/GB

100%

38%

62%

Flash Storage

$40 - $54/GB

32%

6%

Capacity Disk

$1 - $6/GB

Tape Storage

$0.1 - $0.85/GB

Average

~$15,000/TB

Average

~$8,000/TB

Average

~$4,000/TB

60%


Program Agenda








Next Steps


Storage Strategy

• Small/Medium/Large solutions matched to

Customer need

– Low Cost non-clustered

• 71XX for customers with 3-4 TB requirement

– Mid Cost clustered

• 73XX for customers with 7-15TB requirement

– Large Scale NAS

• 74XX for customer up to +28TB requirement

– Large Scale SAN

• Pillar Axiom

해결책

• Transition Support

– Scripted workflows support existing

automation

– Multi-protocol sharing in a single device

• Advanced analytics allow administrators

visibility through out transition lifecycle


ZFS SA Performance Tests Testing Configurations

The tests were run with a 100MB dataset, consisting of the following steps on each client host:

1. Create a /tmp/<hostname>_mnt directory

2. Mount the dir from Sun 7420 to /tmp/<hostname>_mnt

3. Randomly select 1 of 25 source directories (same dataset) and copy to the local host, into /tmp/<hostname>

4. Move /tmp/<hostname> back to the filer into output/<hostname>

5. Unmount filer and remove temporary mount point

Model CPU Memory Read Cache Write Cache Shelves Drives RAID

ZFS 7420 16 cores 128GB 2 x 512GB 8 X 18GB 4 1TB SAS2 72.K 3-way Mirror

ZFS 7420 16 cores 128GB 2 x 512GB

8 X 18GB

4 300GB SAS2 15K 3-way Mirror

NetApp FAS3070 4 cores 8GB n/a n/a 3 500GB SATA RAID-DP

NetApp FAS6070 8 cores 32GB n/a

n/a

5 300GB FC 10K RAID-DP


ZFS SA Performance Tests Test Results

0

100

200

300

400

500

600

1 H

ost

5 H

osts

10 H

osts

25 H

osts

50 H

osts

100 H

osts

200 H

osts

400 H

osts

600 H

osts

800 H

osts

1000 H

osts

1250 H

osts

1500 H

osts

2000 H

osts

2500 H

osts

Se

co

nd

s

0100200300400500600700800900

1000

1 H

ost

5 H

osts

10 H

osts

25 H

osts

50 H

osts

100 H

osts

200 H

osts

400 H

osts

600 H

osts

800 H

osts

1000 H

osts

1250 H

osts

1500 H

osts

2000 H

osts

2500 H

osts

Seco

nd

s

0

20

40

60

80

100

120

1 H

ost

5 H

osts

10

Ho

sts

25

Ho

sts

50

Ho

sts

10

0 H

osts

20

0 H

osts

40

0 H

osts

60

0 H

osts

80

0 H

osts

10

00

Ho

sts

12

50

Ho

sts

15

00

Ho

sts

20

00

Ho

sts

25

00

Ho

sts

Uti

liza

tio

n %

0

5000

10000

15000

20000

25000

30000

35000

40000

1 H

ost

5 H

osts

10

Ho

sts

25

Ho

sts

50

Ho

sts

10

0 H

osts

20

0 H

osts

40

0 H

osts

60

0 H

osts

80

0 H

osts

10

00

Ho

sts

12

50

Ho

sts

15

00

Ho

sts

20

00

Ho

sts

25

00

Ho

sts

NF

S O

ps

/se

c

Average Read Time(sec) Average Write Time(sec)

Peak NFS Ops Peak CPU Utilization


Monitor performance of ZFS storage directly from EM Grid Control along with DB apps

Dynamic real-time visualization of application and storage related workloads

Customizable dashboards

Used extensively by Oracle IT groups

Sun ZFS Storage Appliance

Oracle Enterprise Manager Grid Control


Program Agenda








Next Steps


ZFS Storage in Product Development

• SPARC Ranch

– SPARC Simulation and Testing Environment

• Compute FARM

– Build/Integration, LRG, Metadata, Transaction Results

• Environment Management System (EMS)

– Fusion, eBiz, Siebel, Enterprise, BEA, Agile

• Virtualization

– Dynamic FARM, VM primary OS, VM secondary storage


• 43 ZFS Storage Appliances serve a very large compute grid for CPU and system design

• 1.32 Peta Byte of data in production on ZFS

• More than 4500 file systems

• Better than 2:1 space savings with built-in data compression

• More than 125,000 computing jobs hit ZFS storage every day

• More than 2000 hosts and more than 46K CPUs use ZFS storage every day

SPARC CPU Simulations

Testing in Production: SPARC


• A development and testing environment

• Offloads local development and QA systems

• Enables parallel processing of builds and tests

• Creates test baseline as part of label creation process

• A label is a string, set of files, a snapshot of development process at any given time

• Compute Farm (9,800 hosts) includes both Static and Dynamic Farm systems

– Capacity of 235K compute hours per day

– Completing 90K farm jobs per day

Highlights

Compute FARM

Developer

FARM Scheduler

Oracle

Enterprise Manager

Grid Control

Job Mgmt Partitioning

Priority Match Making

Resource Mgmt

Submit

Notification

ZFS Storage Build, Metadata, Transaction,

Farm Results, Transaction Results,

Source Code ADE Label Servers

Sun x4170Sun x4170Sun 7410c

Sun x4170 FARM systems

Build/Integration,

Automated testing


Compute FARM Metrics

• ZFS storage is 12x faster compared to EMC

storage involving Farm results purge activity

– ZFS: <15 minutes to purge 1TB farm results

– EMC: >180 minutes to purge 1TB farm results

• ZFS storage is 5x faster compared to EMC

storage involving DB test suite execution for

4800 Farm jobs

– ZFS: consume 381 hours to copy over 828GB of

test results from farm hosts to ZFS storage

– EMC: consume 387 hours to copy over 170GB of

test results from farm hosts to EMC storage

EMC storage – Celerra NSX / Symmetrix DMX 2000/1000

Functions: Build/Integration, LRGs, Metadata, Results

Total filers – 8 (480TB raw, 200TB usable) Mirror

Typical FARM filer – Metrics

7410 ZFS storage Average Maximum

CPU 10% 49%

IOPS 14,000 126,000

Throughput 300 MB/s 750 MB/s

Number of volumes per filer 2 4

Volume size 10TB 18TB

Typical FARM – Workload

Daily average writes 18TB 34TB

Daily average purge 18TB 34TB

# of files written / day 25-35 million 40-50 million

Files written < 10K 60% 90%

Files written < 100K 8% 25%

Files written > 100K < 10000K <1% 10%

Storage features used NFS v3


• Environment Management System (EMS) simplifies the process of creating, deploying and managing environments

• EMS can be used to manage different types of environments, such as various releases of Oracle E-Business Suite, Oracle Database, Oracle Fusion Apps, PeopleSoft, Siebel and stripped down version of application environments

• Leverages storage clone technology

Highlights

Environment Management System (EMS)

EMS UI

BigIP

EMS

Database Servers

EMS daemon,

Storage mgmt,

Slot mgmt

EMS

Servers

SSH daemon,

Hosting

Environments

ZFS Storage

Project,

zPool,

Shares

EMS

Grid Servers

EMS

Web Servers

ZFS

Storage

Su n x4170Su n x4170 Sun 7410cSun

x4170


Environment Management System (EMS) Metrics

• ZFS storage handles >500 up to 1200

volumes and beyond, while NetApp’s hard

limit is 500 only

– ZFS: Ability to use 100% of usable capacity

– NetApp: 30-40% capacity unusable in large

filers (42TB/filer)

• ZFS storage is 2x faster w/ 2x Mirror profile

comparing NetApp EMS Fusion

environments:

– Time taken for refresh slot from template

• ZFS – 30:46 mins (LRGV1S64)

• NetApp – 53:20 mins (LRGV1S64)

NetApp storage – FAS6070, FAS3070, FAS3170

Functions: Fusion, eBiz, Siebel, Enterprise, BEA, Agile

Total filers – 28 (2418TB raw, 1528TB usable) RAIDz1, Mirror

Typical EMS filer – Metrics

7410 ZFS storage Average Maximum

CPU 10% 49%

IOPS 8,000 88,000

Throughput 140 MB/s 650 MB/s

No. of volumes/clones per filer 200 500

Volume size 300GB 1TB

Typical EMS – Workload

Total clones on ZFS 2535 2535

Total full clones on ZFS 2894 2894

Clones per filer 85 234

Full clones per filer 100 280

Storage features used NFS v3, Snapshots, Clones

Total EMS environments on ZFS – 5429

http://emsprod.us.oracle.com/EMS/Forms/RequestDetails/process_form?request_id=9715264

http://emsprod.us.oracle.com/EMS/Forms/RequestDetails/process_form?request_id=9715264


Virtualization – Dynamic FARM

• Compute resource – Pre release starvation, Post release

availability

• Ability to switch FARM hosts OS type/version quickly

• Reduction of compute resource loss

• Use of virtualization and 10G network for performance

• Use of automation to achieve consistent configuration

• Use of storage cloning features for rapid deployment

Highlights


Dynamic FARM – Architecture

HDD HDD HDD HDD HDD HDD HDD HDD

Sun x4170-M2

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

Hypervisor

Su

n O

pu

s T

OR

Sw

itc

h

10G 4x10G

2x20G Cisco 6509

Distribution Switch

Label Server / Cache

Misc Infrastructure

Farm Storage

10/20G

Boot VM’s from clones on NAS storage

ZFS 7410 Storage

Clones

VM01

VM02

VM03

VM04

VM..N

VM01

VM02

VM03

VM04

VM..N

Master Image

Linux Windows Solaris

VM01

VM02

VM03

VM04

VM..N


Sun x4170-M2

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

Hypervisor


Sun x4170-M2

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

/scra

tch

& O

S s

wap

Hypervisor

VM12 VM11 VM10 VM09 VM08 VM07 VM06 VM05 VM04 VM03 VM02 VM01 VM12 VM11 VM10 VM09 VM08 VM07 VM06 VM05 VM04 VM03 VM02 VM01 VM12 VM11 VM10 VM09 VM08 VM07 VM06 VM05 VM04 VM03 VM02 VM01 VM12 VM11 VM10 VM09 VM08 VM07 VM06 VM05 VM04 VM03 VM02 VM01


Dynamic FARM – Deployment


What We Have Learned

• 최적의 서버 통합 – “Green Savings” – power, space, cooling

• 최적의 스토리지 구성 옵션 – defined pre-rack configuration

• 실시간 분석 활용 – ability to understand in-depth work load specifics

• 최적의 RAID 구성 – choose profile based on performance vs. capacity

vs. data protection

• 쉬운 데이터 마이그레이션 – Easy migrations from any NFS server to

ZFS Storage (Shadow Migration)

• 더 많은 자원 관리 – Ability to handle high number of volumes,

directories, clones – exceeds other vendors by >5x times


IT Uncovered Massive Benefits


Program Agenda








Next Steps


Introduction: Oracle Cloud Services

Provide backup and recovery of our hosted customers environments

5500+ Oracle Databases

5 Sites worldwide

Protection of ZFS-SA, Exadata, Exalogic, Analytics

RMAN, NDMP, FS Backups

Onsite and Offsite protection

Backup and Recovery


Program Agenda








Next Steps


Cloud Services Backup Infrastructure Oracle on Oracle vs. Legacy Solutions

6x

Fatser

with 4% Fewer Drives

35% Less Floor Space

64% more Data Stored

1/4 Complexity

VS

4x Larger Library

Desinty

45% Less Media


Oracle Cloud Services Backup and Recovery

Reduced Man Power

– Vaulting went from a full time task for 2 people down to a part time task for

1 person

Fault Isolation

– Redundancy built in at more layers then legacy

Security

– More flexible with drive based encryption

– T10000c offers FIPS to be required permanently

– Robust Secure Key Storage and Access

Key Oracle on Oracle Advantages


Traditional Approach

Solving Backup and Recovery Deployment Complexity

Collection of

vendors

First integration

is in our

data center

Complex matrix

of products

Staff focused

on making

it work

Cloud IT’s Approach

Engineered to

work together

One point of

accountability Tested & proven

architecture

Staff focused on

strategic projects


Program Agenda








Next Steps


Oracle Cloud Services Backup and Recovery

Enterprise Tape Library

– SL3000

Tape Drives

– StorageTek T100000c, StorageTek LTO4/5

Encryption

– Oracle Key Manager

Library Management

– StorageTek Automated Cartridge System Library Software (ACSLS)

Centralized Tape Backup Management

– Oracle Secure Backup

플랫폼 구성 요소


High Level Standard Solution Diagram

1. ACSLS Server

Addition

2. Even / Odd

• Drives

• ACSLS/Library

• Servers

4. Double the robotic

moves at once

5. Reduced Drive

Count per Library

3. 50% Fault Isolation


High Level OSB to ACSLS / Library Layout

1. More then 1 Logical

Library per OSB

Domain

2. Even / Odd

• Drives

• ACSLS/Library

• Servers

3. More then 1 OSB

Domain per physical

library


5. Double the robotic

moves at once


Mini Redundant Level Solution Diagram

DEM Base

LIB 1

Mini redundant high

level counts draft 1.2

--scorso

P S1

P S2

FA N

S T AT U S M G T

4 6 4 7 484 5 4 6 4 7 484 3 4 441 4 23 9 4037 3 83 5 363 3 3 431 3 229 302 7 282 5 2 62 3 2 421 2 21 9 2017 1 81 5 161 3 1 411 1 29 1 07 85 63 41 2

CO N

4 5

1U Cable Manager

4948E-FOSB Pri

TOP

FAN

REAR

PS

O K

OK

OK

OK

OK

O K

O K

O K

O K

O K

O K

O K

O K

O K

O K

OK

OK

Su n F i re X42 70

SAT A DV D

1

0 4

5

8

9 DVD

3

2 6

7

4

5 12

14

10 13

11

15

1U Cable Manager

1U Cable Manager

P S 1

P S 2

F AN

S TA T US M G T

46 47 4 84 5 46 4 7 4 84 3 444 1 4 239 4 03 7 3835 3 633 343 1 322 9 3 027 2 825 262 3 242 1 2 219 2 01 7 1815 1 61 3 141 1 1 29 1 07 85 63 41 2

C ON

4 5

1U Cable Manager

Catalyst 2960G Serie s

M O DE

SYS T

RPS

ST AT

D UPL X

SP EED

1X

2 X

1 2 3 4 5 6 7 8 9 1 0

1 1 X

12 X

1 1 1 2

13 X

14 X

1 3 1 4 15 16 1 7 1 8 1 9 20 21 2 2 2 3 24

25 X

26 X

2 5 26 27 2 8 2 9 3 0 3 1 32 33 3 4

35 X

3 6 X

3 5 3 6

3 7X

3 8 X

3 7 3 8 39 40 4 1 42 43 4 4

4 3 X

4 4 X

4 5 4 6 47 48

2 4 X

2 3 X

1U Cable Manager

OPUS

4948E-F

2960OOB Network

OPUS

OSB/Backup SEC

OSB/Backup PRI

OSB Sec

2U Cable Manager

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

34

35

36

41

42

40

38

39

37

26

27

28

29

30

32

33

31

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

34

35

36

41

42

40

38

39

37

26

27

28

29

30

32

33

31

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

GRID

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

34

35

36

41

42

40

38

39

37

26

27

28

29

30

32

33

31

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

34

35

36

41

42

40

38

39

37

26

27

28

29

30

32

33

31

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

GRID

OSB admin server

media server

media server

media server

media server

media server

media server

media server

media server

Aggregation/

Backup

Storage

OSB

Servers

Fiber to zones in DC02

DEM Base

LIB 2

1U Cable Manager

TOP

FAN

REAR

PS

O K

O K

O K

O K

O K

OK

OK

OK

OK

O K

O K

O K

OK

OK

OK

O K

O K

Sun Fi re X 427 0

SA TA D VD1

0 4

5

8

9 DVD

3

2 6

7

4

5 1 2

1 4

1 0 13

1 1

1 5Osb admin server 2

16- drives per

server pair

34 fiber ports to GRID/Patch

34 copper ports to Grid/Patch

Possible future growth to 58 of each type

Mini LIB 1

34 fiber ports to Grid(s)

34 copper ports to Grid(s)

Possible future growth to 58 of each type

Mini LIB 2

Initial 2 media servers + Admin

4 FC SAN to Grid(s)

6 FC IP (10gbe) to Grid(s)

9 Copper (FE+Ilom) to Grid(s)

Encryption

13 Copper (MGMT/FE, SVC, Ilom) to Grid(s)

ACSLS

10 Copper (Mgmt/FE, NFS, Ilom) to Grids(s)

4 FC (SAN to Brocade)

DC2-C14 (Day 1)

1 Admin, All media servers, Encryption

16 FC SAN to Grid(s)

18 FC IP (10gbe) to Grid(s)

40 Copper (FE+Ilom) to Grid(s)

Grid(s) (Future as built)

TOP

FAN

REAR

PS

O K

OK

OK

OK

OK

O K

O K

O K

O K

O K

O K

O K

O K

O K

O K

OK

OK

Su n F ire X42 70

SAT A DVD

1

0 4

5

8

9 DVD

3

2 6

7

4

5 12

14

10 13

11

15

KMA02KMA01

BKP-MGMT

Growth

Local OKM Cluster

Replicates to Remote(s)

ACSLSACSLS

1. Big impact, small

foot print

2. Even / Odd

• Drives

• ACSLS/Library

• Servers

4. Balance slots to

drives as required


5. Interchangeable with

our Enterprise

deployment


Program Agenda








Next Steps


Best Practices Data Availability - Protection

Data Guard

RMAN

ZFSSA: Snapshots

Minutes-Hours

RMAN

ZFSSA: Snapshots

ZFSSA: Replication

Oracle Tape: Onsite

RMAN: Offsite

Oracle Tape: Onsite

Oracle Tape: Offsite

ZFSSA: Replication

Oracle Tape

Business Critical

Data Corruption

User Error protection

User Error

DB Refresh/Copy

Fault Protection

Business Requirements

Fault Protection

Disaster Recovery

Geographic Migration

Remote access to data

Remote Disaster Recovery

Hours Retention Flexibility

Products

Use Case


Best Practices 보안 : 테이프 데이터 암호화

Highly Available

Simple

Secure

Accessible


Best Practices Infrastructure Deployments: OSB, ACSLS, STK Libraries, OKM

Redundant

Scalable

Efficient

Flexible


Documents

Extreme Storage Scale and Efficiency: Lessons from a