On the Performance Variation Motivation in Modern Storage ... · ue.g., ddif=/dev/sde1 Workload: Fileserver On the Performance Variation in Modern Storage Stacks (FAST’17) 0 200

13/2/2017 On the Performance Variation in Modern Storage Stacks (FAST’17)

On the Performance Variation in Modern Storage Stacks

15th USENIX Conference on File and Storage Technologies 2017

Zhen Cao1, Vasily Tarasov2, Hari Prasath Raman1, Dean Hildebrand2, and Erez Zadok1

1Stony Brook University; 2IBM Research – Almaden;

© 2017 Stony Brook University

2

Motivation

24,888 Configurations

23% unstable(> 5% std. dev.)


3

Motivation (cont.)l Stable performance is critical

uBenchmarking§ Inaccurate conclusions

uProduction systems§Clients’ satisfaction

l Performance variation is complicateduVarious sourcesuTime-consuming to measure

l Storage stacks are important contributorsl First systematic study

3/2/2017 On the Performance Variation in Modern Storage Stacks (FAST’17) 4

Backgroundl Measurements for performance

uThroughput§Window size matters

uLatency


5

Background (cont.)Measurements for variations


uRelative Standard Deviation

u (Coefficient of Variation)

u

uUnderstate impacts of outliers

uRelative Rangeu

6





u


uRelative Rangeu

7





u


uRelative Rangeu

�

8

Methodologyl Important parameters

uFile Systems§ Ext4, XFS, Btrfs

uF/S parameters§ Block Size, Inode Size, etc.

uI/O SchedulersuDevices: HDDs, SSDs

l Latin Hypercube SamplinguExplore many-factor systems

3/2/2017

1,700+ Configurations1.5 Years

On the Performance Variation in Modern Storage Stacks (FAST’17)

9

Methodologyl Important parameters

uFile Systems§ Ext4, XFS, Btrfs

uF/S parameters§ Block Size, Inode Size, etc.

uI/O SchedulersuDevices: HDDs, SSDs

l Latin Hypercube SamplinguExplore many-factor systems

3/2/2017

1,700+ Configurations1.5 Years

100+ Configurations1 Month

On the Performance Variation in Modern Storage Stacks (FAST’17) 10

Experiment Settingsl Machines

u1 Intel Xeon quad-core 2.4GHz CPU, 4GB RAMuDevices: SAS, SATA, SSD

l FilebenchuWorkloads

§Mailserver, Fileserver, WebserveruFileset size: 10GB (2.5 × RAM size)uRunning time

§ 2,000s for Mailserver; otherwise 800s

l 10+ runs for each experiment3/2/2017 On the Performance Variation in Modern Storage Stacks (FAST’17)

11

Performance Variation: Overview

0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e

Avg. Throughput (IOPS) (log10)

Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017

Workload: Mailserver



0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



13


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017




0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



15


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017




0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



17


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017




0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



19


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017


HDDl Throughput

u 200 ~ 2,000l Range

u Up to 42%



0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017


HDDl Throughput

u 200 ~ 2,000l Range

u Up to 42%

SSDl Throughput

u 2,000 ~ 20,000

l Rangeu Up to 11%


21


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017


Ext4l Up to 42%l Most above

20%



0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



20%XFS

l Most under 18%


23


0%

10%

20%

30%

40%

100 1000 10000

Rel

ativ

e R

ang

e


Ext4-HDDExt4-SSD

XFS-HDDXFS-SSD

Btrfs-HDDBtrfs-SSD

3/2/2017



20%XFS

l Most under 18%

Btrfsl Most under

15%


0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ange

Sampled Configurations

Mailserver Fileserver Webserver

Performance Variation: Overview (cont.)


25

0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ang

e





0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ang

e





27

0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ange





0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ange





29

0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ange





0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ange





31

0%

10%

20%

30%

40%

1 2 3 4 5 6 7 8 9 10

Rel

ativ

e R

ang

e





Ext4 Case Study

0

200

400

600

baseline

2048-2048-writeback-noatime-noop-SATA

47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017

Workload: Fileserver


33

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017



Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017



max1st quartilemedian3rd quartile

min

35

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017



actual data points

36

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017

l baseline



37

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017

l baselinel +no_lazy

u mkfs.ext4 –E lazy_itable_init=0



0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

38

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017





0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

39

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017



l +umountu sync → umount + mount



0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount


47%

22% 19%

Th

rou

gh

pu

t (I

OP

S)

40

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017






0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount


47%

22% 19%

Th

rou

gh

pu

t (I

OP

S)

41

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017




l +allocu Preserve layout

u e.g., dd if=/dev/sde1



0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount


47%

22% 19%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount

+alloc


47%

22% 19%

2%

Th

rou

gh

pu

t (I

OP

S)

42

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017








0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount


47%

22% 19%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount

+alloc


47%

22% 19%

2%

Th

rou

gh

pu

t (I

OP

S)

43

Ext4 Case Study

0

200

400

600

baseline


47%

Th

rou

gh

pu

t (I

OP

S)

3/2/2017








0

200

400

600

baseline

+no_lazy


47%

22%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount


47%

22% 19%

Th

rou

gh

pu

t (I

OP

S)

0

200

400

600

baseline

+no_lazy

+umount

+alloc


47%

22% 19%

2%

Th

rou

gh

pu

t (I

OP

S)

44

460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number

default alloc. policy

Ext4 Case Study (cont.)

3/2/2017



45

460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number



3/2/2017



460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number



3/2/2017



47

460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number



3/2/2017



460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number



3/2/2017



49

460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Thro

ughput

(IO

PS

)

Block Number



3/2/2017



19%

50


3/2/2017

460

480

500

520

540

560

0 2x107

4x107

6x107

8x107

1x108

1.2x108

Th

rou

gh

pu

t (I

OP

S)

Block Number

default alloc. policys_hash_seed=null



51

Throughput vs. Time

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS

3/2/2017



Window Size: 120s

52

Throughput vs. Time

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS

3/2/2017



Window Size: 120s

53

Throughput vs. Time

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS

3/2/2017



Window Size: 120s

54

Throughput vs. Time

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS

3/2/2017



Window Size: 120s

55

Throughput vs. Time

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS

3/2/2017



Window Size: 120s

56

Throughput vs. Time

3/2/2017


l Ext4u 14% ~ 43%

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS


Window Size: 120s

57

Throughput vs. Time

3/2/2017


l Ext4u 14% ~ 43%

l Btrfsu 2% ~ 23%

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS


Window Size: 120s

58

Throughput vs. Time

3/2/2017


l Ext4u 14% ~ 43%

l Btrfsu 2% ~ 23%

l XFSu 7% ~ 65%

0

300

600

0 0.5 1 1.5 2

Thro

ughp

ut (I

OPS

)

Time (hours)

Ext4 Btrfs XFS


Window Size: 120s

59

Throughput vs. Time (cont.)

3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s

Ext4Workload: Fileserver



3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s



61


3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s




3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s



63


3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s




3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s 0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s

Ext4 XFSWorkload: Fileserver


65


3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s 0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s




3/2/2017

0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s 0

20

40

60

80

100

0 20 40 60 80 100

Cum

. P

ct. of

Tim

e W

indow

s (%

)

Relative Range (%)

60s120s180s240s400s



67

Latency Variation

3/2/2017

0

20

40

60

80

100

10-3

10-1

101

103

Cum

. P

ct. of

Lat

ency

(%

)

Latency (ms) (log10)

read

l Ext4 - HDDu 32%

l Fileserveru createu deleteu openu readu statu write


Latency Variation

3/2/2017

0

20

40

60

80

100

10-3

10-1

101

103

Cum

. P

ct. of

Lat

ency

(%

)


read

l Ext4 - HDDu 32%



69

Latency Variation

3/2/2017

0

20

40

60

80

100

10-3

10-1

101

103

Cum

. P

ct. of

Lat

ency

(%

)


read

l Ext4 - HDDu 32%



Latency Variation

3/2/2017

0

20

40

60

80

100

10-3

10-1

101

103

Cum

. P

ct. of

Lat

ency

(%

)


0

20

40

60

80

100

10-3

10-1

101

103

Cum

. P

ct. of

Lat

ency

(%

)


create read


71

Conclusionsl First systematic study of storage

performance variationsl Characterization from various perspectivesl Randomization in file systemsl Lazy initialization and background activities

contribute to instabilityl We will make datasets public


Future Workl Aged file systemsl More workloads/benchmark tools l More Devices

uNear full-capacity SSDsuPCM, SMR, etc.

l More storage layersuRAID, LVM, etc.



On the Performance Variation in Modern Storage Stacks

Zhen Cao1, Vasily Tarasov2, Hari Prasath Raman1, Dean Hildebrand2, and Erez Zadok1

© 2017 Stony Brook University

Documents

On the Performance Variation Motivation in Modern Storage ... · ue.g., ddif=/dev/sde1 Workload: Fileserver On the Performance Variation in Modern Storage Stacks (FAST’17) 0 200