Yiying ZhangGokul Soundararajan

Mark W. Storer

Lakshmi N. Bairavasundaram

Sethuraman Subbiah

Andrea C. Arpaci-Dusseau

Remzi H. Arpaci-Dusseau

Warming up Storage-level Caches with Bonfire

Does on-demand cache warmup still work ?

2

0.0

0.1

1.0

16.0

256.0

4096.0

65536.0

1990 1995 2000 2005 2010 2015

Me

mo

ry (

Cac

he

) Si

ze (

GB

)

Year

3

< 10 GB

10s of TBs

. . .

+ idle time

Random:6 days

Sequential:2.5 hours

1 TB Cache

0

20

40

60

80

100

0 4 8 12 16 20 24

Re

ad H

it R

ate

(%

)

Time (hour)

Always-warm

Cold

How Long Does On-demand Warmup Take?

* Simulation results from a project server trace

4

• Read hit rate difference between warm cache and on-demand

On-demand

On-demand warmup takes hours to days

• Caches are critical▫ Key component to meet application SLAs

▫ Reduce storage server I/O load

• Cache warmup happens often▫ Storage server restart

▫ Storage server take-over

▫ Dynamic caching [Narayanan’08, Bairavasundaram’12]

To Make Things Worse

5

• Bonfire▫ Monitors and logs I/Os

▫ Load warmup data in bulk

• Challenges▫ What to monitor & log? Effective

▫ How to monitor & log? Efficient

▫ How to load warmup data? Fast▫ General solution

On-demand Warmup Doesn’t Work Anymore What Can We Do?

6

BonfireMonitor

Storage System

Warmup Information

LoggingVolume

I/O

• Trace analysis for storage-level cache warmup▫ Temporal and spatial patterns of reaccesses

• Cache warmup algorithm design and simulation

• Implementation and evaluation of Bonfire

▫ Up to 100% warmup time improvement over on-demand

▫ Up to 200% more server I/O load reduction

▫ Up to 5 times lower read latency

▫ Low overhead

Summary of Contributions

7

• Introduction

• Trace analysis for cache warmup

• Cache warmup algorithm study with simulation

• Bonfire architecture

• Evaluation results

• Conclusion

Outline

8

• MSR-Cambridge [Narayanan’08]

▫ 36 one-week block-level traces from MSR-Cambridge data center servers

▫ Filter out write-intensive, small working set, and low reaccess-rate

Workload Study – Trace Selection

Server Function #volumes

mds Media server 1

prn Print server 1

proj Project directories 3

src1 Source control 1

usr User home directories 2

web Web/SQL server 1

Reaccesses: Read After Readsand Read After Writes

9

Questions for Trace Study

10

Time Space

Relative Q1: What’s the temporal distance?Q3: What’s the spatial distance?

Any clustering of reaccesses?

AbsoluteQ2: When do reaccesses happen

(in terms of wall clock time)?

Q4: Where do reaccesses

happen (in terms of LBA)?

0

50

100

150

12 AM 1 AM 2 AM 3 AM 4 AM 5 AM 6 AM

LBA

5 hours 1 hour

Q1: What is the Temporal Distance?

11

0

20

40

60

80

100

0 24 48 72 96 120 144 168

Am

ou

nt

of

Re

acce

sse

s (%

)

Temporal Distance (Hour)

Time b/w Reaccesses for All Traces

Within an hour: Hourly

23-25 hours: Daily

Q1: What is the Temporal Distance?

A1: Two main reaccess patterns: Hourly & Daily

12

0

20

40

60

80

100

mds_1 proj_1 proj_4 usr_1 src1_1 web_2 prn_1 proj_2 usr_2

Am

ou

nt

of

Re

acce

ses

(%)

HourlyDailyOther

Hourly Dominated

Daily Dominated Other

In an hour, recent blocks more likely reaccessed

Q2: When Do Reaccesses Happen (Wall Clock Time)?

13

0

1

2

3

4

5

6

0 1 2 3 4 5 6 7

Dai

ly R

eac

cess

es

(%)

Time (day)

src11web2

A2: Daily reaccesses at same time every day

Q3: What is the Spatial Distance?

A3: Spatial distance usually small for hourly

sometimes small for other reaccesses

14

0%

20%

40%

60%

80%

100%

mds1 proj1 proj4 usr1 src11 web2 prn1 proj2 usr2Am

ou

nt

of

Rea

cces

ses

(%)

<10MB 10MB-1GB 1-10GB 10-100GB >100GB

Hourly Dominated Daily Dominated Other

• Percentage of 1MB regions that have reaccesses

Q3: Any spatial clustering among reaccesses?

A3: Daily reaccesses more spatially clustered

15

0

20

40

60

80

100

mds_1 proj_1 proj_4 usr_1 src1_1 web_2 prn_1 proj_2 usr_2

Pce

nta

geo

f R

eac

cese

sin

1

MB

Re

gio

n (

%)

Hourly

Daily

Other

• A1 Hourly: Use recently accessed blocks

• A1 and A2 Daily: Use same period from previous day

• A3 Small spatial distance: Size of monitoring buffer is small

Trace Analysis Summary and Implications

Time Space

Relative

A1: Reaccesses have two main

temporal patterns:

within 1 hour, around 1 day

A3: Hourly reaccesses are

close in spatial distance.

Daily reaccesses exhibit

spatial clustering.

AbsoluteA2: Daily reaccesses correlate

with wall clock time

A4: No hot spot of

reaccesses in LBA space

16

• Introduction

• Trace analysis for cache warmup

• Cache warmup algorithm study with simulation

• Bonfire architecture

• Evaluation results

• Conclusion

Outline

17

• Warmup period: Hit-rate convergence time

0

20

40

60

80

100

0 20 40 60 80 100 120

Re

ad H

it R

ate

(%

)

Time

Always-warm cache

New cache

Metrics: Warmup Time

Converge time strictConverge time loose

18

• Storage server I/O load reduction

▫𝐴𝑚𝑜𝑢𝑛𝑡 𝑜𝑓 𝐼/𝑂𝑠 𝑔𝑜𝑖𝑛𝑔 𝑡𝑜 𝑐𝑎𝑐ℎ𝑒

𝑇𝑜𝑡𝑎𝑙 𝐼/𝑂𝑠during convergence time

• Improvement in server I/O load reduction

▫𝑆𝑒𝑟𝑣𝑒𝑟 𝐼/𝑂 𝑙𝑜𝑎𝑑 𝑟𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝐵𝑜𝑛𝑓𝑖𝑟𝑒

𝑆𝑒𝑟𝑣𝑒𝑟𝐼

𝑂𝑙𝑜𝑎𝑑 𝑟𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑂𝑛−𝑑𝑒𝑚𝑎𝑛𝑑

Metrics: Server I/O Reduction

19

• Last-K: Last K regions accessed in the trace

• First-K: First K regions in the past 24 hours

• Top-K: K most frequent regions

• Random-K: Random K regions

Cache Warmup Algorithms

LF

24 Hours

I/Os

Time

RT

cache starts

20

Region: granularity of monitoring and logginge.g., 1MB

• LRU cache simulator with four warmup algorithms

• Convergence time ▫ Improves 14% to 100%

• Server I/O load reduction ▫ Improves 44% to 228%

• In general, Last-K is the best

• First-K works for special case (known patterns)

Simulation Results - Overall

21

• Introduction

• Trace analysis for cache warmup

• Cache warmup algorithm study with simulation

• Bonfire architecture

• Evaluation results

• Conclusion

Outline

22

• Design principles▫ Low overhead monitoring and logging (efficient)

▫ Bulk loading useful warmup data (effective and fast)

▫ General design applicable to a range of scenarios

• Techniques▫ Last-K

▫ Monitors I/O below the server buffer cache

▫ Performance snapshot

Bonfire Design

23

Bonfire Architecture: Monitoring

PerformanceSnapshot

Bonfire Monitor

Storage System

…

123

4

n

5...Warmup

Metadata

BufferCache

LoggingVolume

I/O

I/O Warmup Data

In-memoryStaging Buffer

Data Volumes

I/O

24

Only store warmup metadata: metadata-only

Store warmup metadata and data: metadata+data

Bonfire Architecture: Bulk Cache Warmup

PerformanceSnapshot

Bonfire Monitor

Storage System

…

123

4

n

5...Warmup

Metadata

BufferCache

LoggingVolume

I/O

I/O Warmup Data

Data Volumes

In-mem n n-1 k..

Sorted by LBA

NewCache

Warmup Data

NewCache

25

• Introduction

• Trace analysis for cache warmup

• Cache warmup algorithm study with simulation

• Bonfire architecture

• Evaluation results

• Conclusion

Outline

26

1 week

• Implemented Bonfire as a trace replayer▫ Always-warm, on-demand, and Bonfire

▫ Metadata-only and metadata+data

▫ Replay traces using sync I/Os

• Workloads▫ Synthetic workloads

▫ MSR-Cambridge traces

• Metrics▫ Benefits and overheads

Evaluation Set Up

27

1 2 3 4 5 6 7

On-demand

BonfireAlways-warm

0

20

40

60

80

100

0 1 2 3 4

Re

ad H

it R

ate

(%

)

Num of I/Os (x1000000)

Always-warm

Cold

Bonfire

Benefit Results - Read Hit Rate of MSR Trace

On-demand converges

Bonfire

converges

28

* Results of a project server trace from MSR-Cambridge trace set

CacheStarts

Day 5 Day 6

• Higher read hit rate => less server I/O load

On-demand

0

0.5

1

1.5

2

0 1 2 3 4

Re

ad L

ate

ncy

(m

s)

Num of I/Os (x1000000)

Always-warm

Cold

Bonfire

Benefit Results - Read Latency of MSR Trace

29

• Lower read latency => better application-perceived performance

* Results of a project server trace from MSR-Cambridge trace set

Cache Starts

Day 5 Day 6

On-demand

Overhead Results

30

PerformanceSnapshot

Bonfire Monitor

Storage System

…

Warmup Metadata

BufferCache

LoggingVolume

I/O

I/O Warmup Data

In-memoryStaging Buffer

Data Volumes

123

4

n

5...

In-memoryStaging Buffer

PerformanceSnapshot

256KB & 128MB19MB to 71MB

4.6MB/s to

238MB/s9.5GB to 36GB

9KB/s to

476KB/s

Overhead Results

31

PerformanceSnapshot

Bonfire Monitor

Storage System

Warmup Metadata

BufferCache

LoggingVolume

I/O

I/O Warmup Data

In-memoryStaging Buffer

Data Volumes

123

4

n

5...

Warmup Data

NewCache

256KB & 128MB19MB to 71MB

2 to 20 minutes

PerformanceSnapshot

Proper Bonfire scheme and configuration

…

4.6MB/s to

238MB/s9.5GB to 36GB

9KB/s to

476KB/sIn-memory

Staging Buffer

• Faster cache warmup▫ 59% to 100% improvement over on-demand

• Less storage server I/O load▫ 38% to 200% more reduction than on-demand

• Better application-perceived latency▫ Avg read latency 1/5 to 2/3 of on-demand

• Small controllable overhead

Summary of Results

32

On-demand warmup doesn’t work anymore▫ Warm up terabytes of caches take days

Bonfire and beyond▫ Client-side cache warmup

▫ Application-aware warmup

▫ …

In need for more long big public traces !

Conclusion

33

Thank You

Questions?

34

http://wisdom.cs.wisc.edu/home

http://research.cs.wisc.edu/adsl