Optimal Inter-Object Correlation When Replicating for Availability

Optimal Inter-Object CorrelationOptimal Inter-Object CorrelationWhen Replicating for AvailabilityWhen Replicating for Availability

Haifeng YuNational University of Singapore

Phillip B. GibbonsIntel Research Pittsburgh

Haifeng Yu (National University of Singapore) 2

Multi-object OperationsMulti-object Operations

Data replication for better availability Traditional research focuses on availability of

individual data objects

E.g., individual files or database objects

User-level tasks may access multiple data objects / files: Multi-Object Operations Compile a project / Latex a paper

Aggregation queries for databases


Availability of Multi-Object OperationsAvailability of Multi-Object Operations

Availability of single object is not the same as the availability for multi-object operations: An operation requesting 1,000 objects may observe

nearly 1,000 times higher failure probability

But there’s more..... Our recent experimental study shows that the

assignment from object replicas to machines has critical effects on such availability:

“Availability of Multi-Object Operations” [NSDI’06]


Object Assignment ExampleObject Assignment Example

Compile a project need all 4 files

4 files: A B C D

4 machines:

A B C DA B C DA B C DA B C D

Computing average may tolerate one

missing object

still OKnot OK

which is better?

A B

A B C D

C Dbetter

A B

A C

C D

B D

better


Important Observations from the ExampleImportant Observations from the Example

For individual objects, it does not matter which machines it is assigned Distinguishes from replica placement problems

Same “concentration” / “spread”

Difference is in inter-object correlation: Obj A is “fully correlated” with one obj (B) VERSUS

Obj A is “partially correlated” with two objs (B and C)

which is better?A B

A B C D

C D A B

A C

C D

B D


Practical ImportancePractical Importance

Applicable to almost all replication systems CAN, CFS, Chord, Coda, FARSITE, GFS, GHT,

Glacier, Pastry, R-CHash, RIO, …

Failure probability of TPC-H varies by 4 orders of magnitude under different assignments All with the same storage overhead


Formal ModelFormal Model N objects each with k replicas

Each machine holds l objects (total Nk / l machines)

Machines fail / crash i.i.d.

Object unavailable if all k machines holding it fail

Assignment: Mapping from objects to machines No machine holds multiple replicas of the same obj

Two specific assignments: PTN: Partition objects into N/l groups of size l, and map

each group to k machines

RAND: A uniformly random assignment


Formal Model (continued)Formal Model (continued)

A multi-object operation requests n specific objs This talk assumes n = N; See paper for n < N.

Operation is successful if ≥ t objects are available

t depends on application semantics

We will consider a single (multi-object) operation See paper for more discussion on this…

Availability of the assignment defined as Prob[the operation succeeds]


Previous Experimental ResultsPrevious Experimental Results

0 .9 8 5 0 .9 9 0 .9 9 5 10.985 0.99 0.995 1

1.0

0.1

0.01

0.001 t/n

Several other assignments experimented

(including Chord) fall between

PTN and RAND

RAND

PTN

failure probability (i.e., 1 – availability)


Our GoalOur Goal Limitations of experimental method:

Can only study a small number of assignments

Results only for specific parameter values (e.g., specific machine failure probabilities)

Our goal: Find assignment with best/worst availability among all

possible assignments

This is the first theoretical study on this subject…


Summary of Our ResultsSummary of Our Results

Impossible to remain optimal for all t values

Calculating the availability of an assignment is #P-hard

Best Worst

t = n (general case)

PTN(within small constants)

RAND(within small constants)

t = n (restricted case)

PTN n/3 rings of size 3

t = l + 1 Trivial construction (also RAND

within small constants)

PTN

This talk focuses on t = n only


The General CaseThe General Case

Upper and lower bounds Leveraging Janson’s inequality – a tail approximation

for sum of dependent Bernoulli trialsNot to be confused with Jensen’s inequality

Approaching upper and lower bounds by PTN and RAND Leverage Janson’s inequality a second time

See paper for details…


The Restricted CaseThe Restricted Case

Results for general case Have constants

RAND is a distribution – what is the “structure” of the worst assignment?

We consider significantly restricted scenarios Each object has two replicas

Each machine holds two objects (l = 2)

Corresponds to our example earlier


Using Rings to Represent AssignmentsUsing Rings to Represent Assignments Each assignment corresponds to a set of rings

Ring size from 2 to n

Sum of sizes of all rings is n

Availability uniquely determined by ring sizes

A B

A B C D

C D A B

A C

C D

B D= =

A B

C D

A B

DC

2 rings of size 2 1 ring of size 4


Hill ClimbingHill Climbing

Adjustment step: merge two rings into one or split one ring into two Can always transform an assignment to another

within a finite number of steps

Crux: How does availability change when merging two rings of size x and y (x y) into a ring of size (x+y) ? Theorem: Availability improves iff y is odd.


The Restricted Case: ResultsThe Restricted Case: Results

Theorem: n/2 rings of size 2 is the best. This corresponds to PTN

Theorem: n/3 rings of size 3 is the worst.

What about a single ring of size n?

What about rings of other sizes? See paper for answers…

Parity of ring sizes matter a lot…


Impossibility ResultImpossibility Result

0 .9 8 5 0 .9 9 0 .9 9 5 10.985 0.99 0.995 1

1.0

0.1

0.01

0.001

RAND

PTN

Area bounded by the curve is constant

Impossible to remain optimal under all t values


ConclusionsConclusions Availability of multi-object operation critically

affected by inter-object correlation

First theoretical study of object assignment Best/worst assignments for t = n and t = l + 1

Impossible to remain optimal under all t values

See paper for full results…

Open questions: Other t values?

Erasure coding?

Documents

Optimal Inter-Object Correlation When Replicating for Availability