Self-Management for Large Scale Distributed Systems · Introduction Niche Platform Robust Management Elements Future Work The Problem Autonomic Computing The Goal Outline 1 Introduction

Self-Management for Large Scale DistributedSystems

Ahmad Al-Shishtawy ([email protected])

Advisors:Vladimir Vlassov ([email protected])

Seif Haridi ([email protected])

KTH Royal Institute of TechnologyStockholm, Sweden

The 5th EuroSys Doctoral Workshop (EuroDW 2011)April 10, 2011

IntroductionNiche Platform

Robust Management ElementsFuture Work

Outline

1 Introduction

2 Niche Platform

3 Robust Management Elements

4 Future Work

Self-Management for Large Scale Distributed Systems (A. Al-Shishtawy) 2/25



The ProblemAutonomic ComputingThe Goal

Outline

1 Introduction

2 Niche Platform


4 Future Work





Dealing with Complexity

ProblemAll computing systems need to be managed






ProblemAll computing systems need to be managed






ProblemComputing systems are getting more and more complex






ProblemComplexity means higher administration overheads






ProblemComplexity poses a barrier on further development






SolutionThe Autonomic Computing initiative by IBM






SolutionSelf-Management: Systems capable of managing themselves






SolutionUse Autonomic Managers

Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge






Open Question

How to achieve Self-Management?

Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge Monitor

Analyze Plan

Execute

Autonomic Manager

Knowledge





The Autonomic Computing Architecture

Managed Resource

Touchpoint (Sensors &Actuators)Autonomic Manager

MonitorAnalyzePlanExecute

Knowledge Source

Communication

Manager Interface

Monitor

Analyze Plan

Execute

Touch Point

Autonomic Manager

Managed Resource

Knowledge

Managed Resource

Touch Point

Manager

Interface





The Goal

Large-scale distributed systems

Complex and require self-management

May run on unreliable resourcesMajor sources of complexity:

Scale (resources, events, users, . . . )Dynamism (resource churn, load changes, . . . )

GoalA platform (concepts, abstractions, algorithms. . . ) that facilitatesdevelopment of self-managing applications in large-scale and/ordynamic distributed environment.

A methodology that help us to achieve self-management.





The Goal

Large-scale distributed systems

Complex and require self-management

May run on unreliable resourcesMajor sources of complexity:

Scale (resources, events, users, . . . )Dynamism (resource churn, load changes, . . . )

GoalA platform (concepts, abstractions, algorithms. . . ) that facilitatesdevelopment of self-managing applications in large-scale and/ordynamic distributed environment.

A methodology that help us to achieve self-management.





Research Plan

Self-Management in large-scale distributed systems. Consists of fourmain parts:

Part 1: Touchpoints and feedback loops in distributed systems

Part 2: Robust Management

Part 3: Improve management logic

Part 4: Integrate previous parts in a self-managing system.




Niche OverviewManagement PartRuntime Environment

Outline

1 Introduction

2 Niche Platform


4 Future Work





Niche

Niche is a Distributed Component ManagementSystem

Niche implements the Autonomic ComputingArchitecture for large-scale distributed environment

Niche leverages Structured Overlay Networks forcommunication and for provisioning of basic services(DHT, Publish/Subscribe, Groups, etc.)

Monitor

Analyze Plan

Execute

Touch Point

Autonomic Manager

Managed Resource

Knowledge

Managed Resource

Touch Point

Manager

Interface





Management Part

Management ElementsWatchersAggregatorsManagersExecutors

Communicate through events

Publish/Subscribe

Autonomic Managers (controlloops) built as network of MEs

Sensors and Actuators forcomponents and groups

Actuation APIFunctional Part





Management Part



Publish/Subscribe




Management Part





Management Part



Publish/Subscribe




Management Part

Watcher

Aggreg.

Manager

ExecutorExecutorWatcher Watcher





Management Part



Publish/Subscribe




Management Part

Watcher

Aggreg.

Manager






Management Part



Publish/Subscribe




Management Part

Watcher

Aggreg.

Manager






Management Part



Publish/Subscribe




Management Part

Watcher

Aggreg.

Manager






Runtime Environment

Containers that hostcomponents and MEs

Use a Structured OverlayNetwork for communication

Provide overlay services





Runtime Environment








Runtime Environment




0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031





Dealing with Resource Churn

How to deal with failures?MEs heal the functional partHow to heal failed MEs?

Programmatically in themanagement logicTransparently by theplatform

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031








0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031








0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

??!!




Solution OutlineThe SMART Reconfiguration AlgorithmAutomatic ReconfigurationEvaluation

Outline

1 Introduction

2 Niche Platform


4 Future Work





Robust Management Elements

A Robust Management Element (RME):

is replicated to ensure fault-tolerance

tolerates continuous churn by automatically restoring failedreplicas on other nodes

maintains its state consistent among replicas

provides its service with minimal disruption in spite of resourcechurn (high availability)

is location transparent, i.e., RME clients communicate with itregardless of current location of its replicas




























































Solution Outline

Replicated state machine

An algorithm to reconfigure the replicated state machine. (Weused the SMART algorithm)

Our decentralized algorithm to automate reconfiguration





Solution Outline








Solution Outline








SMART

A B C D E

Admin





SMART

A B C D E

Admin





SMART

A B C D E

Admin

ConfigurationRepository





SMART

A B C D E

Admin


{A,B,C,D,E}

Configuration {A,B,C,D,E}





SMART

A B C D E

Admin


{A,B,C,D,E}






SMART

A B C D E

Admin


{A,B,C,D,E}






SMART

A B C D E X Y

Admin


{A,B,C,D,E}






SMART

A B C D E X Y

Admin


{A,B,C,D,E}






SMART

A B C D E X Y

Admin


{A,B,C,D,E}


{A,X,C,D,E}





SMART

A B C D E X Y

Admin


{A,X,C,D,E}

Configuration {A,X,C,D,E}

{A,X,C,D,E}





Creating a Replicated State Machine (RSM)

Any node can create a RSM. Select ID and replication degree

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F






Any node can create a RSM. Select ID and replication degree

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

RSM ID = 10, f=4, N=32






The node uses symmetric replication scheme to calculate replica IDs

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

RSM ID = 10, f=4, N=32RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2






The node uses lookups to find responsible nodes . . .

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

RSM ID = 10, f=4, N=32RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2

RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2

Responsible Node IDs = 14, 20, 29, 7






. . . and gets direct references to them

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

RSM ID = 10, f=4, N=32RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2

RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2


RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2


Configuration = D, F,I ,B






The set of direct references forms the configuration

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

RSM ID = 10, f=4, N=32RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2

RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2


RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2








The node sends a Create message to the configuration

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4

RSM ID = 10, f=4, N=32RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2

RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2


RSM ID = 10, f=4, N=32

Replica IDs = 10, 18, 26, 2








Now replicas communicate directly using the configuration

SM10 r1

SM10 r2

SM10 r3

SM10 r4

Configuration_1 D F I B1 2 3 4





SMART with Multiple Admins

A B C D E X Y

Admin


{A,B,C,D,E}







A B C D E X Y

Admin Admin2


{A,B,C,D,E}







A B C D E X Y

Admin Admin2


{A,B,C,D,E}







A B C D E X Y

Admin Admin2


{A,B,C,D,E}


{A,X,C,D,E}






A B C D E X Y

Admin Admin2


{A,B,C,D,E}


{A,X,C,D,E}






A B C D E X Y

Admin Admin2


{A,B,C,D,E}


{A,X,C,D,E} {A,B,C,Y,E}






A B C D E X Y

Admin Admin2


{?,?,?,?,?}

Configuration {?,?,?,?,?}







A B C D E X Y

Admin Admin2


{A,B,C,Y,E}

Configuration {A,B,C,Y,E}







A B C D E X Y

Admin Admin2


{A,B,C,D,E}







A B C D E X Y

Admin Admin2


{A,B,C,D,E}


{ ,X, , , }






A B C D E X Y

Admin Admin2


{A,B,C,D,E}

Configuration {A,B,C,D,E}Proposed Changes { ,X, , , }

{ ,X, , , }






A B C D E X Y

Admin Admin2


{A,B,C,D,E}

Configuration {A,B,C,D,E}Proposed Changes { ,X, , , }

{ ,X, , , } { , , ,Y, }






A B C D E X Y

Admin Admin2


{A,B,C,D,E}

Configuration {A,B,C,D,E}Proposed Changes { ,X, , Y, }

{ ,X, , , } { , , ,Y, }






A B C D E X Y

Admin Admin2


{A,B,C,D,E}

Configuration {A,B,C,D,E}Proposed Changes { ,X, , Y, }

New Configuration{A,X,C,Y,E}






A B C D E X Y

Admin Admin2


{A,X,C,Y,E}

Configuration {A,X,C,Y,E}





Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

SM10 r2

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

SM10 r2

SM10 r2 = G

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

SM10 r2

SM10 r2 = G

G1 2 3 4

F

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

G1 2 3 4

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Handling Churn

0 12

3

4

5

6

7

8

9

10

11

12

13

14151617

1819

20

21

22

23

24

25

26

27

28

29

3031

A

B

C

DE

G

H

I

G1 2 3 4

Configuration_2 D G I B1 2 3 4

SM10 r1

SM10 r2

SM10 r3

SM10 r4






Evaluation

Built a prototype implementation of RME

Simulation-based performance evaluation

Focused on the effect of the churn rate and replication degree onrequest critical path and failure recovery

Used the King latency dataset





Request latency for a single client

0

1000

2000

3000

4000

5000

6000

7000

8000

0 1000 2000 3000 4000 5000 6000 7000 8000

Re

qu

est

La

ten

cy (

ms)

Request Number




Outline

1 Introduction

2 Niche Platform


4 Future Work




Improve Management Logic

Apply control theory to distributed systems

Distributed optimization

Reinforcement Learning




Self-Management in Cloud Applications

Study elastic services in the Cloud

Develop self-management techniques for Cloud applications

Integrate all pieces into an elastic storage system




Thank you for careful listening :-)

Questions?


Documents

Self-Management for Large Scale Distributed Systems · Introduction Niche Platform Robust Management Elements Future Work The Problem Autonomic Computing The Goal Outline 1 Introduction