Apache Helix presentation at Vmware

1

Building distributed systems using Helix

Kishore Gopalakrishna, @kishoreg1980h?p://www.linkedin.com/in/kgopalak

h?p://helix.incubator.apache.org Apache IncubaGon Oct, 2012 @apachehelix

Outline

•  Introduc)on •  Architecture •  How to use Helix •  Tools •  Helix usage

2

3

Examples of distributed data systems

4

Single Node

MulG node

Fault tolerance

Cluster Expansion

•  ParGGoning •  Discovery •  Co-‐locaGon

•  ReplicaGon •  Fault detecGon •  Recovery

•  Thro?le data movement •  Re-‐distribuGon

Lifecycle

5

ApplicaGon

Zookeeper

ApplicaGon

Framework

Consensus System

•  File system •  Lock •  Ephemeral

•  Node •  ParGGon •  Replica •  State •  TransiGon

Zookeeper provides low level primiGves. We need high level primiGves.

6

Outline

•  IntroducGon •  Architecture •  How to use Helix •  Tools •  Helix usage

7

8

Terminologies Node A single machine

Cluster Set of Nodes

Resource A logical en/ty e.g. database, index, task

ParGGon Subset of the resource.

Replica Copy of a parGGon

State Status of a parGGon replica, e.g Master, Slave

TransiGon AcGon that lets replicas change status e.g Slave -‐> Master

Core concept -‐ state machine

•  Set of legal states – S1, S2, S3

•  Set of legal state transiGons – S1àS2 – S2àS1 – S2àS3 – S3àS2

9

Core concept -‐ constraints

•  Minimum and maximum number of replicas that should be in a given state – S3àmax=1, S2àmin=2

•  Maximum concurrent transiGons – Per node – Per resource – Across cluster

10

Core concept: objecGves

•  ParGGon placement – Even distribuGon of replicas in state S1,S2 across cluster

•  Failure/Expansion semanGcs – Create new replicas and assign state – Change state of exisGng replicas – Even distribuGon of replicas

11

Augmented finite state machine

12

State Machine

• States • S1,S2,S3

• TransiGon • S1àS2, S2àS1, S2àS3, S3àS1

Constraints

• States • S1à max=1, S2=min=2

• TransiGons • Concurrent(S1-‐>S2) across cluster < 5

ObjecGves

• ParGGon Placement • Failure semanGcs

Message consumer group -‐ problem

13

ASSIGNMENT SCALING FAULT TOLERANCE

PARTITIONED QUEUE CONSUMER

ParGGon management

• One consumer per queue

•  Even distribuGon

ElasGcity

• Re-‐distribute queues among consumers

• Minimize movement

Fault tolerance

• Re-‐distribute • Minimize movement •  Limit max queue per consumer

Message consumer group: soluGon

Offline Online

14

MAX=1 per par))on Start consumpGon

Stop consumpGon

MAX 10 queues per consumer

ONLINE OFFLINE STATE MODEL

Distributed data store

Node 1 Node 3 Node 2

P.4

P.9 P.10 P.11

P.12

P.1 P.2 P.3 P.7 P.5 P.6

P.8 P.1 P.5 P.6

P.9 P.10

P.4 P.3

P.7 P.8 P.11 P.12

P.2 P.1

ParGGon management

• MulGple replicas • 1 designated master

• Even distribuGon

Fault tolerance

• Fault detecGon • Promote slave to master

• Even distribuGon

• No SPOF

ElasGcity

• Minimize downGme

• Minimize data movement

• Thro?le data movement

MASTER

SLAVE

Distributed data store: soluGon

16

COUNT=2

COUNT=1

minimize(maxnj∈N S(nj) ) t1≤5 S

M O

t1 t2

t3 t4 minimize(maxnj∈N M(nj) )

MASTER SLAVE STATE MODEL

Distributed search service


P.3

P.1 P.2

P.4

ParGGon management

• MulGple replicas • Even distribuGon

• Rack aware placement

Fault tolerance

• Fault detecGon • Auto create replicas

• Controlled creaGon of replicas

ElasGcity

• re-‐distribute parGGons

• Minimize movement

• Thro?le data movement

P.5

P.3 P.4

P.6 P.1

P.5 P.6

P.2

INDEX SHARD

REPLICA

Distributed search service: soluGon

18

MAX=3 (number of replicas)

MAX per node=5

Internals

19

IDEALSTATE

P1 N1:M

N2:S

P2 N2:M

N3:S

P3 N3:M

N1:S

20

ConfiguraGon

• 3 nodes • 3 parGGons • 2 replicas • StateMachine

Constraints

• 1 Master • 1 Slave • Even distribuGon

Replica placement

Replica State

CURRENT STATE

•  P1:OFFLINE •  P3:OFFLINE N1 •  P2:MASTER •  P1:MASTER N2 •  P3:MASTER •  P2:SLAVE N3

21

22

EXTERNAL VIEW

P1 N1:O

N2:M

P2 N2:M

N3:S

P3 N3:M

N1:O

23

Helix Based System Roles


P.4

P.9 P.10 P.11

P.12

P.1 P.2 P.3 P.7 P.5 P.6

P.8 P.1 P.5 P.6

P.9 P.10

P.4 P.3

P.7 P.8 P.11 P.12

P.2 P.1

PARTICIPANT

SPECTATORController

Parition routing logic

CURRENT STATE

IDEAL STATE

RESPONSE COMMAND

Logical deployment

24

Outline


25

Helix based soluGon

1. Define 2. Configure 3. Run

26

Define: State model definiGon

•  States –  All possible states –  Priority

•  TransiGons –  Legal transiGons –  Priority

•  Applicable to each parGGon of a resource

•  e.g. MasterSlave

27

S

M O

Define: state model

28

Builder = new StateModelDefinition.Builder(“MASTERSLAVE”);! // Add states and their rank to indicate priority. ! builder.addState(MASTER, 1);! builder.addState(SLAVE, 2);! builder.addState(OFFLINE);! ! //Set the initial state when the node starts! builder.initialState(OFFLINE);

//Add transitions between the states.! builder.addTransition(OFFLINE, SLAVE);! builder.addTransition(SLAVE, OFFLINE);! builder.addTransition(SLAVE, MASTER);! builder.addTransition(MASTER, SLAVE);! !

Define: constraints State Transi)on

ParGGon Y Y

Resource -‐ Y

Node Y Y

Cluster -‐ Y

29

S

M O

COUNT=2

COUNT=1 State Transi)on

ParGGon M=1,S=2 -‐

Define:constraints

30

// static constraint! builder.upperBound(MASTER, 1);!!! // dynamic constraint! builder.dynamicUpperBound(SLAVE, "R");!!! ! // Unconstrained ! builder.upperBound(OFFLINE, -1;

Define: parGcipant plug-‐in code

31

Step 2: configure

32

helix-‐admin –zkSvr <zkAddress>

CREATE CLUSTER

-‐-‐addCluster <clusterName>

ADD NODE

-‐-‐addNode <clusterName instanceId(host:port)>

CONFIGURE RESOURCE

-‐-‐addResource <clusterName resourceName par;;ons statemodel>

REBALANCE èSET IDEALSTATE

-‐-‐rebalance <clusterName resourceName replicas>

33

zookeeper view IDEALSTATE

Step 3: Run

34

run-‐helix-‐controller -‐zkSvr localhost:2181 –cluster MyCluster START CONTROLLER

START PARTICIPANT

zookeeper view

35

Znode content

CURRENT STATE EXTERNAL VIEW

36

Spectator Plug-‐in code

37

38

Helix ExecuGon modes

39

IDEALSTATE

P1 N1:M

N2:S

P2 N2:M

N3:S

P3 N3:M

N1:S

ConfiguraGon

• 3 nodes • 3 parGGons • 2 replicas • StateMachine

Constraints

• 1 Master • 1 Slave • Even distribuGon

Replica placement

Replica State

ExecuGon modes

•  Who controls what

40

AUTO REBALANCE

AUTO CUSTOM

Replica placement

Helix App App

Replica State

Helix Helix App

Auto rebalance v/s Auto

AUTO REBALANCE AUTO

41

In acGon Auto rebalance

MasterSlave p=3 r=2 N=3 Node1 Node2 Node3

P1:M P2:M P3:M

P2:S P3:S P1:S

Auto MasterSlave p=3 r=2 N=3

42


P1:O P2:M P3:M

P2:O P3:S P1:S

P1:M P2:S


P1:M P2:M P3:M

P2:S P3:S P1:M


P1:M P2:M P3:M

P2:S P3:S P1:S

On failure: Only change states to saGsfy constraint

On failure: Auto create replica and assign state

Custom mode: example

43

44

Custom mode: handling failure �  Custom code invoker

�  Code that lives on all nodes, but acGve in one place �  Invoked when node joins/leaves the cluster �  Computes new idealstate �  Helix controller fires the transiGon without viola)ng constraints

P1

N1:M

N2:S

P2

N2:M

N3:S

P3

N3:M

N1:S

P1

N1:S

N2:M

P2

N2:M

N3:S

P3

N3:M

N1:S

Transi)ons

1 N1 MàS

2 N2 Sà M

1 & 2 in parallel violate single master constraint

Helix sends 2 aqer 1 is finished

45

•  At least 2 separate controllers process to avoid SPOF

•  Only one controller acGve •  Extra process to manage •  Recommended for large

size clusters •  Upgrading controller is

easy

Separate •  Embedded controller within

each parGcipant •  Only one controller acGve •  No extra process to manage •  Suitable for small size cluster. •  Upgrading controller is costly •  ParGcipant health impacts

controller

Embedded

Controller deployment

46

Controller fault tolerance

Controller 1

Controller 2

Controller 3

Zookeeper

LEADER STANDBY STANDBY

Zookeeper ephemeral based leader elecGon for deciding controller leader

47

Controller fault tolerance

Controller 1

Controller 2

Controller 3

Zookeeper

OFFLINE LEADER STANDBY

When leader fails, another controller becomes the new leader

48

Managing the controllers

49

Scaling the controller: Leader Standby Model

Controller

Controller

Controller

Cluster

Cluster

Cluster

Cluster

Cluster

Cluster

O L

S

STANDBY

LEADEROFFLINE

50

Scaling the controller: Failure

Controller

Controller

Controller

Cluster

Cluster

Cluster

Cluster

Cluster

Cluster

O L

S

STANDBY

LEADEROFFLINE

Outline


51

Tools

•  Chaos monkey •  Data driven tesGng and debugging •  Rolling upgrade •  On demand task scheduling and intra-‐cluster messaging

•  Health monitoring and alerts

52

Data driven tesGng

•  Instrument – •  Zookeeper, controller, parGcipant logs

•  Simulate – Chaos monkey •  Analyze – Invariants are

•  Respect state transiGon constraints •  Respect state count constraints •  And so on

•  Debugging made easy •  Reproduce exact sequence of events

53

Structured Log File -‐ sample timestamp partition instanceName sessionId state

1323312236368 TestDB_123 express1-md_16918 ef172fe9-09ca-4d77b05e-15a414478ccc OFFLINE




1323312236561 TestDB_123 express1-md_16918 ef172fe9-09ca-4d77b05e-15a414478ccc SLAVE









No more than R=2 slaves Time State Number Slaves Instance

42632 OFFLINE 0 10.117.58.247_12918

42796 SLAVE 1 10.117.58.247_12918

43124 OFFLINE 1 10.202.187.155_12918

43131 OFFLINE 1 10.220.225.153_12918

43275 SLAVE 2 10.220.225.153_12918

43323 SLAVE 3 10.202.187.155_12918

85795 MASTER 2 10.220.225.153_12918

How long was it out of whack? Number of Slaves Time Percentage

0 1082319 0.5

1 35578388 16.46

2 179417802 82.99

3 118863 0.05

83% of the Gme, there were 2 slaves to a parGGon 93% of the Gme, there was 1 master to a parGGon

Number of Masters Time Percentage

0 15490456 7.164960359 1 200706916 92.83503964

Invariant 2: State TransiGons FROM TO COUNT

MASTER SLAVE 55

OFFLINE DROPPED 0

OFFLINE SLAVE 298

SLAVE MASTER 155

SLAVE OFFLINE 0

Outline


58

Helix usage at LinkedIn

59

Espresso

In flight

•  Apache S4 – ParGGoning, co-‐locaGon – Dynamic cluster expansion

•  Archiva – ParGGoned replicated file store – Rsync based replicaGon

•  Others in evaluaGon – Bigtop

60

Auto scaling soqware deployment tool

61

•  States •  Download, Configure, Start •  AcGve, Standby

•  Constraint for each state •  Download < 100 •  AcGve 1000 •  Standby 100

Download

Configure

Start

Active

Standby

Offline

< 100

1000

100

Summary

•  Helix: A Generic framework for building distributed systems

•  Modifying/enhancing system behavior is easy – AbstracGon and modularity is key

•  Simple programming model: declaraGve state machine

62

Roadmap

•  Features •  Span mulGple data centers •  AutomaGc Load balancing •  Distributed health monitoring •  YARN Generic ApplicaGon master for real Gme Apps

•  Stand alone Helix agent

64

website h?p://helix.incubator.apache.org

user [email protected]

dev [email protected]

twi?er @apachehelix, @kishoreg1980

Technology

Apache Helix presentation at Vmware