S-Paxos: Eliminating the Leader Bottleneck

Martin Biely, Zarko Milosevic, Nuno Santos, André Schiper

Ecole Polytechnique Fédérale de Lausanne (EPFL)Switzerland

October 9, 2012

2

Context: State Machine Replication• Consistency among

replicas ensured by• Deterministic service• Same initial state• Same sequence of

requests

• System model• Partially synchronous• Crash stop (max

crashes)

Nuno Santos Context and Motivation

Replicated Service

Service Service Service

Ordering protocol (Paxos)

Clients

3

The Paxos Protocol

Nuno Santos Context and Motivation

• Observation: leader receives and sends more messages than the followers• Potential system bottleneck…

• Paxos is a leader-based protocol• A distinguished process (leader) coordinates the others (followers)

4

Paxos Performance

Nuno Santos Context and Motivation

Experimental settings• JPaxos – implementation of Paxos in Java (protocol shown previously)• n=3, request size=20 bytes, CPU 2x2cores @2.2Ghz

The bottleneck in Paxos is typically the leader

5

Paxos is Leader-centric• Leader-centric protocol

• The leader does considerably more work than the followers• Therefore, the leader is prone to being the system bottleneck

• Paxos and most leader-based protocols are also leader-centric

Nuno Santos Context and Motivation

6

Leader-based vs Leader-centric• Note that leader-based ≠ leader-centric

• Leader-based – algorithmic concept, leader is a distinguished process

• Leader-centric – resource usage, leader is a bottleneck

Nuno Santos Context and Motivation

Question: do leader-based protocols like Paxos must also be leader-centric?

7

S-PAXOS OVERVIEWLeader-based but not leader-centric

Nuno Santos

8

Why Paxos is Leader-centric• Leader does the following

• Receives requests from clients• Coordinates protocol to order requests• Replies to clients

• Followers do much less• Receive client requests from leader• Acknowledge order proposed by leader

• Underlying problem: unbalanced resource utilization• Leader runs out of resources (CPU, network bandwidth)• While followers are lightly loaded

Nuno Santos S-Paxos Overview

9

S-Paxos: A Balanced Paxos Variant• S-Paxos balances workload across replicas

• Leader and followers have similar resource usage• The full resources of all replicas become available to the ordering

protocol

• S-Paxos is leader-based but not leader-centric

• Combines several well-known ideas in a novel way• All replicas handle client communication• All replicas disseminate requests• Ordering done on IDs

Nuno Santos S-Paxos Overview

10

S-Paxos key ideasDistribute client communication

• Commonly used in practice• For instance, ZooKeeper

• But by itself, still leader-centric• Leader runs the ordering protocol on

requests (Phase 2a messages of Paxos) • Followers have to forward requests to

leader• And hence, sends requests to other

followers

Nuno Santos S-Paxos Overview

All replicas handle client communication

11

S-Paxos key ideasDistribute request dissemination

• Note that Phase 2a messages have a dual purpose• Dissemination of requests• Establishing order

• All replicas disseminate requests• Ordering performed on IDs

Nuno Santos S-Paxos Overview

S-Paxos separates dissemination from ordering

12

S-Paxos Architecture and Data Flow

Nuno Santos S-Paxos Overview

13

S-Paxos balances work among replicas• Client communication and request

dissemination usually the bulk of the load• In S-Paxos this task is performed by all

replicas

• Leader still has to coordinate ordering protocol• But IDs are small messages• So leader has minimal additional overhead

Nuno Santos S-Paxos Overview

• Two levels of batching to further reduce load on leader• Dissemination layer: batch client requests and use ordering layer to

order ids of batches• Ordering layer: usual Paxos batching, in this case batches of batch ids.

14

Benefits in the presence of faults• Faster view change

• Since IDs are small, Phase 1 of Paxos completes quickly

• Failures affecting the leader have less impact on throughput• Ordering protocol is interrupted, but dissemination protocol

continues among working replicas• When a correct leader emerges, it can quickly order the IDs of the

requests that were disseminated while there was no leader

Nuno Santos S-Paxos Overview

15

DISSEMINATION LAYER PROTOCOL

Nuno Santos

16

Dissemination Layer Overview• Dissemination layer tasks

1) Receive requests from clients2) Disseminate requests and IDs to all replicas3) Initiate ordering of IDs4) Execute requests in the order established for

IDs

• Challenges• Once an ID is decided, the corresponding request

must remain available in the system• Coordinate view change between ordering and

dissemination layers to ensure that ids are ordered once-and-only once

Nuno Santos Dissemination Layer Protocol

2 2

1

3 4

17

Overview of the ProtocolDisseminating requests• Optimistic implementation of reliable broadcast• When a replica receives a request from a client, it broadcasts <request,ID> • Replicas acknowledge reception of forwarded requests by broadcasting <Ack,ID>

Proposing IDs• Leader proposes an ID once the corresponding request is stable

• That is, when it receives acknowledgements for the ID

Executing requests• Replica must have: request and decision for corresponding ID• If ID decided before request received, poll other replicas for request after a

small delay• Request stable, so at least one correct replica has the request

Nuno Santos Dissemination Layer Protocol

18

PERFORMANCE EVALUATION

Nuno Santos

19

Performance Evaluation• S-Paxos implemented on top of JPaxos, a Java implementation of Paxos

• Experiments compare • JPaxos (leader-centric)• S-Paxos (non leader-centric)

• Testbed: Grid 5000 (helios cluster)• CPU: 2x2-cores @ 2.2Ghz• Network: 1Gbit Ethernet

• Experimental parameters• Request size: 20 bytes• Batch size

• S-Paxos: dissemination layer 1450 bytes, ordering layer: 50 bytes• JPaxos: 1450 bytes

• Null service

Nuno Santos Experimental Evaluation

20

Load Distribution: Average CPU utilization

Nuno Santos Experimental Evaluation

JPaxos S-Paxos

21

Performance with Increasing Number of Clients (n=3)

Nuno Santos Experimental Evaluation

Throughput Response time

22

Scalability

Nuno Santos Experimental Evaluation

Throughput

23

Throughput with crashes

Nuno Santos Experimental Evaluation

Crash of the leader

• Request size: 1KB, Batch size: 8KB,

24

False suspicions

Nuno Santos Experimental Evaluation

• Leader is (wrongly) suspected every 10 seconds

25

Conclusion

Nuno Santos

A leader-based protocol does not need to be leader-centric

S-Paxos: balances the workload across replicas

Benefits• Better performance for the same number of replicas• Better scalability with the number of replicas• Better performance in the presence of faults

26

ADDITIONAL SLIDES

Nuno Santos

27

Discussion

Nuno Santos Dissemination Layer Protocol

• Broadcast of <request,ID>: best effort, no retransmission• Avoids cost of reliable broadcast on requests• Recovering from partial delivery (message loss/crashes):

• Request does not become stable - client timeouts and retransmits• Request becomes stable – after ID is decided, replicas poll other

replicas for request

• Broadcast of <Ack,ID>: retransmission• Ensures that once a request is stable, it will be proposed• Almost free in practice: acks are small and can be piggybacked on

other messages.