Raft assignments

COS 418: Distributed SystemsPrecept 9

Themis Melissaris and Daniel Suo

Agenda

• Generalobservations• Assignment3:anexampleindesigninganimplementation

• Assignment4:expandingontheexample• Assignment5:avoidingcommonpitfalls

2

Onincrementalassignments

• Wehearyou:nothavingsolutionstoearlierassignmentswhenlaterassignmentsdependonthemishard

• Beatingthedeadhorse:– Thisreflectstherealitymoreoftenthannotinsoftwareengineering

– Thisisalsoaforcingmechanismtoreallyunderstandadistributedsystemandhowtomakegooddesignchoices

3

#1reasonforstruggle:repeatinglogic• Someexamples

– Usingmultiplestatevariablesforonestate– HandlingheartbeatandAppendEntries aredifferent(morerelevantforA4)

– StartnewelectionfromCandidateandFolloweraredifferent

– Resettingtimers

4

Wedon’twantourcodetobe

• Rigid:difficulttochange;needtotouchmanyplacestomakesimplechanges

• Fragile:changesbreaksysteminunexpectedways

• Immobile:hardtoreuselogic/code

Theseadjectivescausedpeoplealotofpain!We’llrevisitthroughoutthisprecept

5

Assignment3

6

WhyarewegoingoverA3again?

• Shouldbereview• Wespendalotoftimeinclassdescribingsystems

• Youspendalotoftimeinassignmentsimplementingsystems

• Aquick(andsimpler)exampleofhowwegofromasystemdescriptiontosystemimplementation

• Therearemanypossibleimplementations!7

Reasoningaboutstate

• Assignment3asksustoimplementastatemachine(i.e.,elections)

• Eachraftserverhasa‘state’(Follower,Candidate,Leader)

• Raftservercanchangestateaccordingtocertainrules

8

Reasoningaboutstate

9

Reasoningaboutstate

• Whatadditionalstatemusteachserverhold?– currentTerm– votedFor

• WewillalsoassumeeachserverhasaTimerobject,butthereareotherimplementationstohandletimeouts(e.g.,timeoutloops)

10

Reasoningabouttransitions(Follower)• Howcanwebecomeafollower?

– Whenwefirststart– WhenwereceiveanRPCfromaserverwithahighercurrentTerm

• Followerscanonlybecomecandidates(directly,anyway)

11

Reasoningabouttransitions(Candidate)• Howcanwebecomeacandidate?

– Whenweareafollowerandhaven’treceivedaheartbeatfromaleaderwithintheelectiontimeout

– Whenweareacandidateandhaven’tbeenvotedleaderorheardfromaleaderwithintheelectiontimeout

• CandidatescanbecomeanyofFollower,Candidate,orLeader

12

Reasoningabouttransitions(Leader)• Howcanwebecomealeader?

– Whenwearecandidateandifwereceivevotesfrommajorityofserverswithinelectiontimeout

• LeadercanbecomeFollower– Ifweseeaserverwithahigherterm– Typicallyhappensafterwedieorthereisapartition

13

Whendowechangestate?• Thereisatimeout

– Follower->Candidate– Candidate->Candidate

• WereceiveanRPC– Leader->Follower– Candidate->Follower

• WehandlearesponsetoanRPC– Leader->Follower– Candidate->Leader– Candidate->Follower

14

Ourcodeshouldreflect!• Timingout

– resetTimer• Createatimerifthereisn’tone(i.e.,whenweMake)andstartgoroutine tocall

handleTimer wheneverthereistimeout• Settimeouttoheartbeatintervalifweareleader,torandomizedelectionintervalif

wearenot(note,thesamewhetherweareFollowerorCandidate)

– handleTimer• Ifleader,callsendAppendEntries• Otherwise,becomecandidate(notethislogicisthesameifweareFolloweror

Candidate)

• ReceivinganRPC– RequestVote:specifiedinpaper(don’tneedtoimplementthewhole

thing)– AppendEntries:specifiedinpaper(don’tneedtoimplementthe

wholething)15

Ourcodeshouldreflect!• HandlingRPCresponses

– handleRequestVoteResponse:specifiedinpaper(don’tneedtoimplementthewholething)

– handleAppendEntriesResponse:specifiedinpaper(don’tneedtoimplementthewholething)

• SendingRPCs– sendRequestVote:sendRequestVote inseparategoroutine toeach

server;callhandleRequestVoteResponse onresponse– sendAppendEntries:sendAppendEntries inseparategoroutine to

eachserver;callhandleAppendEntriesResponse onresponse

16

Somedetails(assumingarchitectureinpreviousslides)• Resettingtimer

– Whenwestart– Wheneverwehandletimeout– Wheneverwechangestate

• Locking/unlocking(whendowemodifystate?)– Whenwehandletimeout– WhenwereceiveanRPC– WhenwehandleasingleRPCresponse

• ResettingvotedFor tonull(or-1)– WhenwebecomefollowerexceptinAppendEntries

17

Assignment4

18

High-leveloverview

• Assumearchitecturefromearlierslides• PartI

– Modifyallfunctionsinvolvingvolatilestateorthelog(basicallyeverythingexceptTimerstuff)

• PartII– Correctlyhandlepersistentstate

19

PartI:sendRequestVote

• MayhavealreadydoneforA3• SetRPCarguments

– lastLogIndex:lengthofthecandidate’slog(indexofcandidate’slastlogentry)

– lastLogTerm:ifwehavemorethanoneentry,termofthelastlogentry

20

PartI:RequestVote

• Needtoaddcheckthatthecandidate’slogisatleastasup-to-dateasreceiver’slog

• Seesection5.4.1inoriginalpaperfordetails

21

PartI:handleRequestVoteResponse• Ifwebecomeleader,initializenextIndex andmatchIndex– nextIndex:initializetothelengthoftheleader’slog(leaderlastlogindex+1)

– matchIndex:initializeto0(why?)

22

PartI:sendAppendEntries

23

• Whichlogindexshouldwesendtofollowers?• IfourlastlogindexisgreaterthanorequaltothenextIndex forafollower,sendAppendEntries RPCwithlogentriesstartingatnextIndex

PartI:AppendEntries

24

PartI:handleAppendEntriesResponse

25

PartII:persistingstate• OnlyneedtoreadfrompersistentstorageinMake• PersistwheneverwechangecurrentTerm,votedFor,orlog;

easy,right?• Thisbecomeshardifsimilarlogicissprinkledthroughoutyour

code.BesidesinMake,– logchangesinAppendEntries– votedFor changesduringelectionsandinAppendEntries

receive/handleresponse– currentTerm changeswheneverwebecomeFollower

• Notrequired,butforcompleteness– Shouldpersistbeforechanginginmemory;mostpeopledidnotdo

this(notethatyoudoneedtopersistbeforerespondingtoRPCs!)

26

Somedetails• Locking/unlocking(whendowemodifystate?)

– Whenwehandletimeout– WhenwereceiveanRPC– WhenwehandleasingleRPCresponse– Start,Kill

• log,matchIndex,nextIndex– Youshouldreasonaboutthestateofthesearraysjustaswedidfor

Assignment3!– ManypeoplejuststartedimplementingbytranslatingFigure2into

code;withoutunderstanding,debuggingwillbemuchharder!

27

Assignment5

28

High-leveloverview

• ShouldonlydependonpublicRaftAPI• PartI:implementPut(key,value),Append(key,value),Get(key)– Musthavesequentialconsistency!

• Part2:handlingfailures– Dealwithduplicaterequests

29

common.go• Shouldberelativelyquick!• Whatadditionalfield(s)doweneedtoputinPutAppendArgs?

• WhataboutGetArgs?

30

client.go

• WejustneedtoproperlyconstructtheRPCstotheserver– Get– PutAppend

• TheseshouldfolloweasilyoncewehavetheArg structuresfromcommon.go

31

server.go

• ThehardpartJ

32

Debugging

33

Generaltips

• Godebuggingisn’tgreat• Ifyouuseprintstatements,makesureyouuseunbufferedoutput(i.e.,usestderr)

• Usego’splayground:https://play.golang.org/• Createsubsetsoftheevaluationtests• Testincrementally:

– Thinkaboutinvariantsandcreateappropriatetests

34

Goslowtogofast

35