RemoteProcedureCall

ArvindKrishnamurthy

CourseLogistics

• Everyoneshouldhaveagitlabaccount

• Letusknowifyoudon’thaveone

• MakesureyouhavesignedupforPiazza

• Lab1duenextThursday

• SubmissionthroughCanvas

• BlogpostforFriday’sreading

• SubmissionthroughCanvas

MuddyForeheads

• 𝑛children,𝑘getmudontheirforeheads

• Childrensitincircle.

• Teacherannounces,"Someonehasmudontheirforehead."

• Someone==1ormore

• Nooncanseetheirownforehead

• 𝑘isnot“commonknowledge”

MuddyForeheads

• 𝑛children,𝑘getmudontheirforeheads

• Childrensitincircle.

• Teacherannounces,"Someonehasmudontheirforehead."

• Teacherrepeatedlyasks,"Raiseyourhandifyouknowyouhavemudonyourforehead."

• Whathappens?

MuddyForeheads

• 𝑛children,𝑘getmudontheirforeheads

• Childrensitincircle.

• Teacherannounces,"Someonehasmudontheirforehead."

• Teacherrepeatedlyasks,"Raiseyourhandifyouknowyouhavemudonyourforehead."

• Whathappens?

MuddyForeheads

• 𝑛children,𝑘getmudontheirforeheads

• Childrensitincircle.

• Teacherannounces,"Someonehasmudontheirforehead."

• Teacherrepeatedlyasks,"Raiseyourhandifyouknowyouhavemudonyourforehead."

• Whathappens?

X

Y

MuddyForeheads

• 𝑛children,𝑘getmudontheirforeheads

• Childrensitincircle.

• Teacherannounces,"Someonehasmudontheirforehead."

• Teacherrepeatedlyasks,"Raiseyourhandifyouknowyouhavemudonyourforehead."

• Whathappens?

X

Y

Z

MuddyForeheads(contd.)

• Claim:

– Thefirstk-1timestheteacherasks,allchildrenwillreply“No”

– Thek-thtimealldirtychildrenwillreply“Yes”

• Reasoningbyconsideringcasesandusinginduction:

– k=1:thechildwithamuddyforeheadwillsayyes– k=2:letXandYhavemuddyforeheads

• Eachseesexactlyoneotherpersonwithmuddyforehead

• Inround1,XnoticedYdidn’tsay“Yes”

–PossibleonlybecauseYmusthaveseenachildwithamuddyforehead==>Xmusthavemud

TheMuddyForehead"Paradox"

If 𝑘>1, the teacher didn't say anything anyone didn't already know!

WhyAreDistributedSystemsHard?

• Asynchrony– Differentnodesrunatdifferentspeeds

– Messagescanbeunpredictably,arbitrarilydelayed

• Failures(partialandambiguous)– Partsofthesystemcancrash

– Can’ttellcrashfromslowness

• Concurrencyandconsistency– Replicatedstate,cachedonmultiplenodes

– Howtokeepmanycopiesofdataconsistent?

WhyAreDistributedSystemsHard?

• Performance– Havetoefficientlycoordinatemanymachines– Performanceisvariableandunpredictable– Taillatency:onlyasfastasslowestmachine

• Testingandverification– Almostimpossibletotestallfailurecases– Proofs(emergingfield)arereallyhard

• Security– Needtoassumeadversarialnodes

MapReduceComputationalModel

Foreachkeykwithvaluev,computeanewsetofkey-valuepairs: map(k,v)→list(k’,v’)Foreachkeyk’andlistofvaluesv’,computeanew(hopefullysmaller)listofvalues: reduce(k’,list(v’))→list(v’’)

Userwritesmapandreducefunctions.Frameworktakescareofparallelism,distribution,andfaulttolerance.

MapReduce(orMLor…)Architecture

• ScheduleracceptsMapReducejobs– findsaMapReducemasterandsetofavailworkers

• Foreachjob,MapReducemaster<array>– farmstaskstoworkers;restartsfailedjobs;syncstaskcompletion

• Worker<array>– executesMapandReducetasks

• Storage<array>– storesinitialdataset,intermediatefiles,endresults

RemoteProcedureCall(RPC)

Arequestfromtheclienttoexecuteafunctionontheserver.– Totheclient,lookslikeaprocedurecall

– Totheserver,lookslikeanimplementationofaprocedurecall

RemoteProcedureCall(RPC)

Arequestfromtheclienttoexecuteafunctionontheserver.• Onclient– Ex:result=DoMap(worker,i)– Parametersmarshalledintoamessage(canbearbitrarytypes)– Messagesenttoserver(canbemultiplepkts)– Waitforreply

• Onserver– messageisparsed– operationDoMap(i)invoked– Resultmarshalledintoamessage(canbemultiplepkts)– Messagesenttoclient

RPClibrary

ReaddataDeserializeargs

Transport

CSE461

RPCimplementation

DoMap(worker, i) Map(worker, i)

RPClibrary

SerializeargsOpenconnectionWritedata

ReaddataDeserializereply

SerializereplyWritedata

Transport

OS

TCP/IPwrite

OS

TCP/IPreadTCP/IPwriteTCP/IPread

x xx

RPCvs.ProcedureCall

• Whatisequivalentof:– Thenameoftheprocedure?

– Thecallingconvention?– Thereturnvalue?– Thereturnaddress?

RPCvs.ProcedureCall

Binding– Clientneedsaconnectiontoserver– Servermustimplementtherequiredfunction– Whatiftheserverisrunningadifferentversionof

thecode?

Performance– procedurecall:maybe10cycles=~3ns– RPCindatacenter:10microseconds=>~1Kslower

– RPCinthewidearea:millionsoftimesslower

RPCvs.ProcedureCall

Failures–Whathappensifmessagesgetdropped?

–Whatifclientcrashes?

–Whatifservercrashes?

–Whatifservercrashesafterperformingopbutbeforereplying?

–Whatifserverappearstocrashbutisslow?

–Whatifnetworkpartitions?

Semantics

• Semantics=meaning

• reply==ok=>???

• reply!=ok=>???

Semantics

• Atleastonce(NFS,DNS)– true:executedatleastonce– false:maybeexecuted,maybemultipletimes

• Atmostonce– true:executedonce– false:maybeexecuted,butnevermorethanonce

• Exactlyonce– true:executedonce– false:neverreturnsfalse

AtLeastOnce

RPClibrarywaitsforresponseforawhile

Ifnonearrives,re-sendtherequest

Dothisafewtimes

Stillnoresponse--returnanerrortotheapplication

Non-replicatedkey/valueserver

ClientsendsPutkv

Servergetsrequest,butnetworkdropsreply

ClientsendsPutkvagain– shouldserverrespond"yes"?– or"no"?

Whatifopis“append”?

DoesTCPFixThis?

• TCP:reliablebi-directionalbytestreambetweentwoendpoints– Retransmissionoflostpackets

– Duplicatedetection

• ButwhatifTCPtimesoutandclientreconnects?– BrowserconnectstoAmazon

– RPCtopurchasebook

– WifitimesoutduringRPC

– Browserreconnects

Whendoesat-least-oncework?

• Ifnosideeffects– read-onlyoperations(oridempotentops)

• Example:MapReduce

• Example:NFS– readFileBlock– writeFileBlock

AtMostOnce

ClientincludesuniqueID(UID)witheachrequest– usesameUIDforre-send

ServerRPCcodedetectsduplicaterequests– returnpreviousreplyinsteadofre-runninghandlerifseen[uid]{r=old[uid]}else{r=handler()old[uid]=rseen[uid]=true}

SomeAt-Most-OnceIssues

HowdoweensureUIDisunique?– Bigrandomnumber?

– CombineuniqueclientID(IPaddress?)withseq#?

– Whatifclientcrashesandrestarts?CanitreusethesameUID?

– Inlabs,nodesneverrestart

– Equivalentto:everynodegetsnewIDonstart

WhenCanServerDiscardOldRPCs?

Option1: Never?Option2:uniqueclientIDsper-clientRPCsequencenumbersclientincludes"seenallreplies<=X"witheveryRPCOption3:onlyallowclientoneoutstandingRPCatatimearrivalofseq+1allowsservertodiscardall<=seqLabsuseOption3

WhatifServerCrashes?

Ifat-most-oncelistofrecentRPCresultsisstoredinmemory,serverwillforgetandacceptduplicaterequestswhenitreboots– DoesserverneedtowritetherecentRPCresultsto

disk?– Ifreplicated,doesreplicaalsoneedtostorerecent

RPCresults?

InLabs,servergetsnewaddressonrestart– Clientmessagesaren’tdeliveredtorestartedserver