Embed Size (px)
Citation preview
Distributed SagasA Protocol for Coordinating Microservices
Caitie McCaffreyDistributed Systems Engineer
caitiem.com
@caitie
Service ServiceServiceMonoliths
Microservices & NoSQL
2015
2015
“Application-level Mechanisms for maintaining
database integrity”
Reserve a Hotel Front End
Reserve a Hotel Front End
Feral Concurrency Control Mechanisms
Reserve a Hotel Front End
Feral Concurrency Control Mechanisms
Reserve a Car Front End
Reserve a Flight Front End
Trips Front End
Trips Front End
Trips
Trips Front End
Trips
Feral Concurrency Control Mechanisms
Trips Front End
Trips
Feral Concurrency Control Mechanisms
Trips Front End
Trips
Feral Concurrency Control Mechanisms
Front End
Trips
Architecture
Death Star Architectures
Can We Do Betterthan Feral Concurrency Control?
2012
2012
“Spanner is Google’s scalable, multi-version, globally distributed, and
synchronously-replicated database”
2015
2015
“The biggest barrier…is that consistency mechanisms must
integrate across many stateful services”
Two Phase Commit
2PC: Prepare Front End
Trips
Propose Book Hotel
Propose Book Car
Propose Book Flight
2PC: Prepare Front End
Trips
Vote Book Hotel
Vote Book Car
Vote Book Flight
2PC: Commit Front End
Trips
Commit/Abort Book Hotel
Commit/Abort Book Car
Commit/Abort Book Flight
2PC: Commit Front End
Trips
Done
Done
Done
2PC: Commit Front End
Trips
Done
Done
Done
Doesn’t Scale•O (N^2) Messages in the worst case
•Coordinator is a Single Point of Failure
•Reduced Throughput
Distributed SagasA Protocol for Coordinating Microservices
1987
1987
“Sagas are Long Lived Transactions [in a single relational Database]”
1987
“A Saga is a Long Lived Transaction that can be written as a sequence of transactions
that can be interleaved.
All transactions in the sequence complete successfully or compensating transactions
are ran to amend a partial execution.”
Distributed SagasA Protocol for Coordinating Microservices
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
and Compensating Requests
Cancel Hotel Cancel Car Cancel Flight Refund Money
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
and Compensating Requests
Cancel Hotel Cancel Car Cancel Flight Refund Money
that represent a single business level action
Distributed SagaRequests
RequestsCan Abort
Book Car
RequestsCan Abort
Book Car
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book CarTimeout
RequestsMust Be Idempotent
Book Car
RequestsMust Be Idempotent
Book Car
Distributed SagaCompensating Requests
Compensating RequestsSemantically undoes the effect of a request
Cancel CarCompensating
RequestsCannot Abort
Cancel CarCompensating
RequestsCan Not Abort
Cancel CarCompensating
RequestsMust Be Idempotent
Compensating Requests
Must Be Commutative with Requests
Book CarCancel Car
Cancel CarBook Car
is the same as
Compensating Requests
Book Car
Must Be Commutative with Requests
Compensating Requests
Book CarTimeout
Must Be Commutative with Requests
Compensating Requests
Book Car
Book Car
Must Be Commutative with Requests
Compensating Requests
Book Car
Book Car
Cancel CarMust Be Commutative with Requests
Compensating Requests
Book Car
Book Car
Cancel CarMust Be Commutative with Requests
Compensating RequestsRequests
Idempotent Idempotent
Commutative
Can Not Abort
Can Abort
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
No Atomicity
No Isolation
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
No Atomicity
No IsolationBook Hotel
Book Car
Visible before Saga Completes
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Distributed Sagadefining a
Directed Acyclic GraphDistributed Saga Car FlightHotel
Payment
Start Saga
End Saga
Name: Hotel
Request: Book Hotel
Compensating Request: Cancel Hotel
Status: Not Completed
VertexDistributed Saga
Start & End Vertices Distributed Saga
Start Saga
End Saga
Car FlightHotel
Payment
Start Saga
End Saga
Distributed Saga Logfault-tolerant & highly available
Saga Execution Coordinator
SECCar FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Distributed Saga Executing a
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Book Trip Request
SEC
Car FlightHotel
Payment
Start Saga
End Saga
SECSaga
Log
Start Saga
Book Trip Request
Car FlightHotel
Payment
Start Saga
End Saga
SECSaga
Log
Done
Start Saga
Book Trip Request
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECStart Hotel
Start Saga
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECDone
Start Saga
Start Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECBook Hotel
Request
Start Saga
Start Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SEC
Book Hotel Response
Start Saga
Start Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECEnd Hotel
Start Saga
Start Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECDone
Start Saga
Start Hotel
End Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECStart Car
Start Saga
Start Hotel
End Hotel
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECDone
Book Car Request
Start Saga
Start Hotel
End Hotel
Start Car
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Book Car Response
End Car
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start CarDone
End Car
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start CarStart Flight
End Car
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
DoneEnd Car
Start Flight Book Flight Request
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Book Flight Response
End Flight
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Done
End Flight
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Start Payment
End Flight
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Done
End Flight
Start Payment
Payment Request
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Payment
End Flight
Start Payment
Payment Response
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
Done
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
End Saga
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
Done
End Saga
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
End Saga
Distributed Saga Failure of a
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
SECBook Car Request
Start Saga
Start Hotel
End Hotel
Start Car
Start Flight
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Book Car Response
Abort CarStart Car
Start Flight
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
DoneStart Car
Start Flight
Abort Car
Car FlightHotel
Payment
Start Saga
End Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
DoneStart Car
Start Flight
Abort Car
Rollback Recovery
Car FlightHotel
Payment
End Comp Saga
Start Comp Saga
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Payment Log Entries?Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
{ }Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Car Log Entries?
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, Abort }
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Hotel Log Entries?
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, End }
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, End }
Cancel Hotel Request
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Cancel Hotel Response
Comp Hotel
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Done
Comp Hotel
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Flight Log Entries?
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
{ Start }
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
{ Start }
Book Flight Request
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
End Flight
Book Flight Response
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
Cancel Flight Request
End Flight
Start Comp Saga
End Comp Saga
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Comp Flight
Cancel Flight Response
Start Comp Saga
End Comp Saga
End Flight
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
Comp Flight
Start Comp Saga
End Comp Saga
End Flight
Comp Flight
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
End Saga
Start Comp Saga
End Comp Saga
Comp Flight
End Flight
Comp Flight
Car FlightHotel
Payment
Saga
Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
End Saga
Start Comp Saga
End Comp Saga
Comp Flight
End Flight
Comp Flight
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Saga Execution CoordinatorRecovering from
Failure
Front End
SEC
Start Saga
End Saga
Saga
Log
Front End
SEC
Start Saga
End Saga
Saga
Log
Front End
SEC
Start Saga
End Saga
Saga
Log
Front End
SEC
Saga
Log
Start Saga
End Saga
with Distributed SagasIsolation of Complex Code
Fron
Tr
Complex Code Lives Everywhere
Front
SECSa
ga L
og
Complex Code Lives Here
Complex Code Lives Here
Fron
Tr
with Distributed SagasModular Services
Front
SECSa
ga L
og
Only Flight Booking
Flight Booking & Feral Concurrency
Control Mechanisms
Fron
Tr
with Distributed SagasService Composition
Front
SECSa
ga L
og
Just Add A New Saga! New Service &
Feral Concurrency Control Mechanisms
Distributed Sagas Makes Building & Modifying Microservices Easier
Thank you@caitie
