Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Moderná orchestrácia microslužieb
stanislav.ondac@[ferratum|gmail].com
linkedin.com/in/stanislav-ondac
@StanislavOndac
FE-Classification: Confidential\Anyone
1. Introduction
2. Use-Case Description
3. Choreography scenarios
4. Orchestration scenarios
5. Orchestration Live demo
6. Lessons learned
14 years in industry
Roles:
– Current: • Software Architect, • Team Lead, • Deputy Chief Architect
– Past:• Developer : C++, J2EE• DB Admin (Informix)• IT analyst/consultant• Enterprise Architect
FE-Classification: Confidential\Anyone
Consumer loans
Businessloans
Mobile Bank
years of profitable growth
Active & formercustomers
Countries
9M 2018revenue
Year-on-year revenue growth
Founded Helsinki 2005
EU Banking Licence
Frankfurt Prime Standard
Year-on-year EBIT growth
FE-Classification: Confidential\Anyone
• IT Hub located in Bratislava
• ~133 internal employees
• https://ferratumit.sk
• Microservices implemented in Java/Scala/TypeScript
• All deployed in OpenShift container platform
• Average latency in milliseconds to process a record within our Kafka Streams/ODS infrastructure
• Months of development to pilot production deployment
• Started August 2017
• Pilot production August 2018
• Java and Spring Boot as main backend stack
• Scala and Kafka Streams for data streams
• Avro message serialization
• Camunda BPMN for process orchestration
• Apollo GraphQL / TypeScript as API GW
FE-Classification: Confidential\Anyone
Microservices• Microservice per Aggregate Root – not that small
– Customer
– Application
– Loan
• Microservices per different responsibilities
– Validations
– Basic CRUD logic
– Façade
– Wrapper
FE-Classification: Confidential\Anyone
Case study – Online onboarding• Customer want’s loan for a car
• Simplified page flow would look like this:
Lead page
• Check duplicity• Basic Policy checks• Create Lead• Create Consents (marketing/GDPR…)
Verification Page
• User internal or external service for verification• Create Customer• Create User Account
Financial Page
• Use 3rd party service • Policy Checks• Scoring
Offer Page Confirmation Page
• Confirm Application• Create Loan• Create Contracts
• Store Offer• Read product configuration for the offer
FE-Classification: Confidential\Anyone
Lead Page
• Check duplicity
• Basic Policy checks
• Create Lead
• Create Consents (marketing/GDPR…)
• Create Application
FE-Classification: Confidential\Anyone
Verification Page
• User internal or external service for verification
• Create Customer
• Create User Account
• Update Application
FE-Classification: Confidential\Anyone
Financial Page
• Use 3rd party service
• Policy Checks
• Scoring
• Update Application
FE-Classification: Confidential\Anyone
Offer Page
• Read product configuration for the offer
• Store Offer
• Update Application
FE-Classification: Confidential\Anyone
Confirmation Page
• Confirm Application
• Create Loan
• Create Contracts
• Update Application
FE-Classification: Confidential\Anyone
What is long running?
Seconds Minutes Days, Weeks
“instant” flows
Automated flows
Manual flows
State handling, persistence, timeouts ,
FE-Classification: Confidential\Anyone
Online Onboarding• List of MicroServices
Customer Management
Consent Management
Application Management
Loan Handling
ScoringRisk
Management
External Registry
…
FE-Classification: Confidential\Anyone
Orchestration vs ChoreographyOrchestration
•Central brain to guide and drive the process/flow
• Invoking and combining the services
• Like conductor in the orchestra
Choreography
• Inform each microservice of it’s job
• Let it work out the details
• Like dancers in the ballet
FE-Classification: Confidential\Anyone
Sequence flows for Lead Page
• Let’s start with choreography
FE-Classification: Confidential\Anyone
FE-Classification: Confidential\Anyone
FE-Classification: Confidential\Anyone
FE-Classification: Confidential\Anyone
22
FE-Classification: Confidential\Anyone
Transformation from events to
commands
Commands instead of events
Risk ManagementWeb
Risk Management
Customer Management
loanRequested
policyChecksExecuted
leadCreated
executePolicyChecks
createLead
updateApplication
Customer Management
Application Management
Risk Management
Customer Management
Application Management
FE-Classification: Confidential\Anyone
Who should do transformation?
• Existing domain microservice (implicit orchestrator)
– Most extensive used one
– With most logic for the given domain
– With most data for the given workflow
– In our case?
• Application Management
• New Orchestration Microservice (explicit orchestrator)
– No business logic, only flow logic
– Additional component – but improved separation of concerns
Transformation from events to
commands
FE-Classification: Confidential\Anyone
Asynchronous
Orchestration
FE-Classification: Confidential\Anyone
Synchronous
Orchestration
FE-Classification: Confidential\Anyone
Orchestration vs Choreography
• Choreography: distributed decision making
– Every service knows what to do next…Or knows what to wait for
– It couples services together
– Bigger risk of cyclic dependencies (they depend on each other)
– Process partially works even when some microservices are down – no single point of failure
– Validations and state management are challenging
– Need for investing in monitoring and tracing features
– Compensations (order of rollback operations) are complicated
• Orchestration: centralized decision making
– Coordination component = state machine, workflow engine
– Process doesn’t work when orchestration component is down
– Risk of putting too much logic into orchestrator
– Change the flow (in one place) is much easier
– Other microservices don’t have to include state management
FE-Classification: Confidential\Anyone
DeCentralized Orchestration
Onboarding Domain
Service 1 Service 2
Service N
Orchestration Service
Payments Domain
Service 1
Service 2
Service N
Collections Domain
Service 1 Service 2
Service N
Orchestration Service
Aggregate Root
Document management Domain
Service 1 Service 2
Service N
Orchestration Service
FE-Classification: Confidential\Anyone
What others think about that? (1)
• Sam Newman – Building microservices (2015)– Choreography “is significantly more decoupled. If some other service needed to reach to the creation of a
customer, it just needs to subscribe to the events…”
– Orchestration “can become too much of a central governing authority. It can become the hub in the middle of a web, and a central point where logic starts to live. I have seen this approach result in a small number of smart “god” services telling anemic CRUD –based services what to do”.
– “The downside is, that the explicit view of the business process we see is now only implicitly reflected in our system”
• What does it mean for us?– Choreography is supposed to be without significant coupling
• From operational point of view – only if use asynchronous communication
• From change management point of view (logical dependencies) – you are still creating coupling
– Don’t put too much business logic into orchestration component
– Don’t make others services just dummy CRUD services
– Whole process is not explicitly visible
FE-Classification: Confidential\Anyone
What others think about that? (2)
• Netflix – Conductor– With peer to peer task choreography, we found it was harder to scale with growing business needs and complexities.
Pub/sub model worked for simplest of the flows, but quickly highlighted some of the issues associated with the approach:
• Process flows are “embedded” within the code of multiple application.
• Often, there is tight coupling and assumptions around input/output, SLAs etc, making it harder to adapt to changing needs.
• Almost no way to systematically answer “how much are we done with process X”?
• What does it mean for us?
– If you have more complex flows in your business logic – use orchestration instead of choreography
– If you have complex flow logic important for your business – you have to know exactly what is going on there
FE-Classification: Confidential\Anyone
Onboarding BPMN (only part)
FE-Classification: Confidential\Anyone
Whole process – only onboarding domain
FE-Classification: Confidential\Anyone
How to do DeCentralizedOrchestration
• Orchestration per domain
– Analyze your domain – according to the DDD
– Analyze data flows within your domain
• Orchestrator is not central brain with dummy CRUD services
– Orchestrator masters the flow logic
– Other microservices master their corresponding business logic (i.e. scoring) – thus no violation of “smart endpoints, dumb pipes” principle from M.F.
• No Central Orchestration tool for the whole architecture/company
• Lightweight orchestration tool/library
– Embeddable into your application (i.e. spring boot app)
FE-Classification: Confidential\Anyone
Why workflow engine for orchestration
• The engine does all state handling and provides the required features for long running flows (monitoring, timers, versioning, …).
• You gain visibility of the flow which might be interesting during:
– requirements engineering (how to implement this?),
– development (what do I have to implement right here?),
– Testing (what it should do?) and
– operations (is everything running smooth? where exactly do we have problems?).
• You also get additional tools for advanced use cases. As an operator you could for example adjust the state for a certain flow in case of errors.
FE-Classification: Confidential\Anyone
Domain Service 1
How today’s modern workflow engines works• First generation– e.g.Camunda
–“some” scalability• Optimistic locking
• Clusters
• Client-server approach
•New generation – zeebe, Netflix conductor– High scalability – scales horizontally
– Pub-sub approach
– Append only logs…
Worklowengine
Domain Service 1 Domain
Service 2
Perform command
Perform command
Shared RDBMS
Store state
messaging platform
Worklow engine
Task queueOrchestrator
Task queue
Domain Service 2
Create tasks
Pool for tasks Update task status
FE-Classification: Confidential\Anyone
Orchestration is not only about workflow engines
• If you need state:
– Workflow Engine (Camunda, Netflix Conductor, JBPM…)
• If you don’t need state (orchestrating multiple services in one synchronous call):
– Camel
– Spring Integration
– Lightweight ESB
FE-Classification: Confidential\Anyone
Orchestration/Choreography vs Organization/Teams• If most of your projects are about implementing complex flows, orchestration can help also from
organizational perspective
– With choreography – you have to coordinate all the teams, which with lot’s of microservices is not an easy task…
– With orchestration – most changes are done in components responsible for orchestration (others are called by commands) , and only some changes in microservices which are being called
• From Project management point of view it’s easier to manage
• On the other hand – team responsible for orchestration is under bigger pressure (calling all the other components in the right way)
FE-Classification: Confidential\Anyone
Workflow engine as a monitor for your choreography
FE-Classification: Confidential\Anyone
Demo
FE-Classification: Confidential\Anyone
Lessons Learned
FE-Classification: Confidential\Anyone
What we did well
•By not putting flow logic into every microservice, we are able to build new flows quickly – changes mostly in one component – calling other microservices as needed
•Looks like a lot of compensation logic is not needed for now – we are not so big
•Our workflow/process engine contains only flow logic – all business logic is in other microservices
–But there were a lot of attempts ☺
•BPMN as the notation was a success
FE-Classification: Confidential\Anyone
Challenges
•Process engine as the stateless service was not the best choice because of the frequent writes into the main entity (Application)
–Think about your domain logic constantly
•Default workflow monitoring UI is not sufficient for business people
–More complex UI needed (extend, or build a new one)
FE-Classification: Confidential\Anyone
stanislav.ondac@[ferratum|gmail].com
linkedin.com/in/stanislav-ondac
@StanislavOndac
FE-Classification: Confidential\Anyone
Backup
FE-Classification: Confidential\Anyone
Sometimes orchestration is not needed at all• There are use-cases where just listening to the events is completely fine. Our example: Campaigns
Management
– Reading events/messages from kafka coming from different domains (client created, loan created, offer created…)
– Processing them
– Preparing reports to reward our partners
• No need for central orchestration component
FE-Classification: Confidential\Anyone
Challenges of distributed computing• Client has to implement:
– Retries
– Waitings
– Timeouts
– Compensations
• Service has to implement:
– Compensation
– Idempotency