PowerPoint-Prsentation
GOTO Nights Berlin | 03.05.2016 | Christian DegerHighway to
heavenBuilding microservices in the cloud
03.05.16
Christian [email protected]@cdeger
AutoScout24 in Munich, 5 YearsSoftware Engineer, Team Lead,
Architect
2
2,4 Million Vehicles
Who does not know AS24?Listings business, completley digital.Our
focus is the search for the vehicle and everything around that3
17 countries 350 employees 140 in IT
RUSSPLNLDBFAHRROBGUAITRECZ
Marktet leader in some countriesStrong second place in
others.4
2000 Servers2 Data Centers
MTBF optimized
Proven delivery engineHighly optimized, but of last decadeIT
platform supported growth for >6 yearsAvailability = MTBF /
(MTBF + MTTR)Proven agile and lean principles5
Even in AutoScout24 there was resistance against change.We are
not like NetflixThat is to expensiveWe have built a private cloud,
which I would just call virtualization.6
NewCEO
Scout24 was sold end of 2013New CEO Greg Ellis beginning of
2014Are you ready for the future?7
Talent?Do you attract
We got good, agile .NET developers.But from banks and
insurrances.We were not getting the talents for internet
business.8
We started think about our ecosystem and the flywheel that
drives it. .NET small, slowLinux/JVM larger, faster
goodHmm, we are
We are not bad, but we are not great.The new questions triggered
something...10
Rebooteverything
Escape the gravity of status quo.Perhaps its a pattern: Just
rebuild it.
11
Project
Tatsu
Fly at the speed of fear - DisruptiveJapanese dragon: Flying
beastRollercoaster: Sixflags Magic Mountain
Started Nov. 2014 with one team, now at 4 teams.
.NET / Windows to JVM / LinuxMonolith to MicroservicesData
center to AWSDevs + Ops to Collaboration cultureInvolve product
people
Migration strategy
samedirection
First team was tightly aligned.Wanted to the right thing, they
were prepared.When ramping up to more teams15
STRATEGICGOALSGoals of the business
sideARCHITECTURALPRINCIPLESHigh-Level PrinciplesDESIGN AND DELIVERY
PRINCIPLESTactical measuresREDUCE TIME TO MARKETSpeed, Fast
FeedbackCOST EFFICIENCYCollect metrics to allow decisions cost vs.
value.SUPPORT DATA-DRIVEN DECISIONSListen to users and validate
hypothesis.Provide as many relevant metrics & data as
possible.YOU BUILT IT, YOU RUN ITThe team is responsible for
shaping, building, running and maintaining its products. Fast
feedback from live and customers helps us to continuously
improve.ORGANIZED AROUND BUSINESS CAPABILITIESBuild teams around
products not projects. Follow the domain and respect bounded
contexts. Inverse Conway Maneuver.LOOSELY COUPLEDBy default avoid
sharing and tight coupling, except for the big things in common.
Dont create the next monolith.MACRO AND MICRO ARCHITECTUREClear
separation. Autonomous micro services within the rules and
constraints of the macro architecture.AWS FIRSTFavor AWS platform
service over managed service, over self-hosted OSS, over
self-rolled solutions.DATA-DRIVEN / METRIC-DRIVENCollect metrics
from processes and applications. Analyze, alert and act on
them.ELIMINATE ACCIDENTAL COMPLEXITYStrive to keep it simple. Focus
on essential complexity. You build one, you delete one.AUTONOMOUS
TEAMSMake fast local decisions. Be responsible. Know your
boundaries. Share findings.INFRASTRACTURE AS CODEAutomate
everything: Reproducible, traceable and tested.Immutable servers
over snowflake servers.COLLABORATION CULTUREEngineers from all
backgrounds work together in collaborative teams as engineers and
share responsibilities. No silos.BE BOLDGo into production early.
Value monitoring over tests. Recover and learn. Optimize for MTTR
not MTBF.SECURITY, COMPLIANCE AND DATA PRIVACYSecurity must be
included from the beginning and everybodys concern. Keep
data-privacy in mind.CONTAINMENT AND BOUNDARIESAlign blast radius
and vendor lock-in with the boundaries of the organization or
business capabilities.
Version 1.0Icons made by Freepik from www.flaticon.com are
licensed under CC BY 3.0
Make your own. Keep evolving.This is for reference only.
We use this to guide discussions.Poster hangs in every room.The
items on the right support items on the left.
New strategic business goals are coming. Discussions.
Autonomous teamsbusiness capabilitiesorganized around
Have empowered, self organizing cells, to get teams that build
decoupled services.Monolith: Collective code ownership. Does not
scale.Microservices: Pull requests.
Products not projects.Follow the domain and respect bounded
contexts.Make fast local decisions.Be responsible.
17
collaborationculture
DevOps for us is not a team, a role or a tool. It is
collaboration.No silos, no handovers, no fingerpointing.No software
that is hard to operate.No infrastructure nobody needs.
18
We are all engineers!
No more devs and ops -> engineersPut people together in one
stable team. All other experiments failed.Do not fall back into old
behaviours:Zoom in you find 4 devs on one side and 2 ops on the
other -> rotate pairsNot all T-shapes are the same
19
You build it,you run it.
Freedom and Reponsibility.The team is responsible for shaping,
building, running and maintaining its products. Fast feedback from
live and customers helps us to continuously improve.The one who
feels the pain of being woken up at nightthe one to be able to make
the changes-> real resilient services
20
ContinuousDelivery
Fast feedback from real users. Build, Measure, LearnFast
feedback on quality of service.Creating value for user is daily
business.
We started with one release per month......to many releases per
day.
But only code changes were delivered.21
Unlimited Infrastructure with APIs
Many reason to chose AWS:Support business agility.Constant
innovation around higher level services.Avoid undifferentiated
heavy lifting.But APIs allow to do
Infrastructure as commodity.Avoid private or hybrid cloud
22
Service repository contains its infrastructure.Every change goes
through a delivery pipeline.High traceability.Alarms for unexpected
production changes: Changes not through the pipeline.23
ImmutableServers
Treat servers like cattle not like pets.Phoenix servers.No
configuration drift.Security: Alerting on instances that are to
old
Tradeoff cycle time:Current cycle time: commit to production is
20-30 minutes: To slow!
24
Monitoring is the new testing
Tests only run on delivery. Monitoring runs all the time.
Performance.CD Pipelines.Open OpsGenie Alerts.Costs per day.Page
Speed.
Not yet, but planned: Monitor business KPIs.25
Separatecode deploymentfeature releasefrom
No merging of branches anymore.You are not doing CI, when you
are branching.Dynamic Feature Toggles:Canary releases.Product can
switch on a feature for acceptance.
26
How many environments?Which versions on staging?Prod differs
anyway: Load, data,
patternsV2V3V6V5V4V7V5V8EngineerCIDevStagingV1V4Prod
All CD patterns from the book are given!But what happens to CD
with Microservices
Nostagingenvironment
We only maintain one environment where all services integrate:
ProductionBlue/ green or canary releases.Shadow trafficConsumer
Driven Contracts CDCs
Be bold!MTTR over MTBF
28
Event Sourcingone way data highway and data pumps
One way data highwayEvent Sourcing - History of all changes
Design decision: No queries against DC. Data needs to be pushed
into AWSAbility to replay all events from beginning of time.Events
= All changes to a classifiedDatabase change propagation = poor
mans event sourcing, no intent captured
SQS + S3Kinesis + S3Kinesis + DynamoDBSQS + DynamoDBProxy +
DynamoDBDynamoDBEvolution
Fast evolutionWorking code and infrastructure for all options
within 2 weeks
SQS + S3 Inspired by ImmobilienScout24Kinesis + S3 Feedback from
AWSKinesis + DynomaDB Queryable, Storage costs not prohibitive
(7x)SQS + DynamoDB Queue better than LATEST or TRIM_HORIZONProxy +
DynamoDB Changes are collected via queue table in OracleDynamoDB
DynamoDB is global service
How (not) to shareshared nothing as defaultloosely coupledfast
local decisionsvoluntary adoptionexception: macro concerns
Sharing implies dependencies.Share only with good
reason.Availability over shared nothingNo side effects: Story of
shared state in dashingUse over re-useCopy npaste, OSS, libraryFast
local decisions over committeeRe-use only after hardening
Frontend integrationLoosely coupledAutonomous teamHigh
optimization
Shared nothing Jigsaw as lightweight as possible, not: frontend
monolith, portal, behaviourNo shared asset pipeline. Bring your own
asset (aka asset Brotzeit).Autonomous teamsMicroservices imply
frontend integrationSelf contained MS over UI monolithHigh
optimisationGoogle pagespeedCaching
PageSpeed Module
css (page+fragment)js (page+fragment)
ngx_pagespeed
css (page)js (page)
css (fragment)js (fragment)
No shared asset pipeline.
Pagesare accessible via (localised) URLare owned by one
teamcould be cacheableFragmentsare parts of a pagedont know the
original request should send cache headersAssetsShould be combined
and minifiedCachingCloudFront Caching: Caching on edge locations.
Respects Cache Headers from Jigsaw.PageSpeed Caching: Caches
combined assets.Backend Caching: Respects Cache Headers from
microservices.
Infrastructure guildAgree on things to doShare learningsDelegate
implementation to teamsInfrastructure product teams needed?
Avoid disconnect infrastructure and product teams.
Why infrastructure guild?Agree on things to doShare learningsHow
we work:Work is done in the teamsFocus work on infrastructure
needed by team.(Macro decision: Shared infrastructure: Logging,
Monitoring, Security)Teams are responsible - SubsidiarityHow to
handle long running/blocked stories?Keep delegating and resist
temptation to take overDont treat infrastructure story as neglected
childEmpty backlog should be normalHurdle to create new tasks.We
want as few shared infrastructure as possible.Shirky: Institutions
will try to preserve the problem to which they are the
solution.Creates it own tasks.Risk of stories without value or
prioritisation.I have found an old sticky on the floor, let's
implement that for two weeks!
?
STRATEGICGOALSGoals of the business
sideARCHITECTURALPRINCIPLESHigh-Level PrinciplesDESIGN AND DELIVERY
PRINCIPLESTactical measuresREDUCE TIME TO MARKETSpeed, Fast
FeedbackCOST EFFICIENCYCollect metrics to allow decisions cost vs.
value.SUPPORT DATA-DRIVEN DECISIONSListen to users and validate
hypothesis.Provide as many relevant metrics & data as
possible.YOU BUILT IT, YOU RUN ITThe team is responsible for
shaping, building, running and maintaining its products. Fast
feedback from live and customers helps us to continuously
improve.ORGANIZED AROUND BUSINESS CAPABILITIESBuild teams around
products not projects. Follow the domain and respect bounded
contexts. Inverse Conway Maneuver.LOOSELY COUPLEDBy default avoid
sharing and tight coupling, except for the big things in common.
Dont create the next monolith.MACRO AND MICRO ARCHITECTUREClear
separation. Autonomous micro services within the rules and
constraints of the macro architecture.AWS FIRSTFavor AWS platform
service over managed service, over self-hosted OSS, over
self-rolled solutions.DATA-DRIVEN / METRIC-DRIVENCollect metrics
from processes and applications. Analyze, alert and act on
them.ELIMINATE ACCIDENTAL COMPLEXITYStrive to keep it simple. Focus
on essential complexity. You build one, you delete one.AUTONOMOUS
TEAMSMake fast local decisions. Be responsible. Know your
boundaries. Share findings.INFRASTRACTURE AS CODEAutomate
everything: Reproducible, traceable and tested.Immutable servers
over snowflake servers.COLLABORATION CULTUREEngineers from all
backgrounds work together in collaborative teams as engineers and
share responsibilities. No silos.BE BOLDGo into production early.
Value monitoring over tests. Recover and learn. Optimize for MTTR
not MTBF.SECURITY, COMPLIANCE AND DATA PRIVACYSecurity must be
included from the beginning and everybodys concern. Keep
data-privacy in mind.CONTAINMENT AND BOUNDARIESAlign blast radius
and vendor lock-in with the boundaries of the organization or
business capabilities.
Version 1.0Icons made by Freepik from www.flaticon.com are
licensed under CC BY 3.0
Make your own. Keep evolving.This is for reference only.
We use this to guide discussions.Poster hangs in every room.The
items on the right support items on the left.
New strategic business goals are coming. Discussions.
Picture Credits"HotWheels - '69 Ford Torino Talladega by Leap
Kye, licensed under CC BY-ND 2.0Differences between Traditional vs
Next Generation by Simon Wardley under CC BY-SA 3.0Enterprise IT
Adoption Cycle by Simon Wardley under CC BY-SA 3.0And the future is
private by Simon Wardley under CC BY-SA 3.0Leosvel et Diosmani by
Ludovic Pron under CC BY-SA 3.0Spare wheel by Brian Snelson under
CC BY 2.0Wheel clamps Texas by Richard Anderson from Denton, United
States (Boots.) under CC BY-SA 2.0Wandergeselle by Sigismund von
Dobschtz under CC BY-SA 3.0Sharing Sucks (4536747557) by eyeliam
from Portland, United States under CC BY 2.0Traffic Jam by Doo Ho
Kim under CC BY-SA 2.0Puzzling by Bernd Gessler (Own work) CC BY-SA
3.0Amazon16 by Neil Palmer/CIAT under CC BY-SA 2.0
38
Backup
39
Roman Riding40
21st CenturyWhat does a
tech company look like?
41
Great DesignUniversally ConnectedMobile FirstInstant Business
ValueMassive Data InsightHighly Available
GiantsStand on the Shoulders of
We want to leverage the tools of the ecosystem.Innovation for us
no longer comes from previous enterprise suppliers.Oracle, IBM,
Microsoft43
We want to learn and participate from the advances the unicorns
in our industry make.Visited Netflix, LinkedIn, AirBnb, etc.44
Conways Law
organizations which design systems ... are constrained to
produce designs which are copies of the communicationstructuresof
these organizations
Intro to next slide: Inverse Conways Maneuver45
UnifiedLogsCC
Ease of adoptionPart of Base AMI, ConventionsEvent Publisher for
KinesisApplication events over log files.CloudTrail also used for
AlertsUnexpected production changeData LakeEverything is written
S3.Additional consumers anticipatedHeartbeat of the platformNo ssh
to machinesNext step: Move timeseries/ anlytics from ES
Two stacksCash stack meets shiny new stack One company Lights on
in cash stackFeature freezeWhere to build new features?Ease of
integration helps business peopleWW
There is a cash stack!
Dont split into shiny new stack and legacy.Legacy is not used as
a word anymore.Lights on in Cash stack - AWSFeature freeze: Where
to build new features?Ease of integration between new and old stack
helps business people to switch fluently between both worlds.Where
to draw the line - Focus on technology change - Minimum viable
integrationTime to market vs. fast re-platforming (N+1 systems)
