Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Containers @ WrappPowered by AWS EC2 Container Service
JudeD’SouzaSolutionsArchitect@WrappPhone:+46767085740Email: [email protected]
AboutMe…q JudeD’Souza– Stockholm,Swedenß Karachi,Pakistan
q MS– DistributedSystemsKTHRoyalInstituteofTechnology,Stockholm,Sweden
q Passionforbuildinglargescaledistributedsystems
AboutWrapp…Wrapp deliverspersonaloffersandrewardsbasedonwhereyoushop,dineandthebrandsyoufollow.
q Microservice Architecture
q Infrastructureq 13ECSClustersq 50+Instancesq 90+Services
q TechnologyStackq OperatingSystem- Ubuntuq Consul,Registrator,AWS(EC2,ECS,Lambda),Docker,HAProxyq Monitoring– Sensu,Riemannq Logs- Logentries andSumoq Languages- Go,Python,Java
Agenda• ArchitecturebeforeECS
– ServiceOrchestration– ServiceDiscovery– ServiceDeployments– ServiceAvailability&Scalability– ContainerRegistry
• TransitiontowardsECS– Microservice Architecture@Wrapp
• WhyECS?
• NextChallenges
• QA
Howwedidit…BeforeECS(1/3)
ServiceOrchestrationq Viarunlist configs
Oninstanceboot:q Determineautoscaling groupq Pullrunlist forautoscaling groupq Runcontainersspecifiedinconfigq Addthesetosupervisor
misc ops api
miscAutoscaling group
opsAutoscaling group
api Autoscaling group
PULLRUNLISTCONFIGS
SampleRunlist configurationfile
ü ServiceOrchestrationServiceDeploymentsServiceAvailability&ScalabilityContainer Registry
ServiceDeployments
q Find instanceswhereserviceisrunning(viaserf)
q SSHintotheseinstances
q Dorollingdeploy (Terminatecontainer,Launchcontainerwithnewversion)
misc ops api
miscAutoscaling group
- misc-i-abc 10.0.0.1 rewards/http/9090- misc-i-def 10.0.1.1 rewards/http/9090- misc-i-ghi 10.0.2.1 rewards/http/9090- ....- ....
Sampleserfdataforservices
SERFCLUSTER
opsAutoscaling group
apiAutoscaling group
Howwedidit…BeforeECS(2/3)
ü ServiceOrchestrationü ServiceDeploymentsServiceAvailability&ScalabilityContainer Registry
ServiceAvailabilityq Supervisord
ServiceScalabilityq Serviceitselfcan'tscaleq Scaleonly byscalingautoscaling group
misc ops api
miscAutoscaling group
SERFCLUSTER
opsAutoscaling group
apiAutoscaling group
misc
Howwedidit…BeforeECS(3/3)
ContainerRegistryq Hostingourowncontainerregistryq Backedbys3
ü ServiceOrchestrationü ServiceDeploymentsü ServiceAvailability &Scalabilityü Container Registry
TransitiontoECS
ServiceOrchestrationq Runlistsq SupervisorServiceDeploymentsq SupervisorServiceAvailabilityq SupervisorServiceScalabilityq EC2AutoscalingContainerRegistryq Self-hosting
ServiceDiscoveryq Serfq HAProxy
ServiceDiscoveryq Consulq Registratorq HAProxy
ServiceOrchestrationServiceDeploymentsServiceAvailabilityServiceScalability (withlambda)
AWSContainerRegistry
+Servicestats(cpu +memory)
+Servicemonitoring+ServiceProvisioning
WrappMicroserviceArchitecture
PUBLICSUBNET
PRIVATESUBNET
ElasticLoadBalancer
api api api
Microservices
ZONEA ZONEB ZONEC
api.wrapp.com DNS
ServiceScalability(ConceptfromAmazon)
misc
miscAutoscaling group
ScalingAmazonECSServicesAutomaticallyUsingAmazonCloudWatchandAWSLambda:https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/
CloudWatch AlarmSNSTopic AWSLambda
EC2ContainerService
ECSInstances
TriggerCloudWatch alarmfrom ECSMetrics(CPU/
Memory)
TriggerLambdafunctionfrom SNSTopic
+1DesiredCount inService
Service(CPU/Memory)Provisioning
api
emails
rewardsauth
notifications
eventbus
TypicalECSInstance
ü MemoryallocationbyService
ü AllocateCPUUnitsbyService
ü EfficientResourceutilization(Container placement)
ContainerLogging
TypicalECSInstance
q Logspout
q Rsyslog
q LogEntries
Logentries:https://logentries.comLogspout: https://github.com/gliderlabs/logspout
logspout
rsyslog
ContainerslaunchedviaECS
LogrouterforDocker
Forwardslogstologentries
WhyECS?
q OurExperience
q Generally goodservice (ECSisoK)
q Greatsupport
q Wedon’tneed tohosttheseservicesourselvesonAWS
q Alwaysimprovingtomaketheirservicesbetterandbetter
ü Supportsbuilt-inservicediscovery,loadbalancingandrouting
ü WeaveFlux:Supportsbuilt-inservicediscovery,loadbalancingandrouting
ü DockerSwarm:Supportsorchestratingcontainers
ü Fleet - ContainerOrchestration
api
8080
Consulserver-b
Consulserver-a
Consulserver-c
users
8082
auth
8081
users
8082
Registrator Registrator Registrator
ConsulAgentConsulAgentConsulAgent
Key/valuestore
Servicecatalog
10.0.0.1 10.0.0.2 10.0.0.3
ServiceDiscovery@Wrapp(1/2)HAConsulCluster
ServiceDiscovery@Wrapp(2/2)
Consulserver-b
Consulserver-a
Consulserver-c
users
8083
Registrator
ConsulAgent
10.0.0.3
HAProxy
auth
8082
ConsulTemplate
frontend users-inbind users:80mode httpdefault_backend users
backend usersserver Server-a 10.0.0.1:8080 …server Server-c 10.0.0.3:8080 …
####
frontend auth-inbind auth:80mode httpdefault_backend auth
backend authserver Server-a 10.0.0.1:8081 …server Server-c 10.0.0.3:8081 …
>> curl users/v1/users/1{"name": "jude""gender": "male"
}
HAConsulCluster
DNSHack(192.168.x.x)
NextChallenges(1/2)– Shortcomingsperceived…
Built-in(likekube-proxy,weave-proxy)?q ServiceDiscoveryq ServiceRoutingq ServiceLoadbalancing
Approach1:ELBPerservice?q Costly- 90+services,70+ELBs
Approach2:SingleELBforallservices?ü ServiceRoutingworksü ServiceLoadbalancingworksq Nohealthcheckperserviceq NowaytoreturnHTTP503
(butpossibleviahaproxy httpcheck option)
Benefits:q RemovestoolslikeConsulandRegistrator
ElasticLoadBalancer
8080 8081 8082
ECSBoto3:create_service()
Route53: ELB.dns = elb-services
frontend users-inbind users:80mode httpdefault_backend users
backend usersserver users elb-services:8080 …####
>> curl users/ping
SINGLEELBFORMULTIPLESERVICESAPPROACH
NextChallenges(2/2)– Shortcomingsperceived…
Ø Cron schedulingq Setupecs taskstorunoncron schedule
Ø ECSEventstreamq Actionables oncertainevents(whendeploycompletes)
Ø ECSContainerLogging
Ø OptionallyIntegrateECSClusterwithEC2Autoscaling Group
Summary
q TheWrapp ArchitecturebeforeECS
q TransitiontoECS
q ServiceDiscovery@WrappwithConsul,Registrator andHAProxy
q OurperceivedshortcomingsfromECSq Built-inServiceDiscoveryq Cron schedulingonTasksq Eventstream