Embed Size (px)
Citation preview
and containers
Andrew Spyker (@aspyker) - Engineering Manager
Not
About Netflix
● 86.7M members● A few thousand employees● 190+ countries● > ⅓ NA internet download traffic● 500+ Microservices● Many 10’s of thousands VM’s● 3 regions across the world
Netflix has a elastic, cloud native, immutable microservice architecture using full devops built on VM’s!
3
Why are we messing around with containers?
Technical motivating factors for containers
● Simpler management of compute resources
● Simpler deployment packaging artifacts for compute jobs
● Need for a consistent local developer environment
Sampling of realized container benefits
Media Encoding - encoding research development time● VM’s platform to container platform - 1 month vs. 1 week
Continuous Integration Testing● Build all Netflix codebases in hours● Saves development 100’s of hours of debugging
Edge Re-architecture using NodeJS● Focus returns to app development● Simplifies, speeds test and deployment
5
Batch applications
Multi-tenant (cgroups/Mesos) historically used for batch
Linux cgroups
What do batch users want?
● Simple shared resources, run till done, job files
● NOT○ EC2 Instance sizes, autoscaling, AMI OS’s
● WHY○ Offloads resource management ops, Simpler
Introducing Titus
Batch
Job Management
Resource Management & Optimization
Container ExecutionIntegration
Workflow, Data Analysis, Adhoc Upstream Systems
Netflix Batch Job Examples
● Algorithm Model Training (with GPU’s)
Netflix Batch Job Examples
● Media Encoding
● Digital Watermarking
1 1
Netflix Batch Job Examples
Open Connect CDN Reporting
AdhocReporting
● Docker helped generalize use cases● Scheduling required (GPU, elastic)● Initially ignored failures (with retries)● Time sensitive batch came later
Lessons Learned from Batch
Current Container Usage - Batch● 1000’s of container hosts (g2, m4, r3 instances)● 1000’s containers / hour average● Large spikes of CI testing and Digital Watermarking
From day of 10/26● 47K containers● Bursts of 1000
containers in a minute
Service applications
Why Services in containers?
Theory Reality
Developer
Opportunities to evolve our baking
● Java focused supported AMI, baking works well!
● However, wanted to allow○ other stacks to evolve independent of OS updates○ simplified builds (vs. Java and OS based tooling)○ reliable smaller instances for dynamic languages○ ability to develop locally with same image
Services are just long running batch?
ServicesJob Management
Resource Management & Optimization
Container ExecutionIntegration
Service Apps
Batch
19
Nope, not that easy - Titus Details
19
Titus UITitus UI
Docker RegistryDocker Registry
Rhea
containercontainer
container
docker
Titus Agent metrics agent
Titus executor
logging agent
zfs
Mesos agent
docker
RheaTitus API
Cassandra
Titus Master
Job Management & Scheduler
S3
ZookeeperDocker Registry
EC2 Autocaling API
Mesos Master
Titus UI
Fenzo
container
Pod & VPC network drivers
containercontainer
AWSmetadata proxy
Integration
AWS VM’sCI/CD
Services more complex● Services resize constantly and run forever
○ Autoscaling○ Hard to upgrade underlying hosts
● Require IPC integration○ Routable IPs, service discovery○ Ready for traffic vs. just started/stopped
● Existing well defined dev, deploy, runtime & ops tools
Real networking is hard
Multi-tenant
Need IP per container - in VPC
Using security groups
Using IAM roles
Considering network resource isolation
Enabling VPC Networking
No IP, SecGrp A
Task 0
SecGrp Y,Z
Task 1 Task 2 Task 3
Titus EC2 Host VMeth1
ENI1SecGrp=A
eth2
ENI2SecGrp=X
eth3
ENI3SecGrp=Y,Z
IP 1IP 2
IP 3
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
appapp
veth<id>
Linux Policy BasedRouting + Traffic Control
TitusEC2
Metadata Proxy
169.254.169.254IPTables NAT (*)
* **
169.254.169.254Non-routable IP
*
Solutions● VPC Networking driver
○ Supports ENI’s - full IP functionality○ Scheduled security groups○ Support traffic control (resource isolation)
● EC2 Metadata proxy○ Adds container “node” identity○ Delivers IAM roles
Reuse existing infrastructure services
VMVM
EC2
AW
S A
utoS
cale
rVMs
App
Cloud Platform(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
Atlas Eureka Edda
Enable them for containers
VMVM
EC2
AW
S A
utoS
cale
rVMs
App
Cloud Platform(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
VMVM
Atlas
Titu
s Jo
b C
ontro
l
Containers
App
Cloud Platform(metrics, IPC, health)
Eureka Edda
VMVM
BatchContainers
Spinnaker
Deploy based on new images
tags
Basic resource requirements
IAM Roles & Sec Groups per container
Deploy Strategies
Same as VM’s
Easily see health &
discovery
Secure Multi-tenancyCommon to VM’s and tiered security needed● Protect the reduced host IAM role, Allow containers to have specific IAM
roles● Needed to support same security groups in container networking as VM’s
User namespacing● Docker 1.10 - Introduced User Namespaces
● Didn’t work /w shared networking NS● Docker 1.11 - Fixed shared networking NS’s
● But, namespacing is per daemon, Not per container, as hoped● Waiting on Linux
● Considering mass chmod / ZFS clones
Titus Advanced Scheduling
● Support for AZ balancing● Multiple instance types selected based on workload● Elastic underlying common resource pool
○ Bin packing managed transparently across all apps● Hard and soft constraints● Resource affinity and task affinity● Capacity guarantees (critical tier)
34
Fenzo - Keep resource scheduling extensible
Fenzo - Extensible Scheduling Library
Features:● Heterogeneous resources & tasks● Autoscaling of mesos cluster
○ Multiple instance types● Plugins based scheduling objectives
○ Bin packing, etc.● Plugins based constraints evaluator
○ Resource affinity, task locality, etc.● Scheduling actions visibility
Current Container Usage - Service
● Still small ~ 2000 long running containers
● NodeJS Device UI Scripts Apps● Stream Processing Jobs - Flink● Various Internal Dashboards
Questions?
Andrew Spyker (@aspyker) - Engineering Manager
