Amazon ECS Deep Dive

October 25th, 2016

Amazon EC2 Container Service Deep DivePeter Dalbhanjan, Solutions Architect

Agenda

• ECS Infrastructure Setup• ECS Infrastructure Management• PaaS on ECS• Q & A

Amazon ECS Benefits

• Easily Manage Clusters for any scale• Flexible Container Placement• Designed for use with other AWS Services• Extensible

Amazon ECS Infrastructure Setup

• Amazon ECS Cluster• AWS CloudFormation• Amazon ECS CLI• AWS OpsWorks

• Amazon ECR

ECS Cluster Setup with AWS CloudFormation

"Resources" : { "ECSCluster": { "Type": "AWS::ECS::Cluster" }, "ECSAutoScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "VPCZoneIdentifier" : { "Ref" : "SubnetID" }, "LaunchConfigurationName" : { "Ref" : "ContainerInstances" }, "MinSize" : "1", "MaxSize" : { "Ref" : "MaxSize" }, "DesiredCapacity" : { "Ref" : "DesiredCapacity" } }, […] },

Autoscaling Group

ECS Cluster

"ContainerInstances": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "commands" : { "01_add_instance_to_cluster" : { "command" : { "Fn::Join": [ "", [ "#!/bin/bash\n", "echo ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] } } }, […] } } }

ECS Cluster Setup with AWS CloudFormation

Launch Configuration

"taskdefinition": { "Type": "AWS::ECS::TaskDefinition", "Properties" : { "ContainerDefinitions" : [ { "Name": "simple-app", "Cpu": "10", "Essential": "true", "Image":"httpd:2.4", "Memory":"300", "MountPoints": [{ "ContainerPath": "/usr/local/apache2/htdocs", "SourceVolume": "my-vol” }], "PortMappings": [ { "HostPort": 80, "ContainerPort": 80 } ] },

ECS Cluster Setup with AWS CloudFormation { "Name": "busybox", "Cpu": 10, "Command": [ "/bin/sh -c \" while true; do echo '<html> <head> <title>Amazon ECS Sample App</title> <style>..... > /usr/local/apache2/htdocs/index.html ; sleep 1; done\"” ], "EntryPoint": [ "sh", "-c"], "Essential": false, "Image": "busybox", "Memory": 200, "VolumesFrom": [ { "SourceContainer": "simple-app" } ] } ],

ECS Cluster Setup with Amazon ECS CLI

• Simplifies creating, updating, and monitoring clusters and tasks

• Supports Docker Compose

• Available on github https://github.com/aws/amazon-ecs-cli

ECS Cluster Setup with Amazon ECS CLI

# Build cluster and container instances$ ecs-cli scale --size 2 --capability-iam --keypair demo-user

# Create task definition and start tasks$ ecs-cli compose up

# See running tasks$ ecs-cli compose ps

# Start tasks as ECS service$ ecs-cli compose --project-name wordpress-test service start

# See the progress of task state$ ecs-cli compose --project-name wordpress-test service ps

ECS Cluster Setup with AWS OpsWorks

• Update OpsWorks IAM role to allow ecs:* actions

• Add instances to layer (24/7, time-based, load-based)

• Manage security updates, user permission and access

Note:• One ECS Cluster layer per

stack• An ECS Cluster can only be

associated with one stack

Amazon ECR Setup

• You have read and write access to the repositories you create in your default registry, i.e. <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

• Repository names can support namespaces, e.g. team-a/web-app.

• Repositories can be controlled with both IAM user access policies and repository policies.

Amazon ECR Setup

# Authenticate Docker to your Amazon ECR registry> aws ecr get-login> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

# Create a repository called ecr-demo> aws ecr create-repository --repository-name ecr-demo

# Push an image to your repository> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

Amazon ECR Docker Credential Helper

• Available today - https://github.com/awslabs/amazon-ecr-credential-helper

• Place the docker-credential-ecr-login binary on your PATH

• Set the contents of ~/.docker/config.json file to be:{ "credsStore": "ecr-login" }• Push and pull images from ECR without docker login

Amazon ECS Infrastructure Management

• Monitoring and Logging• Automatic Scaling• Service Discovery• Security

Monitoring & Logging

Monitoring with Amazon CloudWatch

• Metric data sent to CloudWatch in 1-minute periods and recorded for a period of two weeks

• Available metrics: CPUReservation, MemoryReservation, CPUUtilization, MemoryUtilization

• Available dimensions: ClusterName, ServiceName

Monitoring with Amazon CloudWatch

Use the Amazon CloudWatch Monitoring Scripts to monitor additional metrics, e.g. disk space:

# Edit crontab> crontab -e

# Add command to report disk space utilization to CloudWatch every five minutes*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --from-cron

CloudWatch Logs with awslogs driver

Amazon CloudWatch Logs

Amazon S3

Amazon Kinesis

AWS Lambda

Amazon Elasticsearch Service

Amazon ECS Store

Stream

Process

Search

CloudWatch Logs driver

Configuring Logging in Task Definition

"containerDefinitions": [ { "memory": 300, "portMappings": [ {

"hostPort": 80, "containerPort": 80 } ],

"entryPoint": [ "sh", "-c" ], "logConfiguration": {

"logDriver": "awslogs", "options": {

"awslogs-group": "awslogs-test", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "nginx" }

}, "name": "simple-app", "image": "httpd:2.4", "command": [ "/bin/sh -c \"echo 'Congratulations! Your application is now running on a

container in Amazon ECS.' > /usr/local/apache2/htdocs/index.html && httpd-foreground\"" ], "cpu": 10 } ], "family": "cw-logs-example"

Monitoring Amazon ECS with Datadog

Monitoring Amazon ECS with Sysdig Cloud

Scaling Amazon ECS

Setup ECS Cluster with AutoScaling

Create LaunchConfiguration • Pick instance type depending

on resource requirements, e.g. memory or CPU

• Use latest Amazon Linux ECS-optimized AMI, other distros available

Create AutoScaling group and set to cluster initial size

AutoScaling your Amazon ECS Cluster

• Create CloudWatch alarm on a metric, e.g. MemoryReservation

• Configure scaling policies to increase and decrease the size of your cluster

AutoScaling your Amazon ECS services

Service Discovery

Service Discovery using ELB

• Automation built using CloudWatch Events, Lambda and Route53 private hosted zones

• Route53 is used as service registry

• Lambda is used to add/remove records based on Service API’s from ECS

• Available on github https://github.com/awslabs/ecs-refarch-service-discovery

Service Discovery using ELB

Service Discovery using DNS• Install an agent

(ecssd_agent.go) on container instances

• The agent registers service name, IP and port into Route53 private hosted zone

• lambda_health_check.py used for cleanup

• Available on github https://github.com/awslabs/service-discovery-ecs-dns

Service Discovery using DNS

Service Discovery with Weaveworks

• DNS interface for cross-host container communication

• Gossip protocol to share grouped updates

• Overlay network between hosts

Service Discovery and Configuration Management with Consul

consul-server

ECS Instance

consul-agent

registrator

ECS Instance

Back end 1

Back end 2

consul-agent

registrator

ECS Instance

Front end

Security

IAM Roles for ECS Tasks{

"family": “signup-app", "taskRoleArn":

"arn:aws:iam::123456789012:role/DynamoDBRoleForTask", "volumes": [],

"containerDefinitions": [{ "environment": [ ... ],

"name": “signup-web",

"mountPoints": [], "image": “amazon/signup-

web", "cpu": 25, "portMappings": [ ... ],

"entryPoint": [ ... ],"memory": 100, "essential": true, "volumesFrom": []

Logging Amazon ECS API with AWS CloudTrail

{ "eventVersion": "1.03", "userIdentity": {…}, "eventTime": "2015-10-12T13:57:33Z", "eventSource": "ecs.amazonaws.com", "eventName": "CreateCluster", "awsRegion": "eu-west-1", "sourceIPAddress": "54.240.197.227", "userAgent": "console.amazonaws.com", "requestParameters": { "clusterName": "ecs-cli" },

Create Cluster event

Logging Amazon ECS API with AWS CloudTrail

"responseElements": { "cluster": { "clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli", "pendingTasksCount": 0, "registeredContainerInstancesCount": 0, "status": "ACTIVE", "runningTasksCount": 0, "clusterName": "ecs-cli", "activeServicesCount": 0 } }, […]

Create Cluster event

Image Vulnerability Scanning with Twistlock

Secrets Management

• Option 1: Task Definition Environment Variables• Easy to get Started• Configuration stored Directly into Task Definition• Version in Immutable Definition; Easy Rollback• Not Great for Secrets

• Option 2: Encrypted DynamoDB or S3• Use Environment Variables to Provide Pointer• Use AWS Encryption Clients to Securely Store• Use VPC-Endpoints, IAM Policies, and IAM Roles to Restrict

Access

Secrets Management

ECS Cluster

Container instance

PaaS on ECS

AWS Elastic Beanstalk

• Elastic Beanstalk uses Amazon ECS to coordinate deployments to multi-container Docker environments

• Dockerrun.aws.json file that describes how to deploy containers.• Takes care of tasks including cluster creation, task definition and

execution

Convox

# Initialize your app and create default manifest> convox init

# Locally build and run your app as declared in the manifest > convox start

# Create app> convox apps create my_app

# Deploy app, output ELB DNS name> convox deploy[...]web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

Remind Empire

• Offers a control layer on top of Amazon ECS that provides a Heroku like workflow

• Any tagged Docker image can be deployed to Empire as an app • When you deploy a Docker image to Empire, it will extract a Procfile

from the WORKDIR• Each process type in the Procfile maps directly to an ECS Service

Remind Empire

• Get started by launching CloudFormation stack • Use the emp client to start developing your app

# tell empire client where it can find the API$ export EMPIRE_API_URL=http://empire-60-LoadBala-…elb.amazonaws.com/

# login to empire using your github credentials $ emp login

# run your first app$ emp deploy remind101/acme-inc:master

# check what’s running$ emp apps acme-inc Jun 15 20:42[...]

Additional Resources

• ECS CLI – http://bit.ly/2eKy3I6• ECR Docker Credential Helper – http://bit.ly/2dD02xo• AutoScaling – http://amzn.to/2eohA2a• ECS integration with ALB to support Dynamic ports and

Path-based routing: http://amzn.to/2exhh07• Service Discovery

• Service Discovery using ELB – http://bit.ly/2dAN6Dw• Service Discovery using DNS – http://bit.ly/2eI831D

Thank you!

Peter Dalbhanjandalbhanj@amazon.com

Amazon ECS Deep Dive

Technology

ECS and ECR deep dive

BRE Deep Dive

The Deep Dive

ECS & ECR Deep Dive - 김기완 솔루션즈 아키텍트 :: AWS Container Day

CUDA Deep Dive

FCoE Deep Dive

Deep dive phonegap

VSL Deep Dive

Mobility Deep Dive

Deep Dive Xamarin.Android

Cassandra - Deep Dive

WCF- BasicHTTPRelay Deep Dive - Microsoft Adapter Deep Dive Contents WCF-BasicHTTPRelay Deep Dive 0 About the Author

Customer Sharing: miiiCasa - Deep Dive into AWS ECS and Spot Instances at Scale

Deep Sea Fishing: A Swordfish Deep -Dive · Deep Sea Fishing: A Swordfish Deep -Dive Richelle Ahlvers Principal Storage Management Architect Broadcom Limited . ... This deep-dive

002 Deep Dive

Netwrix Auditor DEEP DIVE Deep€¦ · DEEP DIVE Netwrix Auditor Deep Dive Insider Threat Detection. Welcome Solutions Engineer, Netwrix ... SharePoint Netwrix Auditor for VMware

Deep dive formatting

Android Deep Dive

DynamoDB Deep Dive

Rails Deep Dive