© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Zero to Sixty: AWS CloudFormation Chetan Dandekar, Senior Product Manager – AWS CloudFormation Capen Brinkley, Software Developer – Intuit November 13, 2013
AWS CloudFormation
creation order? how long do I pause? what errors can I recover from?
Instruction Manual
Instruction Manual
Instruction Manual
Provisioning Script(s)
what environment config and utilities does my script depend on?
can my script be faster? will this script work again?
Model Click Done
AWS CloudFormation
Dev
Test
Staging
Prod
Demos
Regions
Version Control Replicate
Standardization
Service Catalog
AWS CloudFormation Automate
CloudFormation
Build Pipeline
Config
Template Snippets
Code
API
API
SNS
Monitor Progress
Intuit’s CloudFormation Story
Key Takeaways
• How we use CloudFormation to manage large scale applications
• Methodologies and tools you can use to follow a similar path
Infrastructure Design
Template Management
Stack Management
Bootstrapping
Live Community Traffic April 15
Feb. 1
Amazon EC2 Amazon S3
Amazon RDS Elastic Load Balancing
Infrastructure as Code
Auto Scaling Group
Web App
Server
App Tier
Web App
Server
Amazon Route 53
Amazon EC2
Amazon S3
Amazon RDS
Amazon ElastiCache
Amazon CloudFront
Amazon CloudWatch
AWS CloudFormation
AWS IAM
Amazon SQS
Amazon SES Amazon SNS
Service Oriented Architecture
SQS Queue App Tier
Auto Scaling Group
Web App
Server
Web App
Server
Multiple Templates, Loosely Coupled
Multiple Templates, Loosely Coupled
Easy To Reason About
Reusable
Stack Management
Simple Deploy https://github.com/intuit/simple_deploy
attributes clone create deploy destroy environments events execute instances
list outputs parameters protect resources status template update
Simple Deploy Commands attributes clone create deploy destroy environments events execute instances
list outputs parameters protect resources status template update
elb-1
Auto Scaling Group
app-1 (v1.0.0) Blue
Auto Scaling Group
app-2 (v1.1.0) Green
$ simple_deploy environments
Default
lc_preprod_us_west_1
lc_preprod_us_west_2
lc_preprod_us_east_1
PROD_lc_prod_us_west_1_PROD
PROD_lc_prod_us_west_2_PROD
PROD_lc_prod_us_east_1_PROD
$ simple_deploy list \
–-environment lc_preprod_us_west_1
lc-dev-elb-1
lc-dev-app-1
lc-dev-db-master-1
lc-dev-db-parameter-group
simple_deploy create \
–-environment lc_preprod_us_west_1 \
–-name lc-dev-app-2 \
–-template app.json \
–-input-stack lc-dev-elb-1 \
–-input-stack lc-dev-db-master-1 \
–-attribute chef_repo=3f57f9f \
–-attribute app=bcb68de
simple_deploy clone
--environment lc_preprod_us_west_1 \
--source-stack lc-dev-1-app-1 \
--name lc-dev-1-app-2 \
--attribute app=afdac509b \
--attribute chef_repo=a4531e5ff6
simple_deploy destroy
--environment lc_preprod_us_west_1 \
--name lc-dev-1-app-1
Code / CI / Artifact
Simple Deploy
CloudFormation
Autoscaling
Userdata
CloudFormation::Init
Chef
Bootstrapping
> GET http://169.254.169.254/latest/user-data
#!/bin/bash
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init –-stack lc-app-stack
-–verbose --resource InstanceLaunchConfig
--region=us-west-2 –-configsets bootstrap
"UserData": { "Fn::Base64": { "Fn::Join": ["", [
"#!/bin/bash\n",
"yum update -y aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-init --stack “,
{ "Ref": "AWS::StackName" },
" --verbose"
" --resource InstanceLaunchConfig",
" --region=", { "Ref": "AWS::Region" },
" -configsets bootstrap", "\n”
CloudFormation Instance User Data
Configsets Commands
Files Groups
Packages Services Sources Users
CloudFormation::Init Resources
Configsets Commands
Files Groups
Packages Services Sources Users
"configSets”: {
"bootstrap”: [ "create_files",
"install_packages",
"run_chef",
"clean_up” ]
}
"create_files": {
"files": {
"/etc/chef/ohai/hints/ec2.json": {
"content": "{}",
"mode": "000400",
"owner": "root",
"group": "root"
}
}
}
"install_packages": {
"packages": {
"yum”: {
"chef”: [ "11.6.2-1" ]
}
}
}
"run_chef": {
"commands": {
"1_download_chef_repo": { ... },
"2_decrypt_chef_repo": { ... },
"3_extract_chef_repo": { ... },
"4_run_chef": { ... }
}
}
"run_chef": {
"commands": {
"1_download_chef_repo": { ... },
"2_decrypt_chef_repo": { ... },
"3_extract_chef_repo": { ... },
"4_run_chef": { ... }
}
}
"run_chef": {
"commands": {
"run_chef": {
"command": "/usr/bin/chef-solo \
–c /var/chef/config/solo.rb \
–o ", { "Ref", "Role" }
}
}
}
"clean_up" : {
"commands": {
"1_cleanup_files" : {
"command": "rm –rf /var/tmp/chef_repo.tar.gz \
/var/tmp/chef_repo.tar.gz.gpg"
}
}
}
The Climb
What’s New in AWS CloudFormation
Let’s take an example
Scalable Reliable Highly Available
Two Types of Tasks
Develop
Parallel stack processing
Richer template language
Operate Fail-safe stack management
Updates without downtime
Federation and IAM roles
Parallel Stack Processing
Parallel Stack Processing
Richer Template Language
Conditions
Prod
Dev
Conditions "Parameters" : { "Environment" : { "Description" : "Specifies if this a Dev QA or Prod Environment", "Type" : "String", "Default" : "Dev", "AllowedValues" : [ "Dev", "QA", "Prod"] }, }, ... "Conditions" : { "ProdEnvironment" : { "Fn::Equals" : [ { "Ref" : "Environment" }, "Prod" ]} },
Conditions "DBInstance" : { "Type": "AWS::RDS::DBInstance", "Properties": { "DBName" : { "Ref" : "DBName" }, "Engine" : "MySQL", "MultiAZ" : { "Fn::If" : [ "ProdEnvironment", "true", "false" ] }, "DBSnapshotIdentifier" : { "Fn::If" : [ "ProdEnvironment", { "Ref" : "DBName" }, { "Ref" : "AWS::NoValue" } ] }, ... } },
Conditions "DBStorageAlarm" : { "Condition" : "ProdEnvironment", "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmDescription" : "Alarm if db size grows beyond a threshold", "Namespace" : "AWS/RDS", "MetricName" : "FreeStorageSpace", ... } },
Conditions
• Fn::If • Fn::Equals • Fn::Not • Fn::And • Fn::Or
"Conditions" : { ... "ProdOrLoadTestingEnv" : { "Fn::Or" : [ { "Condition" : "ProdEnvironment"}, { "Fn::Equals" : [ ... ]} ] } }
"Fn::If": [{condition}, {value_if_true}, {value_if_false}]
User-Defined Resource Names By default,
• AWS CloudFormation
generates unique resource names
• “prodstack20131113-DBStorageAlarm-19BL0MOXL0TPI”
In addition,
• Flexibility to use custom names and still keep them unique
• “SalesDataStorageAlarm”
Develop
Parallel stack processing
Richer template language
Operate Fail-safe stack management
Updates without downtime
Federation and IAM roles
Fail-Safe Stack Management
Stack Protection
CloudFormation Prod
Dev1
Dev2
Dev3
{ "Effect" : "Allow", "Action" : [ "cloudformation:*" ], "Resource" : "arn:aws:cloudformation:us-west- 2:123456789012:stack/Dev*" }
{ "Effect" : "Allow", "Action" : [ "cloudformation:*" ], "Resource" : "*" }
Stack Protection { "Effect" : "Deny", "Action" : [ "cloudformation:DeleteStack", "cloudformation:UpdateStack" ], "Resource" : "arn:aws:cloudformation:us-west-2:123456789012:stack/productionstack/*" }
Stack Protection "Resources" : { "StackProtectionPolicy" : { "Type" : "AWS::IAM::Policy", "Properties" : { "PolicyName" : "StackProtectionPolicy", "Groups" : [ { "Ref" : "DenyGrp" } ], "PolicyDocument" : { "Statement" : [ { "Effect" : "Deny", "Action" : [ "cloudformation:DeleteStack", "cloudformation:UpdateStack" ], "Resource" : { "Ref" : “AWS::StackId" } }
Resource Protection { "Effect" : "Deny", "Action" : [ "ec2:TerminateInstances" ], "Condition": { "Null": { "ec2:ResourceTag/*cloudformation*" : "true" } }, "Resource" : "*" }
Preventing Updates { "Statement" : [ { "Effect" : "Deny", "Action" : "Update:*", "Principal" : "*", "Resource" : "ResourceType/AWS::RDS::DBInstance" }, { "Effect" : "Allow", "Action" : "Update:*", "Principal" : "*", "Resource" : "*" } ] }
Stack Policy Document
Preventing Updates > aws cloudformation create-stack -–template-url ... --stack-policy-url ...
{ "Statement" : [ { "Effect" : "Deny", "Action" : "Update:Replace", "Principal" : "*", "Resource" : "LogicalResourceId/MyInstance" }, { "Effect" : "Allow", "Action" : "Update:*", "Principal" : "*", "Resource" : "*" ...
Fine Grained Stack Policy Setting Stack Policy
Update without Downtime "WebServerGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "LaunchConfigurationName" : { "Ref" : "LaunchConfig" }, ... }, "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MinInstancesInService" : “2", "MaxBatchSize" : “3", "PauseTime" : "PT20M" } } },
Using AWS CloudFormation with Federated Identities
Corporate identity store
1
User accesses broker
2 User authenticated
Identity broker
3 AWS Management
Console
User redirected to console
Temporary security credentials obtained
4
AWS Security Token Service
User accesses APIs
CloudFormation API and other AWS APIs
4
Network Architects
Application Developers
DB Admins
Calling AWS CloudFormation using IAM Roles
EC2 Instance
IAM Role
1. The IAM role has permissions to call
AWS CloudFormation and provision
underlying resources AWS CloudFormation
2. User or script on the EC2 instance calls CloudFormation to
provision a stack
3. AWS CloudFormation provisions the stack using a
template hosted in an S3 bucket inside the VPC
CloudFormation Stack
Related Resources • http://aws.amazon.com/cloudformation/
• "Fundamentals of CloudFormation" lab in the Self Paced Lab Lounge
• DMG303 - AWS CloudFormation under the Hood
• ARC203 - How Adobe Deploys: Refreshing the Entire Stack Every Time
• DMG209 - Enterprise Management for the AWS Cloud
• Multiple other sessions are presenting CloudFormation samples
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
DMG201