zzConfidential

Terraform Modules and Continuous DeploymentsSan Francisco Infrastructure as Code Meetup

zzConfidential

Your speakers…

• Zane Williamson • Trulia Sr. DevOps• Twitter @zane_williamson• Github @sepulworld

• Marius Ducea• AWS Consultant• Twitter @mariusducea• Github @mdxp

zzConfidential

Overview

1. The Challenge

2. System overview

3. Why we chose Terraform and Packer for Continuous

Deployment

4. Terraform modules and how we are using them

5. Demo a continuous deployment with Terraform and Packer

zzConfidential

The Challenge

• Deploying an array of micro services• Multiple teams• Repeatable Cloud configuration• Peer reviewed infrastructure changes• Tooling flexibility• As simple as possible for non experts

zzConfidential

System Overview

• Cloud Provider is AWS

• Micro-services APIs with ELB endpoints

• Micro-services acting as worker clusters without ELB endpoints

• A common shared backend

zzConfidential

System OverviewTerraform Managed

• Kafka cluster• SQS• s3• RDS• EMR (Hbase)

• VPC, subnets, routes, etc

zzConfidential

Why We Chose Packer to Build Images

1. Dev and ops personnel were already familiar with Packer

2. Allowed us to reuse Puppet modules

3. Multiple outputs (VirtualBox, Docker, AMI, etc)

4. Able to embed in the micro-service code base

5. Easy to call from Jenkins server

zzConfidential

Why we chose Terraform to Manage Infrastructure

1. CloudFormation was limited to AWS services

2. Libraries like Boto and Fog required programming skills and don’t

manage state as well

3. Some developers and operations personnel were already familiar with

Terraform

4. Easy to integrate into Jenkins which is the preferred build and

deployment platform of most of our teams

5. Ability to put the Terraform directly into the code base for each micro-

service

6. Leverage Terraform Modules and the capability to use a Git repo as a

source

zzConfidential

Challenges we faced early on

Multiple teams working with terraform

• code duplication | code drift• management & maintenance• versioning • testing

zzConfidential

Terraform modules

The solution to all our problems?

?

zzConfidential

Terraform modules basics

• Modules are used to create reusable components in Terraform as well as for basic code organization.

• Modules are very easy to create and use.• A module is technically just a folder with some

terraform templates.

zzConfidential

Terraform modules - example ELB module

elb├── CHANGELOG.md├── README.md├── main.tf├── outputs.tf├── test│   ├── aws.tf│   └── test.tf└── variables.tf

zzConfidential

variables.tf

variable "name" { default = "dev-elb"}

variable "subnet_ids" { description = "comma separated list of subnet IDs"}

variable "security_groups" { description = "comma separated list of security group IDs"}

variable "port" { description = "Instance port" default = 80}

zzConfidential

main.tf

resource "aws_elb" "main" { name = "${var.name}"

cross_zone_load_balancing = true subnets = ["${split(",", var.subnet_ids)}"] security_groups = ["${split(",",var.security_groups)}"]

listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.port}" instance_protocol = "http" }. . .

zzConfidential

outputs.tf

output "elb_name" { value = "${aws_elb.main.name}"}

output "elb_id" { value = "${aws_elb.main.id}"}

output "elb_dns_name" { value = "${aws_elb.main.dns_name}"}

output "elb_zone_id" { value = "${aws_elb.main.zone_id}"}

zzConfidential

Module usage

Using modules in Terraform is very similar to built-in resources:

module "elb" { source = “./tf_elb” name = "myelb" port = "80" health_check_url = "HTTP:80/"}

zzConfidential

Module usage - remote git source

The module can live in the same place with the code using it (subfolder) or it can be in a separate repo (recommended).

module "elb" { source = "github.com/sepulworld/tf_elb.git?ref=v0.0.1" name = "myelb" port = "80" health_check_url = "HTTP:80/"}

zzConfidential

Module usage - multiple instances

We can instantiate a module multiple times:

module "elb-frontend" { source = “./tf_elb” name = "frontend" port = "80" health_check_url = "HTTP:80/"}

module "elb-internal" { source = “./tf_elb” name = "internal" port = "8080" health_check_url = "HTTP:8080/health"}

zzConfidential

Module usage - get sources

Finally, before using the module we need to first download it from the source, using terraform get:

terraform get -updateGet: git::https://github.com/sepulworld/tf_asg.git?ref=v0.0.3 (update)Get: git::https://github.com/sepulworld/tf_elb.git?ref=v0.0.1 (update)

zzConfidential

So are modules helping solve our challenges?

Let’s see…

✓ code duplication | code drift ✓ management & maintenance✓ versioning ✓ testing

zzConfidential

Lessons learned

• Use variables for everything; have sane defaults.• Document your modules; we use a changelog to

have a history of all bug fixes and new features.• Use separate git repositories for your modules and

use tags to release new versions of the module. • Test your modules (ideally automatically); we use

terraform validate on all commits and a test run on new releases.

• Be aware that different versions of terraform might behave differently.

zzConfidential

Lessons learned

• Separate your terraform code to minimize the impact of a failure; we use something like:

global (global resources like IAM, cloudtrail, s3, etc.)└ users

development (dev environment)└ core (base resources like vpc, sg, etc.)└ db (persistent storage, rds, etc.)└ microservice1 (resources for services like asg, elb, route53, etc.)

production (prod environment)└ core (base resources like vpc, sg, etc.)└ db (persistent storage, rds, etc.)└ microservice2 (resources for services like asg, elb, route53, etc.)

zzConfidential

Lessons learned

• Use terraform_remote_state to share the state between different terraform runs.

data "terraform_remote_state" "vcp" { backend = "s3" config { bucket = “terraform-state-s3-bucket" key = "dev-vpc-us-west-2" }}

resource "aws_instance" "foo" { # ... subnet_id = "${data.terraform_remote_state.vpc.subnet_id}"}

zzConfidential

Lessons learned

• Terraform new features and improvements added by a new version might break the run on an older version. Always tag new releases of a module that might break older version runs.

• If a new resource or a new argument to an existing one was introduced that will be seen as an error on older versions.

resource "aws_elasticsearch_domain" "es" { elasticsearch_version = “${var.es_version}” # added in 0.7.1domain_name = "${var.es_domain_name}"

zzConfidential

Terraform version manager

Use different versions of terraform (tfenv): https://github.com/kamatama41/tfenv

tfenv install 0.7.7install Terraform 0.7.7get archive from https://releases.hashicorp.com/terraform/0.7.7/terraform_0.7.7_darwin_amd64.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 16.8M 100 16.8M 0 0 7163k 0 0:00:02 0:00:02 --:--:-- 7167kArchive: /tmp/terraform_0.7.7_darwin_amd64.zip inflating: /Users/marius/bin/tfenv/versions/0.7.7/terraformthe installation 0.7.7 was successful!!!

zzConfidential

Future improvements

• Conditionals, conditionals, conditionals…

• Terraform language is very limited and this prevents us

from writing ‘real’ reusable modules; at this time we are

using all sort of count related hacks to overcome

conditional limitations but hopefully we’ll have better

solutions in the future.

• Terraform state locking

• Better handling of multiple versions of Terraform

• Testing improvements

zzConfidential

DemoInfrastructure Timeline

zzConfidential

Challenges and No Silver Bullets

1. Doesn’t work with systems that require code to be in

consistent state. For this a Blue/Green type of deployment is

required.

2. Clean up of old AMIs needs to be handled external to Packer

and Terraform

3. Requires application to be engineered for the Cloud

4. Autoscaling Group failure scenarios don’t automatically

cleanup

zzConfidential

https://github.com/sepulworld/tf_elbhttps://github.com/sepulworld/tf_asghttps://github.com/sepulworld/terraform-examples

Demo links:

Questions!