Continuous Deployment in AWS Lambda

Continuous Deployment in AWS Lambda

Suiting Tseng (@suitingtseng)

Who am Iclass SoftwareEngineer(Human):

def __init__(self):self.interests = [

"AWS"]

self.languages = ["Python","Golang"

]

–KKStream

We back you up to compete against Netflix, Spotify, and KKBOX.

Outline

• AWS Lambda Introduction

• Lambda Deployer

• Adoption in KKStream

What is AWS Lambda?

–AWS

AWS Lambda is a zero-administration compute platform for back-end web developers that runs

your code for you in the AWS cloud and provides you with a fine-grained pricing structure.

–AWS

AWS Lambda runs your back-end code on its own AWS compute fleet of Amazon Elastic Compute Cloud (Amazon EC2) instances across multiple

Availability Zones in a region, which provides the high availability, security, performance, and

scalability of the AWS infrastructure.

Cloud Computing in AWS• Elastic Compute Cloud (EC2)

• Infrastructure as a Service

• Abstract hardware


• EC2 Container Service (ECS)

• Container as a Service

• Abstract OS


• EC2 Container Service (ECS)

• AWS Lambda

• Function as a Service

• Serverless

Lambda Runtime

• Python 2.7

• Java OpenJDK 1.8

• Node.js v0.10.36, v4.3.2

Hello World

#lambda_function.py

deflambda_handler(event,context):

print“Hello,PyCon!”

Event & Context

• Event

• What happen? S3? DynamoDB?

• Context

• How much time remaining?

Execution model

• Synchronous

• Event driven

Execution model

• Synchronous

• Event driven

Sync. Call

• AWS SDK

• AWS API Gateway

Sync. Call

• AWS SDK

• AWS API Gateway

AWS SDK

import boto3

boto3.client("lambda").invoke( FunctionName="hello-world")

Sync. Call

• AWS SDK

• AWS API Gateway

API Gateway Lambda

GET/videos/123

API Gateway Lambda

API Gateway calls Lambda

API Gateway Lambda

Lambda retrieve data from somewhere

API Gateway Lambda

Lambda response

API Gateway Lambda200OK

{

“name”:“ ”,

“characters”:[“ ”...],

……

}

API Gateway Lambda

Lambdascalesautomatically

Execution model

• Synchronous

• Event driven

Event driven

• Event sources

• S3

• Scheduled Event

• DynamoDB

• and many more…

raw thumbnailLambda

Users upload their photo

raw thumbnailLambda

S3 triggers Lambda

raw thumbnailLambda

Lambda resizes the image, and put it to another bucket

S3 event

1 def lambda_handler(event, context):2 s3 = boto3.client('s3')3 4 bucket = event['Records'][0]['s3']['bucket']['name']5 key = event['Records'][0]['s3']['object']['key']6 print "S3 Bucket: %s" % bucket7 print "S3 Key: %s" % key8 ……

Lambda usage

• HTTP API backend

• Worker

• Cronjob

Version and Alias• Version

• 1, 2, 3, 4 ….

• Incremental integer

• Immutable

• Alias

• feature -> 4

• testing -> 8

• Mutable

Outline

• AWS Lambda Introduction

• Lambda Deployer

• Adoption in KKStream

Problem• So you wanna deploy…

• Upload a zip to S3

• Create or update function

• Configure version

• Create or update alias

• And you may have 100+ functions

Our goal

push

Our goal

notify

Deploy pipeline

gitpushoriginmaster

Deploy pipeline

HTTP webhook

Deploy pipeline

1. Build dependency 2. Testing 3. Package 4. Upload to S3

Deploy pipeline

Triggers Lambda Deployer

Lambda deployer is also a Lambda function!

Behind the scene - 1packages/

my-func-1/

my-func-2/

my-func-3/

……


my-func-1/

bfb53884976e70966a9e8a8d5133bcb2e1c06691/

eed8ccf9ea22099cef07b0a2012085cd45e1d3ad/

……


my-func-1/

bfb53884976e70966a9e8a8d5133bcb2e1c06691/

lambda.zip

function.json

function.json{

"runtime":"python2.7",

"handler":"main.lambda_handler",

"timeout":300,

"memory":128

}


my-func-1/

bfb53884976e70966a9e8a8d5133bcb2e1c06691/

eed8ccf9ea22099cef07b0a2012085cd45e1d3ad/

testing

……

Deploy pipeline

Create / update function Publish new version Update alias

deployer.py

class Deployer(object):

def __init__(self): self.s3_client = boto3.client("s3") self.lambda_client = boto3.client("lambda") self.slack = Slacker(open("config/slack-token").read()) self.slack_channel = json.load(open("config/slack.json"))

deployer.py

1 class Deployer(object):2 3 def deploy(self, record):4 bucket = record["s3"]["bucket"]["name"]5 key = record["s3"]["object"]["key"]6 7 lambda_function = LambdaFunction(bucket, key)8 version = self.put_function(lambda_function)9 self.put_alias(lambda_function, version)

main.py

1 import boto3, json2 from deployer import Deployer3 4 def run(event, context):5 d = Deployer()6 print "boto3 version:", boto3.__version__7 print json.dumps(event, indent=2)8 for record in event["Records"]:9 d.deploy(record)

Pros

• Serverless (yes we are lazy…)

• One deployer manages

• many functions

• many stages (using alias)

• Perfectly fits in our workflow

In KKStream

• From 2016/02/01 to 2016/05/31

• 12 developers

• 99 functions

• 4288 deployments

• 1 fork

In KKStream

• Deploy itself 56 times ( code )

• failed several times…

• but very easy to rollback

Future work

• Add event source mapping

• More configurations

• https://github.com/KKStream/aws-lambda-deployer

Q & A

We’re hiring!

Meet us at our booth on 4th floor!

Thank you!