Introducing in-house PaaS　

2016/07/25 SRE Tech Talks

　

Manager, Site Reliability Engineering

Nobutoshi Ogata @nobu666

Self Introduc.on

• Nobutoshi Ogata

• Manager, Site Reliability Engineering

• @nobu666

• ❤ Whiskey, Cat, Heavy Metal

• Entrusted dev.(10y) ➡ GREE

infrastructure devision(3y) ➡ Some

startup(1y) ➡ SmartNews(2015/05-)

SmartNews

18M+downloadsworldwide

TV CM On Air now

Now on sale

Agenda• Structure of an organiza0on

• About SRE

• In-hourse PaaS ~ spaas

• Introduc0on

• Architecture

• Future

Structure of an organiza/on

About SRE in SmartNews• Team members: 2

• @nobu666 / @takus

• Task

• Cross-sec7onal construc7on of the system of SmartNews

• Produc7vity improvement

• Ensure security

Available at: <https://cloudplatform.googleblog.com/2016/07/adventures-in-SRE-land-welcome-to-Google-Mission-Control.html> [Accessed 25/Jul/2016]

Examples of our works• Improve server provisioning flow

• Improve deployment flow

• Construct data pla8orm

• How SmartNews Built a Lambda Architecture on AWS to Analyze Customer Behavior and Recommend Content

• Construct in-house PaaS

in-housePaaS"spaas"

spaas

• SmartNews PaaS

• or Sakamoto PaaS !

• ECS thin wrapper w/Terraform

• A self-hosted PaaS for deploying docker-based applica@on

• Goals

• Easy to deploy

• Easy to operate

Architecture• Load Balancing + Service Discovery

• ELB + Nginx

• Consul + Consul Template

• Log Management

• Fluentd + S3

• Lambda + Kinesis Streams + ElasCcsearch (+ Kibana)

Container Scheduling• Cluster

• spaas-service

• developer's applica3on

• spaas-admin

• administrator's applica3on(oauth-proxy etc)

• consul

spaas-cli

Setup$ spaas-cli images:init --repository foo

Created xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo

You can push your image like this:$(aws --region us-east-1 ecr get-login)docker build -t foo .docker tag foo xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo:0.0.1docker push xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo:0.0.1

Create & config### create service$ spaas-cli create --service foo --image xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo --tag 0.0.1 --port 8080

### if you need public http access (https://foo.smartnews.com)$ spaas-cli config:set --service foo SERVICE_TAGS=web

### if you need internal http access (http://foo.smartnews.internal)$ spaas-cli config:set --service foo SERVICE_TAGS=web-internal

Deployment & rollback$ docker tag foo xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo:0.0.2$ docker push xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo:0.0.2$ spaas-cli deploy --service foo 0.0.2

$ spaas-cli rollback --service foo

Canary deployment# create canary service$ spaas-cli create --service foo-canary --image xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/foo --tag 0.0.2 --port 8080$ spaas-cli config:set --service foo-canary YOUR_ENV=foo

# register canary service as production service$ spaas-cli config:set --service foo-canary SERVICE_NAME=foo SERVICE_TAGS=web

# deploy to production service$ spaas-cli deploy --service foo 0.0.2

# deregister canary$ spaas-cli services:destroy --service foo-canary

Future

• Support deployment pipeline

• Like a spinnaker

• Support priority of Task Definition

• For maintenance / container scheduling

• Support Terminate Hook

• For completely log preserva;on

• Support dedicated host mode

• Support CDN integra;on

Talk more?

• Join our free lunch in Tokyo office !

• Ask me later "

We're hiring!Only two people on Site Reliability Engineering Team !

• スマニューのSite Reliability Engineer募集！

• h1p://about.smartnews.com/en/careers/