Continuous Integration & Continuous Delivery with GCP

Continuous Integration & Continuous Delivery with GCP

https://www.facebook.com/groups/GCPUG.TW/

https://plus.google.com/u/0/communities/116100913832589966421

Google Cloud Platform User Group Taiwan我們是Google Cloud Platform Taiwan User Group。在Google雲端服務在台灣地區展露頭角之後，

有許多新的服務、新的知識、新的創意，歡迎大家一起分享，一起了解 Google雲端服務...

GCPUG透過網際網路串聯喜好Google Cloud的使用者，分享與交流使用 GCP的點滴鑑驗。如果您

是Google Cloud Platform的初學者，您應該來聽聽前輩們的使用經驗；如果您是 Google Cloud Platform的Expert，您應該來分享一下寶貴的經驗，並與更多高手互相交流；如果您還沒開始用

Google Cloud Platform，那麼您應該馬上來聽聽我們是怎麼使用 Google Cloud的！

Hello!I am Cage Chung

I am here because I like to share my experiences.

You can find me at:http://kaichu.io

Outline

◎ App Engine Development

◎ Devops: CI & CD (Docker, gitlab, GCP)

◎ Tips & Study information

1.App Engine DevelopmentLet’s start with the first set of slides

Who is using GAE?

Angry Birds

Khan Academy

Royal Wedding

Snapchat

EA Games

Forbes

Bestbuy

https://cloud.google.com/customers/ Appengine

Google Cloud Platform | Compute

PaasIaaS

ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking

ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking

Saas

ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking

Infrastructure as a Service Platform as a Service Service as a Service

You manage Google manage

GAE Architecture

image: https://cloud.google.com/solutions/architecture/webapp

App Engine APIs

◎ Storage◎ Cloud Storage◎ Cloud SQL◎ Cloud Datastore◎ Memcache◎ Task Queue◎ Cloud Endpoints

Find more https://cloud.google.com/appengine/

App Engine runtime

Python2.7 Java7 Php Go

Ruby Node.js

standard

Flexible Beta

Python2.7/3.4 Java8 Go custom

[Google App Engine Documentation - App Engine — Google Cloud Platform](https://cloud.google.com/appengine/docs)

App Engine | waldo

queue

MetadataWaldo Server

App Engine

Task Queues

Cloud Storage

Optimal-waypoints

managed VMs(python27)

...

endpoints API(OAuth 2)

Object Notific

ation

CloudeDatastore

snapshot

managed VMs(nodejs)

Android App

CSV

upload waypoints

Store/retrieve metadata

waypoint map

snapshot

optimal road trip

RESTful APIs for waypoints list, upload

waypoints

standard

Flexible Beta

[Waldo-gcp](http://www.slideshare.net/cagechung/waldogcp)

2.Devops: CI & CD (Docker, gitlab, GCP)Let’s start with the second set of slides

Devops

DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.[1][2] It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably - DevOps - Wikipedia

[image](https://goo.gl/pMC6UZ)

Devops: CI & CD (Docker, gitlab, GCP)

[image](https://goo.gl/pMC6UZ)

Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops-continuous-integration-delivery-docker-circleci-aws-beanstalk.html)

Simple flow

◎ Python (gae-flask) ○ Local Development○ Local Test

◎ Gitlab◎ Code Review with GitHub(Gitlab) Flow◎ gitlab-ci-multi-runner

○ Local runner exec test docker○ Gitlab runner test docker

◎ Google App Engine○ Google Cloud Source Repositories○ Stackdriver Debugger

Local Development / Test | gae-flask

// clone repo

$ git clone git@gitlab.com:cage1016/gae-flask.git

// make a new virtualenv

$ mkvirtualenv gae-flask

// pip install requirements.test.txt or link env as lib folder

(gae-flask-gitlab-ci) $ pip install -r requirements.testing.txt -t lib

// or

(gae-flask-gitlab-ci) $ pip install git+https://github.com/ze-phyr-us/linkenv.git

(gae-flask-gitlab-ci) $ linkenv env/lib/python2.7/site-packages lib

// local run app engine project

(gae-flask-gitlab-ci) $ dev_appserver.py app.yaml

// local run test

(gae-flask-gitlab-ci) $ python ./runtests.py

Local Development / mian.py

import flask

import requests

app = flask.Flask(__name__)

@app.route("/")

def hello():

return "hello world"

if __name__ == "__main__":

app.run()

Local Development / test_main.py

import pytest

import main

@pytest.fixture

def app():

return main.app

@pytest.fixture

def test_client(app):

return app.test_client()

def test_hello(test_client):

response = test_client.get("/")

assert response.data.decode("utf-8") == "hello world"

Gitlab Project Runner

https://gitlab.com/cage1016/gae-flask/runners

Gitlab Project Variables

gitlab-ci.yml

image: cage1016/gitlab-ci-gcp:v1.5

before_script:

- export CLOUDSDK_CORE_DISABLE_PROMPTS=1

- export CLOUDSDK_PYTHON_SITEPACKAGES=1

- export GCP_PROJECT=gae-lab-001

- export GCP_PROJECT_VERSION=uat

types:

- test

- deploy

test:

stage: test

script:

- sh ./scripts/tests.sh

deploy:

stage: deploy

script:

- sh ./scripts/deploy.sh

only:

master[GitLab Documentation](http://doc.gitlab.com/ee/ci/yaml/README.html#only-and-except)

gitlab-ci.yml (tests.sh)

#!/usr/bin/env bash

virtualenv env

source env/bin/activate

# install test env packages from requirements.testing.txt

pip install -r requirements.testing.txt -t lib

# link env as lib folder if you just execute `pip install -r requirements.testing.txt`

# pip install git+https://github.com/ze-phyr-us/linkenv.git

# linkenv env/lib/python2.7/site-packages lib

# run tests

python runtests.py

gitlab-ci.yml (deploy.sh)

#!/usr/bin/env bash

pip install -r requirements.txt -t lib/

echo $GCLOUD_KEY > key.json

gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json

gcloud --quiet config set project $GCP_PROJECT

gcloud --quiet preview app deploy app.yaml --no-promote --version $GCP_PROJECT_VERSION

Local runner exec test docker

// [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4)

// list and register gitlab-ci-runner

$ gitlab-ci-multi-runner list

Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml

gae-flask-gitlab-ci Executor=docker Token=95d56b26104856b9f3852f2d82b1e5 URL=https://gitlab.com/ci

// prepare a docker-machine for gitlab-ci-multi-runner

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS

runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0

// run a gitlab-ci-multi-runner test via docker

$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY

GCLOUD_ACCOUNT=$GCLOUD_ACCOUNT GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"

// push git

(gae-flask-gitlab-ci) $ git push master

Docker cage1016/gitlab-ci-gcp:v1.5

FROM buildpack-deps:jessie

# remove several traces of debian python

RUN apt-get purge -y python.*

RUN apt-get update && apt-get install -y -qq --no-install-recommends unzip openssh-client python-openssl

# install "virtualenv", since the vast majority of users of this image will want it

RUN pip install --no-cache-dir virtualenv

# Install the Google Cloud SDK.

# archive gcloud sdk https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-105.0.0-

linux-x86_64.tar.gz

ENV HOME /

ENV CLOUDSDK_PYTHON_SITEPACKAGES 1

RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-102.0.0-linux-x86_64.

tar.gz && tar -zxvf google-cloud-sdk-102.0.0-linux-x86_64.tar.gz && rm google-cloud-sdk-102.0.0-linux-

x86_64.tar.gz

RUN google-cloud-sdk/install.sh --usage-reporting=true --path-update=true --bash-completion=true --rc-

path=/.bashrc --additional-components app-engine-java app-engine-python app kubectl alpha beta

...

https://github.com/cage1016/gitlab-ci-gcp/blob/master/Dockerfile

Gitlab Build Status

https://gitlab.com/cage1016/gae-flask/builds

Gitlab Build detail

https://gitlab.com/cage1016/gae-flask/builds/1321684

Push a new branch to Gitlab

https://gitlab.com/cage1016/gae-flask/builds

Google Cloud Source Repositories

[/ - GAE Labs](https://goo.gl/c99Dz8)

Google Cloud Source Repositories repo

$ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default

// generates a file named source-context.json, which contains information about the version of the

source code used to build the application

$ gcloud preview app gen-repo-info-file

// push code to google cloud source repositories

$ git push google master[/ - GAE Labs](https://goo.gl/c99Dz8)

Stackdriver Debugger | Source Code

[Debug - GAE Labs](https://goo.gl/I3K6J2)

Stackdriver Debugger | Debug

[Debug - GAE Labs](https://goo.gl/I3K6J2)

3.Tips & Study informationLet’s start with the third set of slides

“5G Google cloud storage for App

engine for FREE

<project-id>.appspot.comstaging.<project-id>.appspot.com

“App Engine Managed VMs is

renamed to App Engine flexible environment. March 24, 2016

[Python Release Notes - Python — Google Cloud Platform](https://cloud.google.com/appengine/docs/python/release-notes#1.9.35)

“App Engine Admin API

Cloud Source Repositories API Google Cloud Storage

“At the moment, the Go app engine SDK is still based on Go 1.4, even if you use the go1.6beta version for

your app. This means that the SDK doesn't understand 1.6-style vendoring, and so doesn't correctly upload the vendored dependencies.

This will be fixed when 1.6 becomes the stable (go1) version and is used as the basis of the SDK. Stay

tuned!

[gcpug.tw.go/.travis.yml at master · gcpug-tw/gcpug.tw.go](https://github.com/gcpug-tw/gcpug.tw.go/blob/master/.travis.yml)[Vendorization for deployment - Google Groups](https://groups.google.com/forum/#!topic/google-appengine-go/umcPV_4tAm8)

Study information

◎ Gitlab○ [Quick Start](http://doc.gitlab.com/ce/ci/quick_start/README.

html)○ [Configuration of your builds with .gitlab-ci.yml](http://doc.gitlab.

com/ee/ci/yaml/README.html)○ [GitLab.org / gitlab-ci-multi-runner · GitLab](https://gitlab.

com/gitlab-org/gitlab-ci-multi-runner)◎ Deploying to App Engine or Managed VMs from Travis CI

○ [Deploying to App Engine or Managed VMs from Travis CI - YouTube](https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu.be)

○ [GoogleCloudPlatform/continuous-deployment-demo](https://github.com/GoogleCloudPlatform/continuous-deployment-demo)

Study information cont.

◎ GCP○ [Cloud Source Repositories - Private Git Repositories — Google

Cloud Platform](https://cloud.google.com/source-repositories/)○ [Stackdriver Debugger - Production Debugging - Google Cloud

Platform — Google Cloud Platform](https://cloud.google.com/debugger/)

○ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed-vms-lab/)

Thanks!Any questions?

You can find me at:http://kaichu.iocage.chung@gmail.com