How we realized SOA by Python at PyCon JP 2015

How we realize SOA by PythonIn PyCon JP 2015

#PyConJP_C

SOA meaning Service Oriented Architecture

What's SOA

SOA is an approach to create a system

based on small servers separated for small

functions.

SOA

WebAppSearch

Auth

Other

Images

https://www.nginx.com/blog/building-microservices-using-an-api-gateway/

https://www.nginx.com/blog/building-microservices-using-an-api-gateway/

So... is the SOA comfy?

• Yes.

• But... no silver bullet.

Pros

• Separated responsibilities

• Rapid integration

• Flexible scaling

Cons

• Lots of servers

●To create

●To deploy

●To monitor

Can't realize SOA with legacy style

• Direct transmitting

• Manual deploying

• Manual testing

So how we realize SOA

Today you can learn

• Case study of SOA by Python

• Practices to manage lots of services

Today you can't lean

• Step by step guide

• So much about Python's core

Agenda

About Our case Summary

Architecture Creating MonitoringTesting Deploying

Who I am

• Hiroki Kiyohara (hirokiky)

• BePROUD Inc

• Developer, Consultant, Trainer

• Admin of djangoproject.jp

Who we are

• News paper company

●Over 400,000 paid user (Web)

• Migrating to in-house production

Our case

Architecture

Architecture

Architecture

REST API

Architecture Separated from Web

API Gateway

• Proxy for back-end servers

• Handling permissions

Creating

Creating

• Need to create many services

• Share docs

Django

• Easy to create

• Many libraries

• Community and knowledge

DjangoRestFramework

• Framework on Django to create REST API

• Useful modules

• Smart Auth/Authz framework

DRF: Serializer

• Django's Form for REST API

●Nested

●Array of objects

Serializer is really simple

Validating by serializer>>> from search.serializers import BulkPostSerializer>>> BulkPostSerializer(data={... "name": "hiroki",... "posts": [... {"title": "PyCon JP 2015",... "body": "This is awesome event"},... {"title": "How we realize SOA",... "body": "Python is awesome"},... ]... })>>> serializer.is_valid()True>>> serializer.validated_dataOrderedDict([('name', 'hiroki'), ('pos...

cookiecutter

• Template of repository

• Share best practices

. ├── README.md ├── project_name

│ ├── manage.py │ └── project_name │ ├── settings │ │ ├── __init__.py │ │ └── test.py │ ├── urls.py │ └── wsgi.py

├── .elasticbeanstalk/ ├── .coveragerc ├── .gitignore ├── circle.yml ├── Dockerfile ├── Dockerrun.aws.json ├── requirements.txt ├── setup.cfg └── tox.ini

OK so... created

• Many applications

• Share docs

django-rest-swagger

• API docs from docstring and code

• Demo-able API docs

django-rest-swagger

django-rest-swagger

Creating

• Django

• DjangoRestFramework

• cookiecutter

• tox

Testing

Testing

• Handy unit tests

• E2E tests

I know you won't run. MEE TOO

• Complex

• Slow

• Useless

Speed up by django's setting

• On memory sqlite

• Dummy or local cache

• Light hasher

Refer Two scoops of Django

Use nice tools

• tox

• flake8

• ...

tox

• Just type `tox`

• Run several tests in separated virtualenv

[tox]envlist = py34, flake8skipsdist = Truesetupdir = ./myprj/[testenv:py34]deps = coverage -rrequirements.txtsetenv = DJANGO_SETTINGS_MODULE = myprj.settings.testcommands = coverage erase coverage run myprj/manage.py test myprj coverage report[testenv:flake8]basepython = python3.4deps = flake8commands = flake8 myprj


Installing dependencies


Using settings for test


Running tests with coverage


Static analysis

Just...

$ tox

flake8

• Code style check

• Static analysis

flake8

• Don't review about syntax

• Remove tiny and messy code bugs

And more

• responses

• testfixtures

• factory-boy

Of cause, DO NOT

• Accessing outer services

• Setting middle wares locally

E2E testing

• Using requests

• Checking connection of each services

Locust.io

• Load tests by Python

• Distributed clients but aggregated report

Test

• Faster Django setting

• tox

• flake8

• locust.io

Deploying

Reduce costs to deploy

• Automated deploy by ElasticBeanstalk

• Master deploying from CI

Auto deploy

• Master branch is on development env

• Deploying from CI tool

Deploying

Deploying

Deploy It

Deploying

GREEN

Deploying

Push

Deploying

Deploy

Deploying

OK Pulling

Deploying

Deployed

Also from CLI or Web console

• Just type `$ eb deploy`

• Or through the Web

Deploy

• GitHub

• CircleCI

• Docker

• ElasticBeanstalk

Monitoring

A lots of servers to monitor

• Many services, lots servers

• Immutable infrastructure

Monitoring tools

Sentry

• Event (log) aggregation platform

• Manage console of each events

Sentry

• Many events but one notification

• Marked as `Resolved`, it'll come again

Fluentd

• Aggregating access log

• S3 and BigQuery

NewRelic

• Resource monitoring for containers

• And APM is awesome

NewRelic APM

• Analyzing Python application

• Which progress is slow?

Rundeck

• Job scheduler

• Better crond

Rundeck

Monitor

• Sentry

• Fluentd

• NewRelic

• Rundeck

So...

Will SOA be a silver bullet?

No

Pros

• Separated responsibilities

• Rapid integration

• Flexible scaling

But you need to do

• Easy creating

• Handy testing

• Auto deploying

• Relief monitoring

What should we do next?

• Non-blocking

• Nicer container platform

Thank you

• For listening

• For this nice event

Technology

How we realized SOA by Python at PyCon JP 2015