managing scale with lean team leveraging the power of docker and aws

About Postman

3+ million users

1.5+ million mau

30+ member team

Our Stack27+ micro services

10+ mill peak req/hr140+ gb in/daysailsjs on nodejs

dockermanaged by aws

feb 2017

the story of making small big decisions

postman started 2014 ⇢ small team ⇢ frugal operations

early goal was to ship and validate andspend less time on operations

we were already raking up 500k+ downloads of our chrome app by listening to users and

iterating

the sync serviceenabling api collaboration ⇢ SailsJS

at that time, this was the best framework to choose to go to market fastest

we used whatever minimum the framework needed to get started: one

server, mysql and redis

we were trying to validate that collaborative api development was a solution the world needs

time to choose1.5+ mil users ⇢ steady service adoption (growing

mau)

we knew we needed to make devops easier if we can get any actual development done

at all

walk the tightrope of solving short term problems while following the long term

vision

this meant, larger server, load balancer, constant mysql and redis performance tweaking

on came tedious (and manual) deployment, traffic spikes, hung processes for no apparent

reason

beanstalk + dockersolved all pressing problems ⇢ abstraction to be

flexible

choosing docker (even though we did not need it right away) allowed us to build the things that we expected we might need to (without actually building them at

that time)

we needed to auto scale during load and at that time, we felt aws could give us that quickly

application deployment, infrastructure creation, auto scaling and auto healing all in one package

rds + elasticcacheno more resizing disks ⇢ no more tuning

configurations

reliable and automated backups (and restore), encryption of data communication

and storage and hassle free replication

there could be a million points why rds might appear “limiting”, but we adopted our product development mantra into devops as well and…

near zero downtimehot reboot ⇢ live deployments ⇢ live rollback ⇢ alarms +

auto healing

no need to reboot entire server, just the docker image

build and deploy applications as a secondary image and then swap

we were prototyping faster than ever

fridays were back

developer prepares code and environment, thenci tests code using centrally inherited image

ci tested image is orchestrated into production services by beanstalk

POSTMAN STACK WAS BORN

simple stack workscommon code repository structure

code + tests + beanstalk extensions + dockerfile

we inherited from our own docker base image and that enabled us to control on base stack, easily permeate stack changes into all micro-services and test using production base image

on CI

goal was to make our first micro-service a fully portable concept with all info in code repository

we gave ourselves a 48 hour challengecan you release a new production quality service?

soon we reached 2+ million users and added many more services including api documentation service

and api monitoring service

nginx and beanstalk extensions on ec2

Docker

NodeJS + pm2

SailsJS + Hooks

orm + socket.io & express Business Logic

elastic load balancer

auto scaling group

and other AWS resources

managed by Beanstalk

in an onion shell

time to validate10x traffic ⇢ 5+ live deployments/day ⇢ difficult to

debug

centralised and auto-instrumented

loggingcloudwatch & elk with grafana ⇢ SailsJS hooks

easily added to all service via beanstalk extensions and

docker root image

since we made all services on SailsJS and as such one

hook works everywhere

auto-instrumented monitoring

cloudwatch ⇢ beanstalk enhanced health checks ⇢ beanstalk extensions for custom metrics

with beanstalk extensions, one can customise just about anything and we auto added all our alarms,

instance CPU, memory, disk and event-loop monitoring in no time

grafana did all the reporting for us

dogfooding

Use newman (Postman CLI) for APItesting on CI platforms

Use own monitoring service toperform complex api health checks

Use own API documentation serviceto collaborate micro-service development

up nextcentralised service discovery ⇢ decentralised

message bus ⇢ aws inspector + aws waf + aws shield

in short

Solve short-term problems while accounting for long-term applications

Use containerisation or some other way to find a common entry point and

standardisation of all your services

Don’t spend time solving for operational issues that you might never face

@shamasisgetpostman.com