Scaling wix with microservices and multi cloud - 2015

@aviranm

Aviran MordoHead of Engineering

@aviranm

linkedin.com/in/aviran

aviransplace.com

Scaling with Microservices Architecture & Multi-cloud platforms

@aviranm

@aviranm

Wix in NumbersOver 72M users (website builders)

Static storage is >2PB of data

3 data centers + 3 clouds (Google, Amazon, Azure)

2B HTTP requests/day

1000 people work at Wix

@aviranm

Initial Architecture

Built for fast developmentStateful login (Tomcat session), Ehcache, file uploadsNo consideration for performance, scalability and testingIntended for short-term use

Tomcat, Hibernate, custom web framework

Lighttpd(file

serving) MySQLDB

Wix(Tomcat)

@aviranm

The Monolithic GiantOne monolithic server that handled everything

Dependency between features

Changes in unrelated areas of the system caused deployment of the whole system

Failure in unrelated areas will cause system wide downtime

@aviranm

Breaking the System Apart

@aviranm

Concerns and SLA

Data Validation

Security / Authentication

Data consistency

Lots of data

Edit websites

High availability

High performance

Lots of static files

Very high traffic volume

Viewport optimization

Long tail (immutable)

Serving Media

High availability

High performance

High traffic volume

Long tail (mutable)

View sites, created by Wix editor

@aviranm

Wix Segmentation

1. Editor Segment 3. Public Segment2. Media Segment

Networking

@aviranm

HTML Editor

Flash Editor

MSM

Private Media

Public Media

Editor Segment Public Segment

Premium Services

eCommerse

List DB

App Builder

App Store

App Market

Dashboard

Statics/media

Mailer

TimeZone

Public HTML API

Public API (Flash)

MSP

Public Server

HTML Renderer

HTML SEO Renderer

Flash Renderer

Flash SEO Renderer

Sitemap Renderer

Robots.txt Renderer

User Server

Template Viewer

ContactsHUBActivit

y

Site Members

Provided Mailing Service

Comments

Snapshoter

User Pref

Feed Me

Shout-out Hotels

PETRI

Site Pref

Dist LoggerSlicer

eCom Renderer

eCom Cart

eCom Checkout

eCom Catalog

eCom Orders

Payment Facade

Account Info

HTML API

HTML Embeder

BlogMobile

@aviranm

TRADE-OFFIt is all about

@aviranm

Microservices GuidelinesEach service has its own DB schema (if one is needed)

Only one service should write to a specific DB table(s)

There may be additional read-only services that directly accesses the DB (for performance reasons)

Services are stateless

No DB transactions

Cache is not a building block, but an optimization

@aviranm

Microservices TradeoffsEach service has its own DB schema (if one is needed)

Gain - Easy to scale microservices based on service level concerns Tradeoff – system complexity, performance

Only one service should write to a specific DB table(s)Gain - Decoupling architecture – faster development Tradeoff – system complexity / performance

May have additional read-only services that accesses the DBGain - Performance gainTradeoff - coupling

Services are statelessGain - Easy to scale out (just add more servers)Tradeoff - performance / consistency

No DB transactionsGain - Better DB performance, easier to scaleTradeoff - system complexity

@aviranm

1. Editor Segment

@aviranm

Editor ServerImmutable JSON pages (~3M / day)

Site revisions

Active – standby MySQL cross datacenters

Editor Server

MySQL Active Sites

MySQL Archiv

e

@aviranm

Find Your Critical Path

@aviranm

Protect The DataDB outage with fast recovery = replication

Data poisoning/corruption = revisions / backup

Make the data available at all times = data distribution to multiple locations / providers

@aviranm

Browser

Editor Server

GCS

MySQL Active Sites

MySQL Archiv

e

Saving Editor DataWixMedia(Amazon)

WixMedia(Google)

Save Page(s)

200 OKUpload

Save Page

DC replication

Notify

MySQL Archiv

e

MySQL Active Sites

S3

WixMedia(DC-1)

@aviranm

Browser

Editor Server

GCS

MySQL Active Sites

MySQL Archiv

e

WixMedia(Amazon)

WixMedia(Google)

Save Page(s)

200 OKUpload

Save Page

DC replication

Notify

MySQL Archiv

e

MySQL Active Sites

S3

WixMedia(DC-1)

Self Healing Process

@aviranm

No DB TransactionsSave each page (JSON) as an atomic operation

Page ID is a content based hash (immutable/idempotent)

Finalize transaction by sending site header (list of pages)

Can generate orphaned pages, not a problem in practice

@aviranm

2. Media Segment (WixMP)

@aviranm

Wix Media Platform (WixMP)

Eventual consistent distributed file system(2PB user media files)

Dynamic media processing

Multi datacenter aware

Automatic fallback cross DC

Run on commodity servers & cloud

@aviranm

T

Google Cloud

Prospero – Wix Media Manager

get image.jpg

First fallback

Second

fallback

If not in

CDN

Amazon

x36Tx36

Tx32

Austin

CDN

@aviranm

3. Public Segment

@aviranm

Public Segment RolesRouting (resolve URLs)

Dispatching (to a renderer)

Rendering (HTML,XML,TXT) Public

Server

HTML Rendere

r

HTML SEO

Renderer

Flash Renderer

Sitemap Rendere

r

Robots.txt

Renderer

www.example.com

Flash SEO

Renderer

@aviranm

Public SLAOur goal: 99% response time <100ms at peak traffic

@aviranm

Publish SitePublish site header (a map of pages for a site)

Publish routing table

Publish site header / routes (CQRS)Editor Segment Public Segment

@aviranm

Built For SpeedMinimize out-of-process hops (2 DB, 1 RPC)

Lookup tables are cached in memory, updated every few minutes

Denormalized data – optimize for read by primary key (MySQL)

Minimize business logic

@aviranm

How a Page Gets RenderedBootstrap HTML template that contains only data

Only JavaScript imports

JSON data (site-header + dynamic data)

No “real” HTML view

@aviranm

Offload rendering work to the browser

@aviranm

The average Intel Core i750 can push up to 7 GFLOPS without overclocking

@aviranm

Why JSON?Easy to parse in JavaScript and Java/Scala

Fairly compact text format

Highly compressible (5:1 even for small payloads)

Easy to fix rendering bugs and cross browsers issues (just deploy a new code)

@aviranm

Minimum Number of Public Servers Needed to Serve 66M Sites

4

@aviranm

Public SLABe Available 99.999%

@aviranm

Serving a Site – Sunny Day

Archive

CDN WixMP

Browserhttp://

example.wix.com

Store HTML to cache

HTTP Request

Notify site view

LB

Public

Renderer

HTML

Resources / Media

HTTP Request

Failure Points

@aviranm

Serving a Site – DC Lost

Archive

CDN WixMP

Browserhttp://

example.wix.com

LB

Public

Renderer

LB

Public

RendererChange DNS

HTTP Request

@aviranm

Serving a Site – Public Lost

Archive

Browserhttp://

example.wix.com

LB

Public

Renderer

Get Cached HTML Version

HTMLHTTP Request

LB

Public

Renderer

Fallback to 2nd DC

@aviranm

Living in the BrowserCDN WixM

P

Browserhttp://

example.wix.com

LB

Public

Renderer

Editor Pages

Fallback

JSON / Media

HTMLHTTP Request Fallbac

k

@aviranm

SummaryIdentify concerns and SLA for different parts of the system

Build redundancy in critical path (for availability)

De-normalize data (for performance)

Minimize out-of-process hops (for performance)

Take advantage of client’s CPU power

@aviranm

@aviranm

http://engineering.wix.com

http://goo.gl/THbU2K@WixEng

We’re hiringwww.wix.com/jobs

Q&A