Evoloution of Ideas

EVOLUTIONJesper Richter-Reichhelm (@jrirei)

BackendClient

BackendClient

BackendClient

BackendClient

Game Evolution

Iterative improvements

Wooga team

FE Dev

BE Dev

Art

Product

“You build it, you run it. - Werner Vogels

BackendClient

Monster World

SQL

app

lb

SQL

Browser

sql sql

appapp

Things we did

Many small optimisations

Bigger databases…

… many, many more of them

app

slave

master master

slave

1. create new DBs

2. setup replication

3. start using masters

4. cut replication

5. truncate data

BA

AB

AB

AB

AB

app

slave

master master

slave slave

master master

slave

AC

AC

BD

BD

AC

AC

BD

BD

Stateless approach

1+ million users / day

10+ billion requests / month

~10 DB writes / request

> 100,000 DB writes / second

http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

Things we did

Many small optimisations

Bigger databases…

… many, many more of them

… and finally the right thing!

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

app appapp app app app app app app app app appapp app app







SQL

EBS EBS EBS EBS

lb lblblb

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd








SQL

EBS EBS EBS EBS

lb lblblb

38 single points of failure

flickr.com/photos/qubodup/9244593352

http://flickr.com/photos/photomonkey/12746989

Why did we survive?

Dev Ops

Tech Evolution

Spreading and mutating ideas

Wooga teams

Freedom

Magic Land

Server Database

Server Database

One Game Session

Stateful approach

1+ million users / day

4 sessions / user / day

1 DB write / session

~50 DB writes / second

Server

sessionsessionsessionsession

Server


Server


S3

http://www.flickr.com/photos/aigle_dore/

http://www.flickr.com/photos/aigle_dore/

Tech Stack

2009

Ruby

LAMP

statelessBB BI

MW

Redis HH

LAMP

statelessBB BI

Ruby MW MW 2

Ruby MW MW 2

2010

LAMP

statelessBB BI

Redis HH

Ruby MW MW 2

2010

Erlang

statefulML

2011

Redis

Ruby

LAMP

statelessBB BI

MW

HH

MW 2

Erlang

statefulML

offline

DD

Erlang

statefulML

2011

Redis

Ruby

Riak

Javascript

ev. Ruby

1 doc

LAMP

statelessBB BI

MW

HH

MW 2

PI

PP

MWM

2012

Erlang

Redis

Ruby

stateful

Riak

Javascript

offline

ev. Ruby

1 doc

JRuby

shared

LAMP

multiplay

statelessBB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

GX

FT

KB

BI 2

2013

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

…

2014

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

… …

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

… …

…

…

…2015

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services2015

MySQLPHP

PHP MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

PHP MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

Wooga’stoolbox

Why does that work?

Reproduction

flickr.com/groups/bestfavoriterelayrace/

Rotating jobs


Selection and recombination

Exchanging knowledge

flickr.com/photos/wili/233621595


Being transparent

flickr.com/photos/marcomagrini/698692268


Mutation

Freedom

Having trust

Evolution

Company Evolution

There is more…

Switch to Mobile

PI

MWM PV

Switch to Mobile HTML5

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

JS

GX

BI 2

DIV

HTML5

Obj. C

Apportable

ext. Android

BB BI

MW

HH

MW 2

ML

Switch to Mobile

DD

PI

PP

MWM PV

JS

GX

BI 2

DIV

HTML5

Obj. C

Apportable

ext. Android

SD / DYN

… …

…

…

…

AAUnity

BB BI

MW

HH

MW 2

ML

Switch to Mobile

Hit Filter

1,000 new mobile games per week

$-

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

1 3 5 7 9 11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45

47

49

51

53

55

57

59

61

63

65

67

69

71

73

75

77

79

81

83

85

87

89

91

93

95

97

99

… 50 1 2 3 4 5 6 7 8 9 10 … 100 …200

Top Grossing Rank

Top ranks reap most revenues

Estimated gross sales per day

Should have stopped earlier

Question Regularly

… …

…

…

…

GX

JS

DIV

AA

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

FT

KB

BI 2

Hit FilterSD / DYN

Hit Filter

What does that mean?

Evolution

…is the key for survival

Repeatedly create hits

Bubble island Monster World Diamond Dash Pearl’s Peril Jelly Splash

Dev Ops

Embrace Evolution!

Questions?

Jesper Richter-Reichhelm (@jrirei)

Questions?

Jesper Richter-Reichhelm (@jrirei)

Nothing is better than

Centralised decision making

Top down decisions

Building a monolith

… if you want maximum efficiency!

Wooga is different

Wooga wants

The ability to quickly adapt to change Survive disruptions and benefit from them

To attract strong people We give transparency and trust, we want impact

Resiliency and robustness Long-term perspective vs. short-term goals

<Reproduction>

Reproduction

<Selection and recombination>

Recombination

<Mutation>

Mutation

EvolutionReproduction Recombination Mutation

2009 New Business Model

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

… …

…

…

…New Business Model

IAP

ad game

2015 Embracing Android

… …

…

…

…

DD

PI

PP

MWM PV

JS

GX

BI 2

DIV

Embracing Android HTML5

Obj. C

Apportable

ext. Android

SDDYN

Unity

BB BI

MW

HH

MW 2

ML

Android

AA

Major milestones

2009 Business Model

2011 Mobile

2013 Hit Filter

2015 Embracing Android

Technology

Evoloution of Ideas