78
ARCHITECTURE EVOLUTION AT WOOGA How does your company learn new things? Jesper RichterReichhelm, @jrirei

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

  • Upload
    wooga

  • View
    4.130

  • Download
    3

Embed Size (px)

Citation preview

Page 1: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

ARCHITECTURE  EVOLUTIONAT  WOOGA

How  does  your  company  learn  new  things?

Jesper  Richter-­‐Reichhelm,  @jrirei

Page 2: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Page 3: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Our  games  all  look  the  same

Flash  client Backend

Page 4: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

But  the  scale  is  interesFng

14  billion  requests  /  month

Page 5: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

But  the  scale  is  interesFng

14  billion  requests  /  month

Page 6: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

But  the  scale  is  interesFng

14  billion  requests  /  month

>100,000  DB  operaFons  /  second

Page 7: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

But  the  scale  is  interesFng

14  billion  requests  /  month

>100,000  DB  operaFons  /  second

>50,000  DB  updates  /  second

Page 8: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Architecture  EvoluFon  at  Wooga

The  Start

The  Next  Step

Best  of  Two  Worlds

Page 9: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

Page 10: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

Page 11: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

Page 12: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

A  basic  setup  using  sharding  worked  fine

app app app

lb

MySQL

MySQL

slave slave

Page 13: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

A  basic  setup  using  sharding  worked  fine

app app appapp app

lb

MySQL

MySQL

slave slave

Page 14: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

A  basic  setup  using  sharding  worked  fine

app app appapp appapp app app app

lb

MySQL

MySQL

slave slave

Page 15: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

250K  daily  users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life  was  good

Page 16: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

250K  daily  users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life  was  good NO  MORE

Page 17: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slaveslave

Page 18: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slave

MySQL

MySQL

slaveslave slave

Page 19: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slave

MySQL

MySQL

MySQL

MySQL

slave slaveslave slave

MySQL

MySQL

slaveslave slave

Page 20: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

At  500K  daily  users  we  were  at  a  dead  end

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Page 21: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

OUCH!

http://www.flickr.com/photos/billue_the_bear/

Page 22: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

MySQL

Page 23: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

MySQL Redis

Page 24: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

256  GB  data

10%  writes

MySQL Redis

Page 25: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

60  GB  data

50%  writes

256  GB  data

10%  writes

MySQL Redis

Page 26: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Redis  saved  the  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Page 27: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Redis  saved  the  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Page 28: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

We  now  have  more  than  2  million  users  /  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Page 29: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

We  now  have  more  than  2  million  users  /  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

AWS  servicedisrupFonin  Ireland

Page 30: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slaveslave slave

Page 31: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slave

MySQL

slave

MySQL

MySQL

slave slave slave slave

Page 32: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slave

redis redis

slave slave

redis

slave

MySQL

slave

MySQL

MySQL

slave slave slave slave

Page 33: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

OUCH!

http://www.flickr.com/photos/wolfsavard/

Page 34: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Architecture  EvoluFon  at  Wooga

The  Start:  Ruby  +  AutomaFon

The  Next  Step

Best  of  Two  Worlds

Page 35: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 36: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 37: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 38: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 39: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 40: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateless  servers  and  DBs

Server Database

Page 41: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

“Stateless  applica,on  servers  guarantee  one  thing:

Page 42: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

“Stateless  applica,on  servers  guarantee  one  thing:

The  data  is  never

where  you  need  it!”Paolo  Negri,  Developer  @  Wooga

Page 43: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  and  DBs

Server Database

Page 44: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  and  DBs

Server Database

Page 45: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  and  DBs

Server Database

Page 46: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  and  DBs

Server Database

One  Game  Session

Page 47: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  and  DBs

Server Database

One  Game  Session

Page 48: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Page 49: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Page 50: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Page 51: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

session

Page 52: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

sessionsessionsessionsession

Page 53: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

Page 54: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 55: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 56: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 57: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 58: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 59: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Page 60: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

S3

Page 61: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

With  stateful  server  the  DB  is  less  used

0

7,500

15,000

22,500

30,000

database  operations  /  sec

Ruby  Stateless Erlang  Stateful

Page 62: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

With  stateful  server  the  DB  is  less  used

0

7,500

15,000

22,500

30,000

database  operations  /  sec

Ruby  Stateless Erlang  Stateful

700

Page 63: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Deploying  with  a  stateful  server

In  order  to  bring  up  a  new  version

Page 64: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Deploying  with  a  stateful  server

In  order  to  bring  up  a  new  version

Just  deploy  it

Hot  code  replacement  is  great!

Page 65: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

Page 66: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

Very  few  SPOFs-­‐ ...  and  those  are  easy  to  recover

Page 67: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

Very  few  SPOFs-­‐ ...  and  those  are  easy  to  recover

TransacFonal  logic-­‐ Invariants  instead  of  explicit  error  handling

Page 68: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

NICE!

http://www.flickr.com/photos/aigle_dore/

Page 69: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Architecture  EvoluFon  at  Wooga

The  Start:  Ruby  +  AutomaFon

The  Next  Step:  Erlang  +  S3

Company  Values

Page 70: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

We’ve  learned  to  value

A  good  value  system

Page 71: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Small  teams                              over   Big  teams

We’ve  learned  to  value

A  good  value  system

Page 72: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Small  teams                              over   Big  teams

We’ve  learned  to  value

   

CollaboraFon                        over Compe??on

A  good  value  system

Page 73: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Small  teams                              over   Big  teams

We’ve  learned  to  value

 

Generalists                                over Specialists

   

CollaboraFon                        over Compe??on

A  good  value  system

Page 74: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Small  teams                              over   Big  teams

We’ve  learned  to  value

 

Generalists                                over Specialists

 

Effort  reducFon                over Cost  reduc?on

   

CollaboraFon                        over Compe??on

A  good  value  system

Page 75: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Small  teams                              over   Big  teams

We’ve  learned  to  value

 

Generalists                                over Specialists

 

InnovaFon                                  over Risk  mi?ga?on

 

Effort  reducFon                over Cost  reduc?on

   

CollaboraFon                        over Compe??on

A  good  value  system

Page 76: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

It works!

Page 77: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

It works!

Be fast, be bold!

Page 78: Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

QuesFons?

Jesper  Richter-­‐Reichhelm@jrirei

slideshare.net/woogawooga.com/jobs