Saturday, 27th July, 2013

Introduction and Background

Sajit Vasudevan, Technical Architect, Cognizant

Technology Solutions

Nirmalya Sengupta, Software Craftsman and Coach,

CeeZone Consulting

Is to show that

Complex projects can be executed by

following Scrum

Adaptability to emerging architectural

needs is achievable

Adoption of Scrum mindset helps in

succeeding

Is to give some concrete examples

Overview

Project Background

Dutch company – Online lottery giant, decided to enter Online Poker market

Scope: Building a complete offering of online betting Poker game environment (first version in 12 months)

Technology choices: Java, C++, GWT,Test Scripting, Linux, MySQL, RabbitMQ, XUnit, Cruisecontrol, Artifactory, Maven, RedDwarf

Application: a bird’s eye view

Player Portal

Game Server

Lobby Server

Database

RabbitMQ

Tournament

Server

Poker Lobby – showing all table types, tournament types etc

Poker lobby – Tables list

Game table

Servers must be ...

Robust

Network ready

Support for ‘Statefulness’

Easy to interface with external systems

Highly threaded

Minimum supportable concurrent player base: ~20000

Logged in players don't always play but add load

Speed and Order of message exchange between Client

and Server are of essence

Crucial Challenges

Inherently asynchronous

Arbitrarily fired Timers affect software's behaviour

Players may decide to sit out and re-join anytime

Network connection with player can drop anytime: game

must continue

Randomness must be satisfactorily built in and proven

(legal requirement)

Game History must be retained (legal requirement)

Multiple simultaneous tables, Multiple simultaneous

tournaments

Example of asynchronous behaviour

P1

P2 P4

P3

step1. P2 plays

step2. P3 sits-out /

gets disconnected

NEW HAND!

step3. Hand over!

ASYNCHRONOUS

MESSAGE!

Now P3 is

informed of

new Hand

allowed to re-

join

Tricky to test software which..

is inherently asynchronous

is supposed to support many users, while

maintaining performance

is asking testers to be good/bad Poker players

is affected indeterminately by Timers (arbitrariness)

requires preservation of history of every move

is expected to display minimum latency always

`

Infrastructure - revisited

Player Portal

Game Server

Lobby Server

Database

RabbitMQ

Tournament

Server

.

.

.

.

.

.

.

.

. . .

.

BOTs

Human Players

JMX Console

Some initial Architectural decisions Game logic implemented as a handcrafted FSM

Game Server separate from Tournament Server

All components inside a Game/Tournament Server

communicate through messages

{ Game | Tournament | Lobby } server exchange

information through Database

Clients ‘pull’ information from Lobby Server

Poker Lobby – showing all table types, tournament types etc

Problems popped up: Pig headed or headless chicken

Understanding of tournament server was clearer over

time

Performance bottleneck:

Identified earlier that lobby would be heavily loaded

Clients ‘pulling’ information from Lobby Server was a

bottleneck

Exchanging information through Database was

difficult

Plugging in 3rd party authentication mechanism was not

straightforward

Architectural adaptations led to

Replacing Game Server with Tournament Server

We realised that every game played can be considered to be a

part of ‘default’ tournament with no competition

Game/Tournament and Lobby Servers decoupling

through a Queue

Lobby Server keeps its own database

Lobby Server gets latest update from queue

Previous updates can be forgotten – saves memory and

processing

Some architectural changes made..

Replaced Game Server with Tournament Server

We realised that every game played can be considered to be a

part of ‘default’ tournament with no competition

Game/Tournament and Lobby Servers decoupled

through a Queue

Lobby Server keeps its own database

Lobby Server gets latest update from queue

Previous updates can be forgotten – saves memory and

processing

Building functionality

Significant focus on unit testing

Regular and frequent builds

By not having adequate automated tests, we paid the

price in

Incorrectly calculating how to divide money at

stake, amongst players

Incorrectly recording the HandHistory of games

(both of the above, legal requirements and hence,

extremely important)

Debugging effort and cost was huge

Our Scrum experience reminds us that

Collaboration between product management,

developers, testers and architect is crucial

Changes from the product management could be

accommodated at any point of time

Architectural alterations can be made to the initial

architecture

Brickbats