Upload
amazon-web-services
View
109
Download
1
Tags:
Embed Size (px)
DESCRIPTION
For Killzone Mercenary and Killzone ShadowFall, Guerrilla Games and SCE Online Technology Group switched from an inflexible, classic 3-tier architecture to one using well understood tools and AWS technologies. This switch allowed them to deliver more title-specific features without losing the benefits of sharing code between titles easily. This session covers problems in the previous architectures, how they fundamentally changed how they built game servers, some of the problems they faced while rebuilding from the ground up, and rabbit holes they got stuck down—both generally, and specific to their use of AWS Elastic Beanstalk, Amazon DynamoDB, and other AWS services. They show how they were able to react quicker to the changing needs of the title, make last-minute feature changes, whip up servers at a moment's notice, and switch out implementations when a new AWS feature came along that would deliver performance or cost improvements.
Citation preview
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
(re)Building Killzone’s Servers How We Used AWS in Killzone: Shadow Fall and
Killzone: Mercenary
Tim Darby, Sony Computer Entertainment Europe
November 14, 2013
Bio
Aims
What is Killzone?
A brief history of Killzone’s servers
Killzone • Serving core multiplayer
experience
• Centrally developed backend tech
• No title-specific backend code
• C/C++
• Regional real-time gaming servers
• Centralized core servers
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
KZ : Liberation • Original real time backend plus …
• Java 3-tier backend for community features
• Shared tech again
• Customization at presentation layer, plus configuration
• Plugins for score processing and tournament progression in ‘singleton’
• Leaderboards sat on the DB
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
Killzone 2 • Applying lessons learned from Killzone :
Liberation
• Materialized views added to leaderboard backend
• Battle Replay using shared file system (NFS)
• Plugins in ‘singleton’ servers for Tournament progression
• Additional webapp delivering XML for website
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
Killzone 3 • Iteration on previous backend
(java and C++)
• Redone leaderboards from Materialized Views to MV + TSDB
• Plugins to middle tier
• C++ side matchmaking server with title-specific MM rules plugin
• And it wasn’t just us feeling the constraints…
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
A Busy Year
September 2013 November 2013
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
Re-arch Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
• Separately versioned
shared components
• Independent game
projects
• Maven + Jenkins
• Dynamic Scaling
• Use of AWS services
Mercenary • Relatively straightforward REST API
• Frontend, Scheduler, Admin Elastic Beanstalk applications
• Multiple Beanstalk Environments per app (staging, or dev envs)
• Both “SingleInstance” and “ELB” Beanstalk Env types used
• PSN functionality used for real-time elements
– Matching, NAT Traversal, P2P
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
ShadowFall • Straight REST, with addition of
Comet style push messaging
• Similar topology for Elastic
Beanstalk elements to KZM
• Addition of real-time servers
• Hybrid deployment – part Sony
hosting, part AWS
Killzone
(2004)
Killzone : Liberation
(2006)
Killzone 2
(2009)
Killzone 3
(2011)
Killzone : Mercenary
Killzone : ShadowFall
(2013)
Use of AWS Services
Scheduled Jobs • Scheduler webapp simple spring-quartz
• Generates jobs, posts to Amazon SQS
• Main webapp runs SQS receivers
• Scheduled job receivers spawn worker threads per job
• Wrappers around these jobs provide – Amazon CloudWatch duration/error tracking
– Distributed job management
• Admin app provides ability to manually schedule/fire jobs, and view distributed job state info
Score Posting #1 • Score posts pushed to SQS for async
processing
• Listeners on component for game
history store triggered on successful
write to Amazon DynamoDB
• Individual listeners perform many
secondary tasks
• Failure modes in secondary tasks not
easily recovered
Score Posting #2 • Changed game history listeners to
post to secondary game jobs SQS
queue
• Added separate receiver threads for
secondary game jobs SQS queue
messages
• Receivers then run secondary jobs
• Failure modes in secondary jobs now
result in non-confirmation of SQS
message, and re-presentation by SQS
Leaderboards • SimpleDB, transitioned to DynamoDB
when available
• DynamoDB required secondary indices in some form for multiple sort columns on a single board
• Single Table vs. Multiple Table implementations
• Fast ranking a problem for both
• Fast ranking provided by reverse index rank calculator or statistical predictor
Issues and Lessons Learned
Issues • Old-style monitoring of dynamically
scaled environments
• Manual scaling management on DynamoDB
• DynamoDB, leaderboards, and hotspots
• Surprise! Changes to console
• Limits of AWS CloudFormation
Lessons Learned • Appreciate the change in mindset going
from SQL to NoSQL
• Swappable implementations generate
flexibility
• Flexibility around architecture adds
options
• Dynamic scaling/management doesn’t
necessarily stop at server boxes
• Still no silver bullet for leaderboards
Summary
Questions?
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
MBL305