1  

Game Studio Perforce Architecture

Ryan Mensching Senior Network Administrator NetherRealm Studios/WB Games Logo area

2  

The Setup

•  Established Studio with track record •  Mortal Kombat (1992 – 2011) •  DC Comics – Injustice (2013), Arkham City Lockdown

•  AAA Games on multiple platforms •  360 •  PS3 •  WiiU •  Vita •  iOS

3  

Historical Perspective

•  Team has been making games for 20 plus years. •  “Things will get done.” •  Outages cost money, ship dates, and sanity. •  “Crunch” is a reality. •  Multi-Year development cycle, two years •  Used other tools in the past, Source Safe

4  

Statistics

•  Games are Code and Binary Art Assets •  Textures •  Animations •  Cinematics •  Lighting

•  Sync of current project at head is 20G •  Sync of source art assets is 545G

5  

Visuals

•  460+ CL’s per day at peek intervals (1000’s of builds) •  Some 20G of changes per day at peek intervals

6  

Products

•  As a studio we have 3 software products that are being produced at any given time. •  Engine •  Games •  Art Tool/Game Editor

7  

Challenges

•  As IT we had several hurdles to overcome •  Aging Hardware •  Aging processes •  Management woes •  Insight into systems •  Lack of agility to meet business demand

8  

Strategic Goals

9  

Strategic Goals

•  Game development support is a balancing of immediate gratification and resources.

•  IT business driven goals/pillars •  Space •  Speed •  Flexibility •  Reliability •  Management

10  

Space

•  Available, can’t be used if its locked up, i.e. thin provisioning

•  Abstraction, array layer should not need configuration to allocate space and at volume layer should not require reconfig for volume expansion.

11  

Speed

•  Entire VM platform, OS, and mass storage. •  Performance needs to be maintained in relation

to consumption.

12  

Flexibility

•  Works in conjunction with other pillars. •  Stands alone as a goal in games environment. •  Freedom of protocol choices. •  Ease of VM and hardware additions.

13  

Reliability

•  Redundancy should be inherent. •  Should not require special plans or procedures. •  If the reliability path is to complex issues will

arise.

14  

Management

•  IT Glue •  Proper management makes all the pillars come

together. •  Self documenting

15  

Solutions How as an IT organization do we accomplish these goals and vision?

16  

Solutions

•  Solid Layer 2 and 3 core with 10G distribution.

•  Previous experience with VM, Storage over IP and infrastructure testing.

17  

Research

•  Built VM farm with existing hardware and repurposed array for iSCSI.

•  Benchmarks were in the top on the Perforce Benchmark DB.

•  Ran real world project on systems.

18  

19  

Storage Platform

•  Key building block of infrastructure design. •  iSCSI management proved not agile enough,

LUN management chores. Redundancy requires complex configurations.

•  NFS provided equal or better performance.

20  

NFS

•  True thin provisioning, space consumed and given back.

•  No LUN changes. •  Volumes size can be adjusted on the fly. •  Strong VMware support. •  NetApp engine

•  Snapshots •  Native NFS among other protocols

•  Key to P4 revision file storage design.

21  

Virtualization

•  VM hardware needs to be “wide and fast”. •  As much RAM as possible to improve VM operations

and better disk caching. •  Cores should have highest clock speed possible. •  Perforce uses 2 vCPUs. •  Cisco UCS

•  Hardware abstraction •  Native Unified Fabric, 10G •  VM Passthrough

22  

OS

•  RHEL for all Perforce servers and many applications.

•  Linux provides best usage of resources for VM’s and disk caching.

•  Current project Perforce servers have 32GB of RAM.

•  Able to achieve 440+Mb sustained transfer to p4 clients.

23  

Reliability

•  Redundancy at array level with NetApp SnapMirror. •  Volume and VM level snapshots.

•  Provides ability to single file restore directly back to depots/servers.

•  UCS blades multi-homed to core; dual chassis uplinks, dual fabric uplinks, dual core switches.

•  VMware HA clustering •  Perforce uses proxies for load balancing and

checkpoints for integrity. •  Moving to replicas.

24  

Management Tools

•  NetApp OnCommand •  UCS Manager •  VMware

•  Plugins, SMVI

•  Active monitoring of P4 services and disk health.

25  

26  

Conclusion

•  Perforce is very capable of scaling to speed and data intense development.

•  Requires investments in engineering and infrastructure.

•  Each environment is different, but can share philosophies.

•  Abstraction, abstraction, abstraction

27  

Thank you Questions? Ryan Mensching rmensching@wbgames.com