Georg Zoeller, Lead Technical Designer, BioWare Austin Georg
Zoeller, Lead Technical Designer, BioWare Austin About Georg
Zoeller http://www.linkedin.com/in/georgzoeller
[email protected]
Slide 3
Once upon a time. in the Jade Empire QA department 7 player
characters with unique content and quests. 4 male, 3 female player
appearances. 10 dedicated story testers. 9 male and 1 female. Guess
which appearances rarely got used?
Slide 4
The Problem. Full Path coverage is hard to achieve, and even
harder to verify. ~800,000 words of dialog voiced by 144 voice
actors in English alone. 180+ area layouts. 3 classes, 3 races, 2
gender options. 10 companion characters. 6 unique, hour long Origin
Stories. 300+ abilities 300,000+ lines of script code. 3,000
Cutscenes, 5,500 animations, 20,000 models New IP, World, Rules,
Engine. Dragon Age: Origins QA resources are finite and sometimes
volatile. Planning and managing for full path coverage is very time
intensive.
Slide 5
Definition. n. The science and technology of automatic
measurement and transmission of data by wire, radio, or other means
from remote sources, as from space vehicles, to receiving stations
for recording and analysis. Game Development Telemetry automatic
measurement and transmission of data from game executable, build
pipeline and development tools for recording, analysis and workflow
improvement! not from space, yet Telemetry -
Slide 6
Same idea, different goals. Customer facingDeveloper facing
Understand how customers experience the game. Identify and react to
issues post launch. Understand the market, learn for future games.
Already quite popular. Understand how developers use tools and
pipeline. Identify and react to pipeline and workflow issues.
Measure stability, productivity and efficiency of development. Is
Business Intelligence.
Slide 7
Slide 8
All your data. Introducing SkyNet 2 nd Generation homegrown
telemetry system. Serving Dragon Age: Origins, Mass Effect2, SWTOR
On Dragon Age: Origins alone: 1,100,000+ games tracked on 1,141
different machines. Tracked any game of DA:O in the entire EA
network. 250 GB of data gathered in ~38,000,000 datapoints 50,000+
bugs, 4597 unique users on 582 machines. Lets have a look
Slide 9
Slide 10
Know who is running which build, language, executable. Know who
is running which build, language, executable. Drill down into any
running game Drill down into any running game
Slide 11
Chronological / spatial event visualization.
Slide 12
Aggregated spatial visualization (hmp)
Slide 13
Data Visualization Why presentation makes or breaks your
system. Raw data is great... if your day starts with SELECT socks,
pants FROM closet; Engineers demand different detail than managers.
Executive summaries are graphs. Best solution we found:
Drill-down
Slide 14
Report view provides the next level of detail. Report view
provides the next level of detail. Aggregates information for leads
and intermediate managers. Aggregates information for leads and
intermediate managers. Reports are nothing more than SQL
statements. Reports are nothing more than SQL statements. But can
be exported and subscribed to. But can be exported and subscribed
to. Allow further drilldown Allow further drilldown
Slide 15
This view is interesting to programmers, not so much managers.
This view is interesting to programmers, not so much managers.
Incoming callstacks are timestamped, sanitized and hashed. Incoming
callstacks are timestamped, sanitized and hashed. Tells us the
rough# of unique crashes in the system Tells us the rough# of
unique crashes in the system Freshly introduced crashes file an
investigation task to engineering QA. Freshly introduced crashes
file an investigation task to engineering QA. Who can retrieve the
full details by drilling down Who can retrieve the full details by
drilling down
Slide 16
Slide 17
Business Intelligence Great for evaluation outsourcer
performance
Slide 18
Slide 19
Smart Bugs When bugs learned to listen Now lets use the
telemetry data to improve other workflow... Industry Typical
Bugtracker interface Lots of fields Some mandatory Many freeform.
Result: Fields stay empty Random values, Typos Database
pollution.
Slide 20
Better?. Telemetry enabled interface. Retrieves fields from the
server. No typos in database fields. Fewer mandatory fields. Links
to session history, RunInfo, etc. Location aware bugs. Suggests who
should get the bug. Warns of duplicates based on text, build and
location. Vacation Tracker Aware. Smart Categories. But wait, there
is more
Slide 21
Slide 22
95% adoption rate for new interface after 3 months. Significant
Bug count increase (especially non QA). Significant Bug quality
increase. Bugs always have platform and build #. Hardware
information in each bug. Much cleaner bug database, spatial bug
information. Spatial information == great for finding dupes.
Increased overall QA capacity by drafting everyone to help. Much
improved reporting Results.
Slide 23
Slide 24
Achievements For developers? Achievements work! They influence
player behavior through reward. Teach about deeper aspects of the
game. Yield valuable telemetry for developers. Inspire some gamers
to go to great length... and many developers are gamers, so: Yes,
they work for (some) developers too. Can be used to influence how
testers play. Yield valuable balance telemetry for design. Inspire
some developers to play the game. A lot.
Slide 25
Slide 26
Results. Doubled hours of playtime for a month after
introduction Increased bugs filed by both QA and engineers. Got
people competing and excited about the game. Morale boosters.
Taught users about more obscure game features. Some people stayed
addicted and had to get them all. Reward desirable behavior
Slide 27
Slide 28
Getting Started Practical tips and pitfalls. Telemetry systems
dont mix well! Different goals, different needs. Different
technical requirements Start small, dont over-engineer. Be Non
Invasive. Reuse test hooks Make it dead simple to add events.
Create an API and they will come
Slide 29
Our Tech. SkyNet Server PC ClientsPS3 Clients360 Clients
Automated test bots. C# Server Receives Packets (udp) Tracks
clients. Mangles packets into DB. ASP.Net WebApp DB Tools and
Sensors SkyNet API
Slide 30
More Tech Talk. UDP vs TCP? Lightweight, non blocking. We use
one way communication. Telemetry data may get lost. Server software
C# is RAD. Runtime-Compiled packet handlers Dynamically routes
events into tables by type. Protocol Character delimited string.
Only defined in packet handlers. No support for fragmentation.
Slide 31
Useful data sources. Game Client / Server Callstacks, Warnings,
etc. Game Events, Movement, etc. Savegames, Screenshots Performance
Timings Toolset Startup time, Usage patterns. Callstacks, Warnings,
etc. Wikis / Sharepoint. Bugtracker Landesk IP enabled coffee
maker. Twitter Goes well with timeline of PR
Slide 32
Usage patterns.
Slide 33
Slide 34
Endgame Where do we go from here? One database that indexes
information from any available datasource in the company. Makes it
searchable and viewable. And retains historical information so we
can understand, optimize and learn. Push information back into the
game (e.g. warn users when playing a broken build). Something like
this
Slide 35
Thank you. For attending this presentation. Georg Zoeller
[email protected][email protected] Questions, Comments, Feedback? If
you have further questions -or end up implementing your own
development telemetry system, Id like to hear from you especially
if you have ideas for additional data sources.
