Automated Testing In MMP Development Controlling Cost, Risk and Schedule Larry Mellon Austin Game Conference Sept 10-11, 2003

Automated Testing Automated Testing In MMP In MMP

DevelopmentDevelopmentControlling Cost, Risk and Controlling Cost, Risk and

ScheduleScheduleLarry MellonLarry Mellon

Austin Game ConferenceAustin Game ConferenceSept 10-11, 2003Sept 10-11, 2003

Major Functions of Major Functions of AutomationAutomation

Repeatable, Synchronized Input Repeatable, Synchronized Input

DebuggingDebugging Performance TuningPerformance Tuning RegressionRegression

Data ManagementData Management

Test ResultsTest ResultsRaw DataRaw Data

Major Weaknesses of Major Weaknesses of AutomationAutomation

JudgmentJudgment

Rigid Inputs, Rigid AnalysisRigid Inputs, Rigid Analysis

Visual EffectsVisual Effects

Semi-Automated TestingSemi-Automated Testing

(1) (2)

(3)AutomationAutomation Manual Manual

Command Steps Command Steps

Validation Steps Validation Steps

Automation: Automation: ArchitectureArchitecture

Startup&

Control

System Under Test

Test ManagerTest Selection/SetupControl N Clients

RT probes

Repeatable, Sync’edTest Inputs

System Under Test

Scripted Use CasesEmulated User Play SessionsMulti-client synchronization

Collection&

Analysis

System Under Test

Report ManagersRaw Data Collection

Aggregation / SummarizationAlarm Triggers

Automated Testing: Automated Testing: TopicsTopics

Business CaseBusiness Case

TSO Applications & ImplementationTSO Applications & Implementation

WrapupWrapup

QuestionsQuestions

Automated Testing: OverviewAutomated Testing: Overview

Customer satisfactionCustomer satisfaction

Development risk / operational costsDevelopment risk / operational costs

Development / Schedule Development / Schedule RiskRisk

How do we know when we are ready to launch?How do we know when we are ready to launch?

How do we know how fast we are going?How do we know how fast we are going?

Complex, Large-Scale SystemComplex, Large-Scale System

Large Development TeamLarge Development Team

Accelerated, Measurable Accelerated, Measurable DevelopmentDevelopment

Repeatable, Synchronized InputRepeatable, Synchronized Input

Data ManagementData Management

DevelopmentDevelopment

Instant Feedback,Rapid Triage,

History

ManagementManagement

StabilityReference Cases

Summarized, AggregatedResults

Current StatusHistoryTrends

Customer SatisfactionCustomer Satisfaction

StabilityStability

New ContentNew Content

Automated testing keeps existing code working while new code is

being added.

Automated testing keeps existing code working while new code is

being added.

Subscription: Profit & Running Subscription: Profit & Running CostsCosts

Profit…

NewContent

Regression

CustomerSupport

Operations

~ $10per

customer

Automation: Lower Running Automation: Lower Running CostCost

Profit…

Regression

CustomerSupport

Operations

~ $10per

customer

Lower New Content CostLower New Content Cost

Automation: Lower Automation: Lower Running CostRunning Cost

Profit…

CustomerSupport

Operations

~ $10per

customer


Lower Testing CostLower Testing Cost

Automation: Lower Automation: Lower Running CostRunning Cost

Profit…

Operations

~ $10per

customer


Lower Testing CostLower Testing Cost

Happy Customers Don’t CallHappy Customers Don’t Call

Business Case: SummaryBusiness Case: Summary

Tool investment accelerates development, provides tangible

measures of progress

Tool investment accelerates development, provides tangible

measures of progress

Tool investment pays out over entire lifecycle of subscription-based

service

Tool investment pays out over entire lifecycle of subscription-based

service



The Sims OnlineApplicationsImplementation


WrapupWrapup

QuestionsQuestions


TSO: ApplicationsTSO: Applications

Load TestingLoad Testing

QA RegressionQA Regression

StabilityStability

Operations

Development

Load Testing: GoalsLoad Testing: Goals

Expose issues that only occur at scaleExpose issues that only occur at scale Establish hardware requirementsEstablish hardware requirements Establish response is playable @ scaleEstablish response is playable @ scale Emulate true user behaviourEmulate true user behaviour

ClientMetrics

Game Traffic

ResourceMetrics

Debugging Data

Test Driver CPU

Load Control Rig

Server Cluster

Load Testing Team

System Monitors

Internal Probes

Test ClientTest

ClientTest

Client

Test Driver CPU

Test ClientTest

ClientTest

Client

Test Driver CPU

Test ClientTest

ClientTest

Client

Stability Via Monkey TestsStability Via Monkey Tests

Code Repository Compilers

Continual Repetition of Unit / Integration TestsContinual Repetition of Unit / Integration Tests

Reference Servers

Stability Via Monkey TestsStability Via Monkey Tests

Reference Servers

1. Last “known good”2. What works, right now3. Aggregate reporting

1. Last “known good”2. What works, right now3. Aggregate reporting

AgingAging

Moving partsMoving parts

Monkey Test: EnterLotMonkey Test: EnterLot

Non-Deterministic FailuresNon-Deterministic Failures

Critical Path: Unit Critical Path: Unit TestingTesting

Failures on the Critical Path block access to much of the game.Worse, unreliable failures…

Failures on the Critical Path block access to much of the game.Worse, unreliable failures…

enter_house ()

Test Case: Can an Avatar Sit in a Chair?

use_object ()

buy_object ()

buy_house ()

create_avatar ()

login ()

daysBug

Introduced

Development

Checkin

Smoke

Regression

Build

Time to Fix

Cost of Detection

ImpactOn

Others

ImpactOn

Others

Working Code

Candidate Code

Pass / Fail,Diagnostics

Development

Checkin

Smoke

Sniff

RegressionPre-Checkin Regression: don’t let broken code into Mainline.

Pre-Checkin Regression: don’t let broken code into Mainline.





WrapupWrapup

QuestionsQuestions


Test Client

Single, Data Driven Test Single, Data Driven Test ClientClient

Regression Load

Single API

ReusableScripts & Data

Testing feature correctness Testing system performance

Test Client

Data Driven ReportingData Driven Reporting

Regression Load

SingleAPI

ReusableScripts & Data

SingleAPI

ConfigurableLogs & Metrics

Key Game StatesPass/Fail

Responsiveness

Test Client Derived From Real Test Client Derived From Real Client Client

Game Client

View

Logic

Presentation LayerPresentation Layer

Test Client Test Client Game Client Game Client

Scripted Player Scripted Player ActionsActions

Script Engine

State

Game GUI

Client-Side Game Logic

Commands

State

Presentation Layer

What Level To Test At?What Level To Test At?

Game Client

MouseClicks

Presentation Layer

Regression: Too Brittle (pixel shift)Load: Too Bulky

Regression: Too Brittle (pixel shift)Load: Too Bulky

View

Logic

What Level To Test At?What Level To Test At?

Game Client

InternalEvents

Presentation Layer

Regression: Too Brittle (Churn Rate vs Logic & Data)

Regression: Too Brittle (Churn Rate vs Logic & Data)

View

Logic

Gameplay: Semantic Gameplay: Semantic AbstractionsAbstractions

NullView ClientView

LogicPresentation LayerChat Enter Lot

Use Object

…Route Avatar

Basic gameplay changes less frequently than UI or protocol implementations.Basic gameplay changes less frequently than UI or protocol implementations.

Scriptable User Play Scriptable User Play SessionsSessions

Set of Presentation Layer “primitives”Set of Presentation Layer “primitives” State probes: peek/poke arbitrary game stateState probes: peek/poke arbitrary game state Synchronization: Synchronization: wait_until, remote_commandwait_until, remote_command

RecordableRecordable…… Test ScriptsTest Scripts: :

Specific / ordered inputsSpecific / ordered inputs Specific game responsesSpecific game responses

Single user play session, multi-user sessionSingle user play session, multi-user session

Toolkit UsabilityToolkit Usability

Data ManagersData Managers

Pushbutton Use: Developers, Testers,Test Farms, Everywhere

Pushbutton Use: Developers, Testers,Test Farms, Everywhere

Continual Process ImprovementTool UI / feature setFalse positives

Continual Process ImprovementTool UI / feature setFalse positives



WrapupWrapup

QuestionsQuestions


TSO Apps & ImplTSO Apps & Impl

WrapupWrapup

Very Useful, Very ReusableVery Useful, Very Reusable

Development AccelerationDevelopment Acceleration

Lower Operational CostsLower Operational Costs

Cautionary TalesCautionary Tales

Tabula RasaTabula Rasa

Tabula RasaTabula Rasa

PreCheckin SniffTestPreCheckin SniffTest Keep Mainline workingKeep Mainline working

Hourly Stability CheckersHourly Stability Checkers Baseline for DevelopersBaseline for Developers

Easy to Use == UsedEasy to Use == Used

Distribute Test Development & Ownership Across Full TeamDistribute Test Development & Ownership Across Full Team

Load Test: Early & OftenLoad Test: Early & Often Break it before LiveBreak it before Live

Executive SupportExecutive Support Radical shifts in ProcessRadical shifts in Process

Dedicated Tools GroupDedicated Tools Group

Cautionary TalesCautionary Tales

Flexible Design Requires Flexible TestsFlexible Design Requires Flexible Tests

Signal To Noise RatioSignal To Noise Ratio

Defects In The Testing SystemDefects In The Testing System



QuestionsQuestions



Architecture & ApplicationsArchitecture & Applications

Documents

Automated Testing In MMP Development Controlling Cost, Risk and Schedule Larry Mellon Austin Game Conference Sept 10-11, 2003