Reflection for Tools Development Senior Tools Programmer Ubisoft Vancouver Jeremy Walker

Reflection for Tools Development

Senior Tools ProgrammerUbisoft Vancouver

Jeremy Walker

• Understanding the Costs of Tools Development

• The Ultimate Technique for Data Driven Solutions

• A Reflection-Powered Infrastructure for Tools

Table of Contents

1. The Problem

2. Reflection

3. Content Framework

1. The ProblemUnderstanding the Costs of Tools Development

Hard Coded vs. Data Driven Systems

Animation

Hard Coded Animation System

Animation

Animation Tools

Animation Build System

Animation Engine

Animation Serialization

Hard Coded System Data Driven System

What's the Cost of Great Workflows?

Scenes

Scene Editing Tools

Scene Build System

Scene Rendering Engine

Scene Serialization

Animation

Animation Tools

Animation Build System

Animation Engine

Animation Serialization

Sounds

Sound Editing Tools

Sound Build System

Sound Playback Engine

Sound Serialization

SHARED SHAREDSHAREDSHARED SHARED

SHAREDSHARED

SHAREDSHAREDSHARED

SHARED

SHAREDSHAREDSHAREDSHARED

SHAREDSHAREDSHARED

HUD

Animation Sounds

Cut Scenes

Input Front End Cameras Particles Characters

Textures AI Levels Models Shaders Materials Props

Skeletons Music Sequences State Flows Curves Videos Scripts

Scenes

Great Workflows for All Types of Content

SHARED SHAREDSHAREDSHARED SHARED

SHAREDSHARED

SHAREDSHAREDSHARED

SHARED

SHAREDSHAREDSHAREDSHARED

SHAREDSHAREDSHARED

HUD

Animation Sounds

Cut Scenes




Scenes

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

Hard Coded

The Keep-It-Simple Approach

Hard Coded

HUD

Animation Sounds

Cut Scenes




Scenes

The Monolithic Engine ApproachMonolithic Engine

Animation, Sounds, Scenes, Input, Front End, Cameras, Particles, Characters, Textures, AI, Levels, Models, Shaders, Materials, HUD, Props, Skeletons, Music, Sequences, State

Flows, Curves, Videos, Scripts, Cut Scenes







The Problem

LowDevelopment

Cost

Stable Tools

ReusableSystems

Great Workflows

AdaptableTo Change

Game andGenre-Specific

Needs

?

For all types of content:• Minimize the cost of developing great workflows.• Design reusable systems while accommodating game-

specific needs.• Develop stable tools that adapt to constant change

during production.

The Solution

The Package Release Cycle

Source Code

Automated Build Process OK?

Release Candidate

yes

no

Testing Process

OK?

yes

no

Release

Artist Gets Package Release

Programmer Modifies Code

Artist Requests New

Features

The Monolithic Package Release Problem

Monolithic Package Source

Code

Monolithic Package Release

Package Release

Package Release

Package Release

Package Release

Package Release

The Benefit of Decoupled PackagesPackage

Source Code

Package Source Code

Package Source Code

Package Source Code

Package Source Code

Animation Tool Release

Sound Tool Release

Game Release

The Lockstep Release Problem

Animation Tool Source Code

Sound Tool Source Code

Game Source Code

Compile TimeDependency

Compile TimeDependency

Lockstep Release Cycle

Run TimeDependency

Run TimeDependency

Decoupled Systems

HUD

Animation Sounds

Cameras Particles Characters

Materials Props

Scenes

Decoupled Systems

HUD

Animation Sounds

Cameras Particles Characters

Materials Props

Scenes

Reflection

Reducing Cost and Complexity

Hardcoded Engine

Hard Coded

Low CostSimple

Hard to ReusePoor Workflow

Monolithic Engine

Animation, Sounds, Scenes, Input, Front End, Cameras, Particles, Characters, Textures, AI, Levels, Models, Shaders, Materials, HUD, Props, Skeletons, Music,

Sequences, State Flows, Curves, Videos, Scripts, Cut Scenes







Medium CostComplex

Reuse All or NothingGreat Workflow

High CostVery ComplexVery Reusable

Great Workflow

Low CostSimple

Very ReusableGreat Workflow

2. ReflectionThe Ultimate Technique

for Data Driven Solutions

From Wikipedia: “…Reflection is the process by which a computer program can observe and modify its own structure and behaviour.”

What is Reflection?

Reflection is Used Everywhere

Case Study: C++ Compilation

C++Compiler

Function Bcalls

Function A

ExposeDiscover

Validate Implement

Reflection

Function Declaration

Binding to Function

Definition

Function AC++Compiler

Reflection for Language Interoperability

Visual BasicCompilerC# Code

ExposeDiscover

Validate Implement

Reflection

Type Definitions

Type Binding

Visual Basic Code

C#Compiler

Reflection for Scripting in Games

C++ AdapterLua Script

ExposeDiscover

Validate Implement

Type Definitions

Type Binding

C++ ClassesScript Adapter

Reflection

Reflection for Serialization in Games

C++ Adapter

Serialized Objects

ExposeDiscover

Validate Implement

Type Definitions

Type Binding

C++ ClassesSerializer Reflection

Common Language Specification

Type Definitions Type BindingReflection

• Type size and alignment• Native types• Classes• Enums

• Allocate objects• Binary representation• Invoke constructor, get/invoke members• Get/set enum value

const Type* type = TypeOf<SimpleVehicle>();const char* name = type->Name();size_t size = type->Size();printf(“Type: %s\n Size: %d\n”, name, size);

Instance vehicle = type->CreateInstance();vehicle.As<SimpleVehicle&>().m_MaxSpeedKPH = 193.12f;for (int i=0; i<type->NumMembers(); i++){

const Member* member = vehicle.GetMember(i);if (member->IsField()){

const Field* field = (Field*)member;if (field->GetType() == TypeOf<float>()){

Instance value = vehicle.GetField(field);printf(“Field: %s\n Offset: %d\n Value: %.2f\n”,

field->Name(), field->Offset(), value.As<float>());}

}}

Example: Print all fields of type “float”

Reflection in Action

Type: SimpleVehicle Size: 4Field: MaxSpeedKPH Offset 0 Value: 193.12

Output

• Macros• Code Parser• Type Definition Language

Popular Approaches to C++ Reflection

Reflection with Macros

class SimpleVehicle : public Entity{public:

DECLARE_TYPE(); float m_MaxSpeedKPH; void Reset(bool useDefaults);

float GetMaxSpeedMPH() const;void SetMaxSpeedMPH(float maxSpeedMPH);

};

//In a separate .CPP file:DEFINE_TYPE(SimpleVehicle)

BASE_CLASS(Entity)FIELD(“MaxSpeedKPH”, m_MaxSpeedKPH)METHOD(Reset)PROPERTY(GetMaxSpeedMPH, SetMaxSpeedMPH)

DEFINE_TYPE_END()

Macros Example Pros• No external toolsCons• Awkward to implement• Hard to debug• Run-time discovery

Reflection with Code Parser

/// [Class]class SimpleVehicle : public Entity{public:

/// [Field(“MaxSpeedKPH”)]float m_MaxSpeedKPH;/// [Method]void Reset(bool useDefaults);/// [Property]float GetMaxSpeedMPH() const;/// [Property]void SetMaxSpeedMPH(float maxSpeedMPH);

};

Code Parser Example Pros• Easier to implement• Compile-time discoveryCons• Slow pre-build step

Reflection with Type Definition Language

class SimpleVehicle : Entity{

float MaxSpeedKPH;void Reset(bool useDefaults);float MaxSpeedMPH { get; set; }

};

Type Definition Language Example Pros• Easiest to implement• No slow pre-buildCons• Can’t reflect existing

classes

/// [Class]class SimpleVehicle : public Entity{public:

/// [Field(“MaxSpeedKPH”)]float m_MaxSpeedKPH;/// [Method]void Reset(bool useDefaults);/// [Property]float GetMaxSpeedMPH() const;/// [Property]void SetMaxSpeedMPH(float maxSpeedMPH);

};

C++ Class<type name=“SimpleVehicle”> <field name=“MaxSpeedKPH” type=“float”/> <method name=“Reset” returntype=“void”> <parameter name=“useDefaults” type=“bool”/> </method> <property name=“MaxSpeedMPH” type=“float” hasget=“true” hasset=“true”/></type>

Types.xml

Exporting Type Definitions for Game

C++ Reflected Types

(game-side)Code Parser

Types.xml

Generated Binder Code

Reflection System(game-side)

<type name=“SimpleVehicle”> <field name=“MaxSpeedKPH” type=“float”/> <method name=“Reset” returntype=“void”> <parameter name=“useDefaults” type=“bool”/> </method> <property name=“MaxSpeedMPH” type=“float” hasget=“true” hasset=“true”/></type>

Types.xml

Exporting Type Definitions for Tools

C++ Reflected Types

(game-side)Code Parser Types.xml Reflection System

(tool-side)C# Proxy

Generator

[ProxyType(“SimpleVehicle”, 0x81c37132)]public partial class SimpleVehicle : Entity{ public float MaxSpeedKPH { get { return this.Instance.GetField(“MaxSpeedKPH”).Get<float>(); } set { this.Instance.GetField(“MaxSpeedKPH”).Set<float>(value); } } } public float MaxSpeedMPH { ... } public void Reset(bool useDefaults) { ... }}

C# Proxy Class

Primary Uses for Reflection in Tools

Types.xml

Reflection SystemSerialization

Generated UI(PropertyGrid, TreeView, etc.)

Client-ServerRemoting

Client-Server Remoting

RemotingClient

Generated C# Proxy Classes

ExposeDiscover

Validate Implement

Reflection

Type Definitions

Type Binding

C# Proxy Generator C++ Adapter

ExposeDiscover

Validate Implement

Reflection

Type Definitions

Type Binding

C++ Game Classes

RemotingServer

Tools (Client) Game (Server)

Type definitions out of sync• Detect type checksum mismatch• Detect problems early• Auto-synchronization of type information• Auto-migration of data

Problems and Workarounds

Tools tightly coupled to the game• Avoid overuse of generated proxy classes• Use generated UI where possible• Use polymorphic proxy classes


Tools Code

Base Class

Derived Class A

Derived Class B

Derived Class C

has proxy

has no proxy

Excessive memory usage• Strip type information based on usage • Auto-detect unused reflected types


• Marshalling events for multi-processor architectures• Client-Server Remoting for Online• Serialization of saved game data

Other Uses for Reflection

3. Content FrameworkA Reflection-Powered Infrastructure for Tools

Content Framework

Build System

Asset Repository

Asset Manager

Asset ExplorerPackage System

Content Framework

Reflection

Tool-side

Game-side

Asset Previewing

3rd Party

(Demo Videos)

• Fast tools development• Great workflows for all types of content• Decoupled systems with improved reusability and

resilience to change

Promising Results

Questions?

Documents

Reflection for Tools Development Senior Tools Programmer Ubisoft Vancouver Jeremy Walker