1/19/2011

1

A Pipeline Architecture forA Pipeline Architecture forParallel Virtual EnvironmentsParallel Virtual Environments

MMVR ‘11 Workshop – February 10, 2011Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Eric Acosta

National Capital Area Medical Simulation CenterUniformed Services University

http://simcen.usuhs.edu

Virtual Environment (VE)Virtual Environment (VE)

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

I/O Devices

Head Trauma Simulator VEHead Trauma Simulator VE

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Virtual EnvironmentsVirtual Environments

VE Modeling

• Geometrical

VE Execution

• Interactions

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Geometrical• Geometry• Visual properties

• Physical• Behavior • Interactions

Interactions• Updates• Rendering

Work

Work

Work

Work

Work

Work

Work

Work

Work

Work

Work

WorkVirtual Environment

Work

Work

Work

Work

Work

Work

Work

Work

Work

Work

Work

Work

MultiMulti--core CPUscore CPUsVirtual Environment

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Pipeline Architecture OverviewPipeline Architecture Overview

• Develop VEs for multi-core CPU systems– VE Modeling

• 3D Objects• Interactions

– Multi-threaded VE Pipeline (execution)• Parallel simulation & rendering

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

1/19/2011

2

VE ModelingVE Modeling

• 3D Objects– Modularized data & algorithms

V3V3D Object

Geometry1) Display2) Behavior

Visual Properties

Graphics Renderer

Behavior Manager

V0 V1

V2

V4

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

VE ModelingVE Modeling

• Interaction Pairs– Pair-wise � parallel processing– Interactions of interest

Interaction Pair 1,2

Collision Detection

Collision Results

Collision Response

Interaction Pair 3,4

Collision Detection

Collision Results

Collision Response

…

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Object 1 Object 2 Object 3Object 2

VE Simulation & Rendering PipelineVE Simulation & Rendering Pipeline

Object 1

Object 2

Interaction Pair1,2Interaction Pair1,2

Object 1

Object 3

Interaction Pair1,3Interaction Pair1,3

Object 1

Object 2

Modeled VEModeled VEObject 3

Object 2

Interaction Pair2,3Interaction Pair2,3

Collision Detection

Collision Response

Object Update

Object Render

Object InteractionsObject Interactions

Object 3

Object Interactions

VE Execution PipelineVE Execution Pipeline

Parallel Pipeline ConceptParallel Pipeline Concept

Object 1

Object 2

Interaction Pair1,2Interaction Pair1,2

Object Render

Collision Detection

Collision Results

Collision Response Object 1 Object

Update Object 1

VE SynchronizationVE Synchronization

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Object 2

Object 1

Object 3

Interaction Pair1,3Interaction Pair1,3

Object 3

Object 2

Interaction Pair2,3Interaction Pair2,3

Object Render

Object Render

Collision Detection

Collision Detection

Collision Results

Collision Results

Collision Response

Collision Response

Object 2

Object 3

Object Update

Object Update

Object 2

Object 3

MultiMulti--threaded Pipeline Overviewthreaded Pipeline Overview

• Thread scheduling– Per thread pipeline � Fixed & inefficient– Thread migration � Dynamic load distribution & balancing

Collision Detection

Collision Response

ObjectUpdate

ObjectRender

Pipeline Thread MigrationPipeline Thread Migration

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

MultiMulti--threaded Pipeline Overviewthreaded Pipeline Overview

• Work flow overview– Thread-safe pipeline queues (synchronization)– Carry-over (no synchronization)

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Collision Detection

Collision Response

ObjectUpdate

ObjectRender

Pipeline Thread MigrationPipeline Thread Migration

CD Queue CR Queue Update Queue

Render Queue

1/19/2011

3

MultiMulti--threaded Pipeline Overviewthreaded Pipeline Overview• Callback mechanism

– Trigger callbacks (VE modeling) – Pipeline remains domain independent

CD Q CR Q Update Render

Collision Detection

Collision Response

ObjectUpdate

ObjectRender

CD Queue CR Queue pQueue Queue

PhysicalBehavior Callback

(3D Object)

Update

Graphics Render Callback

(3D Object)

Render

CollisionDetection Callback

(Interaction Pair)

CD ResultsInteraction Pair

CR Results

CollisionResponse Callback

(Interaction Pair)

Interaction Pair

Pipeline ThreadsPipeline Threads

• Thread pooling

• Generalized pipeline threads– Callbacks � Execute any pipeline stagey p p g

• Autonomous design– Loops until terminated– Execute work at current stage– Embedded scheduling algorithm � current stage– No dedicated thread scheduler required

Thread Scheduling AlgorithmThread Scheduling Algorithm• High-level

Basic flowStart

Work Carried Over?

Yes

YesQueue Empty?

No

Perform work

Move up 1Stage

Yes

No

Next Stage Queue Empty?

Get work from Queue

p y

No

Move Down 1 Stage

Yes

End

No

Carry Over?

MultiMulti--threaded Pipelinethreaded Pipeline

• CD/CR queues large � Synchronization overhead– Eliminate CR queue (carry-over only)– Combine interactions stages

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Collision Detection Queue

Object Render Queue

Object Update Stage

Object Render Stage

3D Object

3DObject

3D Object

CollisionDetection Callback (Interaction Pair)

CollisionResponse Callback

(Interaction Pair)

PhysicalBehavior Callback

(3D Object)

Graphics Render Callback

(3D Object)

Update Render

Collision Detection Stage

Collision Response Stage

3D Object

Object Update Queue

Collision Response Queue

Interaction Pair

Int. Pair(s)

Interaction Pair

Interaction Pair CD Results Interaction Pair CR Results

Interaction Pair

Collision Detection Collision Response

Collision Detection and Response Stage

Object Ready Queue

3DObject

3D Object

Interaction Pair

Interaction Pair CD Results Interaction Pair CR Results

Interaction Pair

Virtual Environment InteractionsVirtual Environment Interactions• Collision Detection

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

1. Broad Phase 2. Narrow Phase

MultiMulti--threaded Pipelinethreaded Pipeline

• Separate broad-phase CD stage

3D Object(s)

• No interactions � Object Update

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Object Ready Queue

Collision Detection Queue

Object Render Queue

Broad PhaseCD Stage

Object Update Stage

Object Render Stage

3D Object

3DObject

3D Object

CollisionDetection Callback (Interaction Pair)

CollisionResponse Callback

(Interaction Pair)

PhysicalBehavior Callback

(3D Object)

Graphics Render Callback

(3D Object)

Interaction Pair CD ResultsInteraction Pair

CR ResultsUpdate Render

3DObjectCollision Detection Collision

Response 3D Object

Object Update Queue

Collision Detection and Response Stage3D Object

Interaction Pair

Int. Pair(s)

3D Object

Interaction Pair

Interaction Pair

Broad PhaseCD Stage

1/19/2011

4

MultiMulti--threaded OpenGL Renderingthreaded OpenGL Rendering

• Multiple rendering threads• Single graphics driver (bottleneck)

• Context switch overhead

Thread 1

Thread 2

Thread 3

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

V0

V2

V3V4

V1

V0

V2

V3V4

V1

V0

V2

V3V4

V1

Graphics DriverV0

V2

V3V4

V1V0

V2

V3V4

V1V0

V2

V3V4

V1

MultiMulti--threaded Pipelinethreaded Pipeline

• Object Render token system• Dual-frame processing � Reduce idle threads

Object Read Collision Detection Object Render Queue

3D Object(s)

Object Update Obj t R d

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Object ReadyQueue

Collision DetectionQueue

Broad PhaseCD Stage

Object Update Stage

Object Render Stage

3D ObjectInteraction Pair 3D Object

3DObject

Object Render QueueFrame i Frame i+1

3D Object

CollisionDetection Callback (Interaction Pair)

CollisionResponse Callback

(Interaction Pair)

PhysicalBehavior Callback

(3D Object)

Graphics Render Callback

(3D Object)

Interaction Pair CD ResultsInteraction Pair

CR ResultsUpdate Render

3DObject

Pipeline Thread

Transition

Pipeline Thread

Transition

Collision Detection Collision Response

3D Object

Object UpdateQueue

Int. Pair(s)

Collision Detection and Response Stage

N ThreadsN Threads 1 Thread1 Thread

Object Render Queue

Modular ArchitectureModular Architecture

Object Interaction Manager

Pipeline Queues

Modeled InteractionsInteraction Pairs

Thread Manager

VE Pipeline Thread

Pipeline

3D Objects

InteractionPairs

InteractionPairs

Window

Main Rendering Context

Context

Rendered Image

Visual Display

Rendered Image

VE Manager

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Object Manager

Visual Properties

Geometry

Graphics Renderer

Physical Behavior

PipelineAlgorithm

Off Screen Rendering

ContextObjectRendering

ObjectUpdating

Device I/O Manager

Haptic Device Interface

Device Input

Force Feedback

HapticDevice

Device InputForce

Feedback

3D Object3D Objects

Multi-rate HapticsMulti-rate Haptics

Experimental ResultsExperimental Results

• Test System– Windows 7 64-bit– 6-core Intel i7-980x (3.33GHz)

6 GB RAM– 6 GB RAM– Nvidia Quadro 5000 (2560 MB)

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Experimental ResultsExperimental Results• Deformable mass-spring particles

– Vary # particles and # threads– Particles

• 42 nodes & 162 springsRandom step size (1 100x per frame)• Random step size (1-100x per frame)

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Deformable Particles ResultsDeformable Particles Results

30

35

40

Throughput1 2 3 4 5 6 SA

Number Objects

0

5

10

15

20

25

100 200 300 400

FPS

Single ThreadedOptimized App.Single ThreadedOptimized App.

1/19/2011

5

Deformable Particles ResultsDeformable Particles Results

5

6

Speedup100 200 300 400 Ideal

1

1.2

Thread Efficiency100 200 300 400 Ideal

Number Threads

0

1

2

3

4

1 2 3 4 5 6

Sp

Number Threads

0

0.2

0.4

0.6

0.8

1 2 3 4 5 6

Ep

Experimental ResultsExperimental Results• Head trauma simulator

– Automated skin retraction step

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

Head Trauma Simulator ResultsHead Trauma Simulator Results

140

160

180

200

Throughput

Number Threads

0

20

40

60

80

100

120

1 2 3 4 5

FPS

Head Trauma Simulator ResultsHead Trauma Simulator Results

4

5

Speedup

0 8

1

1.2

Thread Efficiency

Number Threads Number Threads

0

1

2

3

1 2 3 4 5

Sp

0

0.2

0.4

0.6

0.8

1 2 3 4 5

Ep

ConclusionsConclusions

• Object-centric development– Objects + interactions– Support parallel and sequential algorithms

Parallelization transparent to developer– Parallelization transparent to developer• Near-linear performance• Scales efficiently vs. number cores used

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative

ConclusionsConclusions

• Thread scheduling algorithm– Simple and effective– 1 thread � Sequential pipeline behavior

Work load distribution (no scheduler overhead)– Work load distribution (no scheduler overhead)• Automatic load balancing

– Write application once � scale performance

Developing a Standardized Tri-Service Medical Simulation Platform – The TOPS Initiative