ni.com

Developing a System for Real-Time Numerical Simulation during Physical Experiments in a

Wave Propagation LaboratoryMarch 27, 2014

Darren Schmidt (presenter)Lothar WenzelNI Scientific Research & Big Physics

Dr. Thomas BlumProf. Stewart GreenhalghProf. Johan RobertssonDr. Dirk van ManenMarlies VasmelETH-Zurich – D-EDRW – Geophysics

2ni.com

NI Team• Barry Hutt• Darren Schmidt• Balazs Toth• Lothar Wenzel

ETH Team• Christoph Baerlocher• Dr. Thomas Blum• Prof. Stewart Greenhalgh• Heinrich Horstmeyer• Prof. Johan Robertsson• Dr. Dirk van Manen• Marlies Vasmel• `

The International Team

www.ni.com www.geophysics.ethz.ch

http://www.ni.com/http://www.geophysics.ethz.ch/

3ni.com

Problem: Existing Laboratory Limitations

• Size of physical experiments limit the study of longer wavelengths or lower frequencies

Rock

reflection

• Boundaries of physical experiments introduce undesired reflections

4ni.com

Goals

• A new lab needs to:

• Address existing limitations

• Avoid introducing new barriers

• Allow for applications outside of geophysics

5ni.com

Solution – Virtualize the Physical Experiment

“Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual

simulation.”

Rock

Simulated Environment

6ni.com

Solution – Virtualize the Physical Experiment

“Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual

simulation.”

7ni.com

Solution – Virtualize the Physical Experiment

This new system allows

• Investigation of larger wavelengths

• Full control over wave propagation

Example: ‘Noise’ cancellation for undesired reflections.

8ni.com

Solution – Virtualize the Physical Experiment

This new system allows

• Investigation of larger wavelengths

• Full control over wave propagation

Example: ‘Noise’ cancellation for undesired reflections.

“hear”

9ni.com

Solution – Virtualize the Physical Experiment

This new system allows

• Investigation of larger wavelengths

• Full control over wave propagation

Example: ‘Noise’ cancellation for undesired reflections.

“cancel”

10ni.com

Solution – Virtualize the Physical Experiment

This new system allows

• Investigation of larger wavelengths

• Full control over wave propagation

Example: ‘Noise’ cancellation for undesired reflections.

“cancel”

11ni.com

How?

The physical components of the experiment are:• A 1.5m cube rock sample

1.5m

1.5m

1.5m

Rock

12ni.com

How?

The physical components of the experiment are:• A 1.5m cube rock sample

• A 2m cube of water containing the rock sample

2m

2m

2m

Water

13ni.com

How?

The physical components of the experiment are:• A 1.5m cube rock sample

• A 2m cube of water containing the rock sample

• 1000+ sensors on or close to rock surface

o Sensors are equally distributed across sixrecording surfaces

o Each sensor measures pressure

sensors

14ni.com

How?

The physical components of the experiment are:• A 1.5m cube rock sample

• A 2m cube of water containing the rock sample

• 1000+ sensors on or close to rock surface

o Sensors are equally distributed across sixrecording surfaces

o Each sensor measures pressure

• 1000+ actuators on or close to water surface

o Actuators are equally distributed across sixemitting surfaces

o Each actuator generates acoustic waves

actuators

15ni.com

How?

The physical components of the experiment are:• A 1.5m cube rock sample

• A 2m cube of water containing the rock sample

• 1000+ sensors on or close to rock surface

o Sensors are equally distributed across sixrecording surfaces

o Each sensor measures pressure

• 1000+ actuators on or close to water surface

o Actuators are equally distributed across sixemitting surfaces

o Each actuator generates acoustic waves

16ni.com

How?

The simulation part of the experiment involves:

• Acquisition: 1000+ channels• Data represents pressure

• 16-bit resolution

17ni.com

How?

The simulation part of the experiment involves:

• Acquisition: 1000+ channels

• Computation: Large PDE solver• Based on a pre-computed Green’s functions (two 1K x 1K x 250 ‘matrices’)

• Formulation requires pressure and velocity data

18ni.com

How?

The simulation part of the experiment involves:

• Acquisition: 1000+ channels

• Computation: Large PDE solver

• Control: 1000+ channels• Response represents ‘generator’ values

• 16-bit resolution

19ni.com

How?

The simulation part of the experiment involves:

• Acquisition: 1000+ channels

• Computation: Large PDE solver

• Control: 1000+ channels

• Real-time constraint: 50 ms cycle time• Dictated by the 20 kHz sampling rate

• Includes a complete acquirecomputecontrol cycle

20ni.com

Why …. Water?

So the system has the …

• Time to act

21ni.com

Why …. Water?

So the system has the …

• Time to act• Distance from recording to emitting surface = 25 cm

• Propagation time in water ~ 170 ms

Slo

we

r W

ave

Pro

pag

atio

n

22ni.com

Why …. Water?

So the system has the …

• Time to act

• Ability to react

Slo

we

r W

ave

Pro

pag

atio

n

23ni.com

Why …. Water?

So the system has the …

• Time to act

• Ability to react• Impedance differential impacts actuator performance

• Wave ‘power’ in water is ‘stronger’ than air

Slo

we

r W

ave

Pro

pag

atio

n

24ni.com

Why …. Water?

So the system has the …

• Time to act

• Ability to react• Impedance differential impacts actuator performance

• Wave ‘power’ in water is ‘stronger’ than air

Slo

we

r W

ave

Pro

pag

atio

n

25ni.com

Why …. Water?

So the system has the …

• Time to act

• Ability to react

• Knowledge to respond• Wave propagation in water is well-understood

• Defined by ‘computable’ Green’s functions

26ni.com

Why …. Water?

So the system has the …

• Time to act

• Ability to react

• Knowledge to respond• Wave propagation in water is well-understood

• Defined by ‘computable’ Green’s functions

27ni.com

Why … the Rock?

• Predicting wave propagation in water is ‘easy’• Behavior is known

• Models are based on linear operators

• Many computational optimizations are possible

28ni.com

Why … the Rock?

• Predicting wave propagation in water is ‘easy’

• Predicting wave propagation in rock is ‘hard’• Behavior is under exploration

• Models involve inherently non-linear functions

• Computational complexity is high

29ni.com

Why … the Rock?

• Predicting wave propagation in water is ‘easy’

• Predicting wave propagation in rock is ‘hard’

• As a result, this real-time system:• Leaves the nonlinear behavior to Mother Nature

• Captures the results at the recording surfaces

• Manipulates the physical experiment using known models

30ni.com

Why … Pressure & Velocity?

• Dictated by the physics (i.e. the Green’s functions)

31ni.com

Why … Pressure & Velocity?

• Dictated by the physics (i.e. the Green’s functions)

velocity pressure

32ni.com

Why … Pressure & Velocity?

• Dictated by the physics (i.e. the Green’s functions)

• How Do We Get Velocity?

pressurevelocity

33ni.com

Option 1: Duplicate Recording Surfaces• Benefit

o Compute velocity using six additional recording surfaces

o New sensors measure pressure along the surface normal

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

34ni.com

Option 1: Duplicate Recording Surfaces• Benefit

o Compute velocity using six additional recording surfaces

o New sensors measure pressure along the surface normal

• Work

o Compute the velocity using common difference method

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

35ni.com

Option 1: Duplicate Recording Surfaces• Benefit

o Compute velocity using six additional recording surfaces

o New sensors measure pressure along the surface normal

• Work

o Compute the velocity using common difference method

• Issue

o Sensor data is double and computation is increased.

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

36ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit

o Sensor data is not doubled

o Gaps in one recording surface are filled in the other

o Layout still leverages the surface normal

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

37ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit

o Sensor data is not doubled

o Gaps in one recording surface are filled in the other

o Layout still leverages the surface normal

• Work

o Reconstruct all missing pressure data using interpolation

Challenge 1 – How Do We Get Velocity?

= virtual sensor (for pressure)

= virtual sensor (for velocity)

= sensor (for pressure)

= sensor (for velocity)

38ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit

o Sensor data is not doubled

o Gaps in one recording surface are filled in the other

o Layout still leverages the surface normal

• Work

o Reconstruct all missing pressure data using interpolation

• Issue

o Computation increases.

Challenge 1 – How Do We Get Velocity?

= virtual sensor (for pressure)

= virtual sensor (for velocity)

= sensor (for pressure)

= sensor (for velocity)

39ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout

Option 3: Optimize the Mathematical Model• Benefit

o Staggered layout is accounted for in the numerical simulation

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

40ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout

Option 3: Optimize the Mathematical Model• Benefit

o Staggered layout is accounted for in the numerical simulation

• Work

o Velocity is implicitly computed via a new Green’s function

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

41ni.com

Option 1: Duplicate Recording Surfaces

Option 2: Duplicate Surfaces Using Staggered Sensor Layout

Option 3: Optimize the Mathematical Model• Benefit

o Staggered layout is accounted for in the numerical simulation

• Work

o Velocity is implicitly computed via a new Green’s function

• Issue

o No physical meaning to the new simulation model

Challenge 1 – How Do We Get Velocity?

= sensor (for pressure)

= sensor (for velocity)

42ni.com

Option 3 – Optimize the Mathematical Model

43ni.com

Option 3 – Optimize the Mathematical Model

pressurevelocity

44ni.com

Option 3 – Optimize the Mathematical Model

pressure

45ni.com

Option 3 – Optimize the Mathematical Model

pressure

computed in advance

46ni.com

Option 3 – Optimize the Mathematical Model

The new Green’s function:• Avoids increasing the sensor data• Reduces the computation by 50%

47ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

48ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latencyo Is an issue for data movement within the system

Ethernet USB PCI PCI Express

Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) 132 4000 (x16)

Latency (ms) 1000 (Gigabit)

49ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latencyo Is an issue for data movement within the system

o PCI Express bus offers high throughput and low latency

o Transfers are synchronized using FPGA technology

Ethernet USB PCI PCI Express

Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) 132 4000 (x16)

Latency (ms) 1000 (Gigabit)

50ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

51ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

o Pre-computed Green’s functions data is large– 1000 sensors & 1000 actuators 1K x 1K

– Experiment runs for 250 time steps 250

52ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

o Pre-computed Green’s functions data is large– 1000 sensors & 1000 actuators 1K x 1K

– Experiment runs for 250 time steps 250

1000 {1K x (1K x 250)} sgemv ops

53ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

o Pre-computed Green’s functions data is large

o Consume 4GB of memory

Storage = 4GB

54ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

o Pre-computed Green’s functions data is large

o Consume 4GB of memory

o Computation requires 100TB/s memory bandwidth– Estimated with data movement overhead of 10 ms

System bandwidth = 100TB/s

55ni.com

Challenge 2 – How to Simulate in Real-Time?

Given a cycle time of 50 ms

• Latency

• Bandwidth

o Is an issue for the Green’s functions computation

o Pre-computed Green’s functions data is large

o Consume 4GB of memory

o Computation requires 100TB/s memory bandwidth

o To perform the computation in real-time

– Leverage multiple NVIDIA Tesla K40 GPUs

– Connected via PCI Express

– Optimize the Green’s functions implementation to reducethe number of GPUs required by an order of magnitude

Telsa K40: 288GB/s

56ni.com

GPUs

PCIe Switch

Recording surface

Emitting surface

PXImc

MXI Gen2

MXI Gen2

LabVIEW Timing/Synch

PXI FPGA

Network over PCIe GPU acceleration

Real-Time 3D Simulator Components

PCIe enclosures PCIe switches

57ni.com

Demo: System Simulation & Green’s Functions Computation

Simulation of the System

Sensor Data

Green’s Functions Computation

58ni.com

Special Thanks

NVIDIA

• Jerry Chen

• Cliff Woolley

• Duncan Poole

One Stop Systems

• Jaan Mannik

Dell

• Aron Bowan

59ni.com

References

• Vasmel et al (2013). “Immersive experimentation in a wave propagation laboratory,” J. Acoust. Soc. Am. 134, EL492-EL498.http://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdf

• Big Physics at National Instrument’s http://www.ni.com/physics/

http://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdfhttp://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdfhttp://www.ni.com/physics/