[134] immersive sound vr

Immersive SOUND for VR

오현오 (Henney Oh) GAUDI AUDIO LAB

SOUND is half the experience

Listen

SOUND is half the experience

- George Lucas

Immersion into is a perception of being physically present in a non-physical world.

technology refers to technology that blurs the line between the physical world and digital or simulated world,

thereby creating a sense of immersion.

VR

Immersive

by wikipedia

Virtual Reality

PS 게임_ “하쿠나 마타타”

for

Immersive & Interactive

in Mobile

Head-Mounted Display

VR like HMD

HMD is hot

Facebook, Oculus VR $2bil에인수(Mar. 25, 2014)

Samsung, Oculus와협력발표(May 31, 2014)

Sony, Project Morpheus 발표(Mar. 18, 2014)

Google, Cardboard VR(June 25, 2014)

Samsung Galaxy VR (Oculus + Note 4) 발표(Sep. 3, 2014)

Crescent Bay (Headphones)(Sep. 20, 2014)

Microsoft HoloLens(Jan. 21, 2015)

HTC – Valve, Revive (SteamVR)(Mar. 1, 2015)

LG – plastic cardboard(Feb. 10, 2015)

Apple acqr. Metaio(May 28, 2015)

Google invst. Magic Leap(Oct. 21, 2014)

but, no one seriously thought SOUND

Immersive & Interactive 3D SOUND

Sound and Room

Sound is a Wave

Sound = Longitudinal waveSound is a mechanical wave resulted from the longitudinal motion of the particles of the medium through which the sound wave is moving

Anechoic and Reverberant Room

Room Impulse Response

Direct Sound

sound object

microphone

0 Time

Freq

A

A


DS + 1 reflection (echo)

0 Time

Freq


DS + Early Reflections (Multiple Echoes)

0 Time

B-2案 _ IN SEARCH OF THE ORIGIN 음향 시스템의 기원인 축음기의 트랙으로부터 영감을 얻어 끊이지 않고 플레이되는 GAUDI 의 음향을 표현

장식적이고 복고스러운 개성이 강하여 아이덴티티로서 베스트안은 아님


DS + ER + Late Reverberations

0 Time

Late Reverberation

Direct Sound Early Reflections



Sound and Binaural

Binaural

Widely used to mean “headphone virtualization”

Having or relating to two ears

Binaural Hearing

High Frequency (above 4kHz)

• Inter-aural Level (intensity) Difference

• ILD

Low Frequency (Below 1kHz)

• Inter-aural Time (phase) Difference

• ITD L1 L2

Cone of Confusion

How resolve front-back-above-below ?

?

Head and Outer Ear (pinna)

We can since we are trained !!

Spatial Cues

Azimuth cueCaused by head shape and pinna

ILD, ITD

Elevation cueCaused by shape of pinna

Freq. peak and notch

Range cueLoudness, motion parallax, excess ILD, ratio of direct to reverb

HRTF

HRTF: Head Related Transfer FunctionHRIR: Head Related Impulse Response

HRTF Measurement

Binaural Room Impulse Response

BRIR = RIR + HRIR



Binaural Recording v. Binaural Rendering

Binaural

Binaural Recording

Sound Source(Wave) RIR HRTF Human

Perception

Binaural Rendering

Binaural Recording

Manikin measures binaural sound for “general” human

Binaural Recording

Capture immersive sound there

Binaural Rendering

Synthesize immersive sound here

*hL(t)

hR(t) *HRTF Database

mono

Recording v. Rendering

Binaural Recording Binaural Rendering

OOOOO Complexity O

OOOO Immersive OOO

OO Interactive OOOOO

O Personalization OOOOO

Vs.

Challenges in Implementation

Technical Challenges

1. Interactive in Binaural Recording

2. Expensive to filter in real time

3. Hard to keep both timbre and localization

4. Hard to cover full 3D space with measured HRTF

5. Hard to match to visual content

6. Hard to get ‘one-size-fits-all’ HRTF

B-1案 _ A HOMAGE TO GAUDI 가우디 건축물의 도형으로부터 모티브를 얻어 기하학적인 형태로 문자를 표현

Interactive in Binaural Recording

Issues on 360 binaural recording


Match to visual content


Implementation Example

A-1案

Summary

Immersive sound is NOT conventional 3D soundFor HMD, Immersive and Interactive are essentialBinaural means Headphone VirtualizationBinaural Rendering vs. Binaural RecordingStill open issues

Demo is OPEN in booth #5

Q&A

Thank You

Technology

[134] immersive sound vr