27
Audio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG)

Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

  • Upload
    others

  • View
    17

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Audio Device ClientBetter and Faster Audio I/O on Web

Hongchan ChoiGoogle ChromeWeb Audio API Spec Editor (Audio WG)

Page 2: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Proposal: Audio Device Client

• Work-in-progress! (in Audio CG)

Page 3: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Proposal: Audio Device Client

• Low-level audio I/O

• Better access to hardware

• A dedicated scope runs on RT thread

Page 4: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

"But... why?"

Page 5: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

"To close the App Gap for audio."

Page 6: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Mac OS Linux Windows Web Platform

Core AudioPulseAudio

or Jack(ALSA)

WASAPIor ASIO ?

Page 7: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

"Web Audio API?"

Page 8: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Issues: Web Audio API (around 2013)

• Extensibility (W3C TAG review, 1st round)

- No AudioNode subclassing

- ScriptProcessorNode unfit for purpose

Page 9: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Audio Worklet

Page 10: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Audio Worklet Processor

User codeAudio Worklet Node

audiocallback

processor.js(audio thread)

index.js(main thread)

Page 11: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

"How do I port X with it?"

Page 12: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Case study 1:Audio Worklet + WASM

Page 13: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Problems: Audio Worklet + WASM

• Web Audio API render quantum == 128 sample frames(less than 3ms at 44.1Khz)

• Not suitable for large-scale audio application

Page 14: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Case study 2:AW + SAB + WASM Worker

Page 15: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Worker

SharedArrayBuffer (Audio Data)

Input BufferAudioWorklet

ProcessorOutput Buffer

SharedArrayBuffer (States)

Wait/Wake, Buffer indexes

Page 16: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Problems: AW + SAB + WASM Worker

• Low thread priority of Worker

• Overly complex setup for a basic task

Page 17: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

• Low-level audio I/O

- Isochronous callback-based audio I/O

- Suitable for WASM-powered audio processing

Proposal: Audio Device Client

Page 18: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

• Better access to hardware

- Configurable render block size, sample rate, and I/O channels

- Constraints-based hardware configuration

Proposal: Audio Device Client

Page 19: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

• A dedicated scope runs on RT thread (if permitted)

- No more complex plumbing and thread hops

- For optimum WASM-powered audio processing

Proposal: Audio Device Client

Page 20: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

• Game audio engine

• Pro-audio applications (music production)

• Client-side spatialization (AR/VR)

• Teleconference

Potential Use Cases

Page 21: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Code Examples

Page 22: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Setting up device constraints

/* async scope: main global scope */const devices = await navigator.mediaDevices.enumerateDevices();

// Scenario: device #0 and #2 are audio input and// output devices respectively.const constraints = { inputDeviceId: devices[0].deviceId, outputDeviceId: devices[2].deviceId, sampleRate: 32000, callbackBufferSize: 512, inputChannelCount: 2, outputChannelCount: 6,};

Page 23: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

AudioDeviceClient

/* async scope: main global scope */const client = await navigator.mediaDevices.getAudioDeviceClient(constraints);await client.addModule('my-client.js');client.start();

// when the application ended

client.stop();

Page 24: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

AudioDeviceClientGlobalScope

/* my-client.js: AudioDeviceClientGlobalScope */import Engine from './my-audio-engine.js';

/** * @params {Array<Float32Array>} input * @params {Array<Float32Array>} output */const process = (input, output) => { Engine.process(input, output);};

setDeviceCallback(process);

Page 25: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Design issues: Integration

• WASM

• AudioContext

• WebRTC

Page 26: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

Security/Privacy Concerns

• Real-time priority thread

- Mitigation: ADC only can be spawned by "top-level document".

• Autoplay policy and secure context by default

Page 27: Better and Faster Audio I/O on WebAudio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio API Spec Editor (Audio WG) Proposal: Audio Device Client

github.com/WebAudio/web-audio-cg(explainer, IDL, code examples, and issue tracker)