Voice That Matter 2010 - Core Audio

Audio and OpenAL for iPhone Games

Kevin AvilaRegistered HEX Offender

• About Me• Core Audio• iPhone Services• OpenAL• Tips & Tricks

Introduction

About Me

email: eddienull@me.comtwitter: eddienull

About Me

email: eddienull@me.comtwitter: eddienull

About Me

email: eddienull@me.comtwitter: eddienull

About Me

email: eddienull@me.comtwitter: eddienull

About Me

email: eddienull@me.comtwitter: eddienull

Core Audio

Why?

Core AudioWhy?

"Easy" and "CoreAudio" can't be used in the same sentence. CoreAudio is very powerful,

very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things

out from sample code rather than reading high-level documentation.

–Jens Alfke, coreaudio-api listFeb 9, 2009

• Problem domain is hard

• Performance is hard

• Low latency is hard

• Reusability is hard

Core AudioWhy?

• Doing without would suck

• Slowness would suck

• Latency would suck

• Non-reusability would suck

Core AudioWhy?

Theory

How it Works

Pres

sure

+10

Inte

nsit

y

-10

+1.0f

-1.0f

Time

• Overview of Core Audio• Terminology• Fundamental Concepts

• ASBD• Properties

• Fundamental API• AudioFormat• AudioConverter• AudioFile

Core AudioIntroduction

Terminology

• Sample—a data point for one channel

Core AudioTerminology

Inte

nsit

y

+1.0f

-1.0f

Time

• Frame— The number of samples presented at one time• 1 for mono• 2 for stereo• 4 for quad

Core AudioTerminology

• Packet—a collection of Frames

Core AudioTerminology

The Basics

• AudioSampleType• Used for I/O• 32-bit float (Mac)• 16-bit integer (iPhone)

• AudioUnitSampleType• Used for DSP• 32-bit float (Mac)• 8.24 fixed (iPhone)

Core Audio OverviewThe Canonical Format

• Contains the minimal information needed to describe audio data

• Some formats may not use all of the fields• Unused fields need to be set to zero

Core Audio OverviewAudioStreamBasicDescription - “ASBD”

struct AudioStreamBasicDescription { Float64 mSampleRate; UInt32 mFormatID; UInt32 mFormatFlags; UInt32 mBytesPerPacket; UInt32 mFramesPerPacket; UInt32 mBytesPerFrame; UInt32 mChannelsPerFrame; UInt32 mBitsPerChannel; UInt32 mReserved;};

• Key/Value pair used to describe/manipulate API attributes.

• Scope and element selectors are used by some API to further qualify a property

• The value of a property can be of whatever type the API needs.

• APIs share several common functions• GetProperty, SetProperty, and GetPropertyInfo• AddPropertyListener and RemovePropertyListener

Core Audio OverviewProperties

• An element is the same as a bus

Core Audio OverviewScopes and Elements

input scopeelement 0

output scopeelement 0

output scopeelement 1

global scope

• Provides information about installed codecs

• Fills out ASBDs based on Format ID

• Provides more information about a formatʼs parameters

Core Audio OverviewAudioFormat

• ASBDs can be complicated, let the system do the work for you!

Core Audio OverviewAudioFormat

asbd.mSampleRate = 44100.0;asbd.mFormatID = kAudioFormatMPEG4AAC;asbd.mChannelsPerFrame = 2;

UInt32 propSize = sizeof(AudioStreamBasicDescription);AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL,!! ! ! ! &propSize, &asbd);

• Converts• bit depths• sample rate• interleaving & deinterleaving• channel ordering• PCM <-> compressed/encoded

• Can use all installed codecs

Core Audio OverviewAudioConverter

• Parses a file and provides access to the raw data• Uses properties to query information about the file

• ExtendedAudioFile• High-level access to an audio file• Combines AudioFile + AudioConverter

Core Audio OverviewAudioFile

For Example...

Core Audio OverviewSimple File Reading

1

2

3

// Open the audio fileExtAudioFileOpenURL(fileURL, &inputFile);

// Get the file’s audio data formatAudioStreamBasicDescription inputFileFormat;UInt32 propSize = sizeof(AudioStreamBasicDescription);ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileDataFormat, &propSize, &inputFileFormat);

// configure the output audio format to native canonical format!AudioStreamBasicDescription outputFormat = {0}; outputFormat.mSampleRate = inputFileFormat.mSampleRate;!outputFormat.mFormatID = kAudioFormatLinearPCM;!outputFormat.mFormatFlags = kAudioFormatFlagsCanonical;!outputFormat.mChannelsPerFrame = inputFileFormat.mChannelsPerFrame;!outputFormat.mBitsPerChannel = 16;

UInt32 propSize = sizeof(AudioStreamBasicDescription);AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &propSize, &outputFormat);

Core Audio OverviewSimple File Reading

4

5

6

// Set the desired decode data formatExtAudioFileSetProperty(inputFile, kExtAudioFileProperty_ClientDataFormat, sizeof(outputFormat), &outputFormat);

// Get the total frame countSInt64 inputFileLengthInFrames;UInt32 propSize = sizeof(SInt64);ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileLengthFrames, &propSize, &inputFileLengthInFrames);

// Read all the data into memoryUInt32 dataSize = (inputFileLengthInFrames * outputFormat.mBytesPerFrame); void *theData = malloc(dataSize);AudioBufferList dataBuffer;!dataBuffer.mNumberBuffers = 1;!dataBuffer.mBuffers[0].mDataByteSize = dataSize;!dataBuffer.mBuffers[0].mNumberChannels = outputFormat.mChannelsPerFrame;!dataBuffer.mBuffers[0].mData = theData;

ExtAudioFileRead(inputFile, (UInt32*)&theFileLengthInFrames, &dataBuffer);

iPhone Services

• Audio Sessions• Categories• Interruptions• Routes

• Hardware Acceleration

iPhone Services Overview

Audio SessionFundamental Concepts• Describes an applicationʼs interaction with the audio system

• Represents the current state of audio on the device• Current Settings• Preferences

• State Transitions• Interruptions• Route Changes

• Session Settings• Influences all audio activity

• Except UI sound effects• Current Session Characteristics

Audio SessionSettings & Preferences

• Identify a set of audio features for your application• Mixable with others• Have input or output• Silence on Screen Lock or Ringer Switch

Audio SessionCategories

Audio SessionBasic Setup

1

2

3

// Get the session instanceAVAudioSession *mySession = [AVAudioSession sharedInstance];

// Implement delegates to handle notificationsmySession.delegate = self;

// Establish appropriate category[mySession setCategory:AVAudioSessionCategoryAmbient error:nil];

4// Activate the session[mySession setActive:YES error:nil];

• System forces session to ʻinactiveʼ• Unable to resume until interrupt task is complete

• AVAudioSession delegates• -(void) beginInterruption

• Update UI to reflect non-active audio state

• -(void) endInterruption• Resume audio, update UI

Audio SessionInterruptions

Audio SessionDefining Interruption Delegates

-(void) beginInterruption{if (isPlaying){

wasInterrupted = YES;isPlaying = NO;

}}

-(void) endInterruption{if (wasInterrupted){

[[AVAudioSession sharedInstance] setActive:YES error:nil];[self startSound];! ! !wasInterrupted = NO;

}}

•The pathway for audio signals

•Where is audio output to?

•Where is audio input from?

•“Last in Wins” rule

•Notification when route changes

•Reason why the route changed

AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange, audioRouteChangeListenerCallback, self);

Audio SessionDefining a Property Listener Callback

void audioRouteChangeListenerCallback ( void *inUserData, AudioSessionPropertyID inPropertyID UInt32 inPropertyValueSize, const void *inPropertyValue){ MyAudioController *controller = (MyAudioController *)inUserData;

if (inPropertyID != kAudioSessionProperty_AudioRouteChange) return;

if(controller.isPlaying != NO) {!! NSDictionary *routeChangeDictionary = (NSDictionary*)inPropertyValue;!! SInt32 routeChangeReason = [[routeChangeDictionary objectWithKey: ! ! ! CFSTR (kAudioSession_AudioRouteChangeKey_Reason)] intValue];

if (routeChangeReason == kAudioSessionRouteChangeReason_OldDeviceUnavailable) {!! ! [controller pause]; } }}

• Low CPU cost, low power consumption

• Supported HW decoders:• AAC / HE-AAC• Apple Lossless • MP3• IMA4 (IMA/ADPCM)

• Supported HW encoders:• AAC (3GS)

Audio SessionHardware Accelerated Codecs

// Override our current categories ‘mix with others’ attributeUInt32 value = 1;AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers,

sizeof(value), &value);

Audio SessionEnabling Hardware Acceleration

• Must set “Mix With Others” to false

• Overrides not persistent across category changes

OpenAL

• Powerful API for 3D audio mixing• Designed for games• Cross-platform, used everywhere!

• Models audio in 3D space, as heard by a single listener

• Designed as a compliment to OpenGL• Mimics OpenGL conventions• Uses the same coordinate system

• Implemented using Core Audioʼs 3D Mixer AU

OpenAL

• Context• The spatial environment• Contains the listener

• Source• 3D point emitting audio• Many attributes to control rendering

• Buffer• Container for audio data• alBufferData() - copies data to internal buffers• alBufferDataStatic() - application owns memory

OpenALFundamental Concepts

OpenALArchitecture

OpenAL

iPhone Hardware

OpenAL Device

ContextListener

Buffer Buffer nBuffer Buffer

Source nSource Source

• Only 1 per context• Positionable just like Sources• Represents the userʼs experience in the 3D environment• Orientation described by two Vectors:

• AT = Direction the Listener is facing• UP = Direction pointing up from the top of the Listenerʼs head

OpenALListener

// Orient the Listener facing +ZALfloat listenerOrientation[6] = {!0.0, 0.0, 1.0, // AT!! 0.0, 1.0, 0.0} // UPalListenerfv(AL_ORIENTATION, listenerOrientation);

• Applies to Listener• Applies to Sources (Mono-only)• Cartesian coordinate system

OpenALPositioning

ALfloat sourcePosition[] = {!0.0, 25.0, 0.0}alSourcefv(AL_POSITION, sourcePosition);

ALfloat listenerPosition[] = {!0.0, 2.0, 0.0}alListenerfv(AL_POSITION, listenerPosition);

OpenALCartesian Coordinates

x:0, y:0, z:+1 = Listener facing the Positive Z

—z

+x

+

—

OpenALCartesian Coordinates

x:0, y:0, z:-1 = Listener facing Negative Z

—z

+x

+

—

OpenALBasic Setup

1

2

3

// open an OpenAL DeviceoalDevice = alcOpenDevice(NULL);

// Create a new OpenAL Context (and listener)oalContext = alcCreateContext(oalDevice, NULL);

// Set our new context to be the current OpenAL ContextalcMakeContextCurrent(oalContext);

OpenALCreating Buffers and Sources

4

5

// Create an OpenAL buffer to hold our audio dataalGenBuffers(1, &oalBuffer);

// Fill the OpenAL buffer with dataalBufferDataStatic(oalBuffer, AL_FORMAT_MONO16, audioData, audioDataSize, 44100);

// Create an OpenAL Source objectalGenSources(1, &oalSource);

// Attach the OpenAL Buffer to the OpenAL SourcealSourcei(oalSource, AL_BUFFER, oalBuffer);

Distance Attenuation

• Describes the reduction in volume based on the distance to the listener.

• Set distance model on the context with alDistanceModel()AL_INVERSE_DISTANCEAL_INVERSE_DISTANCE_CLAMPEDAL_NONE

• Configure Source attributesAL_REFERENCE_DISTANCEAL_MAX_DISTANCEAL_ROLLOFF_FACTORAL_SOURCE_RELATIVE

OpenALAttenuation by Distance

OpenALAttenuation by Distance

dB

+20

-20

0

Distance from Listener

// Set the distance model to be usedalDistanceModel(AL_INVERSE_DISTANCE_CLAMPED);

200

OpenALAttenuation by Distance

reference distance

dB

+20

-20

0

Distance from Listener

// Set the Source’s Reference DistancealSourcef(mySource, AL_REFERENCE_DISTANCE, 2.0);

200

OpenALAttenuation by Distance

reference distance

dB

+20

-20

0

Distance from Listener

// Set the Maximum DistancealSourcef(mySource, AL_MAX_DISTANCE, 30.0);

500

maximum distance

OpenALAttenuation by Distance

reference distance

dB

+20

-20

0

Distance from Listener

// Set the Rolloff FactoralSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0);

500

maximum distance

The Doppler Effect

• No Motion = No Doppler• Doppler only describes the warping of sound due to motion

OpenALThe Doppler Effect

OpenALThe Doppler Effect

• The default value is 0.0 (disabled)• enabling has small CPU cost

• The normal value is 1.0

OpenALThe Doppler Effect

alDopplerFactor(1.0);

• Describes the speed of sound in your universe (per second)

OpenALThe Doppler Effect

// 1000 units per secondalDopplerVelocity(1000);

OpenALPutting it all together- (void)initOpenAL{! ALenum!! ! error;

!! device = alcOpenDevice(NULL);! if (device != NULL)! {! ! context = alcCreateContext(device, 0);! ! if (context != NULL)! ! {! ! ! alcMakeContextCurrent(context);

!! ! ALfloat listenerPosition[] = {! 0.0, 2.0, 0.0} alListenerfv(AL_POSITION, listenerPosition);

! ! ! alGenBuffers(1, &buffer);! ! ! if((error = alGetError()) != AL_NO_ERROR)! ! ! ! exit(1);

!! !! ! ! alGenSources(1, &source);! ! ! if(alGetError() != AL_NO_ERROR) ! ! ! ! exit(1);! ! }! }

!![self initBuffer];!![self initSource];}

OpenALPutting it all together- (void) initBuffer{! ALenum error = AL_NO_ERROR;! ALenum format;! ALsizei size;! ALsizei freq;

!! data = MyGetOpenALAudioData(fileURL, &size, &freq);

!! alBufferDataStatic(buffer, AL_FORMAT_MONO16, data, size, freq);

!! if((error = alGetError()) != AL_NO_ERROR)! {! ! NSLog(@"error attaching audio to buffer: %x\n", error);! }! !

}

OpenALPutting it all together- (void) initSource{! ALenum error = AL_NO_ERROR;

! alSourcei(source, AL_BUFFER, buffer);

! alSourcei(source, AL_LOOPING, AL_TRUE);!! float sourcePosAL[] = {sourcePos.x, kDefaultDistance, sourcePos.y};! alSourcefv(source, AL_POSITION, sourcePosAL);

!! alSourcef(source, AL_REFERENCE_DISTANCE, 5.0f);

!!! if((error = alGetError()) != AL_NO_ERROR)! {! ! NSLog(@"Error attaching buffer to source: %x\n", error);! }!

}

• Youʼre now ready to go!

OpenALPutting it all together

alSourcePlay(source);if((error = alGetError()) != AL_NO_ERROR){! NSLog(@"error starting source: %x\n", error);}

Tips & Tricks

• High-Quality laser *pew pew!* and *beeps* unnecessary• Example: ʻafconvert -f caff -d LEI16@8000ʼ

• The more SRCs the less performance you get

Sample Rates

What Next?

Coming soon eventually

The End