Upload
nicole-head
View
27
Download
5
Tags:
Embed Size (px)
DESCRIPTION
“Ah-Ha! Moments”: Where Science Meets Art and Practice in Digital Sound, Part 1. Jennifer Burg. CCLI Workshop Series: “Linking Science, Art, and Practice through Digital Sound” Workshop 1, August 11 and 12, 2008 Wake Forest University. - PowerPoint PPT Presentation
Citation preview
“Ah-Ha! Moments”: Where Science Meets Art and Practice in Digital Sound,
Part 1
CCLI Workshop Series: “Linking Science, Art, and Practice
through Digital Sound”Workshop 1, August 11 and 12, 2008
Wake Forest University
This work was funded by National Science Foundation CCLI grant DUE-0717743 Jennifer Burg PI, Jason Romney Co-PI
Jennifer Burg
“Ah-Ha Moments” are moments when the lights go on, and you see something clearly
for the first time.
The relationships between things Why something is so How something works What something applies to
Often this requires seeing the thing from a different perspective or in context
The Effect of Bit Depth in Quantization
Rounding to discrete quantization levels causes error. In Audition
The error is itself a wave. In MATLAB
Visualizing the Error from Quantization
0 1 2 3 4 5 6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
sine wave
quantized sine wave
error wave
Dithering Add a random amount between -1 and 1
(scaled to the bit depth of the audio file) to each sample before quantizing.
There will be fewer consecutive samples that round to the same amount. Rounding to 0 is the worst thing, causing breaks.
In Audition
Noise Shaping Raise error wave above the Nyquist frequency. Do this by making the error go up if it was
previously down and down if it was previously up. This raises the error wave’s frequency.
The amount added to a sample depends on the error in previous sample. In Audition
iii
ii
iiii
outFinFE
inFoutF
cEDinFinF
__
____ 1
Ah-ha!
Spectral view of quantization noisesith dithering. (Click picture to hear.)
Spectral view of quantization noisewith dithering and noise shaping. Click picture to hear.)
Spectral view of quantization noise. (Click picture to hear.)
original sound file
Frequency Components
The spectral views just shown introduce the idea of frequency components of a sound wave.
The digitized sound wave can be stored in one of two ways: Time domain – a list of values representing the
amplitude of the sound wave at evenly-spaced moments in time
Frequency domain – a list of values representing “how much” of each frequency is present in the wave
The Relationship Between a Frequency Response and an Impulse Response
Frequency Response(how much of each frequency willbe retained after filtering,on a scale of 0 to 1)
Impulse Response(a graph of the values in the convolution mask,in the time domain)
The Fourier transform of the impulse response gives the frequency response.
Impulse Response = Convolution Mask
Filtering can be done in the time domain by applying a convolution mask to the sound samples.
The impulse response is a convolution mask.
How Does Convolution Work to Filter Frequencies of a Sound Wave?
Click to animate
x is a list of sound samples – It’s the digitized sound wave.h is a list of values that constitute the convolution mask – i.e., the filter in the time domain.y is a list of sound samples that constitute the sound wave after it has been filtered.
Impulse Response vs. Frequency Response
impulse response = convolution mask = filter in time domain
frequency response = a graph of what the filter does in terms of how much each frequency is boosted or attenuated
impulse response = inverse Fourier transform of frequency response
frequency response = Fourier transform of impulse response
Frequency Response of Idealized vs. Realistic Low-Pass Filter
0
1
fc0
1
passband stopbandtransitionband
frequency
fra
ctio
n o
f fre
qu
en
cy p
rese
nt i
n fi
ltere
d s
ign
al
fc
idealized low-pass filter realistic low-pass filter
Going from Frequency Response to Impulse Response
If you know what frequency response you want from a filter, how do you get the corresponding impulse response ?
In the ideal, where the frequency response is a rectangular function, the frequency response and impulse response are both Fourier transforms of each other and both inverse Fourier transforms of each other.
To get the impulse response from the idealized frequency response, take the Fourier transform of the frequency response
Creating a Low-Pass Filter You can do it yourself in MATLAB:
Create the filter using the given function, sin(2fc)/n Read in an audio clip Since this is a filter in the time domain, convolve
audio clip with the filter Listen to the result Graph the frequencies of filtered clip against the
unfiltered clip. (Do this by taking the Fourier transform of each first.)
See the demonstration and worksheet for details.
Creating a Vocoder in MATLABCreating a Vocoder in MATLABCreating a Vocoder in MATLABCreating a Vocoder in MATLAB
From http://www.paia.com/ProdArticles/vocodwrk.htm
Creating a Vocoder in MATLABfunction output = vocoder(input1, input2, s, window) input1 = input1'; input2 = input2'; q=(s-window); output = zeros(1,s); for i=1:window/4:q b = i+window-1; input1partfft = fft(input1(i:b)); input2partfft = fft(input2(i:b)); input1fft(i:b) = input1fft(i:b) + abs(input1partfft); input2fft(i:b) = input2fft(i:b) + abs(input2partfft); mult = input1partfft.*input2partfft; output(i:b) = output(i:b)+ifft(mult); end output = output/max(output);end
Demonstration
That lead me to an ah-ha moment this morning!
Audition’s vocoder Comparison of my vocoder and Audition’s
Kinds of Ah-Ha Moments
Ah-ha! I know how that works now! Ah-ha! I know why someone would be
interested in using that! Ah-ha! I know why it matters to know that!
Application Environments:Audition, Logic Pro, Sound Forge,Pro Tools, Sonar, Music Creator,Reason
Hands on work for some real purpose: music, theatre, television, movie-making
How things work: mathematics, algorithms,and technology
How does knowledge of the math and science help to make the real work better?
Interplay Between Science, Art, and Practice
If you put artist/practitioners together with computer scientists, does one group shed light on the work of another?
What kinds of ah-ha moments emerge? Do artist/practitioners create better products if they
understand more about how things work? Computer scientists like to understand how things
work. But would it help them to know why it matters!
If artist/practitioners know who things work, they can be more purposefully experimental (as opposed to “click on things and see what happens”).
They have more power over their tools to be original and creative.
Consider this in the visual arts…
What Jason and I learned in our Digital Sound Production Workshop
Yes, definitely, when music students understand their tools, they use them more effectively.
Yes, definitely, when computer science students see what musicians want from their tools, they have ideas for how to create new and better tools. Also, understanding something about the music sheds light on how to make the tools work.
Ah-ha Moments for Music Students
Rewiring Cakewalk Music Creator to Combining digital audio and MIDI
Demonstration
MIDIdevice (acceptsuser input, turnsit into aMIDI message)
sequencer(e.g.MusicCreator, storesthe MIDI message)
Reason(synthesizessounds or producesthem from samples)
sequencer(sends digital soundto sound card)
Ah-ha Moments for Music Students Multi-band dynamics processing - L3-LL
Multimaximizer plug-in Unprocessed audio Processed audio after using the plug-in
Ah-ha Moments for Music Students In order to understand this tool, students
needed to know something about frequency, dynamic range, and ADRS envelopes
Dynamics Processing Plug-In from Audition
Ah-ha Moments for Music Students Creating and editing MIDI samples
Samplers and synthesizers are not the same. There’s a lot going on in a sample bank. You can edit a sample bank yourself. A MIDI message can mean whatever you want it to
mean. You can create your own sample banks. Loops and samples aren’t the same thing. It’s actually possible to work creatively with loops. You
can edit the samples from which they’re created, or edit the why the loops are put together.
The Instrument Editor in Logic ProThere’s a lot going on in a sample bank, and you can have access to it!
Ah-ha! A MIDI message can mean whatever I want it to mean!
A control change message (or a pitch bend messageor whatever message you choose) can be definedto mean “Go to the next instrument in the EXS24set of instruments.”
Ah-ha! You can make your own sample bank!
Making a sample bank of birds songs in Reason. Then rewiring Cakewalk Music Creatorthrough Reason, using this sample bank,and playing a jazz piece with the bird songsas instruments.
Listen
Then compose your own piece to use the birdSamples!
Listen
Using loops isn’t “cheating.” You can edit
loops and put them together creatively!
Editing loops in Reason
Ah-ha Moments for Computer Science Students
A MIDI message can mean whatever you want it to mean.
It really makes more sense to think of MIDI messages in hexadecimal rather than decimal.
Status Byte
10000000 is the lowest value a status byte can hold.
10000000 = 80 in hex, 128 in decimal
Generally we deal with MIDI bytes in hex because it makes it easier to program.
Slide courtesy of Jason Romney. Thanks.
Status BytesHex Binary Description
Channel Voice Messages
8n 1000b1b2b3b4 Note Off
9n 1001b1b2b3b4 Note On
An 1010b1b2b3b4 Polyphonic key pressure/Aftertouch
Bn 1011b1b2b3b4 Control change
Cn 1100b1b2b3b4 Program Change
Dn 1101b1b2b3b4 Channel pressure/Aftertouch
En 1110b1b2b3b4 Pitch bend change
n=channel number, in hexadecimal
n=channel number, in hexadecimal Slide courtesy of Jason Romney. Thanks.
Data Byte
MIDI data bytes follow a MIDI status byte
Status bytes tell a MIDI device what to do. Data bytes tell the MIDI device how to do it.
Data bytes are bytes with the 8th bit turned off. Consequently, data bytes cannot carry a value larger than 127 (7FH).
Slide courtesy of Jason Romney. Thanks.
Channel Voice MessagesStatus Byte
Data Bytes DescriptionHex Binary
8n 1000b1b2b3b4 0k1k2k3k4k5k6k7
0v1v2v3v4v5v6v7
Note OffK1k2k3k4k5k6k7: note number
v1v2v3v4v5v6v7: note off velocity
9n 1001b1b2b3b4 0k1k2k3k4k5k6k7
0v1v2v3v4v5v6v7
Note Onv1v2v3v4v5v6v7 ≠ 0: velocityv1v2v3v4v5v6v7 = 0: note off
An 1010b1b2b3b4 0k1k2k3k4k5k6k7
0v1v2v3v4v5v6v7
Polyphonic Key Pressure (Aftertouch)v1v2v3v4v5v6v7 : pressure value
Bn 1011b1b2b3b4 0c1c2c3c4c5c6c7
0v1v2v3v4v5v6v7
Control Changec1c2c3c4c5c6c7: control # (0-119)
v1v2v3v4v5v6v7: control value
Cn 1100b1b2b3b4 0p1p2p3p4p5p6p7 Program Changep1p2p3p4p5p6p7: program # (0-127)
Dn 1101b1b2b3b4 0v1v2v3v4v5v6v7 Channel Pressure (Aftertouch)v1v2v3v4v5v6v7: pressure value
En 1110b1b2b3b4 0v1v2v3v4v5v6v7
0v1v2v3v4v5v6v7
Pitch Bend Change LSBPitch Bend Change MSB
n = channel numbern = channel number
Slide courtesy of Jason Romney. Thanks.
A C program for reading MIDI message, converting them to frequencies, and sending
them to the sound card to be played. Programming assignment created for first or
second year programming students, by John Brock while(j){
if(k&0x80){ d1 = fgetc(midi); if(k != 0xC0 && k != 0xD0){ d2 = fgetc(midi); } if(k == 0xFF && d1 == 0x2F) j = 0; if(k == 0x90){ freq = 8.1758*pow(2, d1/12.0); if(d2 != 0){ for(i = 0; i < (shmsz); i++){ samp[i] += sin(freq*(2.0*M_PI/RATE)*i); } ….. and so forth
You can see that Johnis working in hexadecimal.
Hey! Let’s try using an autotuner!
The music student wants to use it. The computer science student wants to make his
own and understand how it works. Ah-ha, they both say! The human voice has
harmonic frequencies, and if you don’t know that, you can’t create a vocoder!
Ah-ha, they both say! Human perception of sound is non-linear! Now I know what that means and why it matters!
Questions for this workshopQuestions for this workshopQuestions for this workshopQuestions for this workshop
How do we relate the science to art and practice? How much science does the artist/practitioner
need? Where are the points where knowing the science
results in better work? How do we change the computer science
curriculum to retain the science but relate it more interestingly to art and practice?
Tell us about your own Ah-ha moments!Tell us about your own Ah-ha moments!Tell us about your own Ah-ha moments!Tell us about your own Ah-ha moments!
We’d love to hear about your ah-ha moments in this workshop – if there are moments when you say,
“Oh, that’s how it works! That makes me think of something that I’d like to do!”