Download pdf - Final Report - University of Floridathe single byte data into two bytes that include an op code. Low pass Filter The data leaving the DAC goes through a low pass 4.77kHz op-amp filter

EEL 4924 Electrical Engineering Design

(Senior Design)

Final Report

25 April 25, 2012

Project Name: Keyboard Cam Team Smision: Donavon Bryan and Sydney Greene

Team Members:

Name: Donavon Bryan Name: Sydney Greene

Project Abstract

For this project we made a keyboard that is controlled by the user's hands and feet. We created a

peripheral that connects to a computer via USB. This device handled all of the note generation playback

and outputs through a 3.5mm audio jack. A camera connected to the host computer is also required.

Controlling the virtual keyboard requires a special board that can be strapped to ones shoes. The board

will light a red LED when the shoe makes contact with the floor. The PC runs a program written with

OpenCV libraries to determine what key has been pressed. The PC sends this data to the microcontroller

through USB. The area that the camera sees is divided into sections that represent keys. Depending on

the section you step on, different notes will play. These notes are digitally sampled tones from various

instruments and other sources such as a piano, guitar, pure sine waves, and a celebrity sound board.

University of Florida EEL 4924—Spring 2012 25-Apr-122 Electrical & Computer Engineering

Page 2/28 Final Report: Keyboard Cam

Table of Contents

Project Abstract………………………………………………………………...…..1

Project Features/Objectives..……………………………………………………….2

Project Architecture…………………………………………………...……………3

Concept/Technology Selection……………………………………………………..6

Cost Objectives……………………………………………………...…………..…7

Division of Labor…………………………………………………………….……8

Project Schedule……………………………………………………………...…….8

Future Work………………………………………………………………...………8

Appendix A: PCB Layouts………………………………………….……….……..9

Appendix B: Microprocessor Code………………………………………………..11

AppendixC: Tracking Code………………………………………………………...

List of Figures and Tables

Figure 1- Block Diagram………………………………………………………….3

Figure 2- Tracking Vision Example……………………………………….………5

Figure 3- Picture of LED Strap……………………………………………………5

Figure 4- Picture of Main Board…………………………………………………..6

Figure 5- Gantt Chart……………………………………………………..……….8

Figure 6- Main Board PCB………………………………………………………..9

Figure 7- LED PCB………………………………………………………….…..10

Table 1- Bill of Materials………………..…………………………………….…..7

Table 2-Member Responsibility table……………………………………………..8



Project Features/Objectives

Ability to play music on an oversized keyboard with ones feet and no bulky mat or peripherals

besides a camera and LED shoe strap.

Different instruments selectable via a hand motion using a special colored glove.

Digitally sampled audio files stored on an SD card. Can easily be expanded to support more

instruments.

3.5mm audio output jack. Audio signal can be sent directly to speakers, mixing board,

headphones.

USB powered. No batteries required!

Real time response. Can play songs with no noticeable latency between key presses

Project Architecture

The solution involved 2 major components: the board that handles all of the audio playback and retrieval

and the player tracking system. Figure 1 illustrates the connection between the two systems and their

basic functions



Figure 1. Basic Block Diagram of the Entire System

Audio Playback: Computer to Microcontroller through USART

After image processing, the PC sends a two byte code. The bottom bit of the high byte will toggle the

instrument when equal to 1. The low byte contains a key in each nibble. The uC determines which key is

new and plays it. If another key was also just recently pressed (within a fifth of a second) a chord will

play.

Microcontroller SPI Communication with SD Card

Depending on the instrument selected and the key pressed, the uC retrieves a sound pattern from the SD

card. If the instrument is a sine wave, it takes the sine wave file and stores it locally to be looped for a

few seconds. For the other instruments, the uC takes the data from the SD card and sends it straight to

the DAC. We chose a SPI frequency that matched up with our audio sampling frequency so that the

audio would play correctly. The audio files were written raw to the SD card in advance by first

converting them from a .wav to a hex array in matlab and then pasting them into HxD, a freeware hex

editor.

Chords are available for the piano and guitar. A chord only plays if you press the second key very soon

after the first. This gives you the ability to quickly transition to a new key when playing a chord is not

your intention. The uC reads chords the same way as regular keys but they are paged in a different

fashion. They were also stored with an algorithm instead of manually. This method was necessary

because you cannot read two memory locations on an SD card at once and the audio files are too large to

be saved to the uC.



Microcontroller USART SPI Mode Communication with DAC

Using the second USART peripheral is Master SPI mode, the uC sends audio to the DAC. It separates

the single byte data into two bytes that include an op code.

Low pass Filter

The data leaving the DAC goes through a low pass 4.77kHz op-amp filter to smooth the jagged edges.

None of our musical notes are about 5kHz so this was a good cutoff.

Audio Amplifier

We used a power op-amp audio amplifier circuit to increase our output. The original DAC signal never

comes near the voltage rails in order to decrease distortion at the filter. The audio amp takes care of the

necessary gain.

Audio Output

We have the option of a through hole speaker and an eighth inch audio jack that you can choose between

with a physical switch.

Player Tracking The first part of this system involves the peripherals for the player’s feet. There is one board for each

foot and they are secured to the player’s foot with Velcro straps. Along the strap is a force sensitive

resistor that should be placed in the middle of the shoe’s bottom. This FSR is connected to one of the

inputs of a comparator circuit that has a set reference voltage on the other input. The resistance of the

FSR is inversely proportional to the force applied and so when it makes contact with the ground the

voltage on the pin will go down. This causes the LED at the output of the comparator to light up. A

diffuser was required for the LED in order to extend the range of play and make it easier for the camera

to detect it at reasonable distances.

The tracking software relies on the OpenCV computer vision library. It first searches the pixel by pixel

and removes all color that is not red. When it encounters a red pixel within the specified threshold it will

place a pixel of max value in that location. The process is illustrated in figure 2.

Figure 2. What the tracking software “sees”

Various smoothing and dilation OpenCV functions are applied to make sure the area of interest is

accentuated.



The program then searches the image for any nonzero pixels and places a box around it. The center of

mass of this box is calculated and compared to the division of keys and the note that was pressed is

determined. This information is then relayed to the board via USB-USART communication. For two feet

the process remains basically the same but extra steps must be taken to ensure that you are finding

distinct feet. This is done by searching the image in opposite directions. The algorithms only ever search

the play area in order to increase performance and it is assumed that the player will only ever play in the

actual play area.

Figure 3. Picture of LED shoe strap



Figure 4. Picture of Final Board

Concept / Technology Selection

This design was guided by minimizing cost while still providing the quality final product. Each

component and system that was used is cheap, cost effective and exactly what our solution required.

The microcontroller we used was an Atmega324p 8 bit microcontroller. This was the heart of the

audio part of our project and it was exactly what we needed. We required a total of 2 SPI ports

and a single USART. The 324p has 1 SPI and 2 USARTs (each configurable as a SPI in Master

mode). Our running clock frequency was fast enough to receive commands, read from SD card

storage, and play notes in real time which was a major concern during the design phase.

SD card storage was chosen because of their relatively cheap cost. It also enabled us to change

the instruments with very little effort and no special hardware or programmers. It can be done

independent of the microcontroller.

OpenCV afforded us a lot of processing power and tools to get the tracking working. It also

helped teach us a lot about image processing algorithms and standards.

The audio filter was an active filter with a cutoff frequency that allowed the the most common



frequencies to pass through.

Our particular LED/shoe board was designed in order to make our system as portable and

convenient as possible. Players can be set up to play the game in a matter of seconds and it does

not hinder the movement or gameplay at all.

Cost Objectives Keyboard CamParts list

Item Quantity Description

Unit Price

Total Cost

LM393 2 dual comparator 2.87 5.74

Atmega324P 1 8 bit microcontroller 5.36 5.36

LT1661 1 10 bit DAC 3.39 3.39

LT1632 1 dual OP-AMP 6.66 6.66

LM386 1 power amplifier 0.91 0.91

Speaker 1 8 Ohm speaker 4.33 4.33

10k Pot 1 resistor 0.92 0.92

10R 1 resistor 0.09 0.09

500R 1 resistor 0.09 0.09

1kR 1 resistor 0.09 0.09

100KR 2 resistor 0.09 0.18

910R 2 resistor 0.09 0.18

Red LED 3 red LED 0.43 1.29

270uF 1 polar capacitor 0.04 0.04

0.047uF 1 ceramic capacitor 0.08 0.08

10uF 3 polar capacitor 0.04 0.12

47nF 1 ceramic capacitor 0.08 0.08

20nF 1 ceramic capacitor 0.08 0.08 SD card Breakout 1 mount/ breakout for SD card 12.95 12.95

FDTI FT232RL 1 serial to parallel chip 4.5 4.5

FSR 2 force sensitive resistor 6.95 13.9

2kR 3 resistor 0.09 0.27

TOTAL

61.25

Table 1. Bill of Materials



Division of Labor

Member Responsibility Table

Item Sydney Greene Donavon Bryan

Audio Playback 100% 0%

Video Processing 0% 100%

USB Power 20% 80%

USB Communication 20% 80%

SD Card Communication 80% 20%

Shoe Lights 0% 100%

Altium 50% 50%

Microcontroller Programming 50% 50%

Table 2. Division of Labor

Project Schedule

Figure 5. Gantt Chart

Future Work

The tracking software could be refined a bit to make it more resilient and robust against the environment.

Target acquisition could be refined a bit and mad more accurate. This could allow for the keyboard to be

implemented like an actual digital keyboard with actual black keys instead of just same sized keys for all

notes.

Ideally the design could be optimized and redesigned with an embedded processor that could free the

project from a computer. This could be done with a higher end processor and efficient openCV code or

even an FPGA. This would require major revisions so the board and possible scrapped of the tracking

code if the FPGA route is taken.

The library of instruments/ sounds could easily be expanded with more time. More elaborate hand

controls could be developed that would allow possibly parameter changing of notes, special effects, and



other commands.

Appendix A

Project PCBs Main PCB

Figure 6. PCB for the main board



LED PCB

Figure 7. LED PCB



Appendix B: Microprocessor Code

Main Code:

#include <avr/io.h>

#include <util/delay.h>

#include <avr/interrupt.h>

#include <avr/pgmspace.h>

#include <SDCARD/SD_routines.h>

#include <SDCARD/SPI_routines.h>

#include <SDCARD/UART_routines.h>

volatile unsigned long startBlock;

volatile unsigned long totalBlocks;

volatile unsigned char note1[122];

volatile unsigned long key1, key2, firstDataSector, rootCluster, totalClusters;

volatile unsigned int bytesPerSector, sectorPerCluster, reservedSectorCount;

unsigned char x, option, error, data, FAT32_active, input, commandhi, commandlow, key1command,

key2command;

unsigned int i, j, y, count, instrument, note1stop, note2stop, note1decay, note2decay, stop,

instrument_change;

unsigned char SD_tone(unsigned long startBlock, unsigned long totalBlocks);

unsigned char USART_Receive();

void USART_Transmit( unsigned char data );

//int y;

//int data;

//char x;

//char count;

//int instrument;

void port_init(void)

{

PORTA = 0x00;

DDRA = 0x01; //initialize PortA.0 as an output, this will be the CS

for the DAC

PORTB = 0xEF;

DDRB = 0xBF; //B6 (MISO from sd card) input, rest outputs

PORTD = 0x00;

DDRD = 0x7A; //D0 (RXD) and D2 (MISO_2) are inputs, rest are outputs

}

void SPI_Init_LowSpeed()

{

SPCR = 0x00;

SPCR |= (1<<SPE)|(1<<MSTR)|(1<<SPR1);//MSB first, Master Mode, Sample on Rising,

Leading Edge, Clock Speed 125kHz



SPSR = 0x00; //clear flags, clear 2x sck multiplier

}

void TimingInterrupts_Init(void)

{

TCCR1A &= 0xFC;

TCCR1A += 0; //CTC mode

TCCR1B &= 0xE0;

TCCR1B += 9; //CTC mode and /8 pre scale (8 + ps); ps = 1, CLK = 8Mhz, ps =2,

CLK = 1Mhz

TIMSK1 |= 2; //local OCF1A interrupt enable

TIFR1 |= 2; //make sure output compare A flag is initially cleared

OCR1A = 251; //we are consistently using 32,000 hz throughout so I'll just set this

once

}

void USART_Init()

{

/* Set baud rate */

UBRR0H = 0;

UBRR0L = 51;

/* Enable receiver and transmitter and recieve complete interrupt */

UCSR0B = (1<<RXEN0)|(1<<TXEN0)|(1<<RXCIE0);

/* Set frame format: 8data, 1stop bit */

UCSR0C = (3<<UCSZ00);

}

//The SD Card uses the second USART in SPI Mode

//RX1=MISO1, TX1=MOSI1, XCK1=SCK1, PD5=SD_CS

//USART SPI clock rate: 4Mhz

void SD_USART_spi_init(void)

{

UBRR1 = 0; //make

sure ubrr is 0

DDRD |= (1<<4); //set

XCK1 to output

UCSR1C = 0xC0; //select Master SPI Mode, sample on rising edge, clock low when idle,

MSB bite first, sample on rising edge

UCSR1B |= (1<<RXEN1)|(1<<TXEN1); //enable receiver and

transmitter

UBRR1 = 0;

//4MhZ operation

PORTA ^=1;

PORTA ^=1;

}



//call this routine to initialize all devices

void init_all(void)

{

cli();

TimingInterrupts_Init();

SD_USART_spi_init();

port_init();

SPI_Init_LowSpeed();

USART_Init();

WDTCSR |= (1<<5)|(1); //set watchdog prescaler to 8s

MCUCR = 0x00;

}

void main(void){

//unsigned char fileName[13];

cli();

instrument = 2;

PORTA |= (1); //Drive CS high

_delay_ms(1000); //wait for stabilization

init_all();

SD_init(); //this method comes from the Dharmani SDcard library, it initializes the sd

card for spi

SPI_MUSIC_SPEED(); //SCK - 62.5kHz

_delay_ms(1);

//set all variables to zero initially

i=0;

key1=0;

key2=0;

note1[0]=0xFF;

note1stop=0;

count=0;

instrument_change=0;

sei();

while(1){ //infinite loop

if(instrument_change){ //this if statement plays a

transition audio clip when an instrument is changed

cli();

if(instrument==0){SD_tone(8300, 105);} //depending on the instrument, the

appropriate declaration is played

else if(instrument==1){SD_tone(8405, 51);}

else if(instrument==2){SD_tone(8456,50);}



else{SD_tone(8600, 94);}

instrument_change=0; //clear the instrument change

variable

sei();

}

if(instrument==0 && !instrument_change) { //for the sine waves, note 1 is looped 200

times

if(i==note1stop){i=0; count++;} //reset the position in the array so

that it wraps around itself

if(note1[i]!=0xFF && count<200){ //if note1==0xFF and i=0 there is no note

DAC_SPI(note1[i]);

i++;

}

}

if(instrument==1 && key1!=0 && count<1 && !instrument_change){ //plays a guitar note once

count++;

stop=0;

//clears the stop variable(SD_tone won't play if set)

startBlock = 12 + (key1-1)*400; //find

the appropriate block for this key (guitar notes take up 400 blocks)

SD_tone(startBlock, 400); //play

the note

}

if(instrument==2 && key1!=0 && count<1 && !instrument_change){ //plays a piano note once

count++;

stop=0;


startBlock = 4812 + (key1-1)*100; //find the

appropriate block for this key (piano notes take up 100 blocks)


the note

}

if(instrument==3 && key1!=0 && count<1 && !instrument_change){ //plays a George Takei sound

byte once

count++;

stop=0;


startBlock = 6012 + (key1-1)*200; //find the

appropriate block for this key (George Takei sounds take up 200 blocks)


the note

}

}

}



ISR(TIMER1_COMPA_vect) //only relevant for DAC_SPI function (only used for

sine waves)

{

cli(); //this interrupt is part of the timing control for the

DAC_SPI function

y = 0; //clears y so DAC_SPI can progress

TIMSK1 |=2;

sei();

PORTA ^=(1<<1); //helps for debugging

}

ISR(USART0_RX_vect) //this is where the bulk of the

keyboard work is done

{

cli();

commandhi=USART_Receive(); //receive commands from PC (2

bytes)

commandlow=USART_Receive();

if(commandhi & 1){ //if the low bit of the first byte is

high, we need an instrument toggle

instrument++; //increment instrument

if(instrument>=4){instrument=0;} //reset to zero if it exceeds the max

instrument_change=1;

}

if(instrument>=4){instrument=0;}

key1command = (commandlow & (0x0F)); //commandlow contains a key2 command in

the high nibble and key1 in the low

key2command = ((commandlow & (0xF0))>>4);

if(key1command>12){key1command=0;} //the keys are numbered 1-12

anything higher is invalid

if(key2command>12){key2command=0;}

//sine waves

if(instrument==0){

if(key1command!=key1 && key1command!=key2){ //if key1 is new

key1=key1command; //update the key status

key2=key2command;

if(key1==0){note1[0]=0xFF;} //if key1 is zero no key is

being pressed

else {update_note(key1-1); count=0;} //load note1 with the key indicated by key1,

reset count

}

else if(key2command!=key2 && key2command!=key1){//if key2 is new

key2=key2command; //update the key status

key1=key1command;

if(key2==0){note1[0]=0xFF;} //if key2 is zero no key is

being pressed



else {update_note(key2-1); count=0;} //load note1 with the key indicated by key2,

reset count

}

else{

key2=key2command; //otherwise, update the key

status anyway

key1=key1command;

}

}

//every other instrument

else {

if(key1command!=key1 && key1command!=key2){ //if key1 is new

key1=key1command; //update keys

key2=key2command;

count=0; //reset count

stop=1; //enable

SD_Tone

}

else if(key2command!=key2 && key2command!=key1){ //if key2 is new

key1=key2command; //update keys

key2=key1command;

count=0; //reset count

stop=1;

//enable SD_Tone

}

else{

key2=key2command; //else update keys

anyway

key1=key1command;

}

}

i=0; //reset vector

start

sei(); //enable

interrupts

}

//this function is modified from the read multiple block function in the sd_routines library here:

http://www.dharmanitech.com/2009/01/sd-card-interfacing-with-atmega8-fat32.html

//this function sends what it reads directly to the DAC

//a SPI write speed of 62.5kHz yields an output to the DAC for a sampling frequency that is

//very close to 32kHz (which is the sampling frequency we used for all our audio)

unsigned char SD_tone(unsigned long startBlock, unsigned long totalBlocks)



{

unsigned char response;

unsigned int i, retry=0;

retry = 0;

response = SD_sendCommand(READ_MULTIPLE_BLOCKS, startBlock); //write a Block command

if(response != 0x00) return response; //check for SD status: 0x00 - OK (No flags set)

SD_CS_ASSERT; //set chip select

sei(); //enable interrupts so that the audio may be

interrupted

while( totalBlocks )

{

retry = 0;

while(SPI_receive() != 0xfe) //wait for start block token 0xfe (0x11111110)

if(retry++ > 0xfffe){SD_CS_DEASSERT; return 1;} //return if time-out

i=0;

while(((data=SPI_receive())!=0xFF) && i<512 && stop!=1){ //stop is set by the USART receive

ISR if a new note is pressed

SD_DAC_SPI(data);

//immediately send data to the DAC

i++;

}

if((i<512) && data==0xFF && SPI_receive()==0xFF){totalBlocks=1;} //the stop token is

0xFF, if you ever see this, the note is over

if(stop==1){totalBlocks=1;}

//end if stop is set

totalBlocks--;

}

SD_sendCommand(STOP_TRANSMISSION, 0); //command to stop transmission

SD_CS_DEASSERT;

SPI_receive(); //extra 8 clock pulses

return 0;

}

//this function is also modified from Dharmani's read multiple block function

void update_note(unsigned long startBlock) //only used for sine waves; loads a note to

ucontroller memory

{

unsigned char response;

unsigned int i, retry=0;

retry = 0;

totalBlocks=1;



response = SD_sendCommand(READ_MULTIPLE_BLOCKS, startBlock); //write a Block command

if(response != 0x00) return response; //check for SD status: 0x00 - OK (No flags set)

SD_CS_ASSERT;

while( totalBlocks )

{

retry = 0;

while(SPI_receive() != 0xfe) //wait for start block token 0xfe (0x11111110)

if(retry++ > 0xfffe){SD_CS_DEASSERT; return 1;} //return if time-out

i=0;

while(((data=SPI_receive())!=0xFF) && i<512){

note1[i]=data; //read data and store to note1

i++;

}

note1stop = i; //remember the end position

SPI_receive(); //receive incoming CRC (16-bit), CRC is ignored here

SPI_receive();

SPI_receive(); //extra 8 cycles

totalBlocks--;

}

SD_sendCommand(STOP_TRANSMISSION, 0); //command to stop transmission

SD_CS_DEASSERT;

SPI_receive(); //extra 8 clock pulses

return 0;

}

//used for all instruments but sine waves, no timing interrupts

void SD_DAC_SPI(char Data)

{

//writes 8-bit data to a DAC using SPI

char temph; //high byte

char templ = Data; //low byte

char tempd = Data;

tempd = tempd>>4;

tempd &= 0x0F; //mask top four bits

temph=(0xF0)|tempd; //high byte contains 4 bit op code and high nibble of data



templ=(templ<<4); //low byte contains bottom nibble of data, zeros for the

final two data points(we're using 8 instead of 10 bit resolution) and zeroes for the don't cares

templ &= 0xF0; //mask bottom 4 bits

PORTA &=~(1); //Drive CS low

USART_SPI_transmit(temph); //send high byte

USART_SPI_transmit(templ); //send low byte

_delay_us(2); //wait for transmission to complete


}

//only used for sine waves, uses timing interrupts (interrupt every 1/32khz)

void DAC_SPI(char Data)

{

//writes 8-bit data to a DAC using SPI

y=1;

char temph; //high byte

char templ = Data; //low byte

char tempd = Data;

tempd = tempd>>4;

tempd &= 0x0F; //mask top four bits

temph=(0xF0)|tempd; //high byte contains 4 bit op code and high nibble of data

templ=(templ<<4); //low byte contains bottom nibble of data, zeros for the

final two data points(we're using 8 instead of 10 bit resolution) and zeroes for the don't cares

templ &= 0xF0; //mask bottom 4 bits

while(y)

{

PORTA |= (1<<1);

}

PORTA &=~(1); //Drive CS low

USART_SPI_transmit(temph); //send high byte

USART_SPI_transmit(templ); //send low byte

_delay_us(2); //wait for transmission to complete


}

//send a byte with USART in SPI Mode

void USART_SPI_transmit(unsigned char data)

{

while ( !(UCSR1A & (1<<UDRE1)) ); // Wait for empty transmit buffer

UDR1 = data; //Start transmission

}

//receive a byte with USART in SPI Mode, not needed for this project

/*unsigned char USART_SPI_receive(void)

{

unsigned char data, status;



UDR1 = 0xFF;

while(!(UCSR1A & (1<<RXC1))); // Wait for incoming data

status = UCSR1A;

data = UDR1;

return(data);

}*/

unsigned char USART_Receive()

{

/* Wait for data to be received */

while ( !(UCSR0A & (1<<RXC0)) );

/* Get and return received data from buffer */

return UDR0;

}

void USART_Transmit( unsigned char data )

{

/* Wait for empty transmit buffer */

while ( !( UCSR0A & (1<<UDRE0)) );

/* Put data into buffer, sends the data */

UDR0 = data;

}

//sets SPI to 62.5kHz

void SPI_MUSIC_SPEED(void)

{

SPCR |= (1<<SPR0); //set clock to fck/16

SPCR &= ~(1<<SPR1);

SPSR &= ~1; //make sure SPI2x is off

}

SD Card Library/ Routines adapted from:





Appendix C: Tracking Code

Tracking.c ---main image tracking program

#include <stdlib.h>

#include <stdio.h>

#include <math.h>

#include <cv.h>

#include <highgui.h>

#include "COM.cpp"

void color(IplImage* img,int thresh,IplImage* colors){

//This function takes an image and a threshold and replaces all non-red pixels with black. Red pixels are

mapped to a value of 255

int i,j,k;

int height,width,step,channels;

int stepr, channelsr;

int temp=0;

uchar *data,*datar;

i=j=k=0;

//cvNamedWindow("original",CV_WINDOW_AUTOSIZE);

//cvNamedWindow("Result",CV_WINDOW_AUTOSIZE);

height = img->height;

width = img->width;

step =img->widthStep;

channels = img->nChannels;

data = (uchar *)img->imageData;

stepr=colors->widthStep;

channelsr=colors->nChannels;

datar = (uchar *)colors->imageData;

for(i=0;i < (height);i++) for(j=0;j <(width);j++){

if(((data[i*step+j*channels+2]) > (thresh+data[i*step+j*channels]))&&

((data[i*step+j*channels+2]) > (thresh+data[i*step+j*channels+1])) &&

data[i*step+j*channels+2]>=150 )

datar[i*stepr+j*channelsr]=255;

else

datar[i*stepr+j*channelsr]=0;

}

cvDilate(colors,colors,0,3);

cvSmooth(colors,colors,2,3,0);

//cvShowImage("original",img);

//cvShowImage("Result",color);

//cvWaitKey();



//return color;

}

CvRect FindBlob(IplImage* img){

//This function takes an image and returns a box around the first blob it finds

int x=300;

int y=300;

int found=0;

int i,j;

int height,width,step;

uchar* data;


height=img->height;

width=img->width;

step=img->widthStep;

cvShowImage("Image",img);

for(i=height-1; i>320;i--){

if(found==1) break;

for(j=0; j<width; j++){

if(data[i*step+j]==255){

x=j;

y=i;

found=1;

break; }

}

}

//printf("x: %d",x);

//printf("y: %d",y);

if(found==0)

return cvRect(0,0,40,40);

return cvRect(x-20,y-30,60,60);

}

CvRect FindBlob2(IplImage* img){

//this function takes an image and returns a rectangle containing the first blob it finds

int x=300;

int y=300;

int found=0;

int i,j;


uchar* data;


height=img->height;

width=img->width;





for(i=320; i<height;i++){

if(found==1) break;

for(j=width-1; j>0; j--){

if(data[i*step+j]==255){

x=j;

y=i;

found=1;

break; }

}

}

//printf("x: %d",x);

//printf("y: %d",y);

if(found==0)

return cvRect(0,0,40,40);

return cvRect(x-20,y-20,60,60);

}

void drawKeys(IplImage* img){

int height=img->height;

int width=img->width;

for(int i=0; i<12;i++){

cvRectangle(img,cvPoint(i*width/12,height*2/3),cvPoint((i+1)*width/12,height),cvScalar(255,2

55,255));

}

}

CvPoint COG(IplImage* img){

//this function takes an image and calculates it center of gravity. It also only affects the ROI but this is

relative

//to the origin of the original image so it must be shifted

CvMoments moments;

double M00;

double M10;

double M01;

CvPoint center;

cvMoments(img,&moments,1);

M00 = cvGetSpatialMoment(&moments,0,0);



center.x = (int)(M10/M00);

center.y = (int)(M01/M00);

return center;

}

char key(CvPoint led){



//this function takes a CvPoint and determines what key has been pressed

int x=led.x;

if(led.x==0 && led.y==0)

return 0;

if(x<=53)

return 12;

else if(x<=106)

return 11;

else if(x<=159)

return 10;

else if(x<=212)

return 9;

else if(x<=265)

return 8;

else if(x<=318)

return 7;

else if(x<=371)

return 6;

else if(x<=424)

return 5;

else if(x<=477)

return 4;

else if(x<=530)

return 3;

else if(x<=583)

return 2;

else if(x<=636)

return 1;

return 0 ;

}

int buttonp(IplImage* img){

//this function monitors the "button" for instrument toggle and determines whether it has been pushed or

not

int i,j;

int count=0;


uchar* data;


height=img->height;

width=img->width;



for(i=50; i<100;i++){

for(j=500; j<550; j++){

if(data[i*step+j]==255)

count++;

if(count==100)



return 1;

}

}

return 0;

}

int main(){

int thresh=60;

int height,width;

int target=0;

char key1=0;

char key2=0;

char key1d=0;

char key2d=0;

char buttontog=0;

int count=0;

//opening the COM port

//HANDLE comhandle=COM_open();

CvCapture* capture = cvCaptureFromCAM(0);

IplImage* img=cvQueryFrame(capture);

IplImage* colors=cvCreateImage(cvGetSize(img),8,1);

cvNamedWindow("Track",CV_WINDOW_AUTOSIZE);

cvNamedWindow("Image",CV_WINDOW_AUTOSIZE);

height=img->height;

width=img->width;

printf("height: %d width: %d ",height,width);

img=cvQueryFrame(capture);

color(img,thresh,colors);

CvRect rect=FindBlob(colors);

CvRect rect2=FindBlob(colors);

//cvRectangle(img,cvPoint(rect.x,rect.y),cvPoint(rect.x+25,rect.y+25),cvScalar(255,255,255));

CvPoint center;//=COG(colors);

CvPoint center2;//=COG(colors);

//cvCircle(colors,center,10,cvScalar(255,255,255));


cvShowImage("Track",colors);

//cvWaitKey();

if(!img)

printf("Could not load image");



while(1){

img=cvQueryFrame(capture);

if(!img) break;

//removing all color but red

cvSmooth(img,img,CV_GAUSSIAN);

color(img,thresh,colors);

rect=FindBlob(colors);

rect2=FindBlob2(colors);

//cvRectangle(img,cvPoint(rect.x,rect.y),cvPoint(rect.x+40,rect.y+40),cvScalar(255,255,0));

//cvRectangle(img,cvPoint(rect2.x,rect2.y),cvPoint(rect2.x+40,rect2.y+40),cvScalar(255,255,255));

/*4 possible values for target

target==0----- no key is being pressed

target==1----- both feet are pressing the same key or only 1 foot playing a note

target==2----- both feet are on different notes*/

if((rect.x==0 && rect.y==0) && (rect2.x==0 && rect2.y==0))

target=0;

else if(rect.x==rect2.x && rect.y==rect2.y)

target=1;

else

target=2;

//drawing the keyboard on the screen

drawKeys(img);

cvRectangle(img,cvPoint(500,50),cvPoint(550,100),cvScalar(255,255,0),CV_FILLED);

int but=buttonp(colors);

if(but==1){

if(count<10){

buttontog=0x0;

count++; }

else {

buttontog=0x1;

count=0; }

}

else

buttontog=0x0;

cvSetImageROI(colors,rect);

center=COG(colors);

cvSetImageROI(colors,rect2);

center2=COG(colors);

cvResetImageROI(colors);

center.x=rect.x+center.x;

center.y=rect.y+center.y;



center2.x=rect2.x+center2.x;

center2.y=rect2.y+center2.y;

if(target==0){

key1=0;

key2=0;

}

if(target==2){

cvCircle(img,center,10,cvScalar(255,255,255));

cvCircle(img,center2,10,cvScalar(255,255,255));

key1=key(center);

key2=key(center2);

if(key1!=key1d || key2!=key2d || buttontog==1){

//COM_write(comhandle,buttontog);

//COM_write(comhandle,((key2<<4)+key1));

printf("buttontog: %d \n",buttontog);

printf("count:%d \n",count);

}

key1d=key1;

key2d=key2;

}


cvShowImage("Track",colors);

char c=cvWaitKey(33);

if(c==32) break;

}

cvReleaseImage(&img);

cvReleaseImage(&colors);

}

COM Port Wrapper/ Handler adapted from Examples at:

http://www.ftdichip.com/Support/SoftwareExamples/CodeExamples/VC.htm

http://www.ftdichip.com/Support/SoftwareExamples/CodeExamples/VC.htm