Research Article Sensorimotor Piano System for People with

Research ArticleSensorimotor Piano System for People with Disabilities

Tobias Blumenstein1 Varvara Turova1 Ana Alves-Pinto1 and Reneacutee Lampe12

1Research Unit of the Buhl-Strohmaier Foundation for Cerebral Palsy and Paediatric NeuroorthopaedicsOrthopedic Department Klinikum rechts der Isar Technical University of Munich Ismaninger Str 22 81675 Munich Germany2Markus Wurth Professorship Technical University of Munich Munich Germany

Correspondence should be addressed to Renee Lampe reneelampetumde

Received 9 June 2016 Accepted 12 October 2016

Academic Editor Nicola Donato

Copyright copy 2016 Tobias Blumenstein et al This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited

A sensorimotor training system that facilitates learning to play piano was developed and tested The system consists of threecommunicating units The first unit comprises two pianos an E-piano with a MIDI output for the teacher and an acoustic oran E-piano for the pupil The pupilrsquos piano is supplied with an LED bar that illuminates the key to be struck The second unit is acontroller providing the interface between the teacherrsquos piano and the LED bar The third unit consists of two pairs of gloves onefor the teacher and one for the pupil The teacher gloves have integrated pressure sensors at every fingertip The pupilrsquos gloves havevibration motors and LEDs at every fingerThe pressure sensed on the teacherrsquos glove is transmitted to the corresponding finger onthe pupilrsquos glove via the vibration motors and LEDs such that the pupil knows which finger should strike which key AdditionallytwoOLEDdisplays showing the notation of the note played by the teacher can be attached to the left and right pupilrsquos gloves Initiallydeveloped for people with cerebral palsy the sensorimotor system can support the learning also to all those with sensory cognitiveand space perception impairments

1 Introduction

The difficulties when learning to play the piano are wellknown and have been experienced by many people intheir childhood or youth The coordination of fingers theobservance of rhythm the ability to strike the right keys andespecially the deciphering of an abstract text of music notesare demanding tasks

For people with motor impairments and learning dis-abilities this can be an almost hopeless task Especially forindividuals with cerebral palsy learning to play a musicinstrument is a great challenge Cerebral palsy is a multipledisability where depending on the extent of brain damage notonly are motor areas of the brain affected but also cognitiveand sensory impairments disturbances of space perceptionbody image and language and sometimes epilepsy exist [1]

Music therapy is often offered in institutions for peo-ple with mental and physical disabilities It is frequentlyperformed as a group exercise on Orff instruments andVeeh-harps Other instruments such as drums triangle andxylophone are also pleasant to use and simple to operate

even for people with motor impairments Such therapiesfacilitate expressing feelings and improve communicationskills enhancing the self-esteem of people with disabilities[2]

Positive effects of playing piano on the recovery ofhand fine motor skills and the development of neuroplasticchanges have been reported in patients with acquired braininjury especially in stroke patients [3ndash6] However the effectsof playing piano on hand motor function or in initiatingneuroplastic changes in patients with brain injuries occurringduring development as in cerebral palsy have been rarelyaddressed in the literature One likely contributor to this isthe often coexistence of multiple disabilities that limit theability to learn the instrument Nevertheless playing piano isparticularly advantageous for individuals with cerebral palsysince pianos need not be held in contrast to for examplea violin Moreover immediate auditory feedback receivedwhen striking a piano key stimulates the patient to correcthis own performance

As reported in [7] regular piano lessons can contribute toimproving handmotor skills in children and adolescents with

Hindawi Publishing CorporationJournal of SensorsVolume 2016 Article ID 9825905 7 pageshttpdxdoiorg10115520169825905

2 Journal of Sensors

cerebral palsy Moreover learning to play the piano can alsoproduce changes in brain network connectivity [8] Howeverduring the study several difficulties were observed Learningabstract notes was difficult because of learning disabilitiesCognitive deficits caused difficulties in reading note text sothat playing by notes was hard fhe differentiation of fingerswas furthermore insufficient Hand sensor abilities wereoften disturbed which was expressed in improper use of thefingers In addition participants had problems with spatialorientation at the piano To help overcome these difficultiesand facilitate independent training of piano playing a versatilepiano training system was developed and is here described

Pianos with lighted keys are already commercializedHere music pieces are programmed and built into them sothat the respective keys are illuminated while playing thepiece The user only has to strike the keys in the prescribedsequence There are also a number of software programs thatsupport the learning of piano playing For instance pianokeys of aMIDI player are highlighted in color while the pieceis played from a computerThese piano learning tools provideonly a visual aid and do not take into account the difficultiesin recognizing and differentiating the fingers the deficitsin eye-hand coordination impaired spatial orientation andother sensorimotor disabilities Moreover only pieces can belearned that are already stored in the piano or which areavailable as MIDI files on the Internet

This paper addresses the technical development of a flex-ible piano training system aimed to compensate the deficitsdescribed above This system initially developed for peoplewith cerebral palsy can be used also in neurorehabilitationsettings Furthermore it can support the learning of pianoplaying in other medical conditions involving also sensoryand cognitive deficits

2 Technical Description of the System

The piano system consists of three units that communicatewith each other (see Figure 1) The first unit comprises theteacherrsquos piano the pupilrsquos piano and an LED barThe secondunit is a controller which provides an interface between thetwo pianosThe third unit consists of two pairs of gloves onefor the teacher and another for the pupilThe aim is to transferthe fingering executed by the teacher as well as the sequenceof piano keys played to the pupil

21 Unit 1 Pianos and LED Bar The teacherrsquos piano is an E-piano with 88 keys and a MIDI (Musical Instrument DigitalInterface) output The pupilrsquos piano is an acoustic or an E-pianowith also 88 keysThe LEDbar is placed onto the pupilrsquospiano keyboard to see the use of the keys

When a key is struck a MIDI command consisting of 3bytes is sent to the controllerThe first byte contains the statuscommand (note on note off) and the MIDI channel Thesecond byte describes the note and the third one the strengthwith which the key is struck This information describescompletely when and which button was pressed and released

The keys played by the teacher are presented to the pupilvia the LED bar placed over the pupilrsquos pianoThe bar consists

of 88 WS2812b which are arranged to match the size ofkeyboard (see Figure 2)The LED lights are RGB LEDs with abuilt-in drive circuitThey are connected in series so that onedata line is sufficient for all LEDs LEDs placed over the whitekeys of the piano have red color while those over the blackkeys have green color The redgreen combination makes iteasier to distinguish at first glance which type of the keyis being used By using an appropriate combination of redgreen and blue intensity values (from 0 to 255) practicallyany color can be chosen Each piano key is illuminated withonly one LED in a punctiformmanner (see Figure 2(a)) suchthat it is easy to identify the keys that should be struckThebarconsists of seven segments one for each octave which can beconnected to each other and remain aligned close together viaa magnetic plug This LED bar design is intended to facilitatethe transport of the bar it can be split into its parts and thenfolded together (Figure 2(b))

22 Unit 2 Control Unit Interpretation of the MIDI com-mands and control of the LED bar are done via the con-trol unit The main component is an Arduino Mega 2560microcontroller which has 4UART (Universal AsynchronousReceiver Transmitter) interfaces for serial communication(see Figure 1) One of the interfaces is connected throughan optocoupler to the MIDI output of the teacherrsquos pianoThe data received are interpreted and then transformed intoa signal to control the LED bar placed over the pupilrsquospiano NeoPixel library of Adafruit Industries is used forcommunication between the Arduino and the LED bar Onlyone digital output is required to control an arbitrary numberof LEDs The settings for each individual LED are sent via aresistor through the digital input of the first LEDof the seriesThe information is sent in reverse order (ie last LED first)and transferred from one component to the next Since everysingle LED has its own built-in drive circuit no additionalhardware is necessary to control the LEDs

The control unit can also simultaneously drive 6 LEDbars which would allow the signal from the teacherrsquos pianoto be transmitted to 6 different pianos simultaneously thatis it would allow group lessons

23 Unit 3 Gloves Special gloves have been developed forthe teacher and the pupil The teacher gloves have integratedpressure sensors located at every fingertip The pupil gloveshave a vibration motor and an LED at every finger Thesignal produced by a pressure sensor of the teacher gloves istransferred to the pupil gloves (to the vibration motor and tothe LED light of the corresponding finger) so that the pupil isinformed which finger to which key should be applied

Each teacher glove is a tight-fitting thin leather glove(Figure 3) Sensors FSR 400 (Interlink) were incorporatedin the gloversquos fingertips These sensors show a decrease inresistance from more than 10MΩ unloaded to less than10 kΩ with an increase in force applied to the surface of thesensor They were connected in series with a 150 kΩ resistorThe pressure sensor and resistor form a voltage divider withwhich the change in the resistance of the sensors can bemeasured The sensors were connected to the digital inputs

Journal of Sensors 3

Control unit

Arduino Mega

Atmega2560

LED bar

E-pianowith MIDI

Dig

ital I

OUA

RT

NeoPixel

Dig

ital I

O

vib

mot

oramp

LED

Teacher glove

XbeeSeries 1

Dig

ital I

O

Pres

sure

sens

or

XbeeSeries 1

Digital IO

SPI

display

MIDI

Translator

Pupil glovePupil piano

Teacher piano

Unit 1 Unit 2 Unit 3

88 times WS2812b

128 times 64 OLED

Figure 1 Block diagram of the sensorimotor piano system Indicated here are the units 1 (pianos) 2 (control unit) and 3 (gloves)

(a) (b)

Figure 2 (a) LED bar indicating piano keys to be struck (b) LED bar split into parts for easy transport

of a Xbee Series 1 module (Digi International) that allows forwireless communication to other modules

The module is powered by a mobile power pack andconfigured so that it transmits the states of the pressuresensors to a second module every 10msThis second moduleis located alongwith vibrationmotors andLEDs on the pupilrsquosgloveThepupilrsquos gloveswere developed fromcommon cyclistgloves These have the fingertips free and therefore do nothamper piano playing Each finger of the glove has a vibrationmotor and an LED that are activated by the Xbee moduleApplying force on a sensor in the teacherrsquos glove sets thedigital outputs of the pupils Xbee module to ldquohighrdquo statewhich activates the corresponding LED and vibration motor

The pupilrsquos glove is designed as follows All discreteelectronic elements such as LEDs vibrationmotors andXbeemodule are embedded with silicone in a single unit (seeFigure 4) Silicone provides good stability but also elasticityso that the electronic add-on can be attached to a glove ofany size In order to increase the visibility of individual LEDs

they can be uncovered before embedding in silicone In thiscase a plexiglass disk is used as protection against damageAlternatively transparent silicone can be usedThe electronicadd-on is fixed on the glove using either magnets or snapfasteners

24 Additional Features The following extra features andadditionalmodules can be addedused with the sensorimotorsystem described above

241 Note Display Two displays indicating with letters thenotes played by the teacher can be connected to the controlunit The 1310158401015840 128 times 64 pixel OLED displays by AdafruitIndustries were used here They were controlled by anSSD1306 driver chip using SPI (Serial Peripheral Interface)for the communication with the control unit MIDI datareceived from the teacherrsquos piano are used to identify thekeys struck For example if the teacher plays the note C the


Figure 3 Teacherrsquos glove with integrated pressure sensors

(a) (b)

Figure 4 (a) Electronic add-on with LEDs vibration motors and Xbee module for pupilrsquos glove powered using a mobile power pack (b)pupilrsquos glove with electronic add-on

letter ldquoCrdquo appears on the display (Figure 5(a)) One display isattached to the right glove and another one to the left glove ofthe pupil

242 ldquoSingle-Pianordquo Mode An alternative to using twopianos during the lessons is to have both teacher and pupilseating at one piano only In this ldquosingle-pianordquo mode thepupil seats right of the teacher and the LED bar lights the keysplayed by the teacher one octave above (Figure 5(b)) Thisldquosingle-pianordquomode can be turned on through a switch in thecontrol unit The ldquosingle-pianordquo mode enables simultaneousplaying of the pupil and the teacher on the same piano Thisproximity has the advantage that the teacher can interfere inthe play of the pupil

243 Recording a Musical Piece for Training There is alsothe option of recording the teacherrsquos playing to reproduce itlater To this end the control unit contains an Xbee modulereceiving the data with the fingering of the teacher (seeFigure 6) This module is connected to the Arduino Thenotes are transferred via MIDI output as described aboveThe Arduino contains the USB-to-TTL Serial chip Atmega16U2 which allows communication with a computer via USBThe data transmitted from the Arduino were processed witha program written in LabVIEW The universal IO interfacesoftware VISA 2015 by National Instruments was used as the

interface between hardware and software The data receivedcan be stored locally on the computer Playback of the savednote sequences and fingering runs in reverse order by drivingnow the LED bar and pupil gloveThe program can be run ona PC and also on a Windows tablet This additional featuremakes piano training possible in the absence of the teacher

3 Experimental Testing of the System

The efficacy of the system in supporting the learning ofplaying the piano was tested experimentally on four indi-viduals with cerebral palsy and on eight healthy personsTwo patients had bilateral leg affected cerebral palsy onepatient had unilateral cerebral palsy and one patient hadataxic cerebral palsyThe taskwas to reproduce two sequencesof 10 notes played by the piano teacherThe first sequence wasplayed without the help of the system The second sequencewas presented with the help of the system In both casesthe patients had to reproduce the sequence three times ontheir ownThe system was only used during the presentationSequences were always played three times by the teacher

Individual results of persons with cerebral palsy arepresented in Figures 7(a)ndash7(d) The number of errors versusthe trial number is depicted for sequences presentation withthe system (red line and symbols) and without (blue line andsymbols) For all of them the number of errors (ie wrong


(a) (b)

Figure 5 (a) Translation of notes played by the teacher in letters (in this case ldquoardquo and ldquohrdquo) usingOLEDdisplay (b) training in the single-pianomode

Control unitPianos

Gloves

ComputerLabVIEWLED bar

MIDI out

Digital IO

UART

XbeeXbee

USB-to-TTLSerial

Atmega16U2

NI Visa 2015

Saveand

restore

Arduino Mega

Figure 6 Schematic representation of recording und reproducing option

note played) decreased from the first to the third trial whenusing the systemWhen the systemwas not used the numberof errors remained approximately constant Furthermore theoutcome in the final trial when the system was used wasalways better than when the system was not used

The same task was performed by the healthy participantsApparently the task was too easy for them because practicallyno errors were made when playing with or without thesystem

4 Discussion

People with cerebral palsy have multiple disabilities depend-ing on the extent of areas affected by brain damage Inaddition to motor impairments sensorimotor learning andattention deficits can be present to varying degrees

A piano training system was developed to support peoplewith disabilities to learn to play the piano even without back-ground knowledge of notes and despite the reduced sensoryability of the fingers Playing the piano can be consideredas a rehabilitative therapy since it promotes hand motorfunction Improvement of hand fine motor functions after

learning to play the piano has been reported [7] The systemhere proposed aims additionally to compensate for sensorydeficits It is a good addition to the classical therapeuticprograms in cerebral palsy which are mostly directed toimprove gross motor skills like standing and walking andpreventing contractures of the joints

By means of LED bar which illuminates piano keys thesystem provides the pupil with the information on the keythat must be struck Pianos with lighting keys can currentlybe purchased on themarket However simply having a lightedkeyboard is insufficient to compensate for the sensorimotorand cognitive deficits that can be present in many patientswith cerebral palsy In the sensorimotor system describedhere the punctiform of lighting in the design of the LED barallows easy differentiation of the keys to be struck especiallywith adjacent keys It is also advantageous that the blackand white keys are illuminated with light of different colorsFurthermore the LED bar is suitable for any usual acousticpiano it can be decomposed for comfortable transportationand easily reassembled again The commercially availablepianos contain a limited number of piano pieces for train-ing which may not be suited for a person with multiple


1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)

Figure 7 Test results for four individuals with cerebral palsy Blue and red lines indicate how the sequence was presented to the patient

disabilities The sensorimotor piano system here describedallows the learning of music pieces chosen to fit the pupilrsquosspecific limitations and needs being played by the teacherManyMIDI-Player computer programs offer learning aids bymeans of visualizing the keyboard on a computer screen in asmaller size whereas the LED bar here presented highlightsthe keys on the real piano

MIDI players or illuminated piano keyboards are not yetequipped with a note translator The system here describedcan contain an integrated note translator that makes itpossible for the pupil to see on small displays being placedon the gloves which notes were played This can help to learnabstract notes

The idea to facilitate learning to play piano by transmis-sion of electric signals to the fingers of the pupil was pro-posed for example in [9] In the system developed vibrationsignals were used because they can be easily perceived byindividuals with cerebral palsy and comprise a therapeuticcomponent Children and adults with cerebral palsy haveoften an impaired sensory perception of the fingers Treat-ment methods to improve body awareness through vibrationhave been described in the literature [10ndash12] Whole bodyvibration plates for example Galileo are increasingly beingtested in cerebral palsy [13 14] To promote the perception ofthe individual fingers and reduce spasticity sensory feedbacktraining combined with vibration stimuli has been recentlyproposed [15] The fingers of 7 patients with cerebral palsywere stimulated using small vibration motors integrated in

the fingers of a hand gloveThe pupilrsquos gloves in the presentedtraining system work with a similar principle Additional tovibration motors LEDs are placed in each single finger of theglove The gloves help to recognize the fingers by indicatingthrough vibration and LED lights with which finger the pianokey has to be struck

Normally it requires many hours of practice and expe-rience to acquire an appropriate predicted fingering Thelatter is transmitted here by the teacher by simply playingthe sequence in a piano while wearing special gloves thathave pressure sensors in the tips of the fingers These allowthe recording of the fingering that is then displayed via LEDandor transferred via vibration to the pupilrsquos fingers so thatthe pupil receives the suitable position for the hand andcan play the music piece comfortably The recording of thefingering is easy for the teacher since he solely has to puton his gloves With this technique the teacher can navigatethe pupil during the lesson in real time or another optionby recording his piano playing to a tabletcomputer via thecontrol unit The student can then practice and repeat therecorded fingering at home at his own speed with the help ofthe vibration motors and LEDs The self-developed softwareallows furthermore adjustment of the speed and timing ofvibrations and LED display so that the pupil can repeat thesong in different tempos according to his own individualneeds

The efficacy of the system in supporting the learningof simple sequences of notes was tested experimentally in


a small sample of patients with cerebral palsy and in healthyparticipants Improvements (ie reduction of the number oferrors) could only be observed in the first group This is apositive outcome since this is the target group for the senso-rimotor system proposed Evidence for support of learning inhealthy participants if possible may require the use of longersequences or for longer periods of timeThe results presentedprovide preliminary evidence for the benefits the system candeliver Further evidence remains nevertheless necessary toconfirm the system as therapeutic option in rehabilitationprograms for people with sensorimotor disabilities

By compensating the effects of sensory deficits and learn-ing disabilities this piano systemmay support and encouragepeople with disabilities to learn to play the piano this waycontributing to their inclusion It offers also a new therapeuticapproach that involves motoric sensory and attention skills

5 Conclusion

This work describes a novel piano training system aimed atassisting people with sensorimotor and cognitive disabilitiesto learn to play the piano Compensation of deficits isachieved mainly by means of vibration and visual informa-tion to the pupil The following options are included

(i) Training (individual or collective) with the teacher onseparate pianos

(ii) Training with the teacher on a single piano

(iii) Training without the teacher using records of pianoplaying by the teacher

(iv) Training with the note translator

Competing Interests

The authors declare no conflict of interests

Acknowledgments

The work was partly funded by the German ResearchSociety (DFG project LA 26703-1) the Buhl-StrohmaierFoundation the Friede Springer Foundation theGips-SchuleFoundation the Wurth Foundation the Zeidler ResearchFoundation and Casio Europe

References

[1] E OddingM E Roebroeck andH J Stam ldquoThe epidemiologyof cerebral palsy incidence impairments and risk factorsrdquoDisability and Rehabilitation vol 28 no 4 pp 183ndash191 2006

[2] T Wigram and J D Backer Clinical Applications of MusicTherapy in Developmental Disability Paediatrics and NeurologyJessica Kingsley London UK 1999

[3] E Altenmuller J Marco-Pallares T F Munte and S Schnei-der ldquoNeural reorganization underlies improvement in stroke-induced motor dysfunction by music-supported therapyrdquoAnnals of the New York Academy of Sciences vol 1169 pp 395ndash405 2009

[4] A Rodriguez-Fornells N Rojo J L Amengual P Ripolles EAltenmuller and T FMunte ldquoThe involvement of audio-motorcoupling in the music-supported therapy applied to strokepatientsrdquo Annals of the New York Academy of Sciences vol 1252no 1 pp 282ndash293 2012

[5] M Villeneuve V Penhune and A Lamontagne ldquoA pianotraining program to improve manual dexterity and upperextremity function in chronic stroke survivorsrdquo Frontiers inHuman Neuroscience vol 8 article 662 2014

[6] F T Van Vugt J Ritter J D Rollnik and E AltenmullerldquoMusic-supportedmotor training after stroke reveals no superi-ority of synchronization in group therapyrdquo Frontiers in HumanNeuroscience vol 8 article 315 2014

[7] R Lampe AThienel J Mitternacht T Blumenstein V Turovaand A Alves-Pinto ldquoPiano training in youths with handmotor impairments after damage to the developing brainrdquoNeuropsychiatric Disease and Treatment vol 11 pp 1929ndash19382015

[8] A Alves-Pinto V Turova T Blumenstein A Thienel AWohlschlager and R Lampe ldquofMRI assessment of neuroplas-ticity in youths with neurodevelopmental-associated motordisorders after piano trainingrdquo European Journal of PaediatricNeurology vol 19 no 1 pp 15ndash28 2015

[9] R McGregor Musical teaching device and method PatentUS20060137511 A1 2006

[10] A Katusic and VMejaski-Bosnjak ldquoEffects of vibrotactile stim-ulation on the control ofmuscle tone andmovement facilitationin childrenwith cerebral injuryrdquoCollegiumAntropologicum vol35 no 1 pp 57ndash63 2011

[11] V F Bento V T Cruz DD Ribeiro and J P Cunha ldquoThe vibra-tory stimulus as a neurorehabilitation tool for stroke patientsproof of concept and tolerability testrdquo NeuroRehabilitation vol30 no 4 pp 287ndash293 2012

[12] M Saquetto V Carvalho C Silva C Conceicao and MGomes-Neto ldquoThe effects of whole body vibration on mobilityand balance in children with cerebral palsy a systematicreview withmeta-analysisrdquo Journal of Musculoskeletal NeuronalInteractions vol 15 no 2 pp 137ndash144 2015

[13] O Semler O Fricke K Vezyroglou C Stark and E SchoenauldquoPreliminary results on the mobility after whole body vibrationin immobilized children and adolescentsrdquo Journal of Muscu-loskeletal Neuronal Interactions vol 7 no 1 pp 77ndash81 2007

[14] B-K Lee and S-C Chon ldquoEffect of whole body vibration train-ing on mobility in children with cerebral palsy a randomizedcontrolled experimenter-blinded studyrdquo Clinical Rehabilitationvol 27 no 7 pp 599ndash607 2013

[15] T Blumenstein A Alves-Pinto V Turova S Aschmann ILutzow and R Lampe ldquoSensory feedback training for improve-ment of finger perception in cerebral palsyrdquo RehabilitationResearch and Practice vol 2015 Article ID 861617 7 pages 2015

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics


Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of


Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks



cerebral palsy Moreover learning to play the piano can alsoproduce changes in brain network connectivity [8] Howeverduring the study several difficulties were observed Learningabstract notes was difficult because of learning disabilitiesCognitive deficits caused difficulties in reading note text sothat playing by notes was hard fhe differentiation of fingerswas furthermore insufficient Hand sensor abilities wereoften disturbed which was expressed in improper use of thefingers In addition participants had problems with spatialorientation at the piano To help overcome these difficultiesand facilitate independent training of piano playing a versatilepiano training system was developed and is here described

Pianos with lighted keys are already commercializedHere music pieces are programmed and built into them sothat the respective keys are illuminated while playing thepiece The user only has to strike the keys in the prescribedsequence There are also a number of software programs thatsupport the learning of piano playing For instance pianokeys of aMIDI player are highlighted in color while the pieceis played from a computerThese piano learning tools provideonly a visual aid and do not take into account the difficultiesin recognizing and differentiating the fingers the deficitsin eye-hand coordination impaired spatial orientation andother sensorimotor disabilities Moreover only pieces can belearned that are already stored in the piano or which areavailable as MIDI files on the Internet

This paper addresses the technical development of a flex-ible piano training system aimed to compensate the deficitsdescribed above This system initially developed for peoplewith cerebral palsy can be used also in neurorehabilitationsettings Furthermore it can support the learning of pianoplaying in other medical conditions involving also sensoryand cognitive deficits

2 Technical Description of the System

The piano system consists of three units that communicatewith each other (see Figure 1) The first unit comprises theteacherrsquos piano the pupilrsquos piano and an LED barThe secondunit is a controller which provides an interface between thetwo pianosThe third unit consists of two pairs of gloves onefor the teacher and another for the pupilThe aim is to transferthe fingering executed by the teacher as well as the sequenceof piano keys played to the pupil

21 Unit 1 Pianos and LED Bar The teacherrsquos piano is an E-piano with 88 keys and a MIDI (Musical Instrument DigitalInterface) output The pupilrsquos piano is an acoustic or an E-pianowith also 88 keysThe LEDbar is placed onto the pupilrsquospiano keyboard to see the use of the keys

When a key is struck a MIDI command consisting of 3bytes is sent to the controllerThe first byte contains the statuscommand (note on note off) and the MIDI channel Thesecond byte describes the note and the third one the strengthwith which the key is struck This information describescompletely when and which button was pressed and released

The keys played by the teacher are presented to the pupilvia the LED bar placed over the pupilrsquos pianoThe bar consists

of 88 WS2812b which are arranged to match the size ofkeyboard (see Figure 2)The LED lights are RGB LEDs with abuilt-in drive circuitThey are connected in series so that onedata line is sufficient for all LEDs LEDs placed over the whitekeys of the piano have red color while those over the blackkeys have green color The redgreen combination makes iteasier to distinguish at first glance which type of the keyis being used By using an appropriate combination of redgreen and blue intensity values (from 0 to 255) practicallyany color can be chosen Each piano key is illuminated withonly one LED in a punctiformmanner (see Figure 2(a)) suchthat it is easy to identify the keys that should be struckThebarconsists of seven segments one for each octave which can beconnected to each other and remain aligned close together viaa magnetic plug This LED bar design is intended to facilitatethe transport of the bar it can be split into its parts and thenfolded together (Figure 2(b))

22 Unit 2 Control Unit Interpretation of the MIDI com-mands and control of the LED bar are done via the con-trol unit The main component is an Arduino Mega 2560microcontroller which has 4UART (Universal AsynchronousReceiver Transmitter) interfaces for serial communication(see Figure 1) One of the interfaces is connected throughan optocoupler to the MIDI output of the teacherrsquos pianoThe data received are interpreted and then transformed intoa signal to control the LED bar placed over the pupilrsquospiano NeoPixel library of Adafruit Industries is used forcommunication between the Arduino and the LED bar Onlyone digital output is required to control an arbitrary numberof LEDs The settings for each individual LED are sent via aresistor through the digital input of the first LEDof the seriesThe information is sent in reverse order (ie last LED first)and transferred from one component to the next Since everysingle LED has its own built-in drive circuit no additionalhardware is necessary to control the LEDs

The control unit can also simultaneously drive 6 LEDbars which would allow the signal from the teacherrsquos pianoto be transmitted to 6 different pianos simultaneously thatis it would allow group lessons

23 Unit 3 Gloves Special gloves have been developed forthe teacher and the pupil The teacher gloves have integratedpressure sensors located at every fingertip The pupil gloveshave a vibration motor and an LED at every finger Thesignal produced by a pressure sensor of the teacher gloves istransferred to the pupil gloves (to the vibration motor and tothe LED light of the corresponding finger) so that the pupil isinformed which finger to which key should be applied

Each teacher glove is a tight-fitting thin leather glove(Figure 3) Sensors FSR 400 (Interlink) were incorporatedin the gloversquos fingertips These sensors show a decrease inresistance from more than 10MΩ unloaded to less than10 kΩ with an increase in force applied to the surface of thesensor They were connected in series with a 150 kΩ resistorThe pressure sensor and resistor form a voltage divider withwhich the change in the resistance of the sensors can bemeasured The sensors were connected to the digital inputs


Control unit

Arduino Mega

Atmega2560

LED bar

E-pianowith MIDI

Dig

ital I

OUA

RT

NeoPixel

Dig

ital I

O

vib

mot

oramp

LED

Teacher glove

XbeeSeries 1

Dig

ital I

O

Pres

sure

sens

or

XbeeSeries 1

Digital IO

SPI

display

MIDI

Translator


Teacher piano


88 times WS2812b

128 times 64 OLED


(a) (b)










(a) (b)










(a) (b)


Control unitPianos

Gloves


MIDI out

Digital IO

UART

XbeeXbee

USB-to-TTLSerial

Atmega16U2

NI Visa 2015

Saveand

restore

Arduino Mega




4 Discussion






1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)











5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










Control unit

Arduino Mega

Atmega2560

LED bar

E-pianowith MIDI

Dig

ital I

OUA

RT

NeoPixel

Dig

ital I

O

vib

mot

oramp

LED

Teacher glove

XbeeSeries 1

Dig

ital I

O

Pres

sure

sens

or

XbeeSeries 1

Digital IO

SPI

display

MIDI

Translator


Teacher piano


88 times WS2812b

128 times 64 OLED


(a) (b)










(a) (b)










(a) (b)


Control unitPianos

Gloves


MIDI out

Digital IO

UART

XbeeXbee

USB-to-TTLSerial

Atmega16U2

NI Visa 2015

Saveand

restore

Arduino Mega




4 Discussion






1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)











5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation











(a) (b)










(a) (b)


Control unitPianos

Gloves


MIDI out

Digital IO

UART

XbeeXbee

USB-to-TTLSerial

Atmega16U2

NI Visa 2015

Saveand

restore

Arduino Mega




4 Discussion






1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)











5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










(a) (b)


Control unitPianos

Gloves


MIDI out

Digital IO

UART

XbeeXbee

USB-to-TTLSerial

Atmega16U2

NI Visa 2015

Saveand

restore

Arduino Mega




4 Discussion






1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)











5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










1 3Test number

Without system

With system

0

2

4

6

8

10

12N

umbe

r of e

rror

s

2

(a)

Without system

With system

1 3Test number

20

1

2

3

4

Num

ber o

f err

ors

(b)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(c)

Without system

With system

1 3Test number

2012345678

Num

ber o

f err

ors

(d)











5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation












5 Conclusion






Competing Interests


Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation











RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation









Documents

Research Article Sensorimotor Piano System for People with