UsingKinectv2toControlaLaserVisualCueSystemtoImprove ...downloads.hindawi.com/journals/jhe/2019/3845462.pdf · using virtual reality (VR) to assess the possibility of VR integration

Research ArticleUsing Kinect v2 to Control a Laser Visual Cue System to Improvethe Mobility during Freezing of Gait in Parkinsonrsquos Disease

Amin Amini and Konstantinos Banitsas

College of Engineering Design and Physical Sciences Department of Electronics and Computer EngineeringBrunel University London London UK

Correspondence should be addressed to Amin Amini aminaminibrunelacuk

Received 19 November 2018 Revised 3 January 2019 Accepted 22 January 2019 Published 20 February 2019

Guest Editor Heenam Yoon

Copyright copy 2019 AminAmini and Konstantinos Banitsas+is is an open access article distributed under the Creative CommonsAttribution License which permits unrestricted use distribution and reproduction in anymedium provided the original work isproperly cited

Different auditory and visual cues have been proven to be very effective in improving the mobility of people with Parkinsonrsquos(PwP) Nonetheless many of the available methods require user intervention and so on to activate the cues Moreover onceactivated these systems would provide cues continuously regardless of the patientrsquos needs +is research proposes a new indoormethod for casting dynamicautomatic visual cues for PwP based on their head direction and location in a room +e proposedsystem controls the behavior of a set of pantilt servo motors and laser pointers based on the real-time skeletal informationacquired from a Kinect v2 sensor +is produces an automatically adjusting set of laser lines that can always be in front of thepatient as a guideline for where the next footstep would be placed A user interface was also created that enables users to controland adjust the settings based on the preferences +e aim of this research was to provide PwP with an unobtrusiveautomaticindoor system for improving their mobility during a Freezing of gait (FOG) incident +e results showed the possibility ofemploying such system which does not rely on the subjectrsquos input nor does it introduce any additional complexities to operate

1 Introduction

Freezing of gait (FOG) is one of the most disablingsymptoms in Parkinsonrsquos disease (PD) that affects its suf-ferers by impacting their gait performance and locomotionFOG is an episodic phenomenon that introduces irregu-larities in the initiation or continuation of a patientrsquos lo-comotion and usually occurs in later stages of PD wherepatientsrsquo muscles cannot function normally and appear to bestill when they are trying to walk [1ndash4] +is makes FOG oneof the most intolerable symptoms that not only affects PDsufferers physically but also psychologically as it makesthem almost completely dependent on others for their basicand daily tasks Consequently the patientrsquos quality of lifedecreases and the healthcare and treatment expendituresincrease as does the cost of the injuries caused [1] It hasbeen estimated that about 50 of PwP experience FOGincidents [5] Moreover it has been proven that visual andauditory cues can have a positive impact on the subjectrsquos gait

performance during a FOG incident [6ndash8] Visual cues suchas laser lines can act as a sensory guidance trick that providesan external trigger which in turn can initiate movement[7]

+ere has been much research conducted towardsimplementing apparatus and systems that can provide visualand auditory cues for PwP In work done by Zhao et al [9] awearable system based on modified shoes was developed inorder to cast a laser-based visual cue in front of PwP +esystem consisted of a 3D printed add-on that included a redlaser line projector and pressure sensors that detect thestance phase of a gait cycle and turn the laser pointer on+eunit provided the option to adjust the distance between thelaser light strip and the subjectrsquos foot for the optimal ef-fectiveness depending on the userrsquos preferences +e re-search provided a simple yet effective approach towardsproviding visual cues for PwP with locomotion issuesNonetheless like any other approaches this too has somelimitations such as the constant need to carry the shoe add-

HindawiJournal of Healthcare EngineeringVolume 2019 Article ID 3845462 8 pageshttpsdoiorg10115520193845462

on the batteries needed for the device charging the bat-teries and remembering to switch them on

In another attempt [10] researchers evaluated the effectof visual cues using two different methods including asubject-mounted light device (SMLD) and taped step lengthmarkers It was concluded that using laser projections basedon SMLD have promising effects on the PwPrsquos locomotionand gait performance +e method required patients to weara SMLD that some patients might find inconvenient to haveor even impractical in some situations Moreover SMLDsystems have stability issues and steadiness difficulties due tothe subjectsrsquo torso movements during a gait cycle As ex-pected the visual cues must be constantly enabled during agait cycle regardless whether they are needed or not

In [11] although the SMLD method was employedresearchers added the 10 seconds on-demand option to theldquoconstantly onrdquo visual cue casting +is system was moresophisticated consisting of a backpack having a remotelycontrollable laptop that made the subjectsrsquo mobility evenmore troublesome

In other attempts [12 13] a different approach wasimplemented by using virtual cues projected on a pair ofgoggles that is only visible to the patient In [14] the effect ofreal and virtual visual cueing was compared and it wasconcluded that real transverse lines casted on the floor aremore impactful than the virtual counterparts Nonethelessusing virtual cueing spectacles (VCS) eliminates theshortcomings in other techniques such as limitations inmobility steadiness and symmetry VCS have also theadvantage of being used in an external environment whenthe patient is out and about

Moreover several research studies have been conductedusing virtual reality (VR) to assess the possibility of VRintegration for Parkinsonrsquos related studies [15ndash20] None-theless as the VR technology blocks patientsrsquo view andmakes them unable to see their surroundings the usage ofthis is limited to either rehabilitation by implementingexercise-based games FOG provoking scenarios or theassessment of patientsrsquo locomotion rather than real-timemobility improvement using cues

Although they are effective to some extent these at-tempts tend to restrict the user either by forcing them tocarry backpacks or wear vests containing electronics ormaking them rely on conventional approaches such asattaching laser pointers to a cane [21] or laser add-on forshoes

+e hypothesis of this study on the other hand is topropose a different technique casting parallel laser lines as adynamic and automatic visual cuing system for PwP basedon Kinect v2 and a set of servo motors suitable for indoorenvironments As Kinect has been proven to be a reliabledata feed source for controlling servo motors [22 23] theKinect camera was chosen for real-time depth data feed forthis study +is paper also examines the possibility of usingthe Kinect v2 sensor for such purposes in terms of accuracyand response time

+is research uses subjectrsquos 3D Cartesian location andhead direction as an input for servo motors to cast visualcues accordingly +is eliminates the need of the user

intervention or trigger and at the same time the need tocarry or wear any special equipment Despite this approachbeing limited to environments equipped with the proposedapparatus it does not require any attachments or reliance onPwP themselves something that can be beneficial in manyscenarios +e system comprises a Microsoft Kinect v2 a setof panttilt servo motors alongside a microcontroller basedon Arduino Uno and two laser line laser pointers A two-lineprojection was chosen so that the second traversed laser linecould be used to indicate a set area for which the next stephas to land +e system was tested in different conditionsincluding a partially occluded scene by furniture to simulatea living room

2 Methods

During the initial testing phase 11 healthy subjects wereinvited consisting of both males and females ranging fromages 24ndash31 with the age mean of 27 and SD of 234 a meanheight of 17445 cm (6868 inch) and SD of 831 cm(327 inch) ranging from 163 to 187 cm (6417 to 7362 inch)+ey were asked to walk in predefined paths 12 paths persubject walking towards the camera and triggering a sim-ulated FOG incident by imitating the symptom while havingthe Kinect camera positioned at a fixed location +e sub-jectsrsquo skeletal data were captured and analyzed by the Kinectcamera in real-time +e software was written in C usingthe Kinect for Windows SDK version 20141019000 +eroom that was used for conducting the experiments con-sisted of different pieces of living room furniture to mimic apractical-use case of the device +is not only yields morerealistic results but also tests the system in real-life scenarioswhere the subject is partially visible to the camera and not allthe skeletal joints are being tracked To test and compare theKinect v2rsquos accuracy in determining both vertical andhorizontal angles according to the subjectrsquos foot distance tothe Kinect camera and body orientation eight Vicon T10cameras (considered as the gold standard) were also used tocapture the subjectrsquos movements and compare those with themovements determined by the Kinect +e Vicon camerasand the Kinect v2 captured each session simultaneouslywhile the frame rate of the recorded data from the Viconcameras was down-sampled to match the Kinect v2 at ap-proximately 30 frames per second

At a later stage and following an ethical approval therewas a recruitment of 15 PwP (with the collaboration ofParkinsonrsquos UK) to test the system and provide feedback+is research was published separately in [22] +e more in-depth analysis and information with regard to this focusgroup can also be checked via [24]

21 Kinect RGB-D Sensor Microsoft Kinect v2 is a time-of-flight (TOF) camera that functions by emitting infrared (IR)lights on objects and upon reflection of the lights back to theIR receiver it constructs a 3Dmap of the environment wherethe Z-axis is calculated via the delay of receiving IR light [25]Kinect v2 introduced many features and improvementscompared to its predecessor such as 1080p and 424p

2 Journal of Healthcare Engineering

resolution at approximately 30 frames per seconds for itsRGB and depthIR streams respectively as well as a widerfield of view [26] +e ability to track 25 joints of six subjectssimultaneously enables researchers to employ Kinect v2 asan unobtrusive human motion tracking device in differentdisciplines including rehabilitation and biomedicalengineering

22 Angle Determination +e Kinect v2 was used to de-termine the subjectsrsquo location in a 3D environment andlocalize the subjectrsquos feet joints to calculate the correcthorizontal and vertical angles for servo motors To de-termine the subjectrsquos location Kinect skeletal data were usedfor jointsrsquo 3D coordinate acquisition A surface floor can bedetermined by using the vector equation of planes +is isnecessary to automate the process of calculating the Kinectrsquosheight to the floor that is one of the parameters in de-termining vertical servo angle

Ax + By + Cz + D 0 (1)

where A B and C are the components of a normal vectorthat is perpendicular to any vector in a given plane and D isthe height of the Kinect from the levelled floor x y and z

are the coordinates of the given plane that locates the floor ofthe viewable area and are provided by the Kinect SDK AxBy Cz and D are also provided by the Kinect SDK once aflat floor is detected by the camera

For vertical angle determination a subjectrsquos 3D feetcoordinates were determined and depending on which footwas closer to the Kinect in the Z-axis the system selects thatfoot for further calculations Once the distance of the se-lected foot to the camera was calculated the vertical angle forthe servo motor is determined using the Pythagorean the-orem as depicted in Figure 1 +e subjectrsquos skeletal jointsrsquodistance to the Kinect on the Z-axis is defined in a right-handed coordinate system where the Kinect v2 is assumedto be at origin with a positive Z-axis value increasing in thedirection of Kinectrsquos point of view

In Figure 1 a is the Kinectrsquos camera height to the floorthat is the same as variable D from equation (1) and c is thehypotenuse of the right triangle which is the subjectrsquos se-lected foot distance to the Kinect camera in the Z-axis θ isthe calculated vertical angle for the servo motor Note thatwe have considered the position offsets in the X and Y axesbetween the Kinect v2 camera and the laser pointersservomotors in order to have the most accurate visual cueprojection

Our experiments showed that the Kinect v2 determines ajointrsquos Z-axis distance to the camera by considering its Y-axisvalue ie the higher the value of a jointrsquos Y-axis is to thecamerarsquos optical center the further the distance it has to thecamera in the Z-axis +is indicates that unlike the Kinectrsquosdepth space the Kinect skeletal coordinate system does notcalculate Z-axis distance (Figure 1 variable c) in a per-pendicular plane to the floor and as a result the height of thepoints that in this case are joints are also taken intoconsideration

In case of a joint being obstructed by an object forexample a piece of furniture the obstructed jointsrsquo 3DCartesian coordinate location tracking was compensatedand predicted using ldquoinferredrdquo state enumerate a built-infeature in the Kinect SDK By implementing the ldquoinferredrdquojoint state a joint data was calculated and its location wasestimated based on other tracked joints and its previouslyknown location

Figure 2 shows the Kinect v2 accuracy in determining asubjectrsquos joint (left foot) distance to the camera in Z-axiscompared to a gold standard motion capture device (ViconT10) It was concluded that Kinect v2 skeletal data acqui-sition accuracy was very close (9809) to the industrystandard counterpart +e random noise artifacts in thesignal were not statistically significant and did not affect thevertical angle determination

+e subjectrsquos body direction that determines the requiredangle for the horizontal servo motor can be yielded throughthe calculation of rotational changes of two subjectrsquos jointsincluding left and right shoulders+e subjectrsquos left and rightshoulder jointsrsquo coordinates were determined using skeletaldata and then fed to an algorithm to determine the bodyorientation as follows

servo angle 90 plusmn sinminus1 |shoulderAminus shoulderB|1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 (2)

In Figure 3 d is the Z-axis distance difference to thecamera between the subjectrsquos left and right shoulders

Once d based in the equation (2) was calculated theangle for the horizontal servo motor can be determined bycalculating the inverse sine of θ Depending on whether thesubject is rotating to the left or right the result would besubtracted or added fromto 90 respectively as the hori-zontal servo motor should rotate in reverse in order to castlaser lines in front of the subject accordingly

23 FOG Detection In previous studies the authors haveimplemented the process of FOG detection in [27] using thegait cycle and walking pattern detection techniques [26 28]Once the developed system detects a FOG incident it willturn the laser pointers on and start determining the ap-propriate angles for both vertical and horizontal servomotors After passing a user-defined waiting threshold ordisappearance of the FOG incident characteristics thesystem returns to its monitoring phase by turning off the

c

b

a Wal

l

θ

Floor

SubjectKinect servo motors

and laser pointers

Figure 1 Vertical angle determination

Journal of Healthcare Engineering 3

laser project and servo motors movements Figure 4 showsthe GUI for the developed system application

+e left image shows a Parkinsonrsquos disease patient im-itator during his FOG incident+e right window shows thatthe subject is being monitored and his gait information isbeing displayed to healthcare providers and doctors As itcan be seen in the ldquoFOG Statusrdquo section displayed in thebottom rectangle the system has detected a FOG incidentand activated the laser projection system to be used as avisual cue stimulus +e circled area shows the projection oflaser lines in front of the subjects (according to the distancefrom their feet to the camera) and their body direction +edeveloped system also allows further customization in-cluding visual cue distance adjustments in front of thepatient

24 Serial Connection A serial connection was needed tocommunicate with the servo motors controlled by theArduino Uno microcontroller +e transmitted signal by thedeveloped application needed to be distinguished at thereceiving point (the Arduino microcontroller) so each servomotor can act according to its intended angle and signalprovided We have developed a multipacket serial datatransmission technique similar to [29] +e data was labeledat the transmitter side so the microcontroller can distin-guish and categorize the received packet and send appro-priate signals to each servo motor +e system loops throughthis cycle of horizontal angle determination every 150ms+is time delay was chosen as the horizontal servo motordoes not need to be updated in real-time due to the fact that asubject is less likely to change hisher direction in very short

005

115

225

335

445

5

1 21 41 61 81 101 121 141 161 181 201 221

Dist

ance

(m)

Frames

Vicon T10Kinect v2 foot joint Z axis distance

Kinect v2Vicon T10

Figure 2 Subjectrsquos left foot distance to the camera in Z-axis using Kinect v2 and Vicon T10

R_shoulder

L_shoulder

θd

Subject

Kinect servo motorsand laser pointers

Projectedlaser lines

Figure 3 Horizontal angle determination (note that Kinect sees a mirrored image thus shoulders are reversed)


intervals +is ensures less jittery and smoother movementsof horizontal laser projection +e vertical servo motormovement was less prone to the jitters as the subjectrsquos feetare always visible to the camera as long as they are notobstructed by an object

25 Design of the Prototype System A two-servo system wasdeveloped using an Arduino Uno microcontroller and twoclass-3B 10mW 532 nm wavelength green line laser pro-jectors as shown in Figure 5(a) green laser lines have beenproven to be most visible amongst other laser colors used asvisual cues [30] A LCD display has also been added to thedesign that shows all the information with regard to verticaland horizontal angles to the user Figure 5(a) shows the laserline projection system attached to the tiltpan servo motorsFigure 5(b) shows the top view of the prototype systemincluding the wiring and voltage regulators Figure 5(c)shows the developed prototype system used in the experi-ment at different angles including the Kinect v2 sensor pantilt servo motors laser pointers and the microcontroller

3 Results

Figure 6 demonstrates the calculated vertical angle based onthe subjectsrsquo feetjoint distance to the Kinect camera in Z-axis +e right foot has been omitted in the graph forsimplicity

As Figure 6 demonstrates the system provided highlyaccurate responses based on the subjectrsquos foot distance to thecamera in Z-axis and the vertical servo motor angle

Subjects were also asked to rotate their body in front ofthe Kinect camera to test the horizontal angle determinationalgorithm and as a result the horizontal servo motorfunctionality Figure 7 shows the result of the calculatedhorizontal angle using equation (2) for the left and rightdirections

Figure 7 shows how the system reacts to the subjectrsquosbody orientation Each subject was asked to face the camerain a stand-still position while rotating their torso to the leftand to the right in turns As mentioned before the hori-zontal angle determination proved to be more susceptible to

noise compared to the vertical angle calculation +is is dueto the fact that as the angle increases to more than 65 de-grees the shoulder farthest away would be obstructed by thenearer shoulder and as a result the Kinect should com-pensate by approximating the position of that jointNonetheless this did not have any impact on the perfor-mance of the system

Overall the entire setup including the Kinect v2 sensortiltpan servo motors laser projectors microcontroller andLCD except the controlling PC will cost about pound13700making it much more affordable than other less capablealternatives available on the market

4 Discussion

A series of pantilt servo motors have been used alongsidelaser line projectors to create a visual cuing system whichcan be used to improve the mobility of PwP +e use of thesystem eliminates the need to carry devices helping patientsto improve their mobility by providing visual cues +eimplemented system has the ability to detect FOG using onlythe Kinect camera ie fully unobtrusive and provide dy-namic and automatic visual cues projection based on thesubjectrsquos location without the patientrsquos intervention as op-posed to other methods mentioned It was observed that thissystem can provide an accurate estimation of the subjectrsquoslocation and direction in a room and cast visual cues in frontof the subject accordingly +e Kinectrsquos effective coveragedistance was observed to be between 15 and 4meters (59and 15748 inch) form the camera which is within the rangeof the area of most living rooms making it an ideal device forindoor rehabilitation and monitoring purposes To evaluatethe Kinect v2rsquos accuracy in calculating the vertical andhorizontal angles a series of eight Vicon T10 cameras werealso used as a golden standard Overall the system proved tobe a viable solution for automatic and unobtrusive visualcuesrsquo apparatus Nonetheless there are some limitations tothis approach including the indoor aspect of it and the factthat it requires the whole setup including the Kinect servosand laser projectors to be included in the most communedareas of a house such as the living room and the kitchen

Figure 4 Graphical user interface for the developed software


Additionally during the experimentation the Kinectrsquos si-multaneous subject detection was limited to only one per-son Nevertheless Kinect v2 is capable to detect sixsimultaneous subjects in a scene However the laser pro-jection system in order to work properly should only aim atone person at a time +e developed system has the ability toeither lock on the first person that comes into the coveragearea or distinguish the real patient based on the locomotionpatterns and ignore other people Despite that the afford-ability and ease of installation of the system would still makeit a desirable solution should more than one setup need to be

placed in a house Moreover the use of a single Kinect wouldlimit the systemrsquos visibility and visual cue projection as well

5 Conclusion

+e results of this research showed a possibility ofimplementing an automatic and unobtrusive FOG moni-toring and mobility improvement system while beingreliable and accurate at the same time +e systemrsquos mainadvantages such as real-time patientrsquos monitoring im-proved locomotion and patientrsquos mobility and unobtrusive

0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames

Vertical servo motor anglefoot joint Z axis distance

Vertical servo angleLeft foot

Figure 6 Vertical servo motor angle in relation to the subjectrsquos foot joint distance to the Kinect camera in Z-axis

(a) (b)

(c)

Figure 5 +e developed prototype of the automatic visual cue system (a) +e two step motors controlling the horizontal and verticalalignment of the system (b) A top view of the Kinect v2 combined with the microcontroller and voltage regulators (c) A view of theprototype system in action


and dynamic visual cue projection make it in overall adesirable solution that can be further enhanced for futureimplementations

As a next step one could improve the systemrsquos cov-erage with a series of this implemented system to beinstalled in PwPrsquos houses to cover most of the communalareas or areas where a patient experiences the FOG themost (ie narrow corridors) One could also investigatethe possibility of using such systems attached to a circularrail on a ceiling that can rotate and move according to thepatientrsquos location this removes the need for extra setup ineach room as the system can cover some additional areasMoreover by coupling the system with other availablesolutions such as laser-mounted canes or shoes patientscan use the implemented system when they are at homewhile using other methods for outdoor purposes +isrequires integration at different levels such as a smart-phone application and visual cues in order for thesesystems to work as intended Finally the systemrsquos formfactor can be made smaller to some extent by removing theKinectrsquos original casing and embedding all the equipmentin a customized 3D printed enclosure which makes itmore suitable for a commercial production

Data Availability

+e gait analysis data used to support the findings of thisstudy are restricted by the Brunel University London EthicsCommittee in order to protect patient privacy Data areavailable fromCEDPS-Researchbrunelacuk for researcherswho meet the criteria for access to confidential data

Conflicts of Interest

+e authors claim no conflicts of interest in the subjectmatter or materials discussed in this manuscript

References

[1] B R Bloem J M Hausdorff J E Visser and N Giladi ldquoFallsand freezing of gait in Parkinsonrsquos disease a review of twointerconnected episodic phenomenardquo Movement Disordersvol 19 no 8 pp 871ndash884 2004

[2] S Donovan C Lim N Diaz et al ldquoLaserlight cues for gaitfreezing in Parkinsonrsquos disease an open-label studyrdquo Par-kinsonism amp Related Disorders vol 17 no 4 pp 240ndash2452011

[3] Y Okuma ldquoFreezing of gait in Parkinsonrsquos diseaserdquo Journal ofNeurology vol 253 no S7 pp vii27ndashvii32 2006

[4] L V Kalia and A E Lang ldquoParkinsonrsquos diseaserdquo 4e Lancetvol 386 no 9996 pp 896ndash912 2015

[5] S Choi H Jung G Yoon and B C Kim ldquoFactors associatedwith freezing of gait in patients with Parkinsonrsquos diseaserdquoNeurological Sciences vol 40 no 2 pp 293ndash298 2018

[6] P Ginis E Nackaerts A Nieuwboer and E HeremansldquoCueing for people with Parkinsonrsquos disease with freezing ofgait a narrative review of the state-of-the-art and novelperspectivesrdquo Annals of Physical and Rehabilitation Medicinevol 61 no 6 pp 407ndash413 2018

[7] A Nieuwboer ldquoCueing for freezing of gait in patients withParkinsonrsquos disease a rehabilitation perspectiverdquo MovementDisorders vol 23 no S2 pp S475ndashS481 2008

[8] P A Rocha G M Porfırio H B Ferraz andV F M Trevisani ldquoEffects of external cues on gait parametersof Parkinsonrsquos disease patients a systematic reviewrdquo ClinicalNeurology and Neurosurgery vol 124 pp 127ndash134 2014

[9] Y Zhao S Ramesberger U M Fietzek L T DrsquoAngelo andT C Luth ldquoA novel wearable laser device to regulate stridelength in Parkinsonrsquos diseaserdquo in Proceedings of 35th AnnualInternational Conference of the IEEE Engineering in Medicineand Biology Society (EMBC) pp 5895ndash5898 Osaka JapanJuly 2013

[10] G N Lewis W D Byblow and S E Walt ldquoStride lengthregulation in Parkinsonrsquos disease the use of extrinsic visualcuesrdquo Brain vol 123 no 10 pp 2077ndash2090 2000

[11] R Velik U Hoffmann H Zabaleta J Felix M Masso andT Keller ldquo+e effect of visual cues on the number and

0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames

Horizontal servo motor angle

Turning leftTurning right

Figure 7 Horizontal servo motor angle changes according to the subjectrsquos body orientation and direction during a test


duration of freezing episodes in Parkinsonrsquos patientsrdquo inProceedings of 2012 Annual International Conference of theIEEE Engineering in Medicine and Biology Society pp 4656ndash4659 San Diego CA USA September 2012

[12] T A Kaminsky B J Dudgeon F F BillingsleyP H Mitchell and S J Weghorst ldquoVirtual cues and func-tional mobility of people with Parkinsonrsquos disease a single-subject pilot studyrdquo Journal of Rehabilitation Research andDevelopment vol 44 no 3 pp 437ndash448 2007

[13] J McAuley P Daly and C Curtis ldquoA preliminary in-vestigation of a novel design of visual cue glasses that aid gaitin Parkinsonrsquos diseaserdquo Clinical Rehabilitation vol 23 no 8pp 687ndash695 2009

[14] H J Griffin R Greenlaw P Limousin K Bhatia N P Quinnand M Jahanshahi ldquo+e effect of real and virtual visual cueson walking in Parkinsonrsquos diseaserdquo Journal of Neurologyvol 258 no 6 pp 991ndash1000 2011

[15] M Gilat J M Shine S J Bolitho et al ldquoVariability ofstepping during a virtual reality paradigm in Parkinsonrsquosdisease patients with and without freezing of gaitrdquo PLoS Onevol 8 no 6 Article ID e66718 2013

[16] A C Lo V C Chang M A Gianfrancesco J H FriedmanT S Patterson and D F Benedicto ldquoReduction of freezing ofgait in Parkinsonrsquos disease by repetitive robot-assistedtreadmill training a pilot studyrdquo Journal of Neuro-Engineering and Rehabilitation vol 7 no 1 p 51 2010

[17] AMirelman I Maidan and J E Deutsch ldquoVirtual reality andmotor imagery promising tools for assessment and therapy inParkinsonrsquos diseaserdquo Movement Disorders vol 28 no 11pp 1597ndash1608 2013

[18] V Robles-Garcıa Y Corral-Bergantintildeos N Espinosa et alldquoEffects of movement imitation training in Parkinsonrsquos dis-ease a virtual reality pilot studyrdquo Parkinsonism amp RelatedDisorders vol 26 pp 17ndash23 2016

[19] M I R Severiano B S Zeigelboim H A G TeiveG J B Santos and V R Fonseca ldquoEffect of virtual reality inParkinsonrsquos disease a prospective observational studyrdquoArquivos de Neuro-Psiquiatria vol 76 no 2 pp 78ndash84 2018

[20] D Yelshyna M F Gago E Bicho et al ldquoCompensatorypostural adjustments in Parkinsonrsquos disease assessed via avirtual reality environmentrdquo Behavioural Brain Researchvol 296 pp 384ndash392 2016

[21] K Swim4e Effect of a Laser Guided Cane on Stride Length inPatients with Parkinsonrsquos Disease University of Toledo To-ledo OH USA 2010

[22] A Amini K Banitsas and W R Young ldquoKinect4FOGmonitoring and improving mobility in people with Parkin-sonrsquos using a novel system incorporating the Microsoft Kinectv2rdquo Disability and Rehabilitation Assistive Technologypp 1ndash8 2018

[23] F Foroughi and P Zong ldquoControlling servo motor angle byexploiting Kinect SDKrdquo International Journal of ComputerApplications vol 116 no 18 pp 1ndash6 2015

[24] A Amini Using 3D sensing and projecting technology toimprove the mobility of Parkinsonrsquos disease patients PhDthesis Brunel University London London UK 2018

[25] A Corti S Giancola G Mainetti and R Sala ldquoAmetrologicalcharacterization of the Kinect v2 time-of-flight camerardquoRobotics and Autonomous Systems vol 75 pp 584ndash594 2016

[26] A Amini K Banitsas and J Cosmas ldquoA comparison betweenheuristic and machine learning techniques in fall detectionusing Kinect v2rdquo in Proceedings of 2016 IEEE InternationalSymposium on Medical Measurements and Applications(MeMeA) pp 1ndash6 Benevento Italy May 2016

[27] A A M Bigy K Banitsas A Badii and J Cosmas ldquoRec-ognition of postures and freezing of gait in Parkinsonrsquos diseasepatients using Microsoft Kinect sensorrdquo in Proceedings of 7thInternational IEEEEMBS Conference on Neural Engineering(NER) pp 731ndash734 Montpellier France April 2015

[28] A Amini K Banitsas and S Hosseinzadeh ldquoA new techniquefor foot-off and foot contact detection in a gait cycle based onthe knee joint angle usingMicrosoft Kinect v2rdquo in Proccedingsof 2017 IEEE EMBS International Conference on Biomedical ampHealth Informatics (BHI) pp 153ndash156 Orland FL USAFebruary 2017

[29] B Evans Beginning Arduino Programming Apress BerkeleyCA USA 2011

[30] L Bunting-Perry M Spindler K M Robinson J NoorigianH J Cianci and J E Duda ldquoLaser light visual cueing forfreezing of gait in Parkinson disease a pilot study with maleparticipantsrdquo Journal of Rehabilitation Research and Devel-opment vol 50 no 2 pp 223ndash230 2013


International Journal of

AerospaceEngineeringHindawiwwwhindawicom Volume 2018

RoboticsJournal of

Hindawiwwwhindawicom Volume 2018


Active and Passive Electronic Components

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawiwwwhindawicom

Volume 2018

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Control Scienceand Engineering

Journal of



Journal ofEngineeringVolume 2018

SensorsJournal of



RotatingMachinery


Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation


Hindawi

wwwhindawicom Volume 2018

Advances in

Multimedia

Submit your manuscripts atwwwhindawicom

on the batteries needed for the device charging the bat-teries and remembering to switch them on

In another attempt [10] researchers evaluated the effectof visual cues using two different methods including asubject-mounted light device (SMLD) and taped step lengthmarkers It was concluded that using laser projections basedon SMLD have promising effects on the PwPrsquos locomotionand gait performance +e method required patients to weara SMLD that some patients might find inconvenient to haveor even impractical in some situations Moreover SMLDsystems have stability issues and steadiness difficulties due tothe subjectsrsquo torso movements during a gait cycle As ex-pected the visual cues must be constantly enabled during agait cycle regardless whether they are needed or not

In [11] although the SMLD method was employedresearchers added the 10 seconds on-demand option to theldquoconstantly onrdquo visual cue casting +is system was moresophisticated consisting of a backpack having a remotelycontrollable laptop that made the subjectsrsquo mobility evenmore troublesome

In other attempts [12 13] a different approach wasimplemented by using virtual cues projected on a pair ofgoggles that is only visible to the patient In [14] the effect ofreal and virtual visual cueing was compared and it wasconcluded that real transverse lines casted on the floor aremore impactful than the virtual counterparts Nonethelessusing virtual cueing spectacles (VCS) eliminates theshortcomings in other techniques such as limitations inmobility steadiness and symmetry VCS have also theadvantage of being used in an external environment whenthe patient is out and about

Moreover several research studies have been conductedusing virtual reality (VR) to assess the possibility of VRintegration for Parkinsonrsquos related studies [15ndash20] None-theless as the VR technology blocks patientsrsquo view andmakes them unable to see their surroundings the usage ofthis is limited to either rehabilitation by implementingexercise-based games FOG provoking scenarios or theassessment of patientsrsquo locomotion rather than real-timemobility improvement using cues

Although they are effective to some extent these at-tempts tend to restrict the user either by forcing them tocarry backpacks or wear vests containing electronics ormaking them rely on conventional approaches such asattaching laser pointers to a cane [21] or laser add-on forshoes

+e hypothesis of this study on the other hand is topropose a different technique casting parallel laser lines as adynamic and automatic visual cuing system for PwP basedon Kinect v2 and a set of servo motors suitable for indoorenvironments As Kinect has been proven to be a reliabledata feed source for controlling servo motors [22 23] theKinect camera was chosen for real-time depth data feed forthis study +is paper also examines the possibility of usingthe Kinect v2 sensor for such purposes in terms of accuracyand response time

+is research uses subjectrsquos 3D Cartesian location andhead direction as an input for servo motors to cast visualcues accordingly +is eliminates the need of the user

intervention or trigger and at the same time the need tocarry or wear any special equipment Despite this approachbeing limited to environments equipped with the proposedapparatus it does not require any attachments or reliance onPwP themselves something that can be beneficial in manyscenarios +e system comprises a Microsoft Kinect v2 a setof panttilt servo motors alongside a microcontroller basedon Arduino Uno and two laser line laser pointers A two-lineprojection was chosen so that the second traversed laser linecould be used to indicate a set area for which the next stephas to land +e system was tested in different conditionsincluding a partially occluded scene by furniture to simulatea living room

2 Methods

During the initial testing phase 11 healthy subjects wereinvited consisting of both males and females ranging fromages 24ndash31 with the age mean of 27 and SD of 234 a meanheight of 17445 cm (6868 inch) and SD of 831 cm(327 inch) ranging from 163 to 187 cm (6417 to 7362 inch)+ey were asked to walk in predefined paths 12 paths persubject walking towards the camera and triggering a sim-ulated FOG incident by imitating the symptom while havingthe Kinect camera positioned at a fixed location +e sub-jectsrsquo skeletal data were captured and analyzed by the Kinectcamera in real-time +e software was written in C usingthe Kinect for Windows SDK version 20141019000 +eroom that was used for conducting the experiments con-sisted of different pieces of living room furniture to mimic apractical-use case of the device +is not only yields morerealistic results but also tests the system in real-life scenarioswhere the subject is partially visible to the camera and not allthe skeletal joints are being tracked To test and compare theKinect v2rsquos accuracy in determining both vertical andhorizontal angles according to the subjectrsquos foot distance tothe Kinect camera and body orientation eight Vicon T10cameras (considered as the gold standard) were also used tocapture the subjectrsquos movements and compare those with themovements determined by the Kinect +e Vicon camerasand the Kinect v2 captured each session simultaneouslywhile the frame rate of the recorded data from the Viconcameras was down-sampled to match the Kinect v2 at ap-proximately 30 frames per second

At a later stage and following an ethical approval therewas a recruitment of 15 PwP (with the collaboration ofParkinsonrsquos UK) to test the system and provide feedback+is research was published separately in [22] +e more in-depth analysis and information with regard to this focusgroup can also be checked via [24]

21 Kinect RGB-D Sensor Microsoft Kinect v2 is a time-of-flight (TOF) camera that functions by emitting infrared (IR)lights on objects and upon reflection of the lights back to theIR receiver it constructs a 3Dmap of the environment wherethe Z-axis is calculated via the delay of receiving IR light [25]Kinect v2 introduced many features and improvementscompared to its predecessor such as 1080p and 424p




Ax + By + Cz + D 0 (1)










11138681113868111386811138681113868 (2)




c

b

a Wal

l

θ

Floor


and laser pointers






005

115

225

335

445

5

1 21 41 61 81 101 121 141 161 181 201 221

Dist

ance

(m)

Frames


Kinect v2Vicon T10


R_shoulder

L_shoulder

θd

Subject







3 Results







4 Discussion






5 Conclusion


0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames




(a) (b)

(c)





Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia




Ax + By + Cz + D 0 (1)










11138681113868111386811138681113868 (2)




c

b

a Wal

l

θ

Floor


and laser pointers






005

115

225

335

445

5

1 21 41 61 81 101 121 141 161 181 201 221

Dist

ance

(m)

Frames


Kinect v2Vicon T10


R_shoulder

L_shoulder

θd

Subject







3 Results







4 Discussion






5 Conclusion


0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames




(a) (b)

(c)





Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia





005

115

225

335

445

5

1 21 41 61 81 101 121 141 161 181 201 221

Dist

ance

(m)

Frames


Kinect v2Vicon T10


R_shoulder

L_shoulder

θd

Subject







3 Results







4 Discussion






5 Conclusion


0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames




(a) (b)

(c)





Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia




3 Results







4 Discussion






5 Conclusion


0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames




(a) (b)

(c)





Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia




5 Conclusion


0

05

1

15

2

25

3

35

4

0

10

20

30

40

50

60

70

80

1 51 101 151 201 251 301 351

Dist

ance

(m)

Ang

le (d

egre

e)

Frames




(a) (b)

(c)





Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia




Data Availability




References












0

20

40

60

80

100

120

1 31 61 91 121 151

Ang

le (d

egre

e)

Frames




























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia

























RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia




RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia


Documents

UsingKinectv2toControlaLaserVisualCueSystemtoImprove ...downloads.hindawi.com/journals/jhe/2019/3845462.pdf · using virtual reality (VR) to assess the possibility of VR integration