Proceedings of the 7th RSIInternational Conference on Robotics and Mechatronics (ICRoM 2019)November 20-21 , 2019 , Tehran, Iran

Skill Assessment Using Kinematic Signatures:Geomagic Touch Haptic Device

N. S. Hojati" , M. Motaharifar" , H. D. Taghir ad" , A. Malekzadeh"*Advanced Robotics and Automated Syst ems (ARAS) , Industrial Control Cent er of Exc ellence (ICCE) ,

Faculty of Electrical Engineering, K.N. Toosi University of Technology, Tehran, Iran .Email: negarhojati@email.kntu.ac.ir.motah arifar@email.kntu.ac.ir.taghirad @kntu. ac.ir

+Faculty of Mathematics, K.N. Toosi University of Technology, Tehran, Iran. Em ail: malekzadeh@kntu. ac.ir

Abs tract-The aim of t his paper is t o develop a practi calskill assessment for some designed experimental tasks, retrievedfrom Minimally Invasive Surgery. The skill evaluation is veryimportant in surgery t raining, especially in MIS. Most of thepr evious studies for skill ass essment methods are limited in theHidden Markov Model and some frequ ency transforms, such asDiscret e Fourier transform, Discret e Cosine Transform and etc.In this paper , some features have been ext racted from time­frequency analysis with t he Discret e Wavelet Transform andt emporal signa l ana lysis of some kinematic metrics whi ch werecompute d from Geom agic Touch kinematic data. In addition,the k-n earest neighbors classifier are employed to detect skilllevel based on ext racted features. Through cross-validationresul ts, it is demonstrat ed that t he proposed methodology hasappropriate accuracy in skill level det ection.

Index Terms-Robotics-Assisted Minimally InvasiveSurgery, skill assessment , feature extraction, time-frequencyanalysis, t emporal signal analysis.

1. IntroductionAssessment of surgical skills during the training process

of novice surgeons is an essent ial part of MinimallyInvasive Surgery (MIS). MIS is a kind of surgical techniquein which surgical instruments are insert ed through smallincisions. This sur gical pro cedure needs a new set ofskills for a surgeon to perform MIS's techniques. There­fore, there is a need for a new training and assessmentmechanism to evaluate novice surgeon before ente ring theoperating room [1].

Glob al rating scales (GRS) and Objective StructuredAssessment of Technical Skill (OSATS) , which is a com­bination of GRS and checklists are st andard practicalskill assessment methods. They are curr ently in use inmedical education [2] . OSATS includes 9 metrics such asrespect of t issue, t ime and motion, instrument handling,and etc . In surgical skill assessment through OSATS , theexpert supervises the surgical pro cedure with a cameraand det ermines the score of each metrics for each trainee.In the end, the total score of the GRS and the checklistis present ed as the final grade of the trainee. Theseperformance metrics and the method of scoring is st illsubj ective [3], in the sense that it just depends on t heresult of surgery; it is t ime-consuming and there is nofeedback during learning complex tasks [4], [5].

Using haptic devices in surgical training and assessmentis a new attit ude in t his branch of medicine. Hapti c

devices involve physical contact between computer anduser , usually through an input / output device, such as ajoysti ck or dat a gloves. Hap tic devices highly facilit atesurgical t raining on account of the fact that they providean appropria te perception of the surgical environment fornovice surgeons . Furthermore, the capability of recordingpositi on and force signals make haptic t echnology suitablefor skill assessment, especially in MIS. By increasin g t heamount of Robot-Assisted Minimally Invasive Surgery(RAMIS) around t he world , applying RAMIS in surgicaltraining and surgical skill assessment has been noticed[6], [7]. RAMIS makes enable to drive position-based andforce-based metrics from the recorded data to obtain aquantitative and automat ic scoring syste m [8]-[10].

Recent approaches for automate d surgical skill assess­ment are based on Hidden Markov Mod el (HMM) [11]and applying classification methods t o global movementfeatures [12]. Recommended feature extraction methodsin previous works are Sequential Motion Texture (SMT) ,Discrete Fourier Transform (DFT), Discrete Cosine Trans­form (DCT) and Approximate Entropy (ApEn) [13] thatdo not contain local information about the movement. Per­formin g linear discriminant ana lysis with cross-validationt o examine the corr elation between the skill level andthe select ed metrics is another effort in this context [14].According to the results mentioned in [14], this methodhas less accuracy to detect different skill levels.

Although t he above studies investigat ed key novelconcepts in skill assessment , non e of t hem have noticedto discrete wavelet t ime-frequency analysis. The mostimportant superiority of wavelet t ransform t o Fouriertransform is that it provides the possibility of simpleanalyzing intense and abrupt behaviors in hand movement[15]. Therefore, it is inferred that the discret e wavelettransform is an appropriate method for recognizing themotion pat tern of people wit h different skill level.

This investigation presents an automated skill assess­ment method based on kinemati c metrics that pursue t hepro gress of individuals in simple tasks. For this purpose, asensorimotor task st ation is designed using the GeomagicTouch haptic device, and classification analysis on t heext racte d features from kinematic metrics is employed. Todetermine how well they could predict the practi cal skill

978-1-7281-6604-9/19/$31.00 ©20 19 IEEE 186

progression process, k-nearest neighbors (KNN) classifieris employed.

The rest of the paper is organized as follows. The meth­ods for data analyzing described in Section II. Section IIIpresents our experimental setup and modules. The resultof the classifier based on selected features is presented inSection IV. Finally, the paper ends up with conclusion inSection V.

and ADNP have two important advantages. The mostimportant one is that regardless of speed variation inpath traverse, they can find the positional differencebetween the participant's location and the desired point.Another one is that these methods provide the possibilityof comparison of two paths with different time duration.In ADNP, the type of path is not important accordinglythis method can be implemented for each curved path.

II. METHODS

Temporal signal analysis: Straight line path andcircle path are examined in two different methods oftemporal analysis, which are described respectively. Instraight line path Mean Square Error (MSE) is calculatedin N windows that first , we divide recorded X and Yto N windows, as well as X»: Next, the mean of eachwindow of X , Y, and x; is computed (X , YandXr )

which by placing X in the line equation and calculationthe essential Y (YE), MSE is computed as follows:

where N is the number of windows, YE figures out fromplacing X in the line equation and Y is the vector of theaverage of each window of Y.Computing MSE based on the mentioned method dependson the number of windows. It is important to choose theproper N due to the number of data samples.Calculating MSE for circle path is different from thestraight line. Therefore, this research proposes a newapproach named Accumulated Distances of NearestPoints (ADNP), based on finding the nearest point ofthe circle to the mean of each window. Same as givenabove, divide X and Y to N windows and then meanof each window is computed. The nearest point of thecircle to each mean is found and finally, all distancesare accumulated. MSE with divide data to N windows

(3)

(2)

1 nV=-"(xi- M )2

n-lL...ti=l

Feature extraction, preprocessing and classification:The features listed above are calculated as follows:Mean: Average value of each coefficients vector

where n is number of samples and x is data sequence.Variance: Variance is computed by averaging the sum ofsquare distance of observed value from the mean

Discrete wavelet transform: The most importantsuperiority of wavelet transform to Fourier transformis that it provides the possibility of non-stationarysignal components analysis. While wavelet analysis offerssimultaneous localization in time and frequency domain,classical Fourier transform of a signal does not containlocal information of data. One of the most importantobjectives of extending the wavelet transform is analyzingthe seismic behavior in a short period of a signal [15].Therefore, it seems that the wavelet transform is anexcellent choice for our problem because participantshave a tremor in their hands, especially in the earlystages of training. Discrete wavelet transform has beenused in heart disease classification to classify betweennormal and abnormal heart [16] and more applications inimage data processing like texture classification by imagedata [17], classification of magnetic resonance imagesof the human brain as either normal or abnormal [18].Accordingly, we suggest DWT as an efficient methodto extract fundamental features from an unfamiliarbehavior.

With several derivations of recorded X and Y, otherkinematic metrics such as speed, acceleration magnitude,and jerk are computed. Haar wavelet transforms in 4levels for each metric is computed, which for the inclusiveprocessing all have been taken. Haar wavelet is a sequenceof square-shaped functions that together form a waveletfamily. From each level of a DWT, some features suchas mean, variance, mean of energy, maximum energyand, the maximum amplitude have been extracted fromthe coefficients vector which we consider mean of eachfeature among four levels.

(1)N

1 " - 2MSE = N L...t(YE - Y)i=l

A. Data analyzingIn order to have a quantitative assessment of practical

skill, some kinematic metrics are employed, including thetool position at any moment, tool-tip speed, accelerationmagnitude and, jerk.

Our overall strategy is to consider some features , basedon Discrete Wavelet Transform (DWT) coefficients ofcomputed metrics to classify the skill level during thetraining process. Based on experiments performed ateach module, from the first set of each participant wechose one or more tests as a novice . From second andthird sets, some tests were taken as an intermediate andfrom the fourth set of each participant, one or more testsselected as an expert. Study the results of experimentsshow that most of the participants after four or five testsets, obtain proper ability to do the tasks with theirnon-dominant hand.

187

Mean of energy: The Energy of a signal defined as thesum of the square signal. Mean of energy is the averageof energy.

(4)

Maximum energy: It is the highest value of the energyvector.Maximum amplitude: It is the peak value of the xsequence.

Altogether, 21 features are extracted from DWT co­efficients of three metrics (tool position, speed of tool­tip, acceleration magnitude and, jerk). Notably, all of theextracted features are not essential for classification andit is necessary to eliminate the redundant features beforepassing them onto the classifier. In order to estimate theoptimum amount of features, performance of the classifierwith different features is evaluated. Finally, the onesthat have the highest average performance of classifyingrespective task are picked up.

We use a simple k-nearest neighbor (KNN) to classifyskill level based on extracted features . In order to provethe correctness and accuracy of the classifier, the k-foldcross-validation setup is employed.

III. EXPERIMENTAL SETUP

The experiments were performed at ARAS lab, Fac­ulty of Electrical Engineering, K. N. Toosi Universityof Technology. The participants were members of ARASlab, Postgraduate and Ph.D. students (three females andfour males) in the age between 23 to 30. Except for twoparticipants, the rest were right-handed.

At the beginning of the tests procedures, both exper­iment modules were explained for participants and theywere asked to take part in experiments several days. Ineach set of tests, both modules were tested for five times.To better see the progression process, participants wereasked to do all tasks with their non-dominant hand. Ourexperimental setup consists of a Geomagic Touch hapticdevice , a plate which placed exactly in front of the hapticdevice that was fixed by shields and, a monitor to see thecursor relative to the handle of the haptic device, see Fig .l.

The Geomagic Touch system provides realistic 3DTouch sensation for any application. Its workspace is160W x 120H x 70Dmm. Its position sensing has 6 degreesof freedom, x , y, andz through digital encoders and Roll ,pitch, yaw by linearity potentiometers [19].

Module_I: For the path traversing, there were twopaths with a distinct starting point and endpoint. Partic­ipants were asked to find the cursor related to the haptichandle on the x-y plane. Let us define the coordinates ofthe workspace, with horizontal (X-axis), vertical (Y-axis) ,

Fig. 1: The Experimental setup with Geomagic Touchhaptic device.

depth (Z-axis) components. The location of starting pointand endpoint for a straight-line path were at (0,-10,0)and (4,10,0), respectively. The circular path was startedand ended at (0,-2,0) with a radius of two units. In thecircle path, all participants started with the right sidemovement.

Module_2: The goal is to prepare an unfamiliar en­vironment in which participants must rapidly adapt withvariable opponent force. The presence of an opponent forcein the half of the path led to instrument vibration in theentry point. Through this module, participants learnedhow to control the instrument vibration in entry pointand adapt with variable opponent force . This module isanalogous to what would occur when the surgeon wants toinsert the instrument into the tissue. The circular path wasas the same as Module_1 but with the different startingpoint and endpoint (-2,-25,0). The opponent force wasapplied along the x-axis, toward the outside of the leftsemicircle. Matlab sample rate was chosen O.OO1s in bothmodules.

IV . RESULTS

Fig 2 is a sample of selected tests as the novice, theintermediate and the expert. Desired path is illustratedwith red color and traversed path with blue. Fig 2 (a) isa selected test from first set, Fig 2 (b) is from second setand Fig 2 (c) is from fourth set of experiment test of aleft hand participant.

We observed that as the number of windows increases,MSE and ADNP converge to their original value. We haveconcluded from MSE and ADNP calculation that with thesample rate of O.OOls, the most appropriate number ofwindows is between 100 to 150. However, if the samplerate is lower, the number of windows is reduced.

Figs. 3 and 4 show subplot results of some of thethirteen extracted features from straight-line path andnine extracted features from circle path in Moduel_l. Ascan be seen in both figures, there is a significant differencebetween the result of each feature and skill levels , inthe sense that these features can discern the skill levelproperly. Fig. 5 shows five extracted features from the

188

Expert

Expert

Expert

B

B

(j) MSE

(h) Maxlilng g

(f) MaxAmpj

(d) MaxAmpg

(b) MeanAmp j

Novice Intermediate Expert

Novice Intermediate

Novice Intermediate

Novice Intermediate Expert

EJ BNovice Intermediate

o

-5

10

0.065

0.06

0.055

0.05

0.045

0.04

0.035

o.03 e...,,,.....,_--,---,--"-::'--,---=~-,-----J

2.5

0.5

3.5

1.5

o0

0.075

0.Q7

0.065

0.06

0.055

0.05

0.045

0.04

E3

E3

Expert

Expert

g

(g) MaxEngp

(c) Varp

(e) MaxAmpj,

(a) MeanAmpj,

(i) Maxflngj

Novice Intermediate

Novice Intermediate Expert

Novice Intermediate Expert

Novice Intermediate

o

o0

o

ox 10-3

-2

0 .2

0.1

-4

0.3

2.5

0.5

1.5

0 .4

0.05

0.04

0.03

0.045

0.035

Fig. 3: Selected features from the straight-line path test(Moduelj l ).

-3' \-3-----:.2:;--~. ,,------;;O - -;-----;;--;X

(b) Int ermed ia t e

> ~ =~O'~~'~~-1

·2

3 - oesiredPath,

_ OT""redp"h.2 ._~

1

>- 0

-1

·2

-3' \-3-----:.2:;--~.,,------;;O - -;-----;;--;X

(a) Novice

> ~E~o'~~~1~d-1

·2

circle path in an unfamiliar environment with opponentforce (Moduel_2) . Features are mean, variance, meanof energy, maximum energy and maximum amplit ude ofdiscret e wavelet transform of kin emat ic met rics. In thesefigures index of each caption refers to relat ed metric.

Selected features were employed in KNN classifier inorder to distinguish skill level. In Moduel_1, the classifierhas 0% k-fold loss for straight -line path and 16% k-fold lossfor circle path. In Moduel_2, circle path with opponentforce has just 12% k-fold loss in skill level det ecti on .Because the extracted feat ures from DWT coefficientscarry important signatures of hand movement , t he loss ofclassifier reduced significant ly. Considering t ime-freq uencyanalysis with DWT and temporal signal ana lysis togetheris another imp ort ant aspect of this st udy. It is also inferredthat whatever t he tasks become complex, fewer feat uresare needed to distinguish skill level. The common feat uresin both mo dules are t he maximum am plitude of speed, t hemaximum energy of speed, t he maximum energy of jerk,ADNP in the circle path and MSE in the stra ight line.Therefore, the ment ioned features are the most import antones in simple and complex task skill level detecti on.

.3_3!;---:_2:;--~_1--;;O - -:--- ---;;------;X

(c) Exper t

F ig. 2: Examples of novice, int ermediate, and expertexpe rimental results (Module_I).

V. CONCLUSIONSIn this paper, a skill level detecti on method is proposed

for a sensorimotor designed task. T he suggested methodis based on t ime-frequency ana lysis wit h discrete wavelett ransform and te mporal signal ana lysis. Some extractedfeatures from DWT coefficients are chosen and app liedto KN N class ifier. Cross-validation results show the effec­t iveness of the proposed approach. Our future work is todevelop a generalized method of skill assessment whichis ab le to score practical skills in more accurate levels fordeveloped experimental test scenarios of MIS. In addit ion,applying our proposed method on clinical expe rimentaltest is our another goal.

189

0.250.055

0.2 0.05

EJ0.045

0.15 0.04

0.1 ~0.035 8Q

0.03 00.05 --'- E=3 0.025 --'-

Novice Intermediate Expert Novice Intermediate Expert

(a) MaxAmpp (b) MaxAmpg

0.080.055

B0.05 0.07

0.045 -;-0.06

0.04 Q -r-0.035 0.05 08 --'-

,0.03

0.04 0~ bd --'-0.025

0.02 0.03Novice Intermediate Expert Novice Intermediate Expert

(c) Max.Ampj, (d) MaxAmpj

x 10·30.12 4

0.1 3.5

0.08 B2.50.06

0.04

EJ1.5 0

0.02 G0

0.....--'-

~0.5

Novice Intermediate Expert Novice Intermediate Expert

(e) MaxEngp (f) MaxEngs

x 10-3 x 10·3

4 7 ...3.5

R6

-r- 52.5

8 4 --.0

1.5 3 0 SB0

2--'- g

0.5 eNovice Intermediate Expert Novice Intermediate Expert

(g) Maxflngj; (h) Maxflngj

0.8

0.6

0.4

0.2

Novice Intermediate Expert

(i) ADNP

Fig. 4: Selected features from the circle path test (Mod­uel_I).

0.18 0.05 --.0.16 0.045

0.14 g 0.04 80.120.035 ..:..

bd0.1

0.08 g 0.03

0.06 0 0.025 0--'-

0.04.

0.02Novice Intermediate Expert Novice Intermediate Expert

(a) MaxAmpp (b) MaxAmpg

x 10-3 x10·312-10

2.5

EJ 8

g ,6 ,

1.5 ,,

8 bJ, 4, 0--'-

0.5 ~

Novice Intermediate Expert Novice Intermediate Expert

(c) MaxEngs (d) Maxlilngj

0.18

0.16 U0.14

0.12

0.1 ,, 0

0.08~

E?0.06

t:;:I0.04

Novice Intermediate Expert

(e) ADNP

Fig. 5: Selected features from Moduel_2 experimental test .

References

[1] J . Rosen , B . Hannaford, C . G . Richards, and M . N. Sinanan,"Markov modeling of minimally invasive surgery based ontool/tissue interaction and forc e/torque signatures for eva lu­ating surgical skills, " IEEE transactions on Biomedical Engi­neering, vol. 48 , no . 5, pp. 579-591, 2001.

[2] B . Goff, L. Mandel, G . Lent z, A. VanBlaricom, A.-M . A.Oelsch lager, D . Lee, A. Galakatos, M. Davies , and P. Nielsen ,"Assessment of resident surgical skills: is t esting feasible? ,"Am eri can journal of obstetrics and gynecology, vol. 192, no. 4,pp. 1331-1338, 2005 .

[3] L. Moody, C. Baber, T . N. Arvanitis, and M . Elliott, "Object ivemetrics for the evaluation of simple surgica l sk ills in rea l and vir ­tual domains ," Presence: t eleoperators & virtual environme nt s ,vol. 12, no. 2, pp. 207-221 , 2003 .

[4] C. E . Reiley, H . C. Lin , D . D. Yuh, and G . D . Hager , "Reviewof methods for objective surgica l sk ill evaluation," Surgica lendoscopy, vol. 25, no . 2, pp. 356-366, 2011.

[5] A . Dubrowski, R. Sidhu, J . Park, and H. Carn ahan , "Quant ifi­cation of motion character istics and forces applied to t issuesduring suturing," The Ameri can journal of surgery, vol. 190 ,no. 1, pp. 131-136, 2005.

[6] A. Iranfar , M. Motaharifar , and H. Taghirad, "A dual-userteleope rated surgery training sch em e based on virtual fixture,"in 2018 6th RSI International Confer ence on Robotics andMechatronics (IcRoM), pp. 422-427, IEEE, 2018.

[7] N. Ahmidi , L. Tao , S. Sefati , Y. Gao, C. Lea , B. B. Haro,L. Zappella , S. Khudanpur, R. Vidal, and G. D. Hager , "Adataset and benchmarks for segmentation and recognition of

190

gestures in robotic surge ry," IEEE Transactions on BiomedicalEng inee ring , vol. 64, no. 9, pp. 2025-2041, 2017 .

[8] S. Cotin, N. Stylop oulos, M. Ot t ensmeyer , P. Neumann, D. Rat­t ne r, and S. Dawso n, "Metrics for laparoscopi c skills traine rs:the weakest lin k!," in Int er nat ional conference on medicalim age computing and computer-assisted int erventi on , pp. 35­43, Springer , 2002.

[9] A. L. Trejos, R. V. Patel, R . A. Malt haner, and C. M . Schlachta,"Developme nt of force-b ased metrics for skills assessment inminimally invasive surge ry," Surgical endoscopy, vol. 28, no . 7,pp. 2106-2119, 2014 .

[10] B. Poursartip, M .-E . LeBel, R. V. Patel, M . D. Naish , andA. L. Trejos, "Energy-based met rics for lapar oscop ic skillsassessment," in 2016 38th Annual Internation al Conferenceof t he IEEE Eng inee ring in Medicine and Biology Society(E MB C ) , pp. 2648-2651, IEEE , 2016.

[11] L. Tao, E . Elha mifar, S. Khudanpur, G. D. Hager , and R . Vidal,"Sparse hidden markov mod els for surgical gesture classificat ionand skill eva luation," in International conference on informationprocessing in computer-assisted inter ventions, pp. 167-177,Springer, 2012 .

[12] M. J. Fard, S. Arner i, R . Darin Ellis, R . B . Chinna rn, A. K.Pandya , and M . D. Klein , "Automated robot-assisted surgicalskill eva luation: Pred icti ve analytics approach ," T he In t erna­ti on al Journal of Med ical Ro botics and Computer Ass istedSurge ry, vol. 14, no . 1, p . e1850, 2018.

[13] A. Zia and I. Essa, "A utomated surgical skill assessme nt in rmist raining ," Int ernati onal journal of computer assisted radiologyand surge ry, vol. 13, no. 5, pp. 731-739, 2018.

[14] F . C. Hu ang, H. Mohamadipanah, F. Mussa-Ivaldi, andC. P ugh, "Combi ning metrics from clinical simulators and sen­sorimotor tasks can reveal t he training background of surgeons,"IEEE Transactions on Biomedical Engineering, 2019.

[15] M. Sifuzzaman, M . Islam, and M. Ali, "A pplication of wave lett ransfo rm and its advantages compare d to fourier t rans form ,"2009 .

[16] G. PatH, K . S. Rao, and K . Satyanarayana, "Heart diseaseclassificat ion using discrete wave let transform coefficients ofisolat ed b ea ts ," in 13th In t erna ti onal Conference on BiomedicalEng inee ring, pp. 60-64, Springer , 2009.

[17] M . Unser, "Texture classification and seg mentation usingwavelet frames," IEEE Transact ions on im age processing, vol. 4,no. 11, pp. 1549-1560, 1995.

[18] S. Chaplot, L. Patnaik, and N. Jagannathan, "Classificationof magnet ic resonance brain images using wave lets as input tosupport vector machine and neural network," Bio me dical signalprocessing and control, vol. 1, no. 1, pp . 86--92, 2006.

[19] "T he touch'P' hapt ic device." ht tps:/ / www.3dsystem s.com /hapt ics-devices/ tou ch /documen ts . Accessed: 2019-02-0 9.

\9\