http://sri.sagepub.com/Surgical Innovation

http://sri.sagepub.com/content/early/2014/11/13/1553350614556364The online version of this article can be found at:

 DOI: 10.1177/1553350614556364

published online 12 November 2014SURG INNOVLaura R. Wingfield, Myutan Kulendran, Andre Chow, Jean Nehme and Sanjay Purkayastha

Cognitive Task Analysis: Bringing Olympic Athlete Style Training to Surgical Education  

Published by:

http://www.sagepublications.com

On behalf of: 

  Institute for Research into Cancer of the Digestive System

can be found at:Surgical InnovationAdditional services and information for    

  http://sri.sagepub.com/cgi/alertsEmail Alerts:

 

http://sri.sagepub.com/subscriptionsSubscriptions:  

http://www.sagepub.com/journalsReprints.navReprints:  

http://www.sagepub.com/journalsPermissions.navPermissions:  

What is This? 

- Nov 12, 2014OnlineFirst Version of Record  

- Nov 13, 2014OnlineFirst Version of Record >>

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Surgical Innovation 1 –12© The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.navDOI: 10.1177/1553350614556364sri.sagepub.com

Reviews

Introduction

Surgical training continues to change and evolve because of restrictions on training hours, which has led to a loss of valuable training time for surgeons.1 This is partly because traditional training has focused primarily on cul-tivating motor skills and practical aspects of the surgical process such as time to complete a procedure, accuracy, and a reduction in errors. Cognitive task analysis (CTA) is a newer approach to surgical training that focuses on decision making and identifies potential procedural errors that are thought to part expert knowledge to trainees in a more efficient manner.

It is well established that the cognitive phase of acquir-ing motor skills cannot be overlooked. Spencer et al2 have highlighted how surgical training is comprised of 75% cognitive ability and 25% mechanical ability. The Fitts and Posner model for learning motor skills describes

cognitive staging as the primary mental process, followed by an associative and an automated stage, which is achieved on mastery of a task.3 Flin et al4 discuss how intraoperative cognitive decisions should be studied and mapped out to fulfill unmet needs in surgical education, especially as they state this knowledge can be lost in the automotive stages, having implications for operative

556364 SRIXXX10.1177/1553350614556364Surgical InnovationWingfield et alresearch-article2014

1University of East Anglia, Norwich, UK2St. George’s Hospital, London, UK3Department of Surgery and Cancer, Imperial College London, UK4Royal Free Hospital, London, UK

Corresponding Author:Sanjay Purkayastha, Department of Surgery and Cancer, Imperial College London, 1022 Queen Elizabeth the Queen Mother Wing (QEQM), St Mary’s Campus, London, UK. Email: s.purkayastha@imperial.ac.uk

Cognitive Task Analysis: Bringing Olympic Athlete Style Training to Surgical Education

Laura R. Wingfield1, Myutan Kulendran, MRCS, PhD2, Andre Chow, MRCS, PhD3, Jean Nehme, MRCS, MS4, and Sanjay Purkayastha, MD, FRCS3

AbstractBackground. Surgical training is changing and evolving as time, pressure, and legislative demands continue to mount on trainee surgeons. A paradigm change in the focus of training has resulted in experts examining the cognitive steps needed to perform complex and often highly pressurized surgical procedures. Objective. To provide an overview of the collective evidence on cognitive task analysis (CTA) as a surgical training method, and determine if CTA improves a surgeon’s performance as measured by technical and nontechnical skills assessment, including precision, accuracy, and operative errors. Methods. A systematic literature review was performed. PubMed, Cochrane, and reference lists were analyzed for appropriate inclusion. Results. A total of 595 surgical participants were identified through the literature review and a total of 13 articles were included. Of these articles, 6 studies focused on general surgery, 2 focused on practical procedures relevant to surgery (central venous catheterization placement), 2 studies focused on head and neck surgical procedures (cricothyroidotomy and percutaneous tracheostomy placement), 2 studies highlighted vascular procedures (endovascular aortic aneurysm repair and carotid artery stenting), and 1 detailed endovascular repair (abdominal aorta and thoracic aorta). Overall, 92.3% of studies showed that CTA improves surgical outcome parameters, including time, precision, accuracy, and error reduction in both simulated and real-world environments. Conclusion. CTA has been shown to be a more effective training tool when compared with traditional methods of surgical training. There is a need for the introduction of CTA into surgical curriculums as this can improve surgical skill and ultimately create better patient outcomes.

Keywordscolorectal surgery, evidence-based medicine/surgery, orthopedic surgery, simulation, surgical education, gastric surgery

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

2 Surgical Innovation

skills teaching by experts to trainees. Furthermore, the cognitive component of skills acquisition has been sup-ported by neuroimaging studies which show neurological changes accompanying the development of manual skills involving the primary motor complex especially in more complex tasks that engage motor and cognitive aspects.5

The theoretical basis underlying CTA was initially developed by military officials to increase learning of complex and often difficult to master technical skills.6 CTA has had proven efficacy in training musicians, pilots, and military personnel. Additionally, the theories behind CTA have been widely adopted within sports psychology where these techniques have been shown to improve per-formance of Olympic athletes.7 To date, there has been no review of the objective performance of CTA in the surgi-cal field. This review aims to identify how CTA improves a surgeon’s performance as measured by technical and nontechnical skills and evidence of improved surgical outcomes.

Methods

Literature Search Strategy

Original studies on CTA were identified by searching the following databases: MEDLINE, PubMed Central, and Cochrane databases on 1 August 2013. No date limita-tions were placed during the literature search. The fol-lowing keywords and phrases were used: “mental rehearsal and surgery,” “surgical team rehearsal,” “cogni-tive training and surgery” with the filter “systematic review,” “cognitive training and surgery” and the filters “systematic review and meta-analysis.” The search was also limited to articles published in English. The refer-ence lists studies retrieved from the above search were further manually reviewed to identify other potentially relevant studies. PRISMA Checklist was consulted to ensure studies were appropriate for inclusion in this review.

Selection Criteria

Studies that specifically analyzed cognitive task training within surgery or procedural-based tasks were evaluated. CTA was defined by researchers as a method to extract knowledge and perform complex procedures, that is, operative procedures or practical medical skills such as central venous lines. Studies were defined as using CTA through step-by-step protocols, checklists, or mental rehearsal only when accompanied with defined steps. Within the reviewed articles, only those that utilized already established or expert analysis of the procedural steps were included. Any overlapping study groups that contained the same data subjects were screened and

excluded. Additionally, inclusion and exclusion of each paper was met by a consensus of LRW and MK and any difference in opinion was discussed with SP, the senior author.

Cognitive Task Analysis Defined•• Process by which expert knowledge is imparted to

trainees by experienced surgeons.•• Inclusive of step-by-step intraoperative protocols,

intraooperative checklists and mental rehearsal only when accompanied with defined steps.

•• Focusing on the mental learning of steps, CTA does not include only manual and/or technical aspects of surgical training.

No limits or exclusions were made on the number of study participants or the specific type of surgery or specialist procedure. Preference was given to meta- analysis (of randomized control trials) and randomized control trials for inclusion within this review. Out of 66 initial studies identified via the search terms and scored using the CTA Research Ranking Scale (see scale description below), 31 articles were excluded following ranking by the CTA Research Ranking Scale and review by the authors as they were literature reviews or com-mentaries (full 66 search results available on online appendix).

One duplicate article was found and excluded. The remaining 34 articles were selected for further full manu-script review. From this group of articles, 21 that focused solely on the manual and/or technical aspects of surgical training without any form of CTA within the study design or highlighted nonsurgical aspects of training were excluded from the review (Figure 1).

Critical Appraisal and Cognitive Task Analysis Research Ranking Scale

A search for a preexisting standard rating system for the validity and strength of literature within this field did not yield any results. A 5-point rating system, the CTA Research Ranking Scale, was developed to indicate the strength of each study and assist in data analysis (Table 1). From the 66 total articles selected for initial review, all were analyzed for continuity purposes using the CTA Research Ranking scale listed following. A more exten-sive and detailed analysis of the final 13 articles selected for review was made including information such as type of assessment, CTA methodology, and fundamental learn-ing principle.

Scoring categories included journal impact score (based on 2012 impact scores), number of research study participants, and study type. Researcher-assigned

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Wingfield et al 3

journal impact scores were based on the following rat-ing scale: journals with an impact factor less than or

equal to 1.5 were assigned a value of 1, journals with an impact score of 1.5 to 2.5 were assigned a value of 2, journals with an impact score of greater than 2.5 to 3.5 were assigned a score of 3, journals with an impact fac-tor greater than 3.5 to 4.5 were assigned a score of 4, and journals with an impact score greater than 4.5 were assigned a score of 5. Similarly, researchers assigned scoring for each study type ranking studies on a 1-to-5 scale. The following rating scale was used: literature reviews/commentaries were assigned a score of 1, case–control/cross-sectional research was assigned a score of 2, cohort studies were assigned a score of 3, randomized control trials were assigned a score of 4, and meta-analysis of randomized control trials were assigned a score of 5. Last, articles were scored from 1 to 5 based on the number of participants in the study. The following ranking system was used: studies with less than 10 participants were assigned a score of 1, studies with greater or equal to 10 to 50 participants were assigned a score of 2, studies with greater than 50 to 100 participants were assigned a score of 3, studies with greater than 100 to 500 participants were assigned a score of 4, and studies with participants greater than 500 were assigned a score of 5.

A validity and impact of research score was garnered by tabulating the average score of all 3 categories. Finally, a meta-analysis of the data was not appropriate because of the heterogeneous nature of the included studies and lack of controlled comparative arm.

31 studies excluded as they werecommentary/literature reviews

21 studies excluded as they onlyfocused on manual/technicaltraining or non-surgical aspects oftraining

1 duplicate study identified and excluded66 initial studies identifiedvia search terms and

scored using CognitiveTask Analysis Research

Ranking Scale

34 studies identified aspotentially approriate andfull manuscript reviewed

13 final studies selected forinclusion

Figure 1. Search strategy.

Table 1. Scoring System.

Journal Impact score (Based on 2012 Impact Factor)

Designated Point Value (1-5 Point Scale)

<1.5 11.5-2.5 22.5-3.5 33.5-4.5 4>4.5 5

Study TypeDesignated Point Value

(1-5 Point Scale)

Meta-analysis of randomized control trial (RCT)

5

RCT 4Cohort study 3Case–control/cross-sectional 2Literature review 1

Number of ParticipantsDesignated Point Value

(1-5 Point Scale)

≥500 5100-499 450-99 310-49 2≤9 1

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

4 Surgical Innovation

Results

Literature Review

The literature review yielded 66 total research articles that were then manually reviewed and 34 full articles were reviewed with study participants ranging in exper-tise level from medical student to attending clinician. Following a secondary round of review of the 34 full articles, 13 were extracted as the most relevant and these included a total of 595 study participants (Table 2).8-23 Within these 13 articles, 6 studies focused on general surgery, 2 focused on practical procedures relevant to surgery (central venous catheterization placement), 2 studies focused on head and neck surgical procedures (cricothyroidotomy and percutaneous tracheostomy placement), 2 studies highlighted vascular procedures (endovascular aortic aneurysm repair and carotid artery stenting), and 1 detailed endovascular repair (abdominal aorta and thoracic aorta).

Participants

There was a mean of 50.09 (range = 15-98) participants from all studies included in the review and there was a propensity toward the evaluation of more junior clini-cians and students. For example, Eldred-Evans et al,11 used medical students with no prior experience on virtual reality simulators or box trainers resulting in a laparo-scopic naïve group. This allowed better control over study participants as they would have all been at the same technical level. Seven studies used medical students or

designated novices as research participants, 4 studies used medical residents (primarily in surgery specializa-tions), and 2 studies used attendings as their intervention group.

Ranked and Most Significant Studies

Using the CTA Research Ranking scale, 10 studies ranked 3 or more within the scoring system, therefore, making them most relevant to the current review. The article by Immenroth et al12 was the only article to score a 4 as it was a randomized control trial, was published in an impactful journal, and had the largest number of study participants (n = 98). The next most relevant articles according to the CTA Research Ranking Scale were by Barsuk et al,9 Sanders et al (2004),14 and Sanders et al (2008),15 which all received a score of 3.67. A prospec-tive randomized control trial of surgical interns (n = 63) over a 1-month period published in an impactful journal received the score of 3.33. This study by DaRosa et al16 focused on checklist-driven intervention based on CTA principles. The work by Willaert’s research group23 also received a score of 3.33 and focused on surgical team rehearsals.

Cognitive Task Analysis Methodology and Fundamental Learning Principle

A number of different CTA interventions were used within the studies, including the following: surgical cur-riculum incorporating CTA principles, checklists of

Table 2. Study Demographics and Specialty.

Authors Year No. of Participants Level Specialty

Arora et al8 2011 18 Novice General surgery (laparoscopic cholecystectomies)Barsuk et al9 2009 92 Resident (second and

third year)Surgical procedure (central venous catheter

placement)Campbell et al10 2011 26 Medical students and

junior residentsCricothyroidotomy

DaRosa et al16 2008 63 Surgical residents (first or second year)

General surgery (laparoscopic cholecystectomy)

Desender et al22 2013 24 Expert surgeons Vascular surgery (endovascular aortic aneurysm repair)

Eldred-Evans et al11 2013 64 Medical students General surgery (laparoscopic)Immenroth et al12 2007 98 Surgical interns General surgery (laparoscopic cholecystectomy)Patel et al13 2012 15 Expert surgeons Endovascular repair (abdominal aorta and thoracic

aorta)Sanders et al14 2004 65 Medical students General surgery (suturing)Sanders et al15 2008 64 Medical students General surgery (suturing)Sullivan et al19 2007 20 General surgical

residentsHead and neck (percutaneous tracheostomy

placement)Velmahos et al20 2004 26 Surgical interns Surgical procedure (central venous catheterization)Willaert et al23 2012 20 Residents Head and neck (carotid artery stenting)

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Wingfield et al 5

surgical procedures, mental rehearsal and visualization, and verbal discussion of appropriate steps. The most commonly used CTA interventions included author-derived curriculums using CTA principles as demon-strated by Velmahos et al,20 Sullivan et al,19 and Patel et al.13 There are no gold standard surgical training curricula currently devised for each of the tasks covered by the lit-erature. Adjunct CTA-based checklists were also a com-mon intervention (Table 3).

Although the terminology mental rehearsal was used by some research groups, on deeper inspection it con-tained similar—if not the same—fundamental learning principles as CTA, including implicit nodal points or steps. For instance Immenroth et al12 used ideomotroic training supervised by a specialized mental trainer. A primer was implemented prior to task completion, which included a list of motor instructions in sequence and a precise list of potential hazards during each nodal phase of the procedure. Similarly, Sanders’s research group14,15 used an imagery rehearsal that consisted of a physician-teacher verbally providing the subjects with instructions in making incisions, suturing, and tying knots. Simultaneous to these instructions, students were asked to imagine the techniques as if they were completing them in a real situation. This educational intervention, used language intended to insight visual images in the participants through core words and phrases, but the prin-ciple behind it was the same as CTA.

Assessment and Study Results

Assessment modalities ranged in complexity from sim-plistic checklists and self-evaluation questionnaires to more sophisticated and complex technical checklists and video review of procedures. More traditional assessments were done with porcine modules used to demonstrate sur-gical skills as well as box trainers used to demonstrate basic laparoscopic skills (Tables 4 and 5). Patel et al13 used a unique approach to assessing surgical skills fol-lowing intervention: through a careful review of operat-ing room (OR) logbooks for errors over a 6-week period. Nine procedures were carried out initially as a control. Prior to the next 6 procedures, a focus group–designed mental imagery protocol was implemented within the OR 5 minutes prior to the start of the procedure. The senior surgeon verbally spoke through all steps of the procedure and checked with OR staff for any points of potential complications. A total of 15 combined open/endovascular procedures were evaluated by a trained observer. Following blinding, 2 independent trained individuals scrutinized the OR logs and reviewed errors, which were classified as “major errors,” including unintentional clip-ping of an artery, and “minor errors” such as a delay at specific steps within the procedure. The researchers went

on to examine the error rates per hours as well as author-designated danger and delay scores indicating areas of high risk within the operative process.

Most studies produced and attempted to validate an assessment tool to examine improvements in surgical proficiency from their CTA intervention, which in turn should result in safer surgery. Within the reviewed litera-ture, confidence and operative knowledge were assessed by questionnaires and multiple choice knowledge exams as seen in the research of Arora et al,8 Campbell et al,10 DaRosa et al,16 Sanders et al,14,15 and Velmahos et al.20 The number of questions used to assess surgical skill dif-fered from a shortened 15-item checklist utilized by Velmahos et al to the more comprehensive 140-point self-efficacy self-inventory used by Campbell et al. For exam-ple, DaRosa et al describe a surgical curriculum aimed at first- and second-year residents for laparoscopic chole-cystectomy. The study consisted of 2 components: an instructional component consisting of 12 critical deci-sions during the operative procedure decided through a semistructured focus group and an evaluative phase. The evaluative phase composed of assessment of both techni-cal and nontechnical skills. Nontechnical skills were assessed through a pre–post open-ended written exami-nation. Content of the examination included knowledge of intraoperative decisions, associated complications and error avoidance, and error rescue strategies. Examples of questions include “immediately after inserting the initial trocar, the surgeon should check for injury of: ______.” Some questions included pictures and questions such as “Which retraction of the gallbladder provides optimal exposure of the cystic duct?” Technical skills were assessed on a porcine model objectively by an external assessor who was blinded to the patients’ allocation. Finally, the pre- and posttest results were validated and scored to assess significant data points.

The Arora research group8 used a unique and validated method of measuring nontechnical skill consistency among their mental practice group via a Mental Imagery Questionnaire (MIQ). The questionnaire was adminis-tered before each participant performed an animal model laparoscopic cholecystectomy. The questionnaire con-tains 8 items that were scored on a 7-point Likert-type scale from 1 to 7, and it assessed the following: mental preparedness to perform a procedure, confidence to per-form this procedure, usefulness of training, visual imag-ery—specifically how well participants could “see” themselves performing the procedures in their mind, how well the participant could “feel” the procedure in their mind, and their overall knowledge of the laparoscopic cholecystectomy. Although this questionnaire was origi-nally developed for use in sports performance, it has sub-sequently been adapted and validated for use in surgery and procedural performance. Within the total 20 study

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

6 Surgical Innovation

Table 3. Assessment Type and Associated Results.

Authors Year LevelCTA

Method Type of Assessment ResultsCTA improved performance

Paper Score

Arora et al8 2011 Novice Mental rehearsal Modified OSATS, MIQ •• Median OSATS score: 20 vs 15, P = .005 for session 1; 20.5 vs 13.5, P = .001 for session 2; 24 vs 15.5, P < .001 for session 3; 25.5 vs 15.5, P < .001 for session 4; and 27.5 vs 19.5, P < .001 for session 5)

Yes 3

•• Superior confidence and operative knowledge as evaluated by the MIQ

•• MIQ correlated with surgical performance (r = 0.47-0.8)

Barsuk et al9 2009 Resident (second and third year)

Surgical instruction using gold standard developed CTA curriculum

Simulator and 27-item CVC skills checklist as well as rates of catheter infection as measured by the hospital infection control committee

84.5% reduction in catheter associated infection rates following CTA intervention. There were fewer infections after the simulator-trained residents entered the intervention ICU (0.50 infections per 1000 catheter-days) compared with both the same unit prior to the intervention (3.20 per 1000 catheter-days) (P = .001)

Yes 3.67

Campbell et al10

2011 Medical students and junior residents

Checklist- Major task list and procedural list

19-point procedural checklist score and 140-point self-efficacy inventory

Procedural knowledge: CTA mean score: 17.75, SD 2.34; control mean score: 15.14, SD 2.48; p= 0.006. Self-efficacy: CTA 126.10 SD 16.9, control 110.67, SD 16.8

Yes 2.67

DaRosa et al16

2008 Surgical residents (first or second year)

Four 90 minute sessions over 6 months: video, discussion on error prevention and rescue strategies, peer coaching with hands-on session. Posttest assessment on session 4

Laparoscopic cholecystectomy, on porcine model, cholecystectomy checklist knowledge (decision making, errors, and technical skills). General open ended pre-, posttest. Box trainer

The intervention group scored significantly higher (mean score 61.0, SD = 6.2, P < .05) on the written posttest than the control group. There were no differences between groups on the practical examination

Yes 3.33

Desender et al22

2013 Expert surgeons

Team rehearsal in laboratory or angiosuite (within 24 hours of actual procedure)

Patient-specific rehearsal using real-life patient 3D anatomy and subjective questionnaire (Likert-type scale 1-5)

•• All team members thought case-specific rehearsal might lead to increased patient safety (median n = 4)

Yes 2.33

•• All team members found the PsR helpful in preparing team members for the actual procedure (median n = 4)

Eldred-Evans et al11

2013 Medical students

Mental rehearsal, Mackay nodal model, 12 steps

Circle cutting task VRS-enhanced group performance 91.7%, mental training–enhanced group performance 87.3%; P < .05

Yes 3

Immenroth et al12

2007 Surgical interns

Ideomotoric training supervised by mental trainer

7-point questionnaire based on mental training. Modified OSATS task–specific checklist (11 nodal points)

OSATS Task Specific Checklist: Test of least significant differences showed significant difference between mental training and practical group (P = .024). No difference between practical training and control group (P = .789). No difference in OSATS global rating scale. Mental training questionnaire: Positive feedback

Yes 4

Patel et al13

2012 Expert surgeons

5 min verbal run based on CTA generated task list through of all procedural steps within the surgery

OR error log pre- and postintervention

Error rates during the endovascular phase were lower after the intervention compared with before (2.5/hour [1.4-6.0] vs 7.6/hour [1.7-9.6], respectively; P = 0.05). During the endovascular phase, danger and delay scores were also lower after the intervention (1.2/error [1.0-2.0] and 1.3/error [1.0-2.3], respectively) compared with before (1.75/error [1.4-2.5] and 2.0/error [1.3-2.5], respectively) (P = .036 and P = .036 for danger and delay, respectively)

Yes 2.33

Sanders et al14

2004 Medical students

Relaxation and verbal instructions by clinical instructor

7-item rating scale of procedure, attitude questionnaire

No significant main effect between group or time using ANOVA

No 3.67

(continued)

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Wingfield et al 7

Authors Year LevelCTA

Method Type of Assessment ResultsCTA improved performance

Paper Score

Sanders et al15

2008 Medical students

15-item rating scale of surgical behavior, 6-item global rating of procedure; 6-item attitude scale (confidence in performance), 10-item prior experience in learning, anxiety inventory, Minnesota Paper Form Board Test

Imagery group significantly did better in final assessment, F(1, 62) = 5.31, P < .03, with no contribution from attitude and anxiety

Yes 3.67

Sullivan et al19

2007 General surgical residents

Cognitive decision points

1 month technical competence: 25-item checklist. Think out loud protocol. 6 month technical checklist and 40-item protocol assessment tool. CTA analysis based on own video at 1 and 6 months.

Significantly higher on technical competence at 1 month (CTA: 43.5 SD 3.7, control 35.2 SD 3.9, P = .001) and 6 months (CTA: 39.4 SD 4.2, control 31.8 SD 5.8, P = .004) Significantly better on think-out loud protocol (CTA: 25.4 SD 5.3, control: 19.2 SD 2.0, P = .004)

Yes 3

Velmahos et al20

2004 Surgical interns

30-minute demonstration by expert using CTA principles

15 item MCQ for knowledge, 14-item checklist for technical competence

•• Course interns scored significantly higher in the repeat test compared with the traditional interns (11 ± 1.86 vs 8.64 ± 1.82, P = .03)

Yes 3

Patient practice over 2-month period

•• Course interns achieved a higher score on the 14-item checklist (12.6 ± 1.1 vs 7.5 ± 2.2, P = .001). They required fewer attempts to find the vein (3.3 ± 2.2 vs 6.4 ± 4.2, P = .046) and showed a trend toward less time to complete the procedure (15.4 ± 9.5 vs 20.6 ± 9.1 minutes, P = .149)

Willaert et al23

2012 Residents 10-session cognitive/technical virtual reality course

Simulator-based metrics including total procedure time (in minutes) and fluoroscopy time (in minutes), and expert-based ratings

•• Performances were better with PsR • vs generic warm-up (GW) or no • warm-up (NW) (PsR = 16.3 ± 0.6 vs • GW = 19.7 ± 1.0 vs NW = 20.9 ± 1.1 • minutes, P = .001)

Yes 3.33

•• PsR significantly improved the quality • of performance as measured by the • expert-based ratings compared to no • warm-up (scores 28 vs 25, P = .020)

Abbreviations: CTA, cognitive task analysis; CVC, central venous catheterization; OR, operating room; OSATS, Objective Structured Assessment of Technical Skills; MIQ, Mental Imagery Questionnaire; PsR, patient-specific simulated rehearsal; VRS, virtual reality simulator.

Table 3. (continued)

participants, 9 were selected for the mental practice inter-vention group and 9 were selected for the control group. Both groups underwent technical skills assessment via a pre- and postintervention (or control) procedure on a vir-tual reality laparoscopic cholecystectomy with assess-ment being analyzed using video Objective Structured Assessment of Technical Skills.

Campbell et al10 administered a pre- and posttest ques-tionnaire to all study participants (not just the interven-tion group as in the Arora group). A total of 26 third-year medical students and postgraduate year 2 and 3 surgery residents were randomized and assigned to the CTA group or control group. Prior knowledge and experience as part of the nontechnical skill set of all participants was tested via a 6-item questionnaire, which included open-ended questions about the actions and decisions required to conduct an open cricothyrotomy procedure. The

questionnaire was scored out of 17 points and answers were validated based on expert surgeon responses. A sep-arate and additional questionnaire was administered to participants following completion of the open cricothy-rotomy procedure on an inanimate model (both CTA and control groups). This self-appraisal questionnaire included 14 items that allowed participants to rate their confidence on a 10-point Likert-type scale. A score of 0 indicated “Cannot do at all,” a score of 5 indicated “Moderately certain I can do,” and 10 indicated that the participant was “Certain I can do.” The questionnaire had a possible total of 140 points that indicated where on the scale of perceived capabilities for the procedure and the cognitive steps of the procedure a participant’s abilities fell.

Velmahos et al20 used a 15-item multiple-choice ques-tionnaire to its study cohort of 26 surgical interns. This

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

8 Surgical Innovation

Table 4. Results.

Authors Year ResultsCTA Improved Performance

Paper Score

Arora et al8 2011 •• Median OSATS score: 20 vs 15, P = .005 for session 1; 20.5 vs 13.5, P = .001 for session 2; 24 vs 15.5, P < .001 for session 3; 25.5 vs 15.5, P < .001 for session 4; and 27.5 vs. 19.5, P < .001 for session 5)

Yes 3

•• Superior confidence and operative knowledge as evaluated by the MIQ •• MIQ correlated with surgical performance (r = 0.47-0.8) Barsuk et al9 2009 84.5% reduction in catheter associated infection rates following CTA

intervention; There were fewer infections after the simulator-trained residents entered the intervention ICU (0.50 infections per 1000 catheter-days) compared with both the same unit prior to the intervention (3.20 per 1000 catheter-days) (P = .001)

Yes 3.67

Campbell et al10

2011 Procedural knowledge: CTA mean score: 17.75, SD 2.34; control mean score: 15.14, SD 2.48; P = .006. Self-efficacy: CTA 126.10 SD 16.9, control 110.67, SD 16.8

Yes 2.67

DaRosa et al16

2008 The intervention group scored significantly higher (mean score 61.0, SD: 6.2, P < .05) on the written posttest than the control group. There were no differences between groups on the practical examination

Yes 3.33

Desender et al22

2013 •• All team members thought case-specific rehearsal might lead to increased patient safety (median n = 4)

Yes 2.33

•• All team members found the PsR helpful in preparing team members for the actual procedure (median n = 4)

Eldred-Evans et al11

2013 VRS-enhanced groups performance 91.7%, mental training–enhanced group performance 87.3%; P < .05

Yes 3

Immenroth et al12

2007 OSATS Task Specific Checklist: Test of least significant differences showed sig difference between mental training and practical group (P = .024). No difference between practical training and control group (P = .789). No difference in OSATS global rating scale. Mental Training questionnaire: positive feedback.

Yes 4

Patel et al13 2012 Error rates during the endovascular phase were lower after the intervention compared with before (2.5/hour [1.4-6.0] vs 7.6/hour [1.7-9.6], respectively; P = .05). During the endovascular phase, danger and delay scores were also lower after the intervention (1.2/error [1.0-2.0] and 1.3/error [1.0-2.3], respectively) compared with before (1.75/error [1.4-2.5] and 2.0/error [1.3-2.5], respectively) (P = .036 and P = .036 for danger and delay, respectively).

Yes 2.33

Sanders et al14

2004 No significant main effect between group or time using ANOVA No 3.67

Sanders et al15

2008 Imagery group significantly did better in final assessment, F(1, 62) = 5.31, P < .03, with not contribution from attitude and anxiety

Yes 3.67

Sullivan et al19

2007 Significantly higher on technical competence at 1 month (CTA: 43.5 SD 3.7, control 35.2 SD 3.9, P = .001) and 6 months (CTA: 39.4 SD 4.2, control 31.8 SD 5.8, P = .004) Significantly better on think-out loud protocol (CTA: 25.4 SD 5.3, control: 19.2 SD 2.0, P = .004)

Yes 3

Velmahos et al20

2004 •• Course interns scored significantly higher in the repeat test compared with the traditional interns (11 ± 1.86 vs 8.64 ± 1.82, P = .03)

Yes 3

•• Course interns achieved a higher score on the 14-item checklist (12.6 ± 1.1 vs 7.5 ± 2.2, P = .001). They required fewer attempts to find the vein (3.3 ± 2.2 vs 6.4 ± 4.2, P = .046) and showed a trend toward less time to complete the procedure (15.4 ± 9.5 vs 20.6 ± 9.1 minutes, P = .149)

Willaert et al23

2012 •• Performances were better with PsR vs generic warm-up (GW) or no • warm-up (NW) (PsR = 16.3 ± 0.6 vs GW = 19.7 ± 1.0 vs NW = 20.9 ± • 1.1 minutes, P = .001)

Yes 3.33

•• PsR significantly improved the quality of performance as measured by • the expert-based ratings compared to no warm-up (scores 28 vs 25, • P = .020)

Abbreviations: CTA, cognitive task analysis; OSATS, Objective Structured Assessment of Technical Skills; MIQ, Mental Imagery Questionnaire; PsR, patient-specific simulated rehearsal; VRS, virtual reality simulator.

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Wingfield et al 9

questionnaire was administered pre- and postintervention and was designed to specifically measure cognitive knowledge and technical know-how on how to insert a central venous catheterization (CVC) and 15 points was the maximum score possible on the test. This technical skills assessment included a pretest questionnaire that was administered to determine the baseline CVC knowl-edge of the interns (both in the control group and the CTA intervention group). The posttest, also testing technical skills, was repeated twice—once immediately after the intern placed a CVC on a patient and at the end of the study (2.5 months later). Compared with the other stud-ies, Campbell et al10 administered one of the posttests immediately following the procedure, which may have resulted in a more accurate participant recall of their cog-nitive knowledge in comparison with other studies where there was a delay (albeit often a short delay) in adminis-tering the posttest.

The Willaert group23 did not use questionnaires but instead looked at metrics gathered by their simulator’s readings and expert-based ratings. This group looked at

the performance of 20 residents to determine if a preop-erative rehearsal on a simulator combined with cognitive guidance from experts improved outcomes in endovascu-lar aortic stent placement. Study endpoints were total pro-cedure time in minutes, fluoroscopy time, and expert-based ratings. Those residents who underwent patient-specific simulated rehearsal showed a significant improvement in the quality of their patient performance as measured by the expert scorings (patient-specific sim-ulated rehearsal = 28 vs virtual reality warm-up = 25 vs no preparation = 25; P = .20).

Finally, Barsuk et al9 examined the performance of 92 internal medicine and emergency medicine residents when inserting a CVC over 32 months. The main end-point of this study looked at the rates of catheter-related bloodstream infection. The technical skills–based pretest included demonstrating CVC insertion using a simulator along with a 27-item CVC skills checklist. Following a training session of 3 hours, which included a video and additional skills training with an emphasis on sterile tech-niques to reduce infection rates, residents underwent a

Table 5. Fundamental Learning Principle.

Authors YearNo. of

Participants Level SpecialtyFundamental Learning

Principle

Arora et al8 2011 18 Novice General surgery (laparoscopic cholecystectomies)

Deductive reasoning, active thinking, and imagery.

Barsuk et al9 2009 92 Resident (second and third year)

Surgical procedure (central venous catheter placement)

Self-efficacy

Campbell et al10 2011 26 Medical students and junior residents

Cricothyroidotomy Self-efficacy

DaRosa et al16 2008 63 Surgical residents (first or second year)

General surgery (laparoscopic cholecystectomy)

Multiple strategies

Desender et al22 2013 24 Expert surgeons Vascular surgery (endovascular aortic aneurysm repair)

Perception of operation based on primer

Eldred-Evans et al11 2013 64 Medical students General surgery (laparoscopic)

Relaxation and guided visualization

Immenroth et al12 2007 98 Surgical interns General surgery (laparoscopic cholecystectomy)

Perception of operation based on primer

Patel et al13 2012 15 Expert surgeons Endovascular repair (abdominal aorta and thoracic aorta)

Perception of operation based on primer

Sanders et al14 2004 65 Medical students General surgery (suturing) Active thinking, imagerySanders et al15 2008 64 Medical students General surgery (suturing) Active thinking, imagerySullivan et al19 2007 20 General surgical

residentsHead and neck

(percutaneous tracheostomy placement)

Behavioral task analysis

Velmahos et al20 2004 26 Surgical interns Surgical procedure (central venous catheterization)

Multiple strategies

Willaert et al23 2012 20 Residents Head and neck (carotid artery stenting)

Perception of operation based on primer

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

10 Surgical Innovation

posttest on the simulator and same 27-item checklist used in the pretest. The rates of infection per 1000 catheter-days were measured monthly by the hospital’s infection control committee, which were in accordance to proto-cols by the National Healthcare Safety Network. In the 16 months prior to the CTA-based intervention, there were 3.20 infections per 1000 catheter-days in the medical intensive care unit and following the intervention there was a significant reduction of infections to 0.50 per 1000 catheter-days (P = .001). Barsuk et al9 highlight the wider arching benefits of CTA curriculum, including a signifi-cant reduction in infection and cost savings. Specifically, the intervention group had a significant 84.5% reduction in catheter associated infection rates. The reduction in catheter-related bloodstream infection rate was calcu-lated to result in a net savings of approximately $700,000.

Although varied assessment techniques were employed by the research groups described above, for this review, we have created a simplistic tool to determine whether the CTA-based intervention in each individual study examined by this review was successful. One author (LRW) independently rated each study as “yes or no” for CTA improving surgical performance based on the individual study’s results and this score was indepen-dently verified by a second author (MK), and any dis-crepancies were moderated by the senior author until a consensus was formed. From this author-derived rating scale, 92.3% of the studies selected for this review showed that CTA improves surgical outcome parameters, including time, precision, accuracy, and error reduction in both simulated and real-world environments (Table 4).

Discussion

Studies have shown that CTA can significantly increase surgical performance and efficiency in a range of levels from medical student up to expert surgeon. Unfortunately, although there is research showing that CTA can improve surgical outcomes, with few exceptions, a task analysis on the majority of common operations has not been com-pleted, nor has emphasis on this methodology been giving by societal or national surgical training organizations.24

Task analysis has been used in procedures of moderate complexity with low technology models but its feasibility in more complex models is yet to be studied. CTA has also been implemented in a range of specialties and on individuals of varied seniority. There is growing evidence to support CTA as an effective training tool when com-pared to more traditional methods of surgical training.

A CTA-based curriculum is especially important in training surgeons as it is estimated that expert surgeons may leave out up to 70% of vital steps when imparting key surgical knowledge.25 This knowledge gap may be attributed to the automation which occurs when surgeons

reach expert level. Although this automated knowledge, occurring with mastering of a task, is desirable for sur-geons it adds a level of difficulty in imparting knowledge to trainees as these experts lose a level of conscious awareness of such procedures.26 Additionally, although surgeons may be experts at their vocation, they may not necessarily be expert teachers, especially following auto-mation in combination with decreased train time.27 CTA methodology may provide a training tool to correct this problem of automation by creating a structured and vali-dated method to assist an expert in passing on procedural steps to trainees. Yates et al28 suggest that further research focus on developing a taxonomy of CTA methods in order to create validated CTA methods. Within the surgi-cal field, such a taxonomy may be especially crucial as a rise in complex cases and level of procedural knowledge is required.

Cognitive task analysis training has clear advantages over some other types of surgical training, including it is easy to administer, is low cost, and has a greater impact on training compared with other motor-based interven-tions. Other training modalities, including virtual reality training and Smartphone applications have also been explored as viable and useful training devices. As medi-cal training has evolved drastically from the 1970s with the advent of the first CO

2 laparoscopic procedure and

basic box trainers, there is a need within surgical training for a next-generation CTA tool, which may expand to incorporate the principles of CTA along with more sophisticated training paradigms. These training tools will be especially effective for the upcoming surgical trainees as they have the ease and portability of being used anywhere and with very limited additional resources. To date, there are very limited resources that fully incor-porate CTA in a validated method. One example of such a training tool is mobile phone applications that integrate learning and CTA technology into surgical training. Examples of such teaching technology currently using CTA include the AO Foundations’ Reference tool and SkinScan. The AO Foundations’ Reference tool incorpo-rates some aspects of CTA by providing procedural steps to its users.29 Touch Surgery is a 3-dimensional applica-tion for smartphones based on real-time validation and feedback. This tool allows surgeons the ease of accessing and running through surgical procedures at any time, while providing a CTA-based curriculum.30 Lastly, SkinScan is an application designed for Apple iOS that uses a 7-point checklist that has implications for use in automatic detection of wounds and ulcers.31

Finally, the high rate of heterogeneity combined with the low number of CTA-specific studies highlights a major gap in surgical education research. Further trials in CTA conducted on a larger, multi-institutional spectrum would add much needed information to surgical training.

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

Wingfield et al 11

It would be well advised to also investigate the use of CTA within more advanced surgical trainees as to date the majority of studies have been solely focused on nov-ice level trainees. This is because it is likely that CTA most benefits novice populations but this paradigm has yet to be proven. Additionally, there is very limited infor-mation on CTA within a number of surgical subspecial-ties such as orthopaedics, plastics, and vascular surgery. With further insight into the cognitive steps behind such procedures, more effective training could be developed to assist surgical training at every level. As time, pressure, and legislative demands continue to mount on trainee sur-geons, new and innovative methods for training need to be investigated.

Author Contributions

LW was responsible for the acquisition of data, analysis of data, drafting of manuscript and critical revision. MK was responsi-ble for data analysis, drafting of manuscript and critical revi-sion. AC was responsible for acquisiton of data, interpretation of data, critical revision and study concept and design. JN was responsible was for acquisiton of data, critical revision and study concept and design. SP was responsible for study concept and design and critical revision.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of inter-est with respect to the research, authorship, and/or publication of this article: SP, AC, and JN are cofounders and directors of the Touch Surgery Application.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

1. Department of Health. The European Working Time Directive for Trainee Doctors—Implementation Update. London, UK: Department of Health; 2009.

2. Spencer F. Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Am Coll Surg. 1978;63:9-12.

3. Shuell TJ. Phases of meaningful learning. Rev Educ Res. 1990;60:531-547.

4. Flin R, Youngson G, Yule S. How do surgeons make intra-operative decisions? Qual Saf Health Care. 2007;16:235-239.

5. Ghilardi M, Ghez C, Dhawan V, et al. Patterns of regional brain activation associated with different forms of motor learning. Brain Res. 2000;871:127-145.

6. Felson DF, Clark RE. Instructional implications of cogni-tive task analysis as a method for improving the accuracy of experts’ self report. In: Clarebout G and Elen J, eds. Avoiding Simplicity, Confronting Complexity: Advances in Studying and Designing (Computer-Based) Powerful

Learning Environments. Rotterdam, Netherlands: Sense; 2006:109-116.

7. Suinn RM. Mental practice in sport psychology: where have we been, where do we go? Clin Psychol Sci Pract. 1997;3:189-207.

8. Arora S, Aggarwal R, Sirimanna P, et al. Mental practice enhances surgical technical skills: a randomized controlled study. Ann Surg. 2011;253:265-270.

9. Barsuk JH, McGaghie WC, Cohen ER, O’Leary KJ, Wayne DB. Simulation-based mastery learning reduces complica-tions during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697-2701.

10. Campbell J, Tirapelle L, Yates K, et al. The effectiveness of a cognitive task analysis informed curriculum to increase self-efficacy and improve performance for an open crico-thyrotomy. J Surg Educ. 2011;68:403-407.

11. Eldred-Evans D, Grange P, Cheang A, et al. Using the mind as a simulator: a randomized controlled trial of mental training. J Surg Educ. 2013;70:544-551.

12. Immenroth M, Bürger T, Brenner J, Nagelschmidt M, Eberspächer H, Troidl H. Mental training in surgi-cal education: a randomized controlled trial. Ann Surg. 2007;245:385-391.

13. Patel SR, Gohel MS, Hamady M, Albayati MA. Reducing errors in combined open/endovascular arterial procedures: influence of a structured mental rehearsal before the endo-vascular phase. J Endovasc Ther. 2012;19:383-389.

14. Sanders CW, Sadoski M, Bramson R, Wiprud R, Van Walsum K. Comparing the effects of physical practice and mental imagery rehearsal on learning basic surgical skills by medical students. Am J Obstet Gynecol. 2004;191:1811-1814.

15. Sanders CW, Sadoski M, van Walsum K, Bramson R. Learning basic surgical skills with mental imagery: using the simulation centre in the mind. Med Educ. 2008;42:607-612.

16. DaRosa D, Rogers DA, Williams RG, et al. Impact of a structured skills laboratory curriculum on surgery resi-dents’ intraoperative decision-making and technical skills. Acad Med. 2008;83(10 suppl):S68-S71.

17. Pauley K, Flin R, Yule S, Youngson G. Surgeons’ intraop-erative decision making and risk management. Am J Surg. 2011;202:375-381.

18. Sullivan ME, Ortega A, Wasserberg N, Kaufman H, Nyquist J, Clark R. Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods? Am J Surg. 2008;195:20-23.

19. Sullivan ME, Brown CV, Peyre SE, et al. The use of cogni-tive task analysis to improve the learning of percutaneous tracheostomy placement. Am J Surg. 2007;193:96-99.

20. Velmahos GC, Toutouzas KG, Sillin LF, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg. 2004;187:114-119.

21. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D. Development of a rating system for surgeons’ non-technical skills. Med Educ. 2006;40:1098-1104.

22. Desender L, Rancic Z, Aggarwal R, et al. Patient-specific rehearsal prior to EVAR: a pilot study. Eur J Vasc Surg. 2013;45:639-647.

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from

12 Surgical Innovation

23. Willaert WI, Aggarwal R, Daruwalla F, et al. Simulated procedure rehearsal is more effective than a preoperative generic warm-up for endovascular procedures. Ann Surg. 2012;255:1184-1189.

24. Cuschieri A. Reducing errors in the operating room: Surgical proficiency and quality assurance of execution. Surg Endosc. 2005;19:1022-1027.

25. Feldon DF, Clark RE. Instructional implications of cogni-tive task analysis as a method for improving the accuracy of experts’ self report. In: Clarebout G and Elen J, eds. Avoiding Simplicity, Confronting Complexity: Advances in Studying and Designing (Computer-Based) Powerful Learning Environments. Rotterdam, Netherlands: Sense; 2006:109-116.

26. Clark RE. Resistance to change: unconscious knowledge and the challenge of unlearning. In: Berliner DC and Kupermintz H, eds. Changing Institutions, Environments and People. New York, NY: Routledge; 2008:75-94.

27. Anderson CI, Gupta RN, Larson JR, Abubars OI. Impact of objectively assessing surgeons’ teaching on effec-tive perioperative instructional behaviors. JAMA Surg. 2013;148:915-922.

28. Yates KA, Feldon DF. Advancing the practice of cognitive task analysis: a call for taxonomic research. Theoret Issues Ergon Sci. 2011:12:472-495.

29. Al-Hadithy N, Gikas PD, Al-Nammari SS. Smartphones in orthopaedics. Int Orthop. 2012;36:1543-1547.

30. Touch Surgery: the iPad app that teaches surgeons how to operate. http://www.guardian.co.uk/artanddesign/archi-tecture-design-blog/2013/feb/27/touch-surgery-ipad-app-surgeons-learn. Accessed August 20, 2013.

31. Wadhawan T, Situ N, Rui H, Lancaster K, Yuan X, Zouridakis G. Implementation of the 7-point checklist for melanoma detection on smart handheld devices. Conf Proc IEEE Eng Med Biol Soc. 2011;2011:3180-3183.

at TEXAS SOUTHERN UNIVERSITY on November 19, 2014sri.sagepub.comDownloaded from