10 bright ideas to set up a validated 4-step curriculum...2017/12/10 · 10 BRIGHT IDEAS TO SET UP A 4‐STEP VALIDATED CURRICULUM 10 When building a curriculum, the following aspects

10 bright ideas to set up a validated 4-step curriculumMay 2015

Contents Introduction1. The Cognitive Component2. Educational Framework3. A suggestion to a 4‐Step Curriculum4. Validate!5. Step 1: Basic Course6. Step 2: MCQ Tests7. Step 3: Engage Instructors in VR8. Overtraining is Good9. Step 4: Residents Can Also Assess in OR10. Motivation Resources Summary

10 BRIGHT IDEAS TO SET UP A 4‐STEP VALIDATED CURRICULUM 2

IntroductionIn modern surgical residency initial skills training has transitioned from the operating room tothe surgical skills laboratory. The effectiveness of simulators in teaching laparoscopictechniques is now widely accepted1. Despite the fact that simulation has shown to deliver a fast and patient‐safe path to technical competence in the operating room, it is still challengingto develop structured training curricula including simulation.

A curriculum encompasses many aspects, such as aims and objectives, learning outcomes, educational strategies, and assessment. The overall aim of any curriculum is to ensure structured teaching of an agreed standard of skills or knowledge within a specific field.

1A few examples: CR Larsen et al, BMJ 2009; NE Seymour et al, Ann Surg 2002; G Ahlberg et al, Am J Surg 2007; TP Grantcharov et al, Br J Surg 2007.


For a surgical curriculum to be successful it requires several elements, and it is important thatit contains a cognitive component, a practice component and subsequent supervised trainingin the actual clinical setting.

This ebooklet aims to help you in building a curriculum by presenting a series of bright ideas, all based on the excellent work done by Dr. Jeanett Strandbygaard, PhD, MD, Dept of Obstetrics and Gynecology, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, Copenhagen University Hospital, Denmark, published in her thesis ”Development and validation of a structured curriculum in basic laparoscopy”.

Have a good read!


1. The Cognitive ComponentHistorically, the operating room has served as the educational setting for novice surgeons, and skills were learned via the Halstedian model (see one, do one, teach one) – this is no longer acceptable due to ethical considerations toward patients, and the advances in educational theory and technology.

However, turning completely to simulators is not the solution. When developing a curriculum for basic laparoscopy it is important to remember that simulators will never do the whole job. Mere surgical skills are not sufficient to ensure proper laparoscopic knowledge; there is alsoneed for a cognitive component. In other words, there needs to be a balance betweenpedagogy and technology.


The challenges when learning to master laparoscopy relate to both psychomotor skills and the perceptual nature. Psychomotor learning theory emphasizes the integration of a cognitivestage – where a novice develops a mental picture of the motor task – as the first step in learning a procedure. The trainee must understand the steps of the operation in the correctorder and must know how to troubleshoot unexpected developments.

Following the cognitive component is the rehearsal of a task in a simulated environment, which then requires both technical and cognitive knowledge. These components, knowledgeand skills, are essential in the development of a technical skills curriculum.

It has actually been demonstrated that cognitive errors, such as a lack of understanding of the correct sequence of steps in an operation, trigger the majority of mistakes in a procedural task rather than technical errors2.

2B Tang et al, Surgery 2005


2. Educational FrameworkIn 1990 Miller described a framework for clinical competence assessment with increasinglycomplex levels of skills performance3. This theory is applicable to skills training and can helpunderstanding the assessment of skills performance:

Knows – the bottom level of the pyramid: didactic skills. Knowledge is a prerequisite for carrying out professional functions.Knows how – procedural knowledge: a physician canapply knowledge in concrete situations (the cognitivecomponent!)Shows how – ability to use this knowledge toperform concrete actions.Does – the actual doing in day‐to‐day practice.

3G Miller, Acad Med, 1990


Perhaps the best‐known evaluation methodology for judging learning processes is Kirkpatrick’s Four Level Evaluation Model. This could be used to help analyse the efficiency ofa curriculum. The model’s four levels:

Reaction level – measures how the participants react to the training or learning activity.Learning level – measures what the participants have actually learned.Behavior level – measures whether what is learned is being applied on the job.Result level – measures whether the application of training is achieving results.

Level 1: Reaction

• Did they like it?Level 2: Learning

• Did they learn it?Level 3: Behavior

• Did they use it?Level 4: Results

• Did it impact the bottom line?


3. Suggestion to a 4-Step CurriculumA curriculum for first‐year residents based on current curriculum literature and practice couldbe outlined as follows:

This curriculum4 matches the four levels of Miller’s theoretical framework: in each step the trainee progresses through the cognitive and practical steps that underlie the following step. Miller’s last level, the does level, does not reveal competence, just performance, so as an addition to the framework, an assessment of performance in a real clinical situation is part ofthe suggested curriculum.Over the following pages we will go through the content of each steps.

1‐day Basic CourseStep 1

Multiple‐Choice Testing

Step 2

StructuredVirtualReality

Simulation Training

Step 3 An OperationStep 4

Knows Knows how Shows how Does


4Strandbygaard et al, Acta Obstet Gynecol Scand, 2014

4. Validate!First of all, before we describe each step of the curriculum, it is important to bear in mind the bottom line question: Will the assessment instrument produce the desired educationaloutcome? There is no point in developing a program that does not demonstrate validity and reliability. Validity is divided into several facets, such as face validity, content validity, construct validity, and predictive validity, and each step of a curriculum needs to be carefully evaluated to fulfilall different demands.And a reliable curriculum produces consistent results under consistent conditions.

Even with great attention to validation, it is not easy to answer the question above – in fact, the most convincing way to justify a curriculum’s existence by setting up a randomized trial with an intervention group and a control group, is not really applicable due to the mountingevidence in favor of simulation training.


When building a curriculum, the following aspects of validation could be considered:

For a good example of a curriculum validation, see Palter and Grantcharov 20135.


Validation parametersFace Subjective expert review of test contentsContent Subjective detailed examination of test contentsConstruct Objective measurement. Degree to which test captures the hypothetical quality it was

designed to measureConcurrent Objective measurement. Extent to which scores on a test and a control instrument are

correlatedPredictive Objective measurement. Extent to which scores predict actual true performance

Reliability parametersInter‐rater Extent to which two different evaluators give the same score in a test performed by a

userIntra‐rater Internal consistency of an evaluator when grading on a given test on different

occasionsTest‐retest Extent to which two different tests made by the same person in two different time

frames give the same result

5Palter VN et al, Ann Surg. 2013

5. Step One: Basic CourseA basic course spanning over 1 day will allow trainees to be introduced to the procedures and gain basic knowledge. Typically, it could include a series of theoretical topics (30 minuteseach) and a chance to train hands‐on on bench models and simulators. Engage chief physicians, preferably in charge of education, and fellows when putting

together the program, have it approved by everyone. If possible, start off with a regional needs analysis.

Give a pilot course before launching the curriculum! Distribute a questionnaire after the course concerning satisfaction with all aspects!

Answers could be collected anonymously. Questions could be scored on a Likert‐type scalefrom one to five, where one represents ’strongly disagree’ and five represents ‘strongly agree’. This goes back to Kirkpatrick’s evaluation model: how do the participants feel after their training experience? High rating suggests good face validity.

Evidence on short, intensive courses improving knowledge and motor skills:G Condous, Aust NZ J Obstet Gynaecol 2009


6. Step Two: MCQ TestsA cognitive knowledge component is important and is expected to provide basicunderstanding of the surgical principles and knowledge in order to handle unexpected events. It has been demonstrated that cognitive skills enhance the ability to correctly execute a surgical task.Create a multiple‐choice questionnaire! MCQs are useful for testing cognitive knowledge and efficient for examination of large groups and large subject areas: The scoring of MCQ:s is time‐saving. It shows parallels to surgical decision‐making: recognition, distinguishing of the correct

solutions, and characteristics of selection, estimation, prediction, and categorization. There are several guidelines providing instructions on MCQs6.

6Examples: Case and Swanson, Constructing written test questions for the basic and clinical sciences, 2nd edn. Natl Board of Medicla Examiners, Philadelphia, 1998, or

T Haladyna et al, A review of multiple‐choice item‐writing guidelines for classroom assessment, ApplMeas Educ, 2002


Strandbygaard et al described the construction of a theoretical test in basic laparoscopy, based on a common structure of MCQ development7. The test consisted of 37 questions, all with one correct answer and three distractors:

7Strandbygaard et al, Surg Endosc, 2012

• A prior need analysis was conducted among 8 educational chief physicians representing all departments in the region in question.

Step 1: Needs analysis of relevant subject.

• A 2‐h conversational interview with four experts in laparoscopy, representing three different university hospitals, was performed to identify relevant basic laparoscopic knowledge for first yearresidents.

Step 2a: Content validity I ‐ Interviews

• Based on the interviews, MCQs were developed and distributed to 8 chief physicians at depts in the region in question. A Delphi method was applied to achieve consensus.

Step 2b: Content validity II – Delphi audit

• The construct validity was tested by comparison of three groups with expected different clinicalcompetence and knowledge levels. The high‐competence group consisted of the chief physicianswho participated in the Delphi evaluation. The intermediary group consisted of first‐year residents, and the low‐competence group consisted of medical students in their final year of med school.

Step 3: Construct validity


Writing effective, creative, and challenging MCQs is difficult and time consuming. Communicate what’s important! Tests should be preceded with analysis of knowledge

gaps, as suggested by Strandbygaard. Selecting the right topics is important. You cannot ask everything. It is of great importance to have content expertise in reviewing the questions. Questions should require examinees to problem solving using clinical judgment and

operational decision making rather than simple recollection. Remember to align with local key curricular goals. A successful MCQ will help achieving

them.


7. Step Three: Engage Instructors in VRThe advantages of virtual reality simulators are well‐established: simulation training gives a positive effect on the learning curve and improvement of basic psychomotor skills in the operating room.

Structured VR simulation training wins on being proficiency‐based – by presenting examineesto exercises where pass levels are based on expert results, individual variations in trainingtime and repetitions do not matter: In contrast to time/volume based training, profiency‐based training allows for self‐paced practice and goal‐directed learning, giving a consistencyof the final results because all examinees are expected to reach the same performancestandard.

Make sure you gain maximum training value by engaging active instructors during VR sessions! It has been shown that instructor feedback increases the efficiency when training a complex operational task on a simulator8. The instructors may not necessarilly be surgicalstaff, as long as they are familiar with procedures and simulators.

8Strandbygaard et al, Annals of Surgery, 2012


8. Overtraining is GoodInterestingly, when studying the impact of instructor feedback on simulation training(reference on last page), Strandbygaard et al found that the control group – receiving no feedback during their simulation training – reached significantly higher performance scores than the intervention group, although at a significantly slower pace. This finding was not unexpected: the performance score is a measure based on time and accuracy and increases withtraining, and in average the control group used twice the amount of time training.

So, although several papers demonstrate that instructorfeedback improves learning of skills, keep in mind thatwhen leaving the trainees to struggle on their own to a certain extent, they may learn more!The power of overtraining or deliberate practice to reach an automated level of knowledgeha also been described well by K Anders Ericsson9.


9Ericsson, Acad Emerg Med. 2008

9. Step Four: Residents Can Also Assess in ORSimulation training must be complementaryrather than supplementary to workplace learning: therefore the last step in the suggested curriculum is an integrated operational step. Such a last step would aim to connect the skills and knowledgeobtained in the preceding three steps to clinicalreality.

Think reward! Another reason for integrating the operational step is to offer residents the opportunity to perform in an operation under supervision after completing proper simulation training – to reward them this door‐opener.

Think engage! With an operational step, you help learners to take active engagement in clinical activity after going through simulation training.


There are several tools for assessment of technical skills, of which the well‐established OSATS is currently accepted as the gold standard: Objective Structured Assessment of TechnicalSkills10. An advantage of it is that it can be done both in real‐time and video‐based, the latterproviding the possibility to assess the performance whenever it is convenient.

With tutor time often being a critical factor, an operation step may be practically difficult toplan. But – do you really need senior physicians to supervise residents?

The answer could be no: it has been shown that chief physicians (having performed 250‐1000 lap procedures) and senior residents (in their fourth year of 5) generated similar performancescores when assessing operations using a laparoscopic procedure‐specific assessment scale11. The study suggests that doctors with different levels of experience can give formative feedback.

10JA Martin et al, Br J Surg, 199711J Oestergaard, Surg Endosc, 2012


10. Motivation ResourcesSo we have a proper training program including a basic course, a knowledge test, a VR training part, and an operation with standardized assessment. Are we all set to go?

‐ No! There are many aspects that must be included to allow a successful curriculum: participants’ motivation, available personnel, and faculty commitment are among the factorsthat play a crucial role in sparking trainee interest in the curriculum.

Money cannot only be spent on equipment – remember to spend it on enthusiastic personnelas well!

One of the strong points in including tests and assessments in the curriculum, is they arepowerful motivators for learning and can encourage residents to study on their own and participate in available educational opportunities. Also, protected training time, i.e. scheduled training during work hours, is usually a big motivator.


SummaryThis text was put together to give you some inspiration to develop a comprehensive trainingprogram for aspiring laparoscopists in their first year of residency. With the four steps described, learners are taken from the basic Knows level to the Does level, according to Miller’s educationaltheory on clinical assessment:

1) A one‐day hands‐on course in basic laparoscopy2) A validated multiple‐choice test3) Structured virtual reality simulation training with instructor feedback 4) An operation with subsequent formative feedback.

We express our gratitude towards Dr Jeanett Strandbygaard for letting us share her and her team’swork at the Dept of Obstetrics and Gynecology, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, Copenhagen University Hospital, Denmark. For more details and a comprehensive reference list, find her complete PhD thesis here.

Thanks for reading.


https://www.rigshospitalet.dk/afdelinger-og-klinikker/julianemarie/enhed-for-simulation/simulationsbaseret-forskning-og-traening/Documents/Phd_Jeanett_Strandbygaard_2012.pdf

About Surgical ScienceSurgical Science offers medical professionals a safe way to practice fundamental technicalskills for laparoscopy and endoscopy before entering an operation theater or procedureroom. While our simulators have provided medical training centers around the world with validated, efficient education and assessment tools for many years, we keep striving to innovate in the field of medical simulation.

Liked what you read? Please do not hesitate to contact us to learn more about simulation possibilities.

www.surgical‐science.com


Version 02