Review of Year 1 Immunology/Virology

• Course occurs in the winter term of Year 1

• Course Directors – Charles Sentman, Ph.D. David

Bzik, Ph.D.

• Course has 44 curricular hours

• Course was last reviewed in May 2012

Recommendations from 2012 Review• Incorporate more active modes of learning/new

methods of pedagogy into the course (i.e. reduce % of lectures)

• Rewrite/create learning objectives for each session to clearly articulate expectations to students

• Revise assessment questions; verify that they link back to major course objectives

• Faculty development is needed for some faculty that are not meeting expectations with regard to teaching

Recommendations from 2012 Review• Reorganize course to create a better flow for the

material; consider matching order of topics used in the text (specific to Immunology)

• Identify at risk students early and provide support, e.g. review sessions (specific to Immunology)

• Update/edit notes; ensure lecture content and notes are in agreement (specific to Virology)

• Provide an opportunity where biologic data are observed, measured and analyzed (specific to Virology)

Course Objectives – ImmunologyThere are 17 course objectives that fulfill Geisel competencies as follows:

• 13 address specific knowledge in the preclinical domain, and seem appropriate

• 4 address communication skills• 3 address components of professionalism, although

only 1 is mapped to competency 4 in Ilios• 1 addresses habit of inquiry/personal improvement

Currently no course objectives are mapped to Geisel competencies 2 (clinical skills) and 6 (health care systems).

Course Objectives – Virology

could map to 4

could map to 4

Course Objectives – VirologyThere are 19 course objectives listed in Ilios that fulfill Geisel competencies as follows:

• 19 address specific knowledge in the preclinical domain, and seem appropriate

• 1 addresses habit of inquiry/personal improvement, but is currently not mapped to competency 5

Currently no course objectives are mapped to Geisel competencies 2, 3, 4 and 6.

could map to 5b

Course Objectives• Course objectives are provided to the students in the

syllabus for both parts of the course, and are clearly written using measurable verbs

• Course objectives correlate well with the content in the Step 1 brochure

• Did not find a published list of objectives by a national organization in the field

Session ObjectivesRecommendation 2012: Rewrite/create learning objectives for each session to clearly articulate expectations to students

• Session objectives are provided at the top of each set of notes (all except sessions on Herpes viruses, Tumor viruses, Poliovirus)

• Most session objectives are written clearly using measurable verbs (all except several immunology sessions that use words like “know” and “understand”)

Objectives addressing LCME topics

• Health of Populations: Course includes discussions of TB and RA, and how they affect the health care system and society; course discusses vaccines in the context of societal needs and problems.

• Basic and Ethical Principles of Clinical and Translational Research: Course includes discussions of cases that occurred prior to the emergence of more modern views regarding basic scientific and ethical principle of clinical and translational research, including ways in which it is conducted, evaluated, explained to patients, and applied to patient care.

Objectives addressing LCME topics

• Gender and Cultural Biases in Students Themselves: Access to vaccination, access to diagnosis, access to treatments, and risk of virus diseases are strongly influenced by gender, racial/ethnic, and cultural biases.

• Instruction in Medical Ethics and Human Values: Not included in the course as this time.

Assessing integration in the curriculum• The curriculum database was used to evaluate the

integration of major topics in the course across the curriculum (e.g. Immunodeficiency, Rheumatoid Arthritis, HIV, Viral Hepatitis, Influenza)

• In the past our curriculum has been weak in issues related to the elderly, so we also assessed whether the course discussed changes of immune function with age

Immunodeficiency• present in two Year 1 courses (Immunology/Virology

and General Pathology)• present in three Year 2 courses (Organ-based Pharm,

SBM Infectious Diseases, SBM Neurology)• present in two Year 4 courses (Advanced Medical

Science, Clinical Pharmacology and Therapeutics)

Lymphoma• present in one Year 1 course (Microbiology)• present in one Year 2 course (SBM Hematology)

Rheumatoid Arthritis• present in one Year 1 course (General Pathology)• present in two Year 2 courses (SBM Bone and CT, On

Doctoring)

HIV• present in two Year 1 courses (Immunology/Virology

and General Pathology)• present in three Year 2 courses (Organ-based Pharm,

SBM Infectious Diseases, SBM Neurology)• present in two Year 4 courses (Advanced Medical

Science, Clinical Pharmacology and Therapeutics)

Viral Hepatitis• present in one Year 1 course (Immunology/Virology)• present in one Year 2 course (SBM GI)

Influenza• present in four Year 1 courses (Immunology/Virology,

Microbiology, General Pathology and HCDS)• present in one Year 2 course (Organ-based

Pharmacology)

Age-related changes in Immunity• searched using “geriatrics”, “elderly”, “aging”, “age

process”• nothing mapped to Year 1 Immunology/Virology• course directors should consider including at least

one of these words in one of their objectives so that this material is detected in a search

Summary regarding Objectives• Course objective listed in ILIOS are provided to the

students; several opportunities exist for mapping existing objectives to additional Geisel competencies

• Most sessions now have objectives that are provided to students; several sessions still need objectives, and several sessions need to rewrite their objectives using measurable verbs

• Searches in ILIOS regarding major topics in the course demonstrated some redundancy with material in other courses; much of this redundancy is planned and appropriate, however the course directors should verify this with other course directors

Course Learning OpportunitiesRecommendation 2012: Incorporate more active modes of learning/new methods of pedagogy into the course (i.e. reduce % of lectures)

Immunology• Lecture 21 hrs. (some are interactive) – 88%• Discussion groups 3 hrs. – 12%Virology• Lecture 20 hrs. (some are interactive) – 100%• Addition of a web-based laboratory

Pedagogy for AY 2012-13 is basically the same as that for AY 2011-12

Course Learning OpportunitiesRecommendation 2012: Identify at risk students early and provide support, e.g. review sessions (specific to Immunology)

• Optional review sessions are offered to the class• Course director contacts students who are struggling

Course Learning OpportunitiesRecommendation 2012: Provide an opportunity where biologic data are observed, measured and analyzed (specific to Virology)

• A web-based laboratory was added to the Virology course

Summary regarding Pedagogy• The percentage of traditional lectures in the course is

higher than is desired (goal 40-50% of course hours); the course director should consider ways to incorporate more active pedagogies into the course

• Opportunities exist for students to discuss unclear material with the faculty (review sessions)

• A web-based laboratory provides an opportunity for students to observe, measure and analyze biologic data

Assessment• Course is divided into Immunology and Virology

components• Immunology (IMM) component

– Two Quizzes (each 10% of course grade)• Each covers 12 sessions• Multiple choice 20-22 questions each

– Small Group Discussion (5% of course grade)• 3 session, each student presents in one session• Presentation by randomly-selected group of 4-5 students• Graded by faculty preceptor with written feedback to students

Assessment• Virology (VIR) component

– Two Quizzes (each 12.5% of course grade)• Each covers 8 sessions (2 additional sessions assessed only on final exam)• Multiple choice, 20 questions each

• Final Exam (50% of course grade)– Covers immunology and virology (equal distribution)– Multiple choice, 80 questions

• Overall Assessment Distribution– Quizzes (4) 45%– Final Exam 50%– Small Group Discussion 5%

Assessment – IMM Question Formats

Immunology Quizzes (Q1 and Q2)• All single best answer with 4 or 5 options• In general, questions are clearly worded• Only 1 of 42 items used a clinical scenario to stage

the question • Three of 42 items used “negative” formats (All of the

following EXCEPT) • Many of the questions used lists of T/F statements,

rather than one best answer lists– A small number of items (3/42) were stated T/F questions (“Which of

the following statements are correct?”)

Assessment – IMM Question Content

Immunology Quizzes (Q1 and Q2)• Most questions assessed recall of facts, and only a

few questions required application of knowledge:Quiz 1 (IMM): ~18/22 assessed

recallQuiz 2 (IMM): ~14/20 assessed

recall• No apparent integration between multiple

session objectives• Coverage is distributed evenly across the

sections

Assessment – IMM Correlation with objectives

Immunology Quizzes (Q1 and Q2)• Questions evaluated correlated well with specific

session objectives• Ex: Quiz 1 (11 sessions + 1 small group), 22 Questions – 21 of 22 questions clearly map to session objectives– 68 session objectives, ~29 covered on quiz– Several questions had more than 1 mapped objective,

although no cross-over between sessions/topics– Minimal overlap (only ~2-3 objectives mapped to more

than 1 question)

Assessment – VIR Question Formats

Virology Quizzes (Q3 and Q4)• All single best answer with 4 or 5 options• In general, questions are clearly worded• 14 of 40 items used a clinical scenario to stage the

question • No questions used “negative” formats (All of the

following EXCEPT) • Many of the questions used lists of T/F statements,

rather than one best answer lists– A small number of items (2/40) were stated T/F questions (“Which of

the following statements are correct?”)

Assessment – VIR Question Content

Virology Quizzes (Q3 and Q4)• Many questions assessed recall of facts; a few

questions required application of knowledge:Quiz 3 (VIR): ~12/20 assessed

recallQuiz 4 (VIR): ~17/20 assessed

recall• No apparent integration between multiple

session objectives• Coverage is distributed evenly across the

sections

Assessment – VIR Correlation with objectives

Virology Quizzes (Q3 and Q4)• Questions evaluated correlated well with specific

session objectives• Ex: Quiz 3 (4 x 2hr sessions), 20 Questions – 20 of 20 questions clearly map to session

objectives– 38 session objectives, ~24 covered on quiz– Few questions had more than 1 mapped objective– Minimal overlap (only ~1-2 objectives mapped to

more than 1 question)

Assessment – Exam Question FormatsFinal exam (includes both IMM and VIR)• All single best answer with 4 or 5 options• Some items were statements (i.e., a symptom,

followed by a list of potential causative agents) without an explicit question

• Only 4 of 80 items used a clinical vignette to introduce the item

• Four of 80 items used “negative” formats (All of the following EXCEPT)

• Many items used lists of T/F statements, rather than one best answer lists

Assessment – Exam Question Content

Final exam (includes both IMM and VIR)• Many of questions assessed recall of facts:

~75% assessed recall~25% assessed application of knowledge

• No apparent integration between multiple session objectives

• Coverage is distributed evenly across the disciplines (IMM and VIR) and sections

Assessment – Correlation with objectives• Balanced treatment of topics (i.e., equal division between IMM

and VIR, and items maps across the COs)• IMM portion of exam (40 items)

– ~ 45 session objectives covered (out of ~144)– Several questions had more than 1 mapped objective, although no cross-

over between sessions/topics– Minimal overlap (only ~2 objectives mapped to more than 1 question)

• VIR portion of exam (40 items)– ~ 46 session objectives covered (out of ~80)– Several questions had more than 1 mapped objective, although no cross-

over between sessions/topics– Minimal overlap (only ~2 objectives mapped to more than 1 question)

Assessment – Small Group Presentations• Students worked in groups (4-5, randomly selected) to

research an assigned immune-related disease condition and lead oral presentations to fellow students

• Format: ~30 min of presentation by group, followed by ~25 min for discussion and questions, driven by a) suggested discussion questions provided to the presenters, and b) questions from their peers

• Students are completely responsible for conducting the sessions

Assessment – Small Group Presentations• Presentations are scored by faculty, based on the following

criteria:– Knowledge of topic– Coverage of immune mechanisms, pathophysiology, epidemiology, diagnosis,

and treatment– Quality of presentation– Professionalism– Response to questions from peers– Teamwork

• Students in the audience provide written feedback to the groups (de-identified and presented in aggregate)

• This activity accounts for 10% of the IMM grade (5% of final course grade)

• All members of the group receive the same grade

Summary regarding Assessment• Quizzes and exam questions were clearly written • All quizzes and the final exam use MCQ

– a fair number of T/F or “except” type questions– Most items assessed recall, although application questions were more prevalent on the

exam– Very few questions used clinical vignette stems

• Students are provided answer keys and access to the quizzes, including detailed explanations of the answers

• Exam provided broad coverage of both IMM and VIR topics with equal distribution

• There was no overlap between IMM and VIR content in the questions; within disciplines, majority of the questions covered a single LO (no integration of topics)

• Small groups provide opportunity for students to develop presentation and discussion skills with feedback from faculty and peers

Measures of

Quality – AAMC

GQ

BASIC SCIENCESGeisel

mean 2008Geisel

mean 2009Geisel

mean 2010Geisel

mean 2011Geisel

mean 2012All schools

means 2012

Behavioral Science 3.1 3.3 3.2 3.3 3.3 3.1Biochemistry 2.7 2.8 2.7 2.5 2.6 2.6Biostatistics/Epidemiology 2.7 2.8 2.9 3.2 3.2 2.8Genetics 2.9 2.8 2.9 2.8 2.8 2.8Gross anatomy/Embryology 3.4 3.5 3.6 3.5 3.6 3.4Histology 2.8 3.0 2.8 2.9 3.1 2.9Immunology 2.8 3.0 2.9 3.0 3.1 3.1Microbiology 3.1 3.4 3.1 3.2 3.3 3.1Neuroscience 3.2 3.2 3.2 3.0 3.0 3.2On Doctoring 3.3 3.3 3.5 3.4 3.5 3.4Pathology 3.1 3.1 3.2 3.1 3.4 3.3Pathophysiology of Disease 3.6 3.7 3.5 3.5 3.5 3.5Pharmacology 3.1 3.5 3.4 3.1 3.1 3.0Physiology 3.5 3.5 3.6 3.6 3.5 3.4

Measures of Quality – AAMC GQ“Indicate how well you think that instruction in Immunology/Microbiology prepared you for clinical clerkships and electives.” [1=poor; 2=fair; 3=good; 4=excellent]

Measures of Quality – Step I2009* 2010* 2011* 2012* Means

09-12

TRADITIONAL CORE DISCIPLINES

Biochemistry 0.50 0.30 0.40 0.30 0.38

Biostatistics/Epidemiology 0.65 0.90 0.73 0.43 0.68

Genetics 0.40 0.30 0.48 0.28 0.37

Gross anatomy/Embryology 0.50 0.35 0.53 0.33 0.43

Histology/Cell Biology 0.53 0.30 0.40 0.37 0.40

Microbiology/Immunology 0.63 0.42 0.41 0.31 0.44

Pathology 0.55 0.35 0.42 0.26 0.40

Pharmacology 0.63 0.15 0.39 0.22 0.35

Physiology 0.67 0.32 0.47 0.38 0.46

*values reported for core disciplines are SD above the US/Can mean for Geisel mean scores

scale [1=poor; 2=fair; 3=good; 4=very good; 5=excellent]

Measures of Quality – Course Reviews

Immunol 2011

(35%)*

Immunol 2012

(96%)*

Virology 2011

(23%)*

Virology 2012

(96%)*

Overall satisfaction of course 1.71 3.19 2.95 3.65

Overall usefulness of lectures 1.70 2.99 2.38 3.58

Overall usefulness of small groups 2.41 2.83 N/A N/A

Overall usefulness of course materials 2.09 3.15 3.45 3.44

Congruence of assessment questions to material emphasized in course 1.87 2.63 2.90 3.02

*student participation rate on course evaluation

Measures of Quality – IMM Course Reviews

• Strengths:sample comments:• “Dr. Mullin's was the highlight of this course. His lectures were dynamic and

clarified the material. I also found the review session led by Dr. Mullins to be very helpful. This was the strongest aspect of the course.”

• “What is GREAT about this course is that you provided page numbers of textbooks and board review questions that correlated with each lecture. Every other course in the school could benefit from this approach.”

• “I really enjoyed how the lecture material mirrored the chapters from the Abbas text book. When I was confused during lecture, the corresponding readings from the book were essential.”

• “In addition to Levinson and the USMLE-type questions in the back of the text…provided most of the material we need to master in a digestible format.”

Measures of Quality – IMM Course Reviews

• Suggestions for Improvement:sample comments:• “Extra information is like overeating. It stresses the system without providing

added nutrient benefit. The right amount of food and information causes the best absorption.”

• “Wade's lectures were very difficult to follow and his notes are long-winded and repetitive. In comparison, Dr. Mullins's notes are clear and concise and his lectures were interesting, easy to follow, and seemed to highlight the most important material.

• “…I often got more confused while reading his lecture notes than if I'd read the textbook.”

• “The material stressed did not feel like the material that was tested.”• “The most frustrating part of the course was the lack of consistency between the

exams and the information presented within the lecture.”

Summary Immunology Measures of Quality• Dr. Mullins: Engaging lectures, organized presentation of materials, concise notes with a

summary, unambiguous expectations for testing• Many students found the hypersensitivity lecture to be very good• Students appreciated Dr. Mullin’s engaging lecture style accompanied by his well organized,

succinct notes• Most students discussed the lack of consistency between professors in their lectures and

materials provided• Students found the textbooks very useful, the questions provided by the Levinson book and

the content in Abbas– This is a double edged sword. Students praised Dr. Mullins notes for being concise,

highlighting key points, and not needing outside resources to understand the material• Students felt that the material tested did not match the material emphasized in class• Students would benefit from a summary at the end of each lecture that reiterated the key

points• Students appreciated that lecture objectives, slides, and textbooks were at times mapped

together, so that when looking at a slide you saw the objective that pertained to that slide or when reading lecture notes there was a list of slides that corresponded to that paragraph of notes

Measures of Quality – VIR Course Reviews

• Strengths:sample comments:• “Dr. Bzik's lecture notes - the bold and underlined words and the concise notes,

made for a very efficient study tool.” • “Summaries given out in the end of the course was helpful in organizing different

viruses in my head.” • “The subject matter is very interesting to me and I liked that the professors

attempted to match their questions to the types of questions asked on the National Boards exam.”

• “I also really appreciated the inclusion of ethical, global health and historical issues into the curriculum.”… “Instead of memorizing all the time you are actually piecing together whys and hows during the process.”

• The material was taught clearly, tested fairly, and was straightforward…I felt that the exams were representative of the material, and that I could get the information easily by watching lecture and using the excel spreadsheet of viruses.”

Measures of Quality – VIR Course Reviews

• Suggestions for Improvement:sample comments:• “The quiz questions were really frustrating. Some of them had multiple right answers, some of them

we really never covered in class. I really didn't think these quiz questions were a good representation of what I learned in class.”

• “It was difficult to grasp which details were important and which were not important to know.”• “…Dr. Witters practice exams, as a readiness blackboard quiz with detailed explanations of why

something is right AND why it is wrong, would help tremendously in focusing in on thinking and rationalizing through questions correctly.”

• “…if we need to read the Levinson material because it will not be taught to us in class but it is quizzable then let us know at the beginning of the class not a couple of days before a quiz.”

• “Not until after I read a few chapters of Levinson, did I realize there are about 10 things we need to know for every virus: viral structure, replication cycle, disease, pathogenesis, transmission, epidemiology, detection, treatment, vaccines. If each lecturer could make an effort to address each of these aspects of the virus in their lecture material, it would make studying for exam much easier.”

• Dr. Leib’s “..slides were very disorganized and haphazard and with no notes to accompany them, there was no way to piece the information together. Some viruses had incubation times, other didn't. Some had treatments, others didn't.”

• There was an incongruity between the expectations of quizzable material versus supplemental material and lectures– “…For each of the three lectures, there was a different emphasis placed on the

value of slides, notes, and the required textbook.”

• Lack of understanding as to what are the key points– Students would appreciate a summary of each lecture of the

most important ideas• Lecture notes that were not proof-read and were out of date, i.e.

didn’t match lecture slides• Scheduling lab assignments in a different week from quizzes• Proof reading quizzes more thoroughly to match class objectives• Lecture notes with slides

Summary Virology Measures of Quality

Summary of Recommendations• Course objectives are clearly written and correlate

well with objectives in the Step I brochure• Some session objectives need to be rewritten;

session objectives need to be provided for every activity in the course

• The issue of redundancy regarding common diseases should be explored by the course directors to ensure that the redundancy is planned

• The course is heavily lecture-based; the course directors need to introduce more engaged/active forms of pedagogy into the course

Summary of Recommendations• Assessment questions have been improved since the

last review of the course, however further revision is needed; students continue to cite frustration with the content on assessments (i.e. the material assessed does not match what is emphasized in the course)

• Expectations regarding the course need to be clearly communicated to students; some improvement is needed in the clarity of course materials