ONLINE ABVP ITEM WRITING GUIDE

Dr. Michael Dutton

Created January 2017

© ABVP

Please use the forward and back buttons to navigate

ITEM WRITING

•  Material needed for this module:

•  Item Writing Guide located at http://www.abvp.com/lib_forms-guides/item_writing_guide.pdf

•  Please download these materials before continuing

ITEM WRITING

•  The purpose of this online program is to acquaint you with ABVP requirements for writing an acceptable exam item

•  You must successfully complete this course to be able to write items for inclusion into the ABVP database.

TESTING YOUR UNDERSTANDING

•  When you signed up to take this course, the ABVP Office provided an Internet link for your test of this material

•  After completing this tutorial, please go online to the Internet link to take a 10 item quiz

•  The quiz is “open book” and you can refer back to this presentation or the downloaded guide

•  A score of 70 or better is required for successful completion of the tutorial

FREQUENTLY ASKED QUESTIONS

•  Why is there a fee for this?

•  The fee pays for a portion of maintaining the ABVP website

•  Why do I need to take this course?

•  You need to successfully complete this course to be able to submit exam items for credit

WHY DO WE WRITE ITEMS?

•  ABVP relies on high quality items to examine new candidates and recertifying Diplomates

•  If you are a recertifying Diplomate, writing exam items is one way to earn Maintenance of Certification (formerly called recertification) points

ITEM WRITING REQUIRES

•  Complex technical skills

•  Sophisticated thinking

•  Understanding of candidates’ likely responses

•  Awareness of candidates’ perceptions

•  Ability to write with a high degree of precision

ITEM WRITING

•  This presentation will cover

•  Cognitive Levels of Items

•  Item Types

•  Writing Suggestions

WRITING AN EXAM ITEM

•  Unique terminology is introduced throughout this document relating to the ‘anatomy’ of an exam item

ITEM COMPONENTS INCLUDE

•  Stem

•  statement, question, chart or graph

•  Options •  all distractors and key

•  Key

•  correct answer

•  Distractors

•  plausible, but clearly incorrect responses

ITEM DEFINITION

•  The entire question or statement including the stem and options

•  Items may be presented in the form of statements rather than questions

STEM DEFINITION

•  The statement, question, chart or graph portion of an item

•  The stem of the item should clearly present ONE central problem or idea

OPTION DEFINITION

•  All offered answers to the item

•  The options are randomized

•  The distractors are plausible, but clearly incorrect

•  The key is the one correct or best answer

•  All ABVP items have three options

•  one (1) key and two (2) distractors

MULTIPLE CHOICE EXAMS

•  Key of a positively worded stem is the best or most appropriate of available responses

•  Distractors •  are incorrect

•  must be comparatively worse than the key

•  need not be absolutely incorrect

•  plausible to persons with less ability

WHY USE MULTIPLE CHOICE?

•  Evidence for validity of each item is required

•  Allows for precise scoring

•  removes subjectivity

•  Validates examinees knowledge

•  each item measures a specific ability

•  aggregate covers a wide range of abilities

WHY ONLY 3 OPTIONS FOR AN ITEM?

•  Rodriguez (Educational Measurement: Issues and Practices. 2005) reported on a meta-analysis of 80 years of studies and found that 3 options is as statistically relevant as 4

•  Evidence has found that 3 options is more appropriate in testing candidates for advanced level certification

RECOGNIZED VETERINARY SPECIALTY

•  The American Board of Veterinary Specialties (ABVS) oversees all AVMA approved veterinary specialties

•  ABVP is a Recognized Veterinary Specialty Organization (RVSO)

•  Each Practice Group within ABVP is termed a Recognized Veterinary Specialty or RVS

•  For example: Avian Practice and Equine Practice are both RVS

EXAMINATION CONSTRUCTION

•  Based on Job Content Analysis (JCA)

•  Details the conditions, activities and modalities that a clinical specialist should know

EXAM BLUEPRINT

•  JCA provides the justification for individual items used on an examination

•  All items are linked directly to this blueprint

•  JCA is different for each Recognized Veterinary Specialty

TESTING SERVICES

•  ABVP has contracted a company called Comira to maintain our item banks

•  Items are categorized in the Comira database by subdomains

•  When submitting an item, you must indicate the most appropriate subdomain for the item

SUBDOMAINS

•  Subdomains are also termed General Conditions in ABVP documents

•  The list is http://www.abvp.com/lib_forms-guides/general_conditions.pdf

•  Examples of Subdomains include

•  Behavioral

•  Gastrointestinal

•  Cardiovascular

COGNITIVE LEVEL OVERVIEW

•  Exam items are classified into 3 cognitive levels.

•  Recall

•  Application

•  Evaluation

COGNITIVE LEVELS

•  Recall •  knowledge of specifics

•  rarely situational dependent •  Application

•  interpreting, calculating, recognizing elements and relationships

•  usually situational dependent

•  Evaluation/Analysis

•  synthesizing information, solving problems, evaluating results

•  almost always situational dependent

ITEM WRITING

•  When submitting an item, in addition to the subdomain, the author must indicate which cognitive level the item best fits

RECALL CHARACTERISTICS

•  These are typically short items

•  Stem is one sentence

•  No variables

•  Options often start with nouns

RECALL ITEMS

•  Ideally ABVP strives for less than 5% recall on the exam

•  These only require memorization and are inappropriate for specialty exams

EXAMPLE - RECALL ITEM

•  The most common endocrinopathy of ferrets is

A. hyperadrenocorticism.

B. insulinoma.

C. hyperthyroidism.

APPLICATION CHARACTERISTICS

•  Any item involving manipulations of formulas are application level

•  Items using graphics or data tables are at least application level

•  The key may vary with the situation

APPLICATION CHARACTERISTICS

•  Moderate length with a stem that may be more than one sentence

•  May include variables

•  age, amount, location

•  Options often start with verbs

EXAMPLE - APPLICATION ITEM

In dogs with diabetes mellitus, the signs of serious psychological depression may resemble

A. the onset of nephropathy.

B. chronic hypoglycemic episodes.

C. chronic hyperglycemia.

EVALUATION/ANALYSIS CHARACTERISTICS

•  Require information synthesis, problem solving, and selection of the best response

•  Key will vary with the situation

•  Examinees must act on problem identified from clues in the stem

EVALUATION/ANALYSIS ITEMS

•  Have the longest stem

•  May require several sentences to convey all relevant information

•  Includes variables

•  Asks examinee to select the “best” action

•  Options typically start with verbs

EXAMPLE – EVALUATION/ANALYSIS ITEM

A 10-year-old female spayed cat was diagnosed with diabetes mellitus two months ago. The cat currently receives 2 units glargine subcutaneously twice daily. The owner does not test for blood glucose or glycosuria at home. Lately the cat has become more listless. This cat may

A. be in remission from the diabetes and a spot blood glucose should be determined.

B. need to be changed to PZI insulin and a fructosamine level should be checked.

C. have acromegaly and a growth hormone level should be determined.

PLEASE NOTE

•  Each key must be referenced from the RVS’s reading list •  Each distractor must be plausible •  Exam takers do NOT need to know normal values.

Normal values must be placed in parenthesis after the given value

•  Except for scientific values (e.g., mmol/dL), do NOT use abbreviations •  Use female spayed as opposed to FS •  We have a significant international applicant pool that

would not know the typical American abbreviations

TYPES OF MULTIPLE CHOICE (MC)

•  One Best Response

•  Complex Multiple Choice

•  Clinical Vignette

ONE BEST RESPONSE MC

•  Most common

•  The examinee needs to select the ‘best answer’ from among these being offered

•  This type can be written as a ‘Direct Question’ or an ‘Incomplete Statement’

EXAMPLE - DIRECT QUESTION

In the treatment of diabetic ketoacidosis, it is important to monitor for which of the following complications?

A. hypokalemia and hypophosphatemia

B. hypercalcemia and hypophosphatemia

C. hypokalemia and hypercalcemia

EXAMPLE - INCOMPLETE STATEMENT

The urethral anchoring structures include the

A. ventral ligament and lateral bilateral ischiocavernosus/ ischiourethralis muscles.

B. ventral ligament, dorsal ligament and bilateral retractor penile muscles.

C. dorsal ligament and the bilateral bulbourethralis muscle.

DIRECT QUESTION ó INCOMPLETE STATEMENT

•  Can reword an item from one subtype to another

•  Example Direct Question

•  What coagulation factors are in the common pathway?

•  Can become incomplete statement

•  In the coagulation pathway, the common pathway includes factors

COMPLEX MULTIPLE CHOICE (CMC)

•  Begin with a stem, followed by three or four true-false statements or elements, and end with three lettered options containing various combinations

•  This question type is highly discouraged by the ABVP

EXAMPLE - CMC

•  Stem •  Which of the following drugs require a DEA license?

•  Elements . . . •  1. Diazepam

•  2. Ketamine

•  3. Acepromazine

•  Options . . . •  A. 1 and 3

•  B. 1 and 2

•  C. 2 and 3

CLINICAL VIGNETTE

•  A direct question or incomplete statement may be preceded by a template of data used to create a clinical scenario called a vignette

•  Clinical vignettes enhance the validity of the exam and focus on important information, allowing examinees to use accumulated data in their reasoning processes

CLINICAL VIGNETTES

•  Data templates may be of varying length and may include some, or all, of the following: age, gender, history, clinical signs, examination findings, diagnostic study results, initial treatment, subsequent findings

EXAMPLE - CLINICAL VIGNETTE

An 18-year-old cat presents with weight loss and inappetence. Creatinine values are found to be 4.9 mg/dL (0.8-2.3 mg/dl), packed cell volume (PCV) is 19% (30-45%) and total protein is 8.2 g/dL (5.9-8.5 g/dl). The anemia is normocytic and normochromic. Erythropoietin treatment fails to increase the PCV after 12 weeks, but the white blood cell count rises from 8,372/microliter to 32,425/microliter (5,500-19,500/microliter). The neutrophil count is 31,078/microliter. Which of the following is the most likely explanation for these findings?

A. The anemia is associated with inflammatory or infectious disease rather than chronic renal failure.

B. The erythropoietin is causing erythroid and myeloid proliferation, but this cat has a maturation defect in the erythroid precursors.

C. Anti-erythropoietin antibodies have developed.

SITUATIONAL SETS

•  Examinees are given a short scenario, or item stem, along with a collection of facts or data. They are then presented with 3 to 5 multiple-choice items regarding recognition and management of the problem. Each item should be able to stand alone in reference to the situation.

•  Each item in the set, while relating to the scenario, remains independent from the other(s) in terms of determining the correct answer. Do not provide a series of interrelated items that supply or suggest the response to another item within the set. Also, avoid a series of items which build upon one another, requiring a correct response to one in order to determine a correct response to another.

SITUATIONAL SET EXAMPLE

A 14-year-old neutered male cat presents with weight loss, tachycardia, and vomiting after eating. Complete blood count and serum biochemistry reveal the following: normals in ()

WBC 18.6 103/uL (5.5-19.5 103/uL), HCT 30% (29-45%), BUN 38 mg/dl (15-34 mg/dl), Cr 2.5 mg/dl (0.8-2.3 mg/dl), ALT 325 IU/L (28-76 IU/L), AST 164 IU/L (5-55 IU/L), and T4 6.8 ug/dl (0.7-5.2ug/dl).

Which of the following is the most appropriate initial treatment?

A. oral methimazole

B. surgical thyroidectomy

C. radioactive iodine

Which of the following is the most appropriate diagnostic step?

A. urine analysis

B. serum bile acids

C. free T4

ITEM WRITING SUGGESTIONS

•  These suggestions are based on feedback ABVP receives from exam item writers and exam candiates.

ITEM WRITING SUGGESTIONS

•  Follow standard rules of grammar

•  Avoid irrelevant sources of difficulty

•  Use an efficient format

•  Eliminate irrelevant clues

•  Use texts and publications from the reading list

•  Provide normal values in international units

THE STEM

•  Use clear and simple language

•  Present a single, clearly formulated problem

•  Avoid the use of negative wording

•  Put as much of the wording into the stem as possible

•  Avoid excessive irrelevant material

•  Include all qualifications needed to select the right answer

THE OPTIONS

•  Select and formulate the distractors with care •  Make sure the key is correct and clearly the best •  Make the distractors plausible to the uninformed or

misinformed

•  Do NOT use the options ‘All of the above’ or ‘None of the above’

•  Make the options independent •  Make all the options grammatically consistent with the

stem and parallel in form

THINGS TO AVOID

NEGATIVELY WORDED STEMS

•  ‘Not’ and ‘Except’ type items are not allowed

•  Encourage measurement error due to candidate confusion

•  Use of these words result in items with unfocused and multiple options

“ALL” OR “NONE OF THE ABOVE”

•  The use of these words allow candidates to respond correctly based on partial information

•  “None of the above” also allows examinees to argue that the best answer is not present

•  Perception is that these options are “fillers” written by someone who couldn’t or wouldn’t write a quality item

AVOID IRRELEVANT INFORMATION

•  Stems should be plain and simple

•  Short sentences rather than long and complex

•  Several small words rather than a few polysyllabic words to describe the situation and ask the question

•  Simple rather than complex computations

AVOID CLUES

•  Placing similar phrases in the stem and key, even including identical words

•  Writing the key in far more technical, detailed language

•  Using modifiers associated with

•  true statements (may, sometimes, usually) for keys

•  false statements (never, all, always) for distractors

AVOID CLUES BY WRITING OPTIONS THAT

•  Have the same meaning

•  both must be incorrect

•  Are mutually exclusive

•  xyz is correct, xyz is incorrect

•  Examinees will have the choice of two parallel options increasing the chance probability of a correct answer to 50%

•  Clues increase measurement error by increasing guessing

AVOID CLUES IN THE ITEM THAT POINT TO THE CORRECT ANSWER

•  Correct responses require little job-related mastery

•  Classroom “smarts” do not necessarily transfer to the job setting

AVOID LANGUAGE CUES

Of the following, the fruit that contains the highest amount of potassium is an

A. mango.

B. banana.

C. apricot.

CLUE PRESENT: ‘an’ would be followed by a word beginning with a vowel.

LANGUAGE CUE CORRECTION 1

Of the following, the fruit that contains the highest amount of potassium is

A. a mango.

B. a banana.

C. an apricot.

This is a little better wording but you are not maximizing the words in the STEM.

LANGUAGE CUE CORRECTION 2

Of the following, the fruit that contains the highest amount of potassium is the

A. mango.

B. banana.

C. apricot.

Notice the words are maximized in the STEM and minimized in the OPTIONS.

LANGUAGE CUE CORRECTION 3

Of the following, the fruit that contains the highest amount potassium is an

A. apple.

B. araza.

C. apricot.

Notice that the options are changed so all begin with a vowel.

AVOID: NON-PARALLEL DISTRACTORS

What is the official state bird of Pennsylvania?

A.  Slippery Rock

B.  Susquehanna

C.  Ruffed Grouse

CLUE PRESENT: not all options are birds.

AVOID: IMPLAUSIBLE DISTRACTORS

What is the primary ingredient in spaghetti sauce?

A. tomato

B. avocado

C. chopped liver

CLUE PRESENT: Chopped liver is an outlier from expected reasonable options.

IMPLAUSIBLE DISTRACTORS CORRECTION

What is the primary ingredient in spaghetti sauce?

a.  basil

b.  cheese

c.  tomato

AVOID: OVERLAPPING OPTIONS

According to the Northeastern Journal of

Respiratory Medicine, what percentage

of the U.S. population over the age of 50

show signs of small airway disease?

A. 20-40%

B. 40-60%

C. 60-80%

OVERLAPPING OPTIONS CORRECTION

According to the Northeastern Journal of

Respiratory Medicine, what percentage

of the U.S. population over the age of 50

show signs of small airway disease?

A. 21-40

B. 41-60

C. 61-80

AVOID: UNDIRECTED STEM

Mycobacterium tuberculosis is

A. the causative bacterial agent of pneumonia.

B. spread by poor hand washing techniques.

C. spread primarily by inhalation of contaminated droplets.

UNDIRECTED STEM CORRECTION

Mycobacterium tuberculosis is spread primarily by

A. inhalation of contaminated droplets.

B. poor hand washing techniques.

C. inappropriate cooking temperatures.

QUESTION: HEPARIN, WHICH IS A COMMONLY USED ANTICOAGULANT, ACTS BY A. DEPRESSING FIBRIN FORMATION. B. MANUFACTURING PROTHROMBIN. C . INCREASING THROMBIN PRODUCTION.

Answer to Pop Quiz: The unnecessary detail given in the stem could cue another item.

Pop Quiz: Name the error

QUESTION: ASSESSMENT A. TAILORS INSTRUCTION TO THE INDIVIDUAL. B. STANDARDIZES INSTRUCTION FOR ALL INDIVIDUALS. C . ALWAYS IMPROVES THE QUALITY OF THE TEACHING PLAN.

Answers to Pop Quiz: • The stem is undirected. (What is the question?). • The word “always” provides a clue that C is a false statement and thus must be a distractor. •  A refers to “tailoring” instruction and B refers to “standardizing” instruction. This draws the candidate to these two options, increasing the chance of guessing correctly to 50%.

Pop Quiz: Name the error

UNDIRECTED STEM CORRECTION 1 WHICH OF THE FOLLOWING I S A PR IMARY PURPOSE OF ASSESSMENT?

A. To standardize instruction.

B. To tailor instruction to the individual.

C. Improve the quality of the teaching plan.

Undirected Stem Correction 2 The primary purpose of assessment is to A.  standardize instruction. B.  tailor instruction to the individual. C.  improve the quality of the teaching plan.

A couple ways to rephrase and improve the question

QUESTION: PATIENT SAFETY DURING DIALYSIS DEPENDS ON TOTAL CONDUCTIVITY OF THE DIALYSATE. “TOTAL CONDUCTIVITY” IS BEST DEFINED AS A . CONDUCTIVITY OF ALL IONS IN A SOLUTION. B . ELECTRICAL CHARGE OF CHLORIDE IONS IN DIALYSATE. C . CONDUCTIVITY OF CATIONS IN SOLUTION.

Answers to the Pop Quiz:

•  Most of first sentence of stem is superfluous to the question.

•  Repeating the term conductivity in the key is a clue, especially when C refers to part of the ions

•  The question asks for a definition.

Pop Quiz: Name the error

QUESTION: THE NORMAL CANINE BODY TEMPERATURE, WHICH REPRESENTS THE BALANCE BETWEEN HEAT PRODUCED AND HEAT LOST, WHEN TAKEN RECTALLY IS APPROXIMATELY A. 100-101.5 DEG F. B . 101-103 DEG F. C . 104-104.5 DEG F.

Answers to the Pop Quiz: • The stem provides unnecessary detail • Overlap exists in A and B •  The question should be directly asked, e.g. “A normal canine rectal body temperature taken in your office is”

Pop Quiz: Name the error

QUESTION: TO COMMUNICATE WELL THROUGH DOCUMENTATION, ENTRIES SHOULD BE A . CLEAR, CONCISE , AND TIMELY. B . CLEAR, COMPLETE, AND LENGTHY. C . CLEAR, COMPLETE, AND ABBREVIATED.

Answers to the Pop Quiz:

• The word “clear” should be added to the stem and removed from each option for more efficiency.

• Could be rewritten as a CMC item

Pop Quiz: Name the error

QUESTION SUBMISSION FORMAT

•  Now you need to get your items to ABVP. Currently we are using a paper based format that is outlined at this link: http://www.abvp.com/lib_forms-guides/sample_question.pdf

•  The ID# is your ABVP number available from the Main Office.

TIME TO TAKE THE TEST

•  The ABVP Main Office will have supplied you with an Internet Address that will link you to a test of the material in the item writing guide and this presentation

•  To receive credit you must score 70% or better on the 10 item test. You can retake the test as needed to successfully complete it

EXAM COMMITTEE APPRECIATION

•  Thank you for helping ABVP develop and maintain quality examinations

•  We are always looking for feedback Please let us know how we can improve this course for future applicants

•  If you have questions, please contact Marisa Hackemann, Executive Director, mhackemann@navc.com