Harnessing the Power of Categories: Using Categories to Improve Exam Questions

FileNewTemplate

Harnessing the Power of Categories: Using Categories to Improve Exam QuestionsDonna K. Czarnecki, BSLead Computer-Based Testing, Data and Core CoordinatorUniversity of Nebraska Medical Center, College of Medicine

1

For Assessment BuildingTo Students

To Faculty & Program

Benefits of CategoriesWork with Focused Set of ItemsFacilitates Review/Improvement Process

There are many benefits to using categories. During the last webinar I discussed using the Categories area to physically build assessments. Today I will focus on their usefulness for reviewing and improving exam questions.2

Reasons to Categorize Manage your ItemsAllows you and your faculty to work with smaller sets of specific questions.No need to duplicate questions in another question folder.

(and your faculty)

3

Questions by CategorySubset of questions from Questions by Folder areaQuestions are tagged with categories to group them together.Think of these groups of questions as being sorted out of the Questions by Folder database.

Questions by category are basically a subset or group of questions from the Folder area.Each institutions categories are based on their specific needs. Here at UNMC we have the illustrated categories plus many more. Think of it as a way to filter or sort out certain questions you want to see or work with.

4

Requires Review

5

Requires Review

At UNMC we are guided by two things when determining if a question meets our formatting requirements. One is the NBME Writing Guide and the other is our own particular institutions preferences.

6

Requires Review

First let me explain that NBME stands for the National Board of Medical Examiners. As you may remember from my introduction, I work with the first and second year medical students at our university.The NBME offers a variety of educational materials and instructional opportunities to help enhance the quality of test items used by educators and assessors in the healthcare professions around the world.The NBME's Item Writing Manual, Constructing Written Test Questions for the Basic and Clinical Sciences, is a resource for faculty members at schools of medicine, schools for healthcare professionals, those invested in the assessment of competence of healthcare providers, and others interested in learning how to write better quality test items.A link to the downloadable manual is listed. As just mentioned, this information is not just limited to use in the healthcare professions so I encourage you to take a look.

7

Requires Review

In addition to the writing guide manual, NBME offers several other item writing services with their tutorials, seminars and workshops.

8

Requires Review

Another major guiding force here at UNMC is the use of Blooms Taxonomy. This topic alone needs its own webinar, which guess what? Has already been done. You can find these webinars on the ExamSoft website and plenty of information on the web. Because of this, I will not get very deep into this topic. Suffice it to say that we try to use higher level taxonomic questions with our students to prepare them for their board exams and licensure.9

Requires Review

I have listed a few examples here of types of questions that do not meet our formatting requirements.The first one Not phrased as a question is exactly that. These are actually question stems that end with a semicolon and continue on into the question foils to complete a sentence.All of the Above and None of the Above questions do not contain a simple answer or response and often cue the student to the correct answer.True/False questions require personal interpretation and may be confusing to studentsThe last example is something specific for our students and institution.

10

Requires Review - Does not meet formatting requirements:

Here is a list of categories we use for inappropriate formatting and a few other categories, some of which are more specific to our institution.There are two other important formatting issues I would like to mention:Remove Negative Phrasing these are questions that contain the word NOT in the question stem or include All of the following EXCEPT:Two Answer Choice Allowed these are questions that were determined after an exam to have had two possible correct answer.11

Requires Review

Here is another major category, and sub categories, regarding the actual statistical performance of the exam questions. It can be broken up into two areas: the actual performance stats and whether or not it was removed from scoring.12

Requires Review

The first thing we look at with our stats is the difficulty. This is the percentage of students that answered the question correctly. If the difficulty is below 0.50, then we look at the discriminator or point biserial. If it is below 0.20 the question is tagged with poor statistical properties, and usually removed from the exam.Negative discriminators also indicate a possible problem and are noted for review.13

Requires Review

Questions that are removed from scoring after an exam are tagged with this information for the appropriate academic year.14

Requires Review - Has poor statistical properties/ was removed from scoring

Here is a screenshot of how we have our categories set up for poor statisticsThe last category is designed to be used by a course director if they need the question looked over by a content expert.15

Requires Review - Review process for faculty

Go through all exam questions and tag if review is required

Unfortunately someone has to go through and tag all of the questions that need to be reviewed. Dont worry, this process is not as bad as it sounds. This task was performed by myself and my colleague once we had imported all of our questions.I would highly recommend that you do this as questions are added to your database. We imported over 4,000 questions in our first year, so this was quite a project to go through all of the questions after that.

16


2. Set up review category for faculty to access

In my last webinar I talked about setting up a construction zone under the Categories area to build assessments. Using a similar method, we created an area for the faculty to review their questions.We set up the main category 2016 M1 Item Review and then set up sub-categories based on the different courses. This allows the appropriate course director the ability to only see and review questions associated with their course.17


3. Sort/search for review categories in appropriate question/course folder

Use the Advanced Search area located in the upper right area of your screen.18



Search for questions with the category Requires Review. Scroll down and select done.19



Select the course folder where all of the questions are located.20



Select All Questions in the status field and then select Search.21


4. Tag questions for inclusion into review category

You will now have a list of all of the questions from that course folder that Requires Review. Select all and add the review category tags to all of the question using the bulk add feature.Make sure to select Update Categories when you are all done.22


5. Provide faculty with location of questions to review (including a deadline)

All of this leads us back to the listing of questions, by course, in the 2016 Review category.This is the location for your faculty to access their questions. Once they click on the review category for their course, they will see all of the question they need to review. Based on this screenshot, if the Neuroscience Course Director click on his course review category, he would see the 129 questions located there.Most importantly, make sure to give the Course Director and/or Faculty a deadline to complete their review.

23

Requires Review Archiving Bad QuestionsFor those questions that are not revised or negotiated to remain the same, we add the word ARCHIVE at the front of the questions title. This indicates to us that the question should not be used on an assessment.By doing this, the question will still remain categorized in the Questions by Folder area, so the faculty can still see it, but it will not be used on the exam in its current state.Eventually the ARCHIVE questions will be removed from the folders, and then eventually deleted from the system.

For those questions that are not revised or negotiated to remain the same, we add the word ARCHIVE at the front of the questions title. This indicates to us that the question should not be used on an assessment.By doing this, the question will still remain categorized in the Questions by Folder area, so the faculty can still see it, but it will not be used on the exam in its current state.Eventually the ARCHIVE questions will be removed from the folders, and then eventually deleted from the system.24

Requires Review CategoriesQuestions by Folder

Here is a diagram to help with the process for those that are more visual.The entire circle shows all of the questions in the ExamSoft database, located in the Questions by Folders area.Inside of the entire circle is a small subset of questions that need to be reviewed.Inside of the review circle are the course subsets.Remember, these are not separate sets of questions. They are just sorted out by categories to make them easier to review for the appropriate person.25

Questions by CategoryQuestions assigned to a category are not a separate set of questions.A question can only be in one folder but many different categories.Do NOT delete questions from category bins, or you will delete them from the database permanently!

Instead, remove the category assignment from a question to remove it from that category bin.

And finally a very important reminder.26

Questions???My contact info:Donna K. Czarnecki, B.S.Lead Computer-Based Testing, Data and Core CoordinatorOffice of Medical Education, College of MedicineUniversity of Nebraska Medical Center402.559.4043 I [email protected]