Test Format

Test Format1The typical teacher will spend one-quarter to one-third of his or her professional time involved in assessment-related activitieswithout question, teachers need to know and understand the principles of sound assessment.(Stiggins, 2004)Introduction22Goals for our session:

To understand what makes for good test items & how to write them.FOCUS: Multiple-Choice Question (MCQ) &Constructing Essay Questions (Based on Blooms cognitive taxonomy)Planning a Test33Use a test matrix or blueprint.

Identify major ideas and skillsrather than specific details.

Use Blooms cognitive taxonomyor something appropriate for your context.Planning a Test(Nilson, 2010)

44The basis of all exam questions should be learning outcomes.Using Blooms Taxonomy can help you clarify what you want students to learn.By setting learning outcomes at different levels of Blooms Taxonomy, you will target exactly what you want students to know, be able to do, or have experienced as a result of learning activities in class.Learning Outcomes: Using Blooms Taxonomy5Blooms Taxonomy

Examples of questions for different levels: http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-bloom.php

6Effective MC questions can be written to cover

Knowledge of terms, facts, principlesComprehensionApplicationAnalysis

Examples of questions for different levels: http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-bloom.php6TYPES OF TEST ITEMS

SELECT TYPESUPPLY TYPEMultiple Choice

MatchingShort Answer Structured ResponseFree ResponseCompletion/ClozeEssayTest Items vs Cognitive Levels of LearningComprehensionKnowledgeApplicationSynthesisAnalysisEvaluationMCQTrue/False (TF)MatchingCompletionShort AnswerMCQShort AnswerEssayStructured Response (MCQ, TF, Likert Scales)Free Response (Essay, Interview Qs, Fill in the blanks)PerformanceMCQShort AnswerEssayFree Response7TEST TYPES - General

Structure Free ResponseAdvantages

Great DepthQuick Scoring

Advantages

Provide full and independent responsesgives a better indication of the competence and intelligence of the testDisadvantages

Limited DepthDifficult to assess higher levels of skillsGuessing/Memorization vs. KnowledgeDisadvantages

the difficulty, bias, and time effort required in grading themthe quality of a free response answer is often determined more by ability with the language in which it is given than competence or intelligenceStructured Response8TYPES OF TEST TYPES

Multiple Choice QuestionsChoose the correct word to complete the sentence.

Cook is ________today for being one of the Britains most famous explorers.recommendedrecognisedremindedRememberedCan be used to test: grammar, vocabulary, reading, listening etc.

Gap-fillingComplete the sentence.

Check the exchange _________to see much your money is worth.Can be used to test: grammar, vocabulary, Very effective at testing listening for specific words.MatchingMatch the word on the left to the word with the opposite meaning.

fat oldyoung talldangerous thinshort safeCan be used to test: Most often used to test vocabulary9TYPES OF TEST TYPES

ClozeComplete the text by adding a word to each gap.

This is the kind________test where a word ________omitted from a passage every so often. The candidate must _______the gaps, usually the first two lines are without gaps.Can be used to test: Very effective for testing grammar, vocabulary & intensive reading

True-FalseDecide if the statement is true or false.

England won the world cup in 1966. T/FCan be used to test: Most often used to test vocabularyOpen QuestionsAnswer the question (after reading or listening.)

Why did John steal the money?

Can be used to test: Very useful for testing any of the four skills. But less useful for testing grammar or vocabularyError CorrectionFind the mistakes in the sentence and correct them.

Manchester United was the more better team on the night.Can be used to test: Useful for testing grammar & vocabulary as well as listening and reading

Etc..... (other Test types)TranslationEssayDictationsSequencing/ordering of words / phrasesSentence construction10

Multiple-Choice Tests1111To assess breadth of learningTo test a variety of levels of learningWhen you have a large number of individuals taking the testWhen you have time to construct the test itemsWhen time is limited for scoringWhen it is not important to determine how well individuals can formulate their own answerWhen you want to prepare individuals for future assessments that use a similar format

When to Use (Clegg & Cashin, 1986)1212Students select the correct answer from alternative responses. Each item has:item stemcorrect or keyed optionseveral distractor options

Format:complete questionincomplete questionMultiple-Choice Tests(Clegg & Cashin, 1986)13131. The capital city of Canada is a. Vancouver b. Montreal c. Toronto*d. Ottawa1. Stem: presents the problem2. Key: correct or best answer3. Distracters or Alternatives: appear to be reasonable answers to the examinee who does not know the content4. Level of difficulty and topic or chapter referenceTerminology: Multiple Choice1231414Clearly defines a problem/question (specific is terrific!)Provide sufficient informationAvoid extra information / clues (a, are, is, etc.)Be grammatically correctAvoid clues that help give away answersAvoid negatives and absolutesConstructing Items -MCQ Stem should:15Be plausible/logical/reasonableBe grammatically consistent with the stemBe approximately the same length or other logical order (alphabetical, numerical, time series..)Make alternatives mutually exclusive (i.e. do not overlap)Provide only one correct answerAvoid all or none of the aboveIncorrect answers (distractors) should not be too obvious but should be justifiably incorrectWhen composing incorrect alternatives, include any misconceptions that students are likely to holdDo not lift a statement exactly from the textbook

Constructing Items MCQ Responses should:16Multiple-choice items can provide:versatile in measuring all levels of cognitive ability,highly reliable test scores,scoring efficiency and accuracy,objective measurement of achievement or ability,a wide sampling of content or objectives,a reduced guessing factor compared with true-false items, different response alternatives which can provide diagnostic feedback.

Advantages(Ory & Ryan, 1993)1717Multiple-choice items:are difficult and time-consuming to construct,lead an instructor to favour simple recall of facts,place a high degree of dependence on the students reading ability and instructors writing ability, andare particularly subject to clueing. (Students can often deduce the correct response by elimination.)Limitations(Ory & Ryan, 1993)1818Objectives at Different LevelsLevel: Knowledge

Objective: State the average effective radiation dose from chest Computed Tomography (CT).Level: Comprehension and Application

Objective: Compare the radiation exposures from different radiologic examinations.

What is the average effective radiation dose from chest Computed Tomography (CT)?1 mSv (millisievert )8 mSv16 mSv24 mSvWhich of the following imaging examinations is associated with the highest effective radiation dose?Abdominal and pelvic multidetector CTCoronary artery multidetector CTConventional pulmonary angiographyDigital pulmonary angiography191. This floor is made of ___________A ciment C cemmentB cemnet D cement

2. Skinner developed programmed instruction in _____. A. 1953 B. 1954C. 1955 D. 1956

Analyse the following sample:20About Essay Questions

21Provide a freedom on response that facilitates assessing higher cognitive behaviors (e.g., analysis and evaluation)

Allow respondent to focus on what they have learned and do not limit them to specific questions

Essay Questions22Types of Essay Questions Extended Response QuestionGreat deal of latitude on how to respond to a question. Example: Discuss essay and multiple-choice type tests.Restricted Response QuestionMore specific, easier to score, improved reliability and validityExample: Compare and contrast the relative advantages of disadvantages of essay and multiple choice tests with respect to: reliability, validity & objectivityEssay Questions 23Make the essay items reflect course objectivesShould measure an objective at Blooms - Synthesis/Creating level Include specific directions such as compare, contrast, define, discuss, formulate, etc...Prepare students for the test and provide essay test taking tipsPlan ahead, and manage your time wisely Have each student answer all the questionsState the number of points each question is worthFor controversial topics, indicate clearly that the student should logically present a position rather than take a specific standLimit the scope of the questionExample: Write a two-page statement defending the importance of conserving our natural resources. (your answer will be evaluated in terms of organisation, its comprehensiveness and the relevance of the arguments presented)Constructing Items: Essay Tests

24Advantages: Measure higher learning levels (synthesis, evaluation) and are easier to construct than objective test itemsStudents are less likely to answer an essay question by guessingRequire superior study methodsOffer students an opportunity to demonstrate their abilities to: Organize knowledgeExpress opinionsFoster creativityEssay Questions2525Better study skills- Students preparing for this type of test are inclined to outline material to draw cause and effect relationships, summarize material and make inferences

DisadvantagesMay limit the sampling of material coveredTends to reduce validity of the testSubjective unreliable nature of scoringhalo effect good or bad students previous level of performanceWritten expressionHandwriting legibilityGrammatical and spelling errorsTime Consuming Essay Items/Questions2626Halo effect - the tendency for an impression created in one area to influence opinion in another area.Write an outline of the key points (use outline to design a rubric analytic or holistic) Determine how many points are to be assigned to the question as a whole and to the various parts within it. If possible, score the test without knowledge of the students name Score all of the answers to one question before proceeding to the next question Consistent standard

Scoring Essay Items 2727The objective test play an prominent role in (Face sheet for students name that can be folded back eliminate both positive and negative halo effects)Research/Term PapersResearch ReviewsReportsCase Studies PortfoliosProjectsPerformances Peer evaluationSimulations Alternative Methods of Assessment 28The main mistakes instructors make when writing essay questions are:

Writing vague, non-specific questions. Asking for too many parts in an answer. Asking questions over material that students did not read or cover in class. Asking questions that are difficult to grade (e.g. asking for students opinions).

Mistakes in Writing Essay Questions29To write an effective essay question:

Decide exactly what you want to know about the students learning. Do you want to know that they can tell you how a complex economic system forms? If so, ask that.Cue students to the material from texts and class that forms the basis of your questions.Put points in parentheses after each component of the question so students will know exactly which parts to answer and how much those parts will count.Use key words so students will understand exactly how you want them to answer.

Tips for Writing an Effective Essay Question30You are going to design a quantitative research project to look at students motivation in classrooms. You want to study ways their past experiences, current expectations, and self- efficacy influence their study behaviors this semester.

Using what you learned from your textbook, list your independent and dependent variables (2 marks);explain for each how you would operationalize that variable (5 marks); Finally, discuss at least two ways you would analyze the data that you collect (10 marks).

What are the reasons this is an effective essay question? Justify your answer.

Tutorial Task

31Brown, H. Douglas, (2010). Language Assessment Principles and Classroom Practices. (2nd ed). New York, Pearson Longman.Clegg, V. L., & Cashin, W. E. (1986). Improving multiple-choice tests. Idea Paper No. 16, Center for Faculty Evaluation and Development, Kansas State University. http://www.idea.ksu.edu/papers/Idea_Paper_16.pdf.Davis, B. G. (2009). Tools for teaching. (2nd ed.). San Francisco, CA: Jossey-Bass.Gronlund, E. Norman, (2006). Assessment of Student Performance. (8th ed). Boston, Pearson Allyn and Bacon.Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334. Nilson, L. B. (2010). Teaching at its best: A research-based resource for college instructors. (3rd ed.) San Francisco, CA: Jossey-Bass.Ory, J.C. & Ryan, K. E. (1993). Tips for improving testing and grading. Vol. 4. Newbury Park: Sage Publications.

Resources3232Svinicki, M. & McKeachie, W. J. (2011). McKeachie's teaching tips: Strategies, research, and theory for college and university teachers. Belmont, CA: Wadsworth, Cengage Learning.University of Oregon, Teaching Effectiveness Program. Writing Multiple Choice Questions that Demand Critical Thinking. Web site: http://tep.uoregon.edu/resources/assessment/multiplechoicequestions/mc4critthink.htmlUniversity of Minnesota, Office of Measurement Services. Writing Multiple Choice Items. Web site: http://oms.umn.edu/fce/how_to_write/multiplechoice.phpUniversity of Texas at Austin, Instructional Assessment Resources. Writing Multiple Choice Items. Web site: http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-write.php

Further Reading3333A good test needs an instrument of quality.

An instrument of quality needs items that are of quality.

Items that are of quality need item builders who are of quality.

DO YOU HAVE ALL THOSE QUALITIES?REFLECTION(Haladyna, Downing & Rodriguez, 2002)3434Develop as many effective choices as you can, but research suggests three is adequate.Make sure that only one of these choices is the right answer.Vary the location of the right answer according to the number of choicesPlace choices in logical or numerical order.Keep choices independent; choices should not be overlapping.ISLMCQ: Writing Item(Haladyna, Downing & Rodriguez, 2002)3535Phrase choices positively; avoid negatives such as NOT.Avoid giving clues to the right answer, such as:specific determiners including always, never, completely, and absolutelychoices identical to or resembling words in the stemconspicuous correct choicepairs or triplets of options that clue the test-taker to the correct choiceridiculous optionsISLMCQ: Writing Item(Haladyna, Downing & Rodriguez, 2002)3636Keep choices homogeneous in content and grammatical structure.Keep the length of the choices about equal.None-of-the-above should be used carefully.Avoid All-of-the-above.Make all distractors logical.Use typical errors of students to write your distractors.Use humor if it is compatible with the teacher and the learning environment.ISLMCQ: Writing Item(Haladyna, Downing & Rodriguez, 2002)3737