Summative Assessment:Summative Assessment:Rubrics and TestsRubrics and TestsEffective Teaching and LearningEffective Teaching and Learning

English Study ProgramEnglish Study Program

FKIP – UNSRIFKIP – UNSRI

AprilApril 201 20144

11

OutcomesOutcomes

Apply a systematic process for creating Apply a systematic process for creating a test blueprinta test blueprint

Identify attributes of effective test Identify attributes of effective test questionsquestions

Explain the advantages and Explain the advantages and disadvantages of different types of test disadvantages of different types of test questions questions

Assess the quality of tests and test itemsAssess the quality of tests and test items Create samples of effective questionsCreate samples of effective questions

22

What type of What type of assessment?assessment? Procedural knowledgeProcedural knowledge Declarative knowledgeDeclarative knowledge

33

Test Writing ProcessTest Writing Process

Planning the testPlanning the test Writing test itemsWriting test items Selecting test itemsSelecting test items Formatting the testFormatting the test Assessing the testAssessing the test Revising the testRevising the test Using the testUsing the test After the testAfter the test

44

Planning the TestPlanning the Test

Content BlueprintContent Blueprint– Learning outcomesLearning outcomes– WeightWeight

LengthLength– Types of itemsTypes of items– Number of itemsNumber of items

55

Test BlueprintTest Blueprint

Learning outcomes Weight

Apply a systematic process for creating a test blueprint

Identify attributes of effective test questions

Explain the advantages and disadvantages of different types of test questions

Assess the quality of tests and test items

Create samples of effective questions

66

Types of ItemsTypes of Items

RecognitionRecognition– True-falseTrue-false– Multiple-choiceMultiple-choice– Multiple-answerMultiple-answer– MatchingMatching– OrderingOrdering

RecallRecall– Short AnswerShort Answer– CompletionCompletion– EssayEssay

77

Average Response Average Response TimeTimeItem Type

True-false 30 seconds

Multiple-choice and Multiple-answer

60 – 90 seconds

Matching and Ordering 30 seconds per response

Short Answer 120 seconds

Completion 60 seconds

Essay 10 – 30 minutes88

Writing Test ItemsWriting Test Items

Simple and direct wordingSimple and direct wording Avoid jargonAvoid jargon Avoid trivia itemsAvoid trivia items Match items to learning outcomesMatch items to learning outcomes Each item has an agreed upon Each item has an agreed upon

correct answercorrect answer Write more questions than you Write more questions than you

will needwill need99

Multiple-choice ItemsMultiple-choice Items

StemStem– Direct questionDirect question– Incomplete statementIncomplete statement

ResponsesResponses– One correct answerOne correct answer– Multiple distractersMultiple distracters

1010

StemStem

Clearly worded Clearly worded One ideaOne idea Avoid the use of negativesAvoid the use of negatives Enough information to answer the Enough information to answer the

questionquestion Direct questions preferredDirect questions preferred Blanks at the end of the stemBlanks at the end of the stem Include words repeated in all responsesInclude words repeated in all responses

1111

ResponsesResponses

3-5 per item3-5 per item Avoid “all of the above” and “none of Avoid “all of the above” and “none of

the above”the above” Grammatically correct with stemGrammatically correct with stem Similar length and structureSimilar length and structure Avoid absolute wordsAvoid absolute words Listed in a logical orderListed in a logical order Mutually exclusive and not overlappingMutually exclusive and not overlapping

1212

DistractersDistracters

PlausiblePlausible Common misconceptionsCommon misconceptions Logical misinterpretationsLogical misinterpretations ClichésClichés Partial answersPartial answers Technical terms or jargonTechnical terms or jargon

1313

ExampleExample

What is the minimum number of What is the minimum number of responses for a multiple-choice responses for a multiple-choice item?item?

A) 2A) 2

B) 3B) 3

C) 4C) 4

D) 5D) 5

1414

Application ExampleApplication Example

What problem exists in the following What problem exists in the following multiple-choice stem:multiple-choice stem:

________ is the most common type of test item.________ is the most common type of test item.

A)A) Absolute words should be avoided in the Absolute words should be avoided in the stem.stem.

B)B) The stem contains more than one idea or The stem contains more than one idea or concept.concept.

C)C) Not enough information is presented to Not enough information is presented to answer the question.answer the question.

D)D) The fill-in the blank should come at the end.The fill-in the blank should come at the end.

1515

Analysis and Analysis and Evaluation Example Evaluation Example StemStemAn instructor was asked the following question: "Briefly list and

explain how you develop a test.” As an answer, this instructor wrote the following:

I begin by going through the chapter and writing questions based on the material in the text and my lectures. Then I decide how many questions I want and select the best questions from the list that I have developed. I format the test and add instructions. After a few days, I review the questions and make any revisions that need to be done and remove any jargon or wording lifted directly from the text. Then I use the test in class. Based on how the class does, I may make changes for the next time I teach the class.

1616

Analysis and Analysis and Evaluation Example Evaluation Example ResponsesResponsesBased on the process described in “Effective Based on the process described in “Effective Classroom Tests,” how would you judge this Classroom Tests,” how would you judge this answer?answer?

A) EXCELLENT (all steps in the right order with correct, A) EXCELLENT (all steps in the right order with correct, clear, and complete descriptions)clear, and complete descriptions)

B) GOOD (all stages correct in the right order, but the B) GOOD (all stages correct in the right order, but the descriptions are not as complete as they should be).descriptions are not as complete as they should be).

C) MEDIOCRE (one or two stages are missing, OR the C) MEDIOCRE (one or two stages are missing, OR the stages are in the wrong order, OR the explanations stages are in the wrong order, OR the explanations are not complete, OR the explanations are are not complete, OR the explanations are irrelevant)irrelevant)

D) UNACCEPTABLE (one or more stages are missing D) UNACCEPTABLE (one or more stages are missing AND the explanations are not complete AND/OR they AND the explanations are not complete AND/OR they are irrelevant) are irrelevant)

1717

Poor Question 1Poor Question 1

Good multiple choice items:Good multiple choice items:A) are easy to writeA) are easy to write

B) can only test memorized contentB) can only test memorized content

C) are better than essay itemsC) are better than essay items

D) there is no such thingD) there is no such thing

E) can test a wide range of contentE) can test a wide range of content

1818

Poor Question 2Poor Question 2

Which of the following characteristics Which of the following characteristics is not true of completion test items is not true of completion test items but is an important distinguishing but is an important distinguishing attribute of matching tests, multiple-attribute of matching tests, multiple-choice questions, and true-false choice questions, and true-false items?items?

A) They are objective test items.A) They are objective test items.B) They require knowledge recognition B) They require knowledge recognition

but not production.but not production.C) Much more difficult to construct.C) Much more difficult to construct.

1919

Poor Question 3Poor Question 3

Which of the following statements is Which of the following statements is FALSE?FALSE?A) Misfeasance is the improperly doing of an A) Misfeasance is the improperly doing of an

illegal act.illegal act.

B) Nonfeasance is improperly doing a legal act.B) Nonfeasance is improperly doing a legal act.

C) Nonfeasance is the failure to do an act that C) Nonfeasance is the failure to do an act that one must do legally.one must do legally.

D) Misfeasance is the failure to PROPERLY do D) Misfeasance is the failure to PROPERLY do an act that one has a duty to perform.an act that one has a duty to perform.

E) None of the above.E) None of the above.

2020

Poor Question 4Poor Question 4

__________ is/are the best method to __________ is/are the best method to determine if students have learned determine if students have learned something.something.

A) Comprehensive ExamA) Comprehensive Exam

B) Homework AssignmentsB) Homework Assignments

C) Pop QuizzesC) Pop Quizzes

D) Research PaperD) Research Paper

2121

Selecting Test ItemsSelecting Test Items

Outcome Weight x # Questions by Outcome Weight x # Questions by Type = #Questions of Each Type for Type = #Questions of Each Type for

OutcomeOutcome

2222

Formatting the TestFormatting the Test

Group items by typeGroup items by type Sort items by increasing difficultySort items by increasing difficulty Add instructionsAdd instructions Review layout and paginationReview layout and pagination Write answer keyWrite answer key

2323

Assessing the TestAssessing the Test

SelfSelf– 2-3 days after writing the test2-3 days after writing the test– ClarityClarity– Clues in items to other itemsClues in items to other items

Non-expertNon-expert– ClarityClarity– Contextual cluesContextual clues

PeerPeer– ContentContent– Weighting to outcomesWeighting to outcomes– Answer keyAnswer key

StudentsStudents– ClarityClarity– ContentContent

2424

Test Taking Test Taking ProceduresProcedures Use of notes or other materialsUse of notes or other materials Time limitsTime limits

2525

After the TestAfter the Test

Item AnalysisItem Analysis Areas for reviewAreas for review Test revisionsTest revisions

2626

ActivityActivity

Write two test questions on any Write two test questions on any topictopic

One question should be an One question should be an example of a good test itemexample of a good test item

One question should be an One question should be an example of a poorly written test example of a poorly written test itemitem

2727

ShareShare

Share your two questions with a Share your two questions with a partnerpartner

Can they determine which is good Can they determine which is good and which is bad?and which is bad?

Can they explain what makes one Can they explain what makes one poorly written?poorly written?

As a team, how can you fix the As a team, how can you fix the poorly written questions?poorly written questions?

2828

IntermissionIntermission

2929

OutcomesOutcomes

Determine what characteristics are Determine what characteristics are important in evaluating student workimportant in evaluating student work

Evaluate rubrics, analytic scales, and Evaluate rubrics, analytic scales, and other evaluation methodsother evaluation methods

Describe the contents of a good Describe the contents of a good rubricrubric

Identify rubrics already in use at Identify rubrics already in use at Baker CollegeBaker College

Begin work on a rubric for a classBegin work on a rubric for a class3030

What is a Rubric?What is a Rubric?

A rubric is a scoring tool or guide that A rubric is a scoring tool or guide that lists the specific criteria and the lists the specific criteria and the ranges for multiple levels of ranges for multiple levels of achievement for a piece of work or achievement for a piece of work or performance. A rubric consists of a set performance. A rubric consists of a set of well-defined factors and criteria of well-defined factors and criteria describing the dimensions of an describing the dimensions of an assignment to be assessed or assignment to be assessed or evaluated.evaluated.

3131

Parts of a RubricParts of a Rubric

Scale (columns)Scale (columns) Dimensions (rows)Dimensions (rows) Criteria descriptions (cells)Criteria descriptions (cells)

3232

Benefits of RubricsBenefits of Rubrics

Communicates the instructor’s Communicates the instructor’s expectations expectations

Streamlines the process for Streamlines the process for feedback to the studentfeedback to the student

Facilitates equitable grading Facilitates equitable grading Standardizes assessment across Standardizes assessment across

different instructorsdifferent instructors

3333

Uses for RubricsUses for Rubrics

PapersPapers PresentationsPresentations ProjectsProjects EssaysEssays HomeworkHomework

Case StudiesCase Studies Participation/Participation/

Class DiscussionClass Discussion PortfoliosPortfolios

3434

Types of RubricsTypes of Rubrics

AnalyticAnalytic– Page 11 Page 11

HolisticHolistic– Page 12 and 13Page 12 and 13

Check ListCheck List– Page 14Page 14

Scoring GuideScoring Guide– Page 15Page 15

3535

Creating a RubricCreating a Rubric

Identify components/outcomes of the Identify components/outcomes of the assignmentassignment

Determine a scaleDetermine a scale Add criteriaAdd criteria Assign pointsAssign points Set component weights (optional)Set component weights (optional) Assess the rubricAssess the rubric Test and reviseTest and revise

3636

ActivityActivity

Split into groups of 3-4 Split into groups of 3-4 Determine team rolesDetermine team roles Select an assignment that needs Select an assignment that needs

a rubrica rubric– Can be a specific assignment, such Can be a specific assignment, such

as a research paper for ENG 102as a research paper for ENG 102– Can be of a more general nature Can be of a more general nature

such as a class presentationsuch as a class presentation3737

Step 1: Identify Step 1: Identify ComponentsComponents List 5 major objectives/outcomes List 5 major objectives/outcomes

of the assignmentof the assignment Write these items as the row Write these items as the row

headers of the sheet providedheaders of the sheet provided

3838

Step 2: Determine a Step 2: Determine a ScaleScale Aim for 3-5 levelsAim for 3-5 levels Can use an odd or even number Can use an odd or even number

of itemsof items Use the headings on the next Use the headings on the next

slide for ideasslide for ideas Write these as column headings Write these as column headings

on the sheet providedon the sheet provided

3939

Potential Column Potential Column HeadingsHeadings Outstanding | Accomplished | Proficient | Developing | Outstanding | Accomplished | Proficient | Developing |

BeginningBeginning Accomplished | Average | Developing | BeginningAccomplished | Average | Developing | Beginning Excellent | Good | Needs Improvement | UnsatisfactoryExcellent | Good | Needs Improvement | Unsatisfactory Exceptional | Acceptable | Marginal | UnacceptableExceptional | Acceptable | Marginal | Unacceptable Expert | Practitioner | Apprentice | NoviceExpert | Practitioner | Apprentice | Novice Professional | Adequate | Needs Work | You’re FiredProfessional | Adequate | Needs Work | You’re Fired Exceeds Expectation | On Target | BeginningExceeds Expectation | On Target | Beginning Exemplary | Competent | DevelopingExemplary | Competent | Developing High | Medium | LowHigh | Medium | Low Outstanding | Proficient | Shows PotentialOutstanding | Proficient | Shows Potential

4040

Step 3: Add CriteriaStep 3: Add Criteria

Create descriptions for each level Create descriptions for each level of performance for each criteria in of performance for each criteria in the cells of the rubricthe cells of the rubric– Bullet pointsBullet points– ParagraphsParagraphs

Write these criteria in the cells of Write these criteria in the cells of the sheet provided the sheet provided

4141

Step 4: Assign PointsStep 4: Assign Points

Assign points for each level of Assign points for each level of performanceperformance

Can use either of the following:Can use either of the following:– Discrete values (5, 4, 3, 2, 1) Discrete values (5, 4, 3, 2, 1) – Ranges (10-9) for each levelRanges (10-9) for each level

Indicate the point value on the Indicate the point value on the sheet provided sheet provided – Normally placed with the scaleNormally placed with the scale

4242

Step 5: Set Component Step 5: Set Component WeightsWeights Allows for different levels of Allows for different levels of

importanceimportance– Spelling/grammar – more or less important Spelling/grammar – more or less important

than content?than content? Determine if weights are necessary for Determine if weights are necessary for

your rubricyour rubric Assign weights accordinglyAssign weights accordingly

– See example on page 17-18 of the See example on page 17-18 of the handouthandout

4343

Step 6: Assess the Step 6: Assess the RubricRubric Assess your rubric using a metarubricAssess your rubric using a metarubric

– See examples on page 11-15 of your See examples on page 11-15 of your handouthandout

Conduct a peer reviewConduct a peer review– Ask one or two other instructors to review Ask one or two other instructors to review

your rubricyour rubric Provide time for student reviewProvide time for student review

– Allow students to ask questions and make Allow students to ask questions and make commentscomments

4444

Group ProjectGroup Project

Trade rubrics with another groupTrade rubrics with another group Assess the rubric using a Assess the rubric using a

metarubric from page 11-15metarubric from page 11-15

4545

DiscussionDiscussion

What metarubric(s) did you use? What metarubric(s) did you use? Why?Why?

What did you see on the other What did you see on the other team’s rubric that you liked?team’s rubric that you liked?

Could you understand the Could you understand the assignment easily by reviewing assignment easily by reviewing the rubric?the rubric?

4646

Step 7: Implement and Step 7: Implement and refinerefine Refine your rubric based on Refine your rubric based on

feedback from other instructors feedback from other instructors and studentsand students

Make notes each time you use Make notes each time you use the rubric for continuous the rubric for continuous improvement purposesimprovement purposes

Share with othersShare with others

4747

Rubric Reliability & Rubric Reliability & ValidityValidity ReliabilityReliability

– ““the likelihood that a given measurement the likelihood that a given measurement procedure will yield the same description procedure will yield the same description of a given phenomena if the measurement of a given phenomena if the measurement is repeated.”is repeated.”

ValidityValidity– ““the extent to which a specific the extent to which a specific

measurement provides data that relate to measurement provides data that relate to commonly accepted meanings of a commonly accepted meanings of a particular concept.”particular concept.”

Babbie, 1986Babbie, 19864848

Reliability RequiresReliability Requires

Instructor should reach same Instructor should reach same conclusion each timeconclusion each time

Different instructors should reach Different instructors should reach similar conclusion (interrater similar conclusion (interrater reliability)reliability)

4949

Interrater ReliabilityInterrater Reliability

Independently score a set of student Independently score a set of student samplessamples

Review responses for consistent and Review responses for consistent and inconsistent responsesinconsistent responses

Discuss and reconcile inconsistenciesDiscuss and reconcile inconsistencies Repeat with second group of samplesRepeat with second group of samples

Maki, 2004, p. 127Maki, 2004, p. 127

5050

Validity RequiresValidity Requires

ReliabilityReliability Comprehensiveness Comprehensiveness

– Cover all outcomes Cover all outcomes EconomyEconomy

– Space is usually limited, so be Space is usually limited, so be selective about what goes into the selective about what goes into the rubricrubric

Balanced scoring and weightingBalanced scoring and weighting5151

End of End of SlidesSlidesAny Any Question?Question?

5252