Embed Size (px)
Citation preview
Tools for Assessing Course Learning Outcomes:
Essay Prompts & Rubrics
Amal BouZeineddine Associate Director
Center for Teaching and learning (CTL)
August 6, 2012
Rubrics
Segment 2
We will :
Describe a rubric;
Analyze the different types of rubrics (descriptive, scoring, analytic, holistic);
Design rubrics based on learning outcomes.
The Learning Outcomes of this Session Segment
What is a Rubric?
A rubric is a printed set of scoring guidelines (criteria) for evaluating work (a performance or a product) and for giving feedback.
Generally, rubrics are put in the form of a chart with an x and y axis of performance criteria and an evaluative range or scale.
Rubrics help guide the evaluator in giving feedback that is descriptive as well as evaluative.
Why are rubrics used?
A good rubric tells the performer (student) and evaluator (teacher):
By what criteria the work will be judged;
What the difference is between good work and weaker work;
How we can make sure our judgments (or scores) are valid and reliable;
How both, performers and judges, can focus their preparation on excellence.
Rubrics may help eliminate comments such as “ I wasn't sure what you wanted.” or “I don't see why I got a "B' instead of a 'A' .”
Rubric Template
Rubric Template (Describe here the task or performance that this rubric is designed to evaluate.)
Beginning 1
Developing 2
Accomplished 3
Exemplary 4
Score
Stated
Objective or
Performance
Description of
identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description of
identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting the
highest level
of
performance.
Stated
Objective or
Performance
Description of
identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description of
identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting the
highest level
of
performance.
Stated
Objective or
Performance
Description of
identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description of
identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting the
highest level
of
performance.
Stated
Objective or
Performance
Description of
identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description of
identifiable
performance
characteristics
reflecting
development
and
movement
toward
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting the
highest level
of
performance.
Sample Levels
Example: Computer Programming Grading Rubric Trait Exceptional Acceptable Amateur Unsatisfactory
Specifications The program works
and meets all of the
specifications.
The program works and
produces the correct
results and displays them
correctly. It also meets
most of the other
specifications.
The program produces
correct results but does
not display them
correctly.
The program is
producing incorrect
results.
Readability The code is
exceptionally well
organized and very
easy to follow.
The code is fairly easy to
read.
The code is readable
only by someone who
knows what it is
supposed to be doing.
The code is poorly
organized and very
difficult to read.
Reusability The code could be
reused as a whole or
each routine could be
reused.
Most of the code could
be reused in other
programs.
Some parts of the code
could be reused in other
programs.
The code is not
organized for
reusability.
Documentation The documentation is
well written and
clearly explains what
the code is
accomplishing and
how.
The documentation
consists of embedded
comment and some
simple header
documentation that is
somewhat useful in
understanding the code.
The documentation is
simply comments
embedded in the code
with some simple header
comments separating
routines.
The documentation is
simply comments
embedded in the code
and does not help the
reader understand the
code.
Delivery The program was
delivered on time.
The program was
delivered within a week
of the due date.
The code was within 2
weeks of the due date.
The code was more
than 2 weeks overdue.
Efficiency The code is
extremely efficient
without sacrificing
readability and
understanding.
The code is fairly
efficient without
sacrificing readability
and understanding.
The code is brute force
and unnecessarily long.
The code is huge and
appears to be patched
together.
Example: Rubric for Class Participation
Exercise 1 Let’s design our first Rubric draft:
• Select a course learning outcome (CLO)
• Identify the traits/dimensions/performance criteria in the CLO which the rubric will be assessing
• Draw the Rubric X axis and Y axis
• Fill out the Criteria column
• Fill out the evaluation levels/range
• Choose one trait/dimension/performance criterion
• Fill out the description cells in the evaluation range for the selected criterion
Make sure the description is observable and measurable
• Share your description with a colleague at your table
What are the types of Rubrics? (1)
Rubrics used for grading are often called "scoring rubrics" and can be considered a form of summative assessment.
Rubrics can also be formative assessment. They can be very effective in improving self-assessment competence and skills such as peer-editing where consistency of judgment is critical.
Example 1
SCALE = 1 point - NOVICE 2 - APPRENTICE 3 - PROFICIENT 4 - DISTINGUISHED
performance
elements are
in this column
descriptors for lowest
performance levels will
be in this column
this column represents
what is the highest
level you expect to see
sample
CLARITY &
COHERENCE
Sentence structure, word
choice, lack of transitions
and/or sequencing of
ideas make reading and
understanding difficult.
Sentence structure
and/or word choice
sometimes interfere
with clarity. Needs to
improve sequencing of
ideas within
paragraphs and
transitions between
paragraphs to make the
writing easy to follow.
Sentences are
structured and words
are chosen to
communicate ideas
clearly. Sequencing
of ideas within
paragraphs and
transitions between
paragraphs make the
writer’s points easy to
follow.
In addition to meeting
the requirements for a
“3,” writing flows
smoothly from one
idea to another. The
writer has taken pains
to assist the reader in
following the logic of
the ideas expressed.
Example 2
Example 3
Clinical Simulation
Advanced 4 pts
Proficient 3 pts
Basic 2 pts
Below Basic 1 pts
Initial Steps 1. Prepares for patient contact ppe/handwashing 2. Introduces him/herself to the patient. 3 identifies his/herself to bedside nurse/doctor
Advanced Performs all initial steps without difficulty that are required in the skill
listed.
Proficient Performs more than three initial steps required in the skill llisted.
Basic Performs less than two initial steps required in the skill listed.
Below Basic Does not perform any of the initial steps required in the skill listed
Patient Assessment
1. Obtain information regarding patients history/ entrance complaint 2. Observe the patients vital signs displayed on the monitor. 3. check position and patencey of EET 4. Chest auscultation
Advanced Recognizes, verbalizes, and demonstrates all assessment steps in a logical manner consistently without difficulty.
Proficient Recognizes, verbalizes, and demonstrates the assessment steps in a logical manner 85% of the time during this check off.
Basic Able to recognize, verbalize, and demonstrate the assessment steps in a logical manner 75% of the time during this check off.
Below Basic Fails to recognize, verbalize, and/or demonstrate the assessment steps in a logical manner less than 50% of the time during this check off.
Rationale Advanced Verbalizes relevant information relating to the skill being checked off on. Answers instructor's questions regarding the skill and its
importance plus the impact to the client without difficulty.
Proficient Able to verbalize relevant information relating to the designatedt skill that is being checked off on. Able to answer more
than four questions regarding the skill and its importance re garding the skill and its importance plus the impact to the client.
Basic Limited ability to verbalize relevant information relating to the designated skill that is being checked off on. Able to answer less
than three questions regarding the skill and its importance plus the impact to the client.
Below Basic Fails to verbalize relevant information relating to the skill being checked off on. Unable to answer instructor's
questions regarding the skill and its importance and/or impact to the client.
Final Steps & Lab Area
Advanced Able to do all of the following: a.verbalize final steps. b. mention documentation of skill. c. mention client response to skill performed. d.clean up after the
Proficient One of the following occurs: Fails to: a. verbalize final steps. b. mention documentation of skill. c.mention client response to skill performed.
Basic Two or more of the following occurs: Fails to: a.verbalize final steps. b.mention documentation of skill. c. mention client response to skill performed.
Below Basic Three or more of the following occurs: Fails to: a.verbalize final steps. b.mention documentation of skill. c.mention client response to skill performed.
What are the types of Rubrics? (2)
Types Purpose/Distinction* Focal Use
Holistic
provide a single score based on
an overall impression of learner
achievement on a task.
To provide overall evaluation
guidelines that clarify how grades
relate to performance/achievement,
such as in course grades
Analytic provide specific feedback along
several dimensions
To break assignments or scores down
into separate components for grading
(description, analysis, grammar,
references, etc.)
Analytic Rubrics
Analytic rubrics identify and assess components of a finished product. It divides a student’s performance into essential dimensions (traits), and each dimension is judged separately. A separate score is given for each dimension or trait considered important for the assessed performance.
PLO-GE/AUB:
Apply drafting and editing strategies to compose work that reflects critical thinking and appropriate understanding of conventions of genre and usage.
Example criteria Exemplary Strong Satisfactory Needs Improvement
Ideas The essay develops fresh
insight that challenges the
readers’ thinking.
The essay demonstrates
mastery of the subject, and
uses appropriate, relevant and
compelling content that shows
the writers understanding and
shaping of the work.
The thesis or focus of the essay
is made clear to readers.
The essay uses appropriate
and relevant content in most
parts to illustrate good
command of the subject.
The essay demonstrates
diligent work and solid
preparation, but not
outstanding originality.
The thesis or focus of the
essay is made clear to the
readers.
Main idea of essay may be
obvious or unimaginative.
Thesis or focus is somewhat
vague.
Essay demonstrates some
knowledge of the subject and
uses appropriate and relevant
content inconsistently.
Essay uses simplistic or
irrelevant or inappropriate
content in most parts of the
work.
Lacks originality and simply
dwells on what is already
common knowledge to
readers.
Essay lacks focus or
purpose.
Organization and
Development
Evidence is relevant, accurate,
well developed, and well
integrated.
Essay demonstrates detailed
attention to successful
execution of conventions,
including organization, style
and format of academic
research.
Evidence is relevant,
accurate, well developed in
most parts of the essay.
Attention to execution of
conventions, including
organization, style, and
formatting of academic
research may be
inconsistent.
Evidence is either irrelevant or
loosely related to the thesis. It
may be partially inaccurate,
underdeveloped or not
integrated.
Essay demonstrates acceptable
attention to conventions of
academic research, such as
organization, style, or format.
Evidence is not relevant or
not developed.
Essay demonstrates little
attention to conventions of
academic research, such as
organization, style, or
format.
Style Essay uses academic English
skillfully to communicate
meaning to readers with
clarity, grace, and concision, in
prose that is virtually error-
free.
Essay uses academic English
skillfully to communicate
meaning to readers with
clarity, though may include
some errors.
Essay uses acceptable academic
English to communicate
meaning to readers with clarity,
though includes many errors.
Essay uses language that
many times impedes
meaning, because of errors
in usage.
Process Research project as a whole
(process and product) shows
evidence and demonstrates
excellent understanding of how
to use composing strategies,
such as drafting, revising,
The research project as a
whole (process and product)
shows some evidence and
demonstrates good
understanding of how to use
composing strategies, such
Research project as a whole
(process and product) shows
little evidence and demonstrates
basic understanding of how to
use composing strategies, such
as drafting, revising, and
Research project as a whole
(process and product) shows
little or no evidence of
development through
revising and editing.
Rubric from Nursing
Rubric from Medical Lab
Medical Assistant clinicl Lab 60 % Assessment of basic clinical skills
Unsatisfactory 60 pts Does not meet performance requirements -Needs much more attention to detail
Satisfactory 70 pts meets performance requirements
Good 85 pts Level of expected performance
Excellent 100 pts above level of expected performance
Infection Control 10 pts Standard precautions in patient care
Unsatisfactory Fails to wash hands <2minutes prior to 1st patient, clear all surfaces of contaminated materials, place contaminated material in waste can
Satisfactory wash hands <2minutes prior to 1st patient, clear all surfaces of contaminated materials, place contaminated material in waste can
Good Perform 2minute hand wash before 1st patient, patients, use of gloves in potential biohazards, clean all surfaces and dispose of contaminated materials, wash hands 1minute after patient exam
Excellent Check all common surfaces upon arrival, perform 2 minute hand wash before seeing 1st patient-, use of proper PPE in potential biohazards, disinfect all surfaces and proper disposal of all contaminated materials and wash hands 1minute after patient exam
Injections 15 pts Demonstrate each of the steps required in administering injections
Unsatisfactory Failed to accurately calculate prescribed dosage. unable to site the six rights
for drug administration. Did not follow safety procedures when handling equipment.
Satisfactory calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety
procedures when handling equipment.
Good calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety
procedures when handling equipment. Inserted needle at the correct angle for each of the injections demonstrated.
Excellent calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety
procedures when handling equipment. Inserted needle at the correct angle for each of the injections demonstrated. Documents injection administration.
CPR Skills 15 pts all procedure do with accordance with the guidelines of the American Heart Association
Unsatisfactory Failed to perform all the Skills required to do Adult, child and infant CPR. failed the Written CPR Test
Satisfactory Adequately performed all the required skills to do Adult, child and Infant CPR. Passed CPR written test with 85% or higher.
Good Performed all the required skills to do Adult, Child and Infant CPR with 85% Accuracy. Passed CPR written test with 85% or higher
Excellent Performed all the required skills to do Adult, Child and Infant CPR with 100% Accuracy. Passed CPR written test with 100%.
Electrocardiogram 20 pts Demonstrate each step required in obtaining a standard 12 lead ECG
Unsatisfactory unable to demonstrate the steps to obtain a standard 12 lead ECG. Could not distinguish any artifacts or how to correct them.
Satisfactory Adequately performed :Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted
Good Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted provider if any abnormalities were
Excellent Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted provider if any abnormalities were
Holistic Rubrics
The focus of a score reported using a holistic rubric is on the overall quality, proficiency, or understanding of the specific content and skills; it involves assessment on a unidimensional level (Mertler, 2001).
Use of holistic rubrics can result in a somewhat quicker scoring process than use of analytic rubrics (Nitko, 2001). This is basically due to the fact that the teacher is required to read through or examine the student’s product or performance only once, in order to get an "overall" sense of what the student was able to accomplish (Mertler, 2001).
Only limited feedback is provided to the student as a result of scoring performance tasks in this manner. A template for holistic scoring rubrics is presented in Table 1.
Holistic Rubric Template
Table 1:
Template for Holistic Rubrics
Score Description
5 Demonstrates complete understanding of the problem. All requirements of task are included in response.
4 Demonstrates considerable understanding of the problem. All requirements of task are included.
3 Demonstrates partial understanding of the problem. Most requirements of task are included.
2 Demonstrates little understanding of the problem. Many requirements of task are missing.
1 Demonstrates no understanding of the problem.
0 No response/task not attempted.
Example
Holistic Rubric for Essay Questions
Response Criteria Rating
Exemplary
Clarity of thought, Complete. Shows understanding of all
processes, reasonable hypothesis or thoughtful questions,
conclusions supportable by data, shows creativity, some
graphic representation of data or concepts.
11
Competent
Clarity of thought, shows understanding of major processes,
includes good hypothesis or questions, draws acceptable
inferences and conclusions, may have graphic representations.
10
Minor Flaws
Completes the assignment, but explanations may be slightly
ambiguous or unclear, may contain some incompleteness,
inappropriateness, or unclearness in representation, hypothesis,
understanding of processes, or conclusions.
8
Nearly
Satisfactory
Begins successfully, but omits significant parts or fails to
complete, may misuse scientific terms, representations may be
incorrect or omitted, incorrect or incomplete in analysis,
inferences and conclusions.
6
Fails to
complete
Assignment and explanation is unclear, or major flaws in
concept mastery, incorrect use of scientific terms,
inappropriate or omitted hypothesis.
4
Unable to
begin
effectively
Product does not reflect the assignment, does not distinguish
what information is needed, restates the question without
making an attempt at a solution.
2
No attempt Does not begin assignment. 0
Exercise 2
Let’s go back to the draft rubric you developed in Exercise 1.
• As a group at each table, select one draft rubric from exercise 1 which the whole group will work on
• Select another criterion from the CLO selected in Exercise 1
• Assign scores for the range/levels you have developed.
• As a group, fill out the description cells
• Put up the group’s rubric on one of the room walls
• Gallery Walk: Move around and check out the other developed draft rubrics.
• Paste a star sticker on the rubric which is clear and assesses the CLO accurately.
Exercise 3
• Let’s develop a holistic rubric.
• Share your Rubric with a colleague at your table.
Summary: Developing a Rubric
Further Rubric Samples
Sample Rubrics
Further Rubric Samples
Some Rubrics Websites
Rubric Samples http://rubrics.kon.org/ Create Your Own Rubrics Online:
RubiStar http://rubistar.4teachers.org/index.php
References and Guides to Rubric Development: http://www.tltgroup.org/resources/Rubrics.htm#CreatingARubric
For Guides to developing Rubrics in support of teaching, learning and self reflection:
This Rubric reference page - http://www.tltgroup.org/resources/Rubrics.htm
Flashlight Resources - TLT Group assessment, evaluation and survey tools – Flashlight Rubric Samples - http://www.tltgroup.org/resources/flashlight/rubrics.htm – Flashlight Online 2.0 - http://www.tltgroup.org/Flashlight/flashlightonline.htm - An online survey tool
that supports rubric construction and assessment
Scoring Rubrics - http://ericae.net/faqs/rubrics/scoring_rubrics.htm - Definitions & Construction
Rubric Basics - http://www.inov8.psu.edu/toolbox/RubricBasics.pdf - Definitions, types, purposes, learner involvement, learning enhancement and rubric use (Schreyer Institute for Innovation in Learning) .
Rubric Builder - https://www.e-education.psu.edu/facdev/id/assessment/rubrics/rubric_builder.html - An interactive web page rubric that can score and give item specific feedback
Rubistar - http://rubistar.4teachers.org/ - Helps you construct online rubrics
Roobrix - http://roobrix.com/ - Converts your rubric scores into percentages.
Waypoint - http://www.subjectivemetrics.com/index.cfm - online, interactive rubrics that let you create tailored narrative feedback for students based on your rubric and, on a larger scale (multiple classes, programs, institution-wide) collect and analyze longitudinal data on student performance
Some of the Consulted References
• http://web.njit.edu/~ronkowit/teaching/rubrics/index.htm
• http://www.tltgroup.org/resources/Rubrics.htm
• http://pareonline.net/getvn.asp?v=7&n=25
• http://seniordesign.engr.uidaho.edu/papers/Assessment_Framework_for_Capstone_Design_Courses.pdf
• http://www.bing.com/search?q=assessment+of+capstone+courses&src=IE-SearchBox&first=11&FORM=PERE
• http://cpr.iub.edu/uploads/Capped%20Off%20Assessment%20Institute%202010%20Kinzie,%20McCormick%20&%20Nelson%20Laird.pdf
• http://users.etown.edu/m/moorerc/capstone.html
• http://phobos.ramapo.edu/~vasishth/Assessing%20Learning%20Outcomes/Kerrigan+GE_Capstones+Assessment.pdf
• http://cgi.stanford.edu/~dept-ctl/tomprof/posting.php?ID=821
• http://www.washington.edu/oea/assessment/departmental/capstones.html
• http://provost.rpi.edu/node/31
• http://www.teachervision.fen.com/teaching-methods-and-management/rubrics/4524.html
End of Segment 2
