View
219
Download
1
Category
Preview:
Citation preview
Assessing Critical ThinkingSummer Critical Thinking
Institute
QEP Team, Faculty Champions, and Academic Roundtables
2008
June 20, 2008 Critical Thinking Institue 2
Assessing Critical Thinking 2008
Critical Thinking
“Beyond the Obvious”
June 20, 2008 Critical Thinking Institue 3
Assessing Critical Thinking 2008
Assessment Basics
Purpose of assessment Creating valid and reliable measures Alignment of goals/measures Use of multiple methods
June 20, 2008 Critical Thinking Institue 4
Assessing Critical Thinking 2008
Assessment Basics
Why do we assess? To see how well we are doing To confirm what we already know To share our progress with others To see where we can improve and change In some cases to demonstrate what does not
work
June 20, 2008 Critical Thinking Institue 5
Assessing Critical Thinking 2008
Assessment Basics
Source: http://www.c-pal.net/course/module2/pdf/Week1_Lesson5.pdf
Why do we assess?
June 20, 2008 Critical Thinking Institue 6
Assessing Critical Thinking 2008
Assessment Basics
Does one size fit all? Assessments need to be valid Assessments need to be reliable
June 20, 2008 Critical Thinking Institue 7
Assessing Critical Thinking 2008
Validity
Does the assessment measure what it is suppose to measure?
“Validation is the process of accumulating evidence that supports the appropriateness of inferences that are made of student responses…” (AERA, APA, & NCME, 1999)
June 20, 2008 Critical Thinking Institue 8
Assessing Critical Thinking 2008
Types of Validity Evidence
Content Related - the extent to which a student’s responses to a given assessment reflect that student’s knowledge of the content area
Construct Related - the extent to which the responses being evaluated are appropriate indicators of the underlying construct
Criterion Related - the extent to which the results of the assessment correlate with a current or future event
Consequential – the consequences or use of the assessment results
June 20, 2008 Critical Thinking Institue 9
Assessing Critical Thinking 2008
Questions to Examine Validity
Content Validity Evidence1. Does the evaluation criteria address
any extraneous content?2. Does the evaluation criteria address
all of the aspects of the intended content?
3. Is there any content addressed in the task that should be evaluated, but is not?
June 20, 2008 Critical Thinking Institue 10
Assessing Critical Thinking 2008
Questions to Examine Validity
Construct Validity Evidence1. Are all the important elements of
the material evaluated through the scoring criteria?
2. Are any of the evaluation criteria NOT relevant to the material?
June 20, 2008 Critical Thinking Institue 11
Assessing Critical Thinking 2008
Questions to Examine Validity
Criterion Validity Evidence1. What are the important components of
the future performance that may be evaluated through the use of this assessment?
2. How does the scoring criteria measure the important components of the future performance?
3. Are there any elements of the future performance that are not reflected in the scoring criteria?
June 20, 2008 Critical Thinking Institue 12
Assessing Critical Thinking 2008
Reliability
Consistency of the assessment scores
Types of reliability… Interrater Reliability – scores vary from
instructor to instructor. Intrarater Reliability – scores vary from a
single instructor from paper to paper A test can be reliable and not valid,
but never valid and not reliable
June 20, 2008 Critical Thinking Institue 13
Assessing Critical Thinking 2008
Reliability Concerns
Reliability1. Are the score categories well defined?2. Are the differences between the score
categories clear?3. Would two independent raters arrive
at the same score for a given student response based on the scoring rubric?
June 20, 2008 Critical Thinking Institue 14
Assessing Critical Thinking 2008
Improving Scoring Consistency
Provide grading rubrics or scoring criteria to students prior to assessment
Grade papers anonymously Use anchor papers to define levels
of proficiency for reference Use multiple scorers Calculate reliability statistics during
training and grading
June 20, 2008 Critical Thinking Institue 15
Assessing Critical Thinking 2008
Assessment Basics
Assessment Purpose Everything needs to align (objectives
through assessment) SPC QEP example
June 20, 2008 Critical Thinking Institue 16
Assessing Critical Thinking 2008
Assessment Basics
Definition
Measurable Learning Outcomes
Operational Elements (KSAs)
Appropriate
Assessment
Measures
June 20, 2008 Critical Thinking Institue 17
Assessing Critical Thinking 2008
SPC Definition
“Critical thinking is the active and systematic process of communication, problem-solving, evaluation, analysis, synthesis, and reflection, both individually and in community, to foster understanding, support sound decision-making, and guide action.”
Definition
Measurable Learning Outcomes
Operational Elements
(KSAs)
Appropriate Assessment Measures
June 20, 2008 Critical Thinking Institue 18
Assessing Critical Thinking 2008
Student Learning Outcomes
In order to link specific and measurable student learning outcomes, SPC’s definition of critical thinking was operationalized.
This provided a more concrete and less abstract linkage or bridge between the student learning outcomes and the definition of critical thinking.
June 20, 2008 Critical Thinking Institue 19
Assessing Critical Thinking 2008
Student Learning Outcomes
Definition
Measurable Learning Outcomes
Operational Elements
(KSAs)
Appropriate Assessment Measures
June 20, 2008 Critical Thinking Institue 20
Assessing Critical Thinking 2008
Student Learning Outcomes
Definition
Measurable Learning Outcomes
Operational Elements
(KSAs)
Appropriate Assessment Measures
June 20, 2008 Critical Thinking Institue 21
Assessing Critical Thinking 2008
Outcomes to Assessments
Student Learning Outcomes were then linked to appropriate assessment instruments
SPC’s QEP contained multiple measures for use in assessing student learning in the area of critical thinking
June 20, 2008 Critical Thinking Institue 22
Assessing Critical Thinking 2008
Outcomes to Assessments
Definition
Measurable Learning Outcomes
Operational Elements
(KSAs)
Appropriate Assessment Measures
June 20, 2008 Critical Thinking Institue 23
Assessing Critical Thinking 2008
Recent Alumni Survey
Question 31: Thinking logically and critically to solve problems Gathering and assessing relevant information Inquiring about and interpreting information Organizing and evaluating information Analyzing and explaining information to others Using Information to solve problems
June 20, 2008 Critical Thinking Institue 24
Assessing Critical Thinking 2008
Employer Survey
Question 3: Use mathematical and computational skills Comfortable with mathematical calculations Uses computational skills appropriately Accurately interprets mathematical data
Question 5: Think logically and critically to solve problems
Gathers and assesses relevant information Inquires and interprets information Organizes and evaluates information Analyzes and explains information to others Uses Information to solve problems
June 20, 2008 Critical Thinking Institue 25
Assessing Critical Thinking 2008
CCSSE
Question 5: During the current school year, how much has your coursework at this college emphasized the following mental activities?
b. Analyzing the basic elements of an idea, experience, or theory d. Making judgments about the value or soundness of
information, arguments, or methods
Question 12: How much has YOUR EXPERIENCE AT THIS COLLEGE contributed to your knowledge, skills, and personal development in the following areas?
e. Thinking critically and analytically
June 20, 2008 Critical Thinking Institue 26
Assessing Critical Thinking 2008
Assessment Basics
Multiple Measures SPC will determine improvement in
students’ critical thinking skills using the multiple measures.
These include standardized direct instruments, authentic assessments, and indirect methods.
June 20, 2008 Critical Thinking Institue 27
Assessing Critical Thinking 2008
Student Assessment Points
Student Assessment Points
Entering Student Survey
SPC
Enrolled Student Survey
Exiting Student Survey
Employer Survey
Recent Alumni Survey
Employment or University
Direct Measures
Indirect Measures
MAPP ARC iSkills
CCSSE
SSI
Capstone (4-year only)
June 20, 2008 Critical Thinking Institue 28
Assessing Critical Thinking 2008
Standardized Direct Instruments
Direct assessments include: CAT - Critical Thinking Assessment Test is designed
to assess and promote the improvement of critical thinking and real-world problem solving skills.
Measure of Academic Proficiency and Progress (MAPP), developed by Educational Testing Services (ETS), is a measure of college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences
The iSkills™ assessment (former ICT Literacy Assessment), developed by ETS, is a comprehensive test of Information and Communication Technology proficiency that uses scenario-based critical thinking tasks to measure both cognitive and technical skills.
June 20, 2008 Critical Thinking Institue 29
Assessing Critical Thinking 2008
Indirect Methods
Student, alumni, employer, faculty, and staff reports, such as end-of-course, institutional, and national surveys and questionnaires, can provide indirect measures that help deepen the interpretation of student learning (Maki, 2004).
June 20, 2008 Critical Thinking Institue 30
Assessing Critical Thinking 2008
Indirect Methods
Indirect methods include: Community College Survey of Student Engagement
(CCSSE), established at UT at Austin, a tool for assessing quality in community college education. CCSSE contains specific survey items intended to assess various Core Operational Elements (KSAs) associated with a student’s critical thinking.
Entering Student Survey, Enrolled Student Survey, Graduating Student Survey, and Recent Alumni Survey are the primary surveys that have been developed to collect student feedback on their experiences.
Employer Surveys are sent out to employers of recent SPC graduates in order to gather information on graduates’ knowledge and behavior.
June 20, 2008 Critical Thinking Institue 31
Assessing Critical Thinking 2008
Authentic Assessments
Authentic assessments serve dual purposes of encouraging students to think critically and of providing assessment data for measuring improved student learning.
June 20, 2008 Critical Thinking Institue 32
Assessing Critical Thinking 2008
Authentic Assessments
Authentic assessments include… Criterion-referenced rubrics. Complex, higher-order
objectives can be measured only by having students create a unique product, whether written or oral [in-class essays, speeches, term papers, videos, computer programs, blueprints, or artwork] (Carey, 2000).
Student Reflection. Written reflection is espoused to have several important benefits: it can deepen the quality of critical thinking, increase active involvement in learning, and increase personal ownership of the new learning by the student (Moon, 1999).
Student Portfolios. Collections of students’ work over a course or a program and can be an effective method of demonstrating student progress in the area of critical thinking (Carey, 2000).
June 20, 2008 Critical Thinking Institue 33
Assessing Critical Thinking 2008
Rubrics
What is a rubric? Scoring guidelines, consisting of
specific pre-established performance criteria, used in evaluating student work on performance assessments
June 20, 2008 Critical Thinking Institue 34
Assessing Critical Thinking 2008
Rubrics
SPC currently uses rubrics in such programs as…
College of Education College of Nursing Paralegal
June 20, 2008 Critical Thinking Institue 35
Assessing Critical Thinking 2008
Assessment Rubric for Critical Thinking (ARC)
June 20, 2008 Critical Thinking Institue 36
Assessing Critical Thinking 2008
Assessment Rubric for CT
ARC was designed to… Enhance the QEP Align with the College’s definition of
critical thinking Be flexible for use in multi-disciplines
June 20, 2008 Critical Thinking Institue 37
Assessing Critical Thinking 2008
Assessment Rubric for CT
ARC is a ‘global’ rubric template developed to provide a snapshot view of how student learning is being affected by the critical thinking initiative.
ARC will be designed to assess a variety of student projects from a critical thinking perspective. For example, students in a composition class may be asked to write a paper on a specific topic.
ARC rubric template will evaluate the student’s use of critical thinking skills in the development of the paper as opposed to specifically evaluating the quality of student’s writing skills.
June 20, 2008 Critical Thinking Institue 38
Assessing Critical Thinking 2008
Assessment Rubric for CT
ARC rubric template will be designed to be flexible enough to address a number of student project modalities including written and oral communications.
The development of a rubric is an iterative process and will be improved and strengthened as it is used more widely; however, the first iteration of the rubric has been developed by the QEP faculty champions.
June 20, 2008 Critical Thinking Institue 39
Assessing Critical Thinking 2008
Rubric Development Process
1. Re-examine the learning objectives to be addressed by the task
2. Identify specific observable attributes your students should demonstrate
3. Describe characteristics at each attribute 4. Write narrative descriptions for each level of
continuum 5. Collect samples of student work 6. Score student work and identify samples that
exemplify various levels 7. Revise the rubric as needed
Repeat as Needed
June 20, 2008 Critical Thinking Institue 40
Assessing Critical Thinking 2008
Assessment Rubric for Critical Thinking
June 20, 2008 Critical Thinking Institue 41
Assessing Critical Thinking 2008
Assessment Rubric for Critical Thinking
June 20, 2008 Critical Thinking Institue 42
Assessing Critical Thinking 2008
Assessment Rubric for Critical Thinking
June 20, 2008 Critical Thinking Institue 43
Assessing Critical Thinking 2008
ARC Assignment Profile
Designed to provide consistency and accuracy in the evaluation of the ARC at the institutional level as well as provide guidelines for the use at the course level
ARC is essentially a ‘tool’ to evaluate critical thinking, but for a tool to be effective it must be in the correct situation or ‘job.’ It would be inefficient to use a machete to conduct heart surgery.
Purpose of the ARC Assignment Profile is to outline the most appropriate course assignment
June 20, 2008 Critical Thinking Institue 44
Assessing Critical Thinking 2008
ARC Assignment Profile
Participating faculty should have one assignment during the course that can be evaluated using the ARC scoring rubric.
Course assignment could be a graded homework assignment or a major assessment for the course.
Course assignment should include all of the elements of the rubric and should be aligned with the task outlined for each element.
Assignments that only evaluate some of the elements or are not aligned with the specific ARC tasks will be considered incomplete and not used in the institutional analysis.
June 20, 2008 Critical Thinking Institue 45
Assessing Critical Thinking 2008
ARC Assignment Profile
Faculty may add additional discipline specific rubric elements (such as grammar and punctuation in a composition class), but must maintain the ARC elements as listed.
June 20, 2008 Critical Thinking Institue 46
Assessing Critical Thinking 2008
ARC Assignment Profile
Students should be provided a copy of the assignment rubric (ARC and any additional discipline specific elements). The specific elements and tasks include:
1. Communication: Define the problem in your own words.2. Analysis: Compare & contrast the available solutions within the
scenario.3. Problem Solving: Select one of the available solutions and defend it
as your final solution.4. Evaluation: Identify the weaknesses of your final solution.5. Synthesis: Suggest ways to improve/strengthen your final solution
(may use information not contained within the scenario).6. Reflection: Reflect on your own thought process after completing the
assignment. a. “What did you learn from this process?” b. “What would you do differently next time to improve?”
June 20, 2008 Critical Thinking Institue 47
Assessing Critical Thinking 2008
ARC Assignment Profile
Evaluating scenario (selected or created) should be stated in such a manner to allow the student to address each of the tasks.
QEP team is willing to assist you with the creation of the scenario or identify possible sources of existing scenario that could be used.
Completed student assignments should include a copy of the scenario, the assignment provided to the student (with the rubric), the students work and the final graded rubric.
June 20, 2008 Critical Thinking Institue 48
Assessing Critical Thinking 2008
Deer Population ScenarioThree teenagers were seriously injured in a car accident when
swerving to avoid a deer in on a two-lane road near a small, rural town in Florida. The residents of the town have seen more and more deer enter the town’s populated areas over recent years. Local law enforcement has been called numerous times this year to remove the animals from backyards and neighborhood streets, and one deer even caused considerable damage as it entered a restaurant in town. The mayor has been charged by the city leaders to keep the town residents safe. Local crops have even been damaged by the animals. Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population. One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas. Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.
June 20, 2008 Critical Thinking Institue 49
Assessing Critical Thinking 2008
Next Steps
Another Scoring workshop will be held this Fall
Pairs of Faculty Champions (scorers) will individually score student work samples and identify samples that exemplify various levels
June 20, 2008 Critical Thinking Institue 50
Assessing Critical Thinking 2008
Next Steps
Faculty Champions (scorers) will complete evaluation forms regarding the validity and reliability of the ARC rubric
Interrater reliability will also be calculated from ARC ratings
June 20, 2008 Critical Thinking Institue 51
Assessing Critical Thinking 2008
Next Steps
Faculty champions will make revisions to the ARC and the assignment profile as needed.
ARC Development Process will be repeated (Steps 5 - 7)
June 20, 2008 Critical Thinking Institue 52
Assessing Critical Thinking 2008
Questions/Next Steps
Assessing Critical ThinkingSummer Critical Thinking Institute
QEP Team, Faculty Champions, and Academic Roundtables
2008
Recommended