Upload
brendon-meadow
View
214
Download
0
Embed Size (px)
Citation preview
1
Developing Tests for Developing Tests for Departmental AssessmentDepartmental Assessment
Deborah Moore, Assessment SpecialistInstitutional Research, Planning, & Effectiveness
University of Kentucky
2
Session OverviewSession Overview
Steps in Test Development and Steps in Test Development and Test AdministrationTest Administration
Process for Evaluating Test and Process for Evaluating Test and ItemsItems
Common DifficultiesCommon Difficulties Strategies for Analysis and ReportsStrategies for Analysis and Reports
3
Improve Student LearningImprove Student Learning
4
Exercise 1 Exercise 1 (10 minutes)(10 minutes)
How do you decide you are going to How do you decide you are going to give a test?give a test?
What do you do to develop a new What do you do to develop a new test?test?
How do you decide what types of How do you decide what types of item formats to use?item formats to use?
How do you develop items?How do you develop items? How long does it take to put the test How long does it take to put the test
together?together?
Under What Conditions Do You Give a Test?Under What Conditions Do You Give a Test?
5
Using a Test for Using a Test for Classroom AssessmentClassroom Assessment
Generally, instructors develop their own Generally, instructors develop their own classroom tests, making all decisions about classroom tests, making all decisions about when and how to construct, administer, score, when and how to construct, administer, score, and report results of tests. and report results of tests.
Construction is often done without formality or Construction is often done without formality or documentation. documentation.
The most frequent use of tests by instructors is The most frequent use of tests by instructors is to assign grades related to individual student to assign grades related to individual student learning.learning.
6
A Test Is a Good Choice When…A Test Is a Good Choice When…
the student must demonstrate acquisition of knowledge the student must demonstrate acquisition of knowledge or ability to process and use knowledgeor ability to process and use knowledge
the student's knowledge about a wide range of content the student's knowledge about a wide range of content is to be evaluated (e.g., survey and capstone courses)is to be evaluated (e.g., survey and capstone courses)
multiple observations of the content-related knowledge multiple observations of the content-related knowledge are needed (e.g., math and foreign languages)are needed (e.g., math and foreign languages)
more resources are available for constructing the test more resources are available for constructing the test than for scoring and reportingthan for scoring and reporting
a large group is being assesseda large group is being assessed
7
Using a Test for Using a Test for Departmental AssessmentDepartmental Assessment
Planning, implementing, and using results Planning, implementing, and using results become a group effort – a shared set of become a group effort – a shared set of decisions and responsibilities. Consensus is decisions and responsibilities. Consensus is emphasized. emphasized.
Additional planning time, communication, and Additional planning time, communication, and record keeping will be needed. record keeping will be needed.
When used for program assessment, test When used for program assessment, test performance is generally used along with other performance is generally used along with other information to describe group achievement information to describe group achievement and is independent of grading.and is independent of grading.
8
Advantages of Advantages of Selected-Response Item Selected-Response Item FormatsFormats
Better content coverageBetter content coverage Higher reliabilityHigher reliability Greater efficiencyGreater efficiency ObjectivityObjectivity Mechanical scoringMechanical scoring
9
Test Development ProcessTest Development Process Define purpose/useDefine purpose/use Outline curriculum & consult goals/objectives Outline curriculum & consult goals/objectives Create test plan Create test plan (aka blueprint or specifications)(aka blueprint or specifications)
Create a pool of itemsCreate a pool of items Critique and revise itemsCritique and revise items Pilot/field test Pilot/field test (e.g. item analysis, reliability/validity studies)(e.g. item analysis, reliability/validity studies)
Set guidelines for test administration, scoring Set guidelines for test administration, scoring procedures, interpretation of scores procedures, interpretation of scores (e.g. develop (e.g. develop norm tables and standard setting procedures)norm tables and standard setting procedures)
10
Exercise 2 Exercise 2 (10 minutes)(10 minutes)
What role does the exam play in the What role does the exam play in the assessment plan for this department?assessment plan for this department?
What type of item formats should be What type of item formats should be used on the comprehensive exam?used on the comprehensive exam?
Review a Sample Departmental PacketReview a Sample Departmental Packet
11
ObjectivesObjectivesRespond in an informed way Respond in an informed way to the form, structure and to the form, structure and aesthetic qualities of artistic aesthetic qualities of artistic and literary works.and literary works.
Explore the interrelationships Explore the interrelationships among historical events and among historical events and intellectual, artistic, literary and intellectual, artistic, literary and philosophical or religious philosophical or religious movements and works.movements and works.
Identify and analyze Identify and analyze similarities, differences and similarities, differences and interrelationships among the interrelationships among the fine arts.fine arts.
Articulate central philosophical Articulate central philosophical and religious questions and the and religious questions and the varying responses to them varying responses to them within different cultures.within different cultures.
Apply appropriate vocabulary Apply appropriate vocabulary and concepts for the and concepts for the description and analysis of description and analysis of artistic, literary, historical and artistic, literary, historical and philosophical or religious philosophical or religious works.works.
Explain how artistic and literary Explain how artistic and literary works from the past and works from the past and present civilizations are present civilizations are individual expressions of individual expressions of cultural, historical and cultural, historical and intellectual forces.intellectual forces.
12
Obj 1 Obj 2 Obj 3 Obj 4 Obj 5 Obj 6 Totals
Theater 1 1 11 3 2 18
Art 5 3 3 4 3 18
Music 5 1 6 1 1 14
Philosophy/ Religion
1 2 3 9 8 23
Anthropology 2 1 1 2 6
Literature 5 4 6 1 4 20
Totals 17 6 30 12 20 14 99
Test Blueprint Test Blueprint (Content X Objective)(Content X Objective)
13
Test Plan Test Plan (Items Within Cells)(Items Within Cells)
Obj 1 Obj 2 Obj 3 Obj 4 Obj 5 Obj 6 Totals
Theater 7 18 1-6,9-12, 16
8, 13, 17
14-15 18
Art 48,50-51, 58, 64
47, 49, 61
53-54, 59
52, 57, 60, 63
56-57, 62
18
Music 42-46 41 35-40 33 85, 89-90, 99
14
Philosophy/ Religion
66 24-25 26, 78-79
65, 67-68, 72-77
19-23, 69-71
18
Anthropology 29, 32 31 27 28, 30 6
Literature 86-88, 94, 98
47, 49, 61
53-54, 59
52, 57, 60, 63
56-57, 62
20
Totals 17 6 30 12 20 14 99
14
Administrative ApproachesAdministrative Approaches
Common exams Common exams
Course-embedded testingCourse-embedded testing
Assessment Center/Assessment Assessment Center/Assessment DaysDays
15
Sources for ItemsSources for Items
ResearchResearch CurriculumCurriculum Real lifeReal life Professional Professional
development development materials & materials & textbookstextbooks
Own final exam Own final exam filesfiles
Comprehensive Comprehensive exams from exams from other other universitiesuniversities
Retired Retired test/item bankstest/item banks
16
Common Difficulties with Test Common Difficulties with Test QualityQuality
Push for higher order thinking often Push for higher order thinking often weakweak
Insufficient consensus on test Insufficient consensus on test plan/blueprintplan/blueprint
Discomfort with various item-writing Discomfort with various item-writing approaches (writer’s block)approaches (writer’s block)
Insufficient editingInsufficient editing
Little or no piloting of itemsLittle or no piloting of items
17
Critiquing ItemsCritiquing Items
Match with test Match with test planplan
Item valueItem value Item qualityItem quality
18
Typical Item Review Typical Item Review QuestionsQuestions
Has each item received a quality appraisal? Has each item received a quality appraisal? Has each item’s content been verified? Has each item’s content been verified? Has each item been classified?Has each item been classified? Has the key been identified?Has the key been identified? Have you edited the items? Have you edited the items? Have items been checked for Have items been checked for
bias/insensitivity?bias/insensitivity? Have you field tested the items?Have you field tested the items?
19
Planning A Departmental Planning A Departmental TestTest
Develop testDevelop test Plan administrationPlan administration Analyses test dataAnalyses test data Report and use results Report and use results Agree on leadership/support rolesAgree on leadership/support roles
20
Exercise 3 Exercise 3 (30 minutes)(30 minutes)
Review the headings of the checklist formReview the headings of the checklist form
List additional headings neededList additional headings needed
List activities that need to be completed List activities that need to be completed under the various headingsunder the various headings
Add a number to the activities to Add a number to the activities to represent the order of completionrepresent the order of completion
Developing a Planning Developing a Planning ChecklistChecklist
21
AnalysesAnalyses
Examine quality of your assessment Examine quality of your assessment tools and procedurestools and procedures
Describe the test resultsDescribe the test results By test plan featuresBy test plan features By meaningful subgroupsBy meaningful subgroups
Combine with other datasetsCombine with other datasets Execute and summarize planned Execute and summarize planned
comparisons linked to research comparisons linked to research designdesign
22
ReportsReports
Determine Determine stakeholders/constituentsstakeholders/constituents Identify specific interestIdentify specific interest
Create a timeline and shareCreate a timeline and share Include a vetting procedureInclude a vetting procedure
23
Testing in the FutureTesting in the Future
Computer based testingComputer based testing Adaptive (CAT)Adaptive (CAT) Self-Adaptive (SAT)Self-Adaptive (SAT)
Computer assisted item generationComputer assisted item generation Automated essay scoringAutomated essay scoring
24
Thank you for your Thank you for your attention.attention.
Deborah Moore, Assessment SpecialistDeborah Moore, Assessment Specialist
101B Alumni Gym101B Alumni GymOffice of Institutional Research, Office of Institutional Research, Planning, & EffectivenessPlanning, & Effectiveness
[email protected]@email.ukyuky..eduedu
859/257-7086859/257-7086
http://www.http://www.ukyuky..eduedu//LexCampusLexCampus//