Upload
zoe-powers
View
218
Download
1
Tags:
Embed Size (px)
Citation preview
August 2007 FFP 1740 1
www.floridastatefirecollege.org
Testing and Evaluation Testing and Evaluation Techniques Techniques
Chapter 7Chapter 7
Florida State Fire College
Ocala, Florida
August 2007 FFP 1740 2
Testing and EvaluationTesting and Evaluation
Note this chapter is reviewed because you will be using some of the techniques involved
When you follow-up with Course Design, the chapter will be covered in much more detail
August 2007 FFP 1740 3
Terminal ObjectiveTerminal Objective
The student will be able to construct, administer, and evaluate an assessment instrument
August 2007 FFP 1740 4
Enabling ObjectivesEnabling Objectives
The student shall be able to:Define the four levels of evaluationDifferentiate between summative and formative evaluationDefine the different kinds of testsDiscuss the difference among the various types of testsList various sources for tests
August 2007 FFP 1740 5
Four Levels of Evaluation Four Levels of Evaluation
Level I – reaction
Level II – learning
Level III – transfer
Level IV – business results
August 2007 FFP 1740 6
ReactionReaction
How did the student react to the class?
Evaluation form
Focus groups (rarely used)
Required by most academic institutions
August 2007 FFP 1740 7
LearningLearning
What has the student learned?
Oral examination
Written test
Skills assessment
August 2007 FFP 1740 8
TransferTransfer
How much did the student retain?
Measured six weeks to six months post coursework – on the job
Based upon tests, observations, surveys, and interviews
Not regularly done within the fire service
August 2007 FFP 1740 9
Business ResultsBusiness Results
An assessment of the financial impact of the training or the return on investment.
Six months to two years post coursework
Most difficult to measureTraining courses do not have business
objectives – ie: reduction in accidentsMethodology for assessment is not yet refined
August 2007 FFP 1740 10
EvaluationEvaluation
“An evaluation is a process of making a value judgment based upon one or more sources”
Evaluation processes examine two components:
Instruction from the teacher
Performance of the student on objectives
August 2007 FFP 1740 11
Purpose of EvaluationPurpose of Evaluation
Provides feedback to students
Provide students gratification and motivation
Measures the effectiveness of the instructor
Measures the effectiveness of the program in meeting objectives
August 2007 FFP 1740 12
Formative EvaluationFormative Evaluation
Ongoing evaluation to change or adapt the program
Compares the objectives to the testing strategy
Occurs during development of course or testPilot testing is formative evaluation
August 2007 FFP 1740 13
Summative EvaluationSummative Evaluation
Typically performed at end of programProvides students feedback on mastery of subjectProvides feedback on effectiveness of teaching strategySummative toolsCourse evaluation formsFinal exams
WrittenPractical
August 2007 FFP 1740 14
TestsTests
Instructor’s last chance to emphasize the important information the student needs to retainWrittenPractical Oral
August 2007 FFP 1740 15
Written TestsWritten Tests
Multiple Choice
True/False
Matching
Completion or fill in the blank
Essay
August 2007 FFP 1740 16
Multiple Choice TestsMultiple Choice Tests
Common method for formal and informal evaluationNormally used for national and state certification examinationsEasy to grade and be objective
Test construction miscuesUsing previous questions informationNegatively worded stemsFill in the blank in middle of stemUsing ‘all of the above” or “none of the above”
August 2007 FFP 1740 17
True/False TestsTrue/False Tests
Limited to two answers, no gray area
Difficult to construct in positive voice
Avoid always or never statements
Useful tool as a study guide
August 2007 FFP 1740 18
Matching TestsMatching Tests
Works best with definitions and terms
Difficult to design
Cautious of multiple matches
Test directions must be clear
August 2007 FFP 1740 19
Completion TestsCompletion Tests
Fill in the blankStatements must be clear as to intent of questionNeed to be grammatically correctBe aware of size of blankAvoid having blank at beginning of sentence
August 2007 FFP 1740 20
Essay TestsEssay Tests
May require long or short answerTime consuming and difficult to gradeRecommended to grade in a group formatRubric is a useful grading toolHand written exams must allow sufficient timeShotgun approach.. As much information as possible in hopes of hitting the target
August 2007 FFP 1740 21
Oral ExamsOral Exams
Requires verbal answers by studentsAdvantagesEvaluate quick reaction of studentAssesses the student thought process
DisadvantagesLimited number of students examined at one timeDifficult to standardizeTime consuming and labor intensiveUnexpected distractionsUnfair emphasis on repeated mistakes
August 2007 FFP 1740 22
Practical ExamsPractical Exams
SituationalDemonstration of a skill in the context of a
scenarioFFII Exam for forcible entry
Rote“Demonstration of steps of performing a skill”FFII Exam donning of SCBA
August 2007 FFP 1740 23
Test ResourcesTest Resources
Written examination resources:NFPAPublishers’ test banksFire textbooksFire textbook instructor guidesTextbooks of practice certification examinationsOn-line and computer-based practice certification testsFire Internet sites
August 2007 FFP 1740 24
Test ResourcesTest Resources
Practical examination resources:NFPAFire Internet sitesFire textbooksFire continuing education programs
August 2007 FFP 1740 25
Test ResourcesTest Resources
Oral examination resources:NFPAFire Internet sites
August 2007 FFP 1740 26
ConclusionConclusion
As part of the learning process, the instructor needs a mechanism to evaluate the
student’s learning and identify whether or not the student is achieving the objectives
and goals of instruction.
August 2007 FFP 1740 27
ConclusionConclusion
EvaluateFour LevelsSummativeFormative
TestsKinds of testDifferences among the various typesSources for testing materials
August 2007 FFP 1740 28
Multiple choice tests should avoid what?“Which choice is not correct”“All of the choices are correct”“None of the choices are correct”Maybe I just broke the rule!!!!!!
August 2007 FFP 1740 29
Which type of test is most commonly used for state or national certifications?True/FalseMatchingMultiple choiceFill in the blank
August 2007 FFP 1740 30
Unexpected distractions would likely be a disadvantage in what type of testing?RoteSituationalWritten Oral
August 2007 FFP 1740 31
Of the four levels of evaluation, which one would likely look at job performance 6 months to a year down the road?Reaction Business resultsLearningTransfer