Upload
seltacademy
View
385
Download
2
Embed Size (px)
DESCRIPTION
The second plenary on day 1 of our TEAM 2014 (testing, evaluation assessment masterclass) held at Meliksah University, Kayseri.
Citation preview
Principles of test design Objectives, measurement qualities, challenges KAYSERI, 29-31 JANUARY 2014
Aims
! Definining our assessment objectives ! Measurement qualities: ‚test usefulness’ ! Testing challenges ! Considering impact
Testing: a definition
! “the systematic gathering of information for the purpose of making decisions” (Weiss 1972)
! “the collection of reliable and relevant information” (Bachman 1990)
! What information? ! What decisions?
Defining our test objectives
1) What kind of test is it going to be? 2) What decisions do we need information for? 3) What abilities are tested? 4) How much detail do we need? 5) How accurate should our results be? 6) Is there any washback effect? 7) What are the practical constraints?
! examinations ! formal tests ! oral tests ! home assignments ! term reports ! continuous assessment ! projectwork, research work ! student self-evaluation
Assessment in the classroom
! for placement or level testing: to decide on the level of learners
! for diagnostic testing: to identify individual strengths or weaknesses
! to evaluate progress ! to evaluate skills for specific needs ! to give marks for performance ! to evaluate and update syllabus and objectives ! also to prepare/train for exams
Purposes of tests
Contrasting categories of ESL tests
Knowledge Performance (or skills)
Subjective Objective
Productive Receptive
Language sub-skills Communication skills
Norm-referenced Criterion-referenced
Discrete-point Integrative
Proficiency Achievement (or progress)
Types of tests (Madsen, 1983)
" A correspondence between language test performance and language use ◦ “In order for a particular language test to be useful for its
intended purposes, test performance must correspond in demonstrable ways to language use in non-test situations.”
" A clear and explicit definition of qualities of test usefulness ◦ “Usefulness = Reliability + Construct validity + Authenticity +
Interactiveness + Impact + Practicality” (Bachman & Palmer, 1996)
Key principles of testing
" consistent across populations " consistent within the same test
irrespective of: ! setting
! marker
! item set When conditions of the test remain unchanged, it always produces essentially the same results.
Qualities of test usefulness: reliability
! Types of validity (Alderson et al, 1995) ! internal validity
! face validity
! content validity
! response validity
! external validity ! concurrent validity
! predictive validity
! construct validity ! meaningful? appropriate? representative?
Qualities of test usefulness: validity
CHECKLIST FOR VALID TESTS Tests what it claims to test?
Best format for what you want to test?
Content relevant to student’s real-life needs?
Items typical/representative of learner use?
Items don’t accidentally test other things?
Test skill, NOT knowledge, creativity or logic?
Clear and appropriate marking criteria?
Construct validity
Testing competence from Language Testing in Practice by Bachman & Palmer (1996, OUP)
Misconceptions ! one best test for a given
situation
! misunderstanding the nature of testing and test development
! unreasonable expectations
! blind faith in measurement
Resulting problems ! inappropriate tests
! failure to meet specific needs
! uninformed use of popular testing methods
! frustration about not finding the perfect test
! loss of faith in self ! testing can only be done by experts
! defending the indefensible (stakeholders’s expectations)
" testing = “real life”? “authentic”? " “communicative language testing”? " defining abilities in a test " selecting tasks and items ! competence? " measurement principles:
can “unidimensional scores for locally independent test items reflect authentic language use”? (Bachman)
" outside factors ! performance?
Challenges in testing