Upload
arwen
View
157
Download
9
Tags:
Embed Size (px)
DESCRIPTION
Validity and Item Analysis. Chapter 4. Validity. Concerns what the instrument measures and how well it does that task Not something an instrument has or does not have Informs counselor when it is appropriate to use the instrument and what can be inferred from the results - PowerPoint PPT Presentation
Citation preview
Validity and Item Analysis
Chapter 4
Validity Concerns what the instrument measures
and how well it does that task Not something an instrument has or does
not have Informs counselor when it is appropriate to
use the instrument and what can be inferred from the results
Reliability is a prerequisite for validity
Movement Away from Traditional Categories of Validity
Traditional types of validity Content-Related
Criterion-Related
Construct-Related
Evidence Based on Instrument Content The degree to which the evidence indicates
the items, questions, or tasks adequately represent the intended behavior domain.
Central focus is typically on how the instrument’s content was determined
Content-related validation evidence should not be confused with face validity
Evidence Based on the Response Processes
Information processing
Examining the response processes used by different groups
Computer simulated performances
Evidence Based on Internal Structure Examining internal structure using
factor analysis
Examining the internal structure of an assessment with groups (e.g., gender, race or ethnicity)
Evidence Based on Relations to Other Variables
Correlational method Convergent evidence/Discriminant evidence Multitrait-multimethod matrix
Prediction or instrument-criterion relationship Concurrent and predictive validity Regression Decision Theory
Validity Generalization
Correlational Method Correlational method-
examine relationship between instrument and other pertinent variables or a criterion
Validity coefficient(s)
ABCScale
Mean ratingson criteria
Tyrone 87 4.5Steve 63 2.4Hannah 75 3.7Jeannie 54 1.6Andy 32 .5Benjamin 44 4.9
Prediction or instrument-criterion relationship Concurrent validity
Predictive validity
Regression
ABC
.... ..
10 20 30 40 50 60 70 80 90 100
1
2
3
4
5
..
… …
… .
. .…
.
. .. .
.
… . .
.. .
..
.Ratings
Regression Line
ABC
.... ..
10 20 30 40 50 60 70 80 90 100
1
2
3
4
5
..
… …
… .
. .…
.
. .. .
.
… . .
.. .
..
.Ratings
a = intercept b = slopeX = score
Standard Error of Estimate
xySS ryest21
The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity.
Decision Theory: Expectancy Table
ABC Scale
Ratings
10 20 30 40 50 60 70 80 90 100
5 1 1 4 5 4 1 1 2 1 2 1 3 1 3 2 2 3 3 2 2 2 3 2 1 2 1 3 1 1
Expectancy Table (cont.)
10 20 30 40 50 60 70 80 90 100
5 1 1 4 54 1 1 2 1 2 1
3 1 3 2 2 3 3 22 2 3 2 1 21 3 1 1
Expectancy Table (cont.)
10 20 30 40 50 60 70 80 90 100
5 1 1 4 54 1 1 2 1 2 1
3 1 3 2 2 3 3 22 2 3 2 1 21 3 1 1
Validity Generalization
A method of combining validation studies to determine if the validity evidence can be generalized Must be a substantial number of studies Meta-analysis
Evidence-based consequences of testing
Social consequences of racial/ethnic group differences
Consider social implications when examining the validation evidence
Conclusion on validation evidence
Gradual accumulation of evidence
Counselor must evaluate the information to determine if it appropriate for which client under what circumstance
Validation evidence should also be considered in informal assessments
Item Analysis Focus is on examining and evaluating
each item on an instrument. Item difficulty:
p = # who answered correctlytotal number
Item discrimination:d = upper % - lower %(or correlational method)
Item Response Theory or Latent Trait Theory
Focus is on each item and on establishing items that measure the individual’s ability or level of a latent trait.
Involves examining the item characteristic function (or characteristic curve) and the calibration of each individual item.
Parameters: Difficulty Slope Basal level
Item Characteristic Curve
4 5 6 7 8 9
1.00
.50
.00
Grade
%