Upload
martina-elfreda-perry
View
227
Download
4
Tags:
Embed Size (px)
Citation preview
Outline
Test bias – definitions The basic issue: group differences What causes group differences? Arguments that tests are not biased Differential item functioning analysis Criterion-related sources of bias
Outline
Other approaches to testing minority groups Chitling test BITCH test SOMPA
Models of test Bias Regression Constant Ratio Cole/Darlington Quota
Test bias – definition
A test is biased if it gives a systematically wrong result when used to predict something.
So, an intelligence test would be biased if, for example, it underestimated one group’s probability of success in a given endeavor.
Test bias – the basic issue
Various groups within society differ in their average scores on some psychological tests.
We don’t know what causes these differences.
What causes group differences?
Some candidate accounts:
Genetics Socioeconomic
factors Caste Culture Stereotype threat
Arguments that tests are not biased
Major tests have been subjected to impressive scrutiny for decades
Enormous resources are devoted to this purpose
Criterion validity has been established very securely for the major intelligence tests – they do predict college and job performance
Arguments that tests are not biased
It is not appropriate to focus on individual items on a test, which some critics of testing do
Items should be drawn from a variety of domains, not all of which will be familiar to anyone
Arguments that tests are not biased
Test developers evaluate tests on the basis of overall patterns of prediction utility
They’re future-oriented, not past-oriented: “How will you do
in college or in a job?”
Not “have you had the opportunity to learn?”
Arguments that tests are not biased
Do you think of test score results as “outcomes” or as “information” (predictors)?
Test developers say, results are the beginning, not the end – they are information that will guide us
Opponents see test results as outcomes
Arguments that tests are not biased
Systematic studies have asked whether biased items produce group differences on tests such as Stanford-Binet and Wechsler tests
These studies found no evidence that group differences disappeared when allegedly biased items were removed
Argument that tests are not biased
Group differences just as large on what is considered the most culture fair test, Ravens Progressive Matrices, as on WAIS
IQ scores have same utility for prediction regardless of race or socio-economic status.
Differential item functioning analysis
In this approach to testing for bias, you first form groups for comparison which are equated on overall test score
Implication: groups are equivalent in overall ability
Then, you look for differences between groups on individual items
Where difference is found, you conclude that the item is biased (since groups are not different on ability)
Differential item functioning analysis
But removing such items does not eliminate group differences
E.g., people depicted in test items may typically be White & male
But changing this has little effect (McCarty, Noble, & Huntley, 1989)
Criterion-related sources of bias
We evaluate criterion validity by looking at correlation between test scores and criterion scores
E.g., SAT scores vs. GPA after 4 years at university
Criterion-related sources of bias
If correlation is good, we use test scores (e.g., SAT) to predict criterion – and make selection decisions
What do we do if the correlation is different for different groups?
This would imply that test scores mean different things for different groups
Criterion-related sources of bias
In this graph, Group B performs better than Group A but the correlation is the same for both
Test scoreC
riter
ion
Group B
Group A
Criterion-related sources of bias
In this graph, the slopes of the lines are the same but the intercepts are different
Equal slopes means equal correlations – that is, equally good predictions
Test scoreC
riter
ion
Group B
Group A
Criterion-related sources of bias
Here, the intercepts are different and the slopes are different, so predictions for Groups A and B would not be equally good
Such cases are rare
Group A
Group B
X1 X2
Criterion-related sources of bias
Major tests, such as SAT and WISC-R, have equal criterion validity for various ethnic groups (e.g, African-American, White, Latino/Latina)
Similar results have been found in other multi-ethnic countries, such as Israel
The Chitling Test (Dove, 1968)
Developed to make a point about testing for information a group is unlikely to have acquired
Questions require a particular form of “street smarts” to answer correctly
No validity data exist for this test
If you want to predict college performance for minority students, this test won’t help
The BITCH test (Williams, 1974)
Task: define 100 words drawn from the Afro-American Slang Dictionary and Williams' personal experience
African-Americans score higher than Whites
Williams argues that this test is analogous to the standard IQ tests, which are also culture-bound
The BITCH test (Williams, 1974)
Problem: there is no reason to accept the claim that this is an intelligence test.
There is no validity evidence – no prediction of any performance
Does not test reasoning skills
May have some value for testing familiarity with African-American culture
SOMPA (Mercer, 1979)
System of Multi-cultural Pluralistic Assessment
Based on idea that what constitutes knowledge is socially-constructed
Mercer also suggested that IQ tests are a tool Whites use to keep minority groups “in their place”.
SOMPA (Mercer, 1979)
Inspired originally in part by over-representation of minority group children in EMR classes in US schools
Mercer: this over-representation resulted from both More medical
problems Unfamiliar cultural
references on tests
SOMPA (Mercer, 1979)
Fundamental assumption: all cultural groups have the same potential on average
On this view, if one cultural group does more poorly than another on a test, that is a fact about the test, not the groups.
SOMPA (Mercer, 1979)
Combines 3 kinds of evaluation:
Medical Health, vision,
hearing, etc. Social
Entire WISC-R Pluralistic
Compare WISC-R scores to those of same community
SOMPA (Mercer, 1979)
Estimated Learning Potentials: WISC-R scores adjusted for socio-economic background
But these ELPs don’t predict school performance as well as the original WISC-R scores
Mercer: ELPs are intended to assess who should be in EMR classes
SOMPA (Mercer, 1979)
A major problem, in my view, is that we don’t know what consequences arise for children who are removed from EMR classes on basis of ELPs
Is what we call these children important? It is if the label has an effect, but data do not show that effect
SOMPA used much less today than it used to be
Regression
Basis – unqualified individualism: Treat each person
as an individual, not as a member of a group
Select people with highest scores for job or college place
Ignores sex, race, other group characteristics
Leads to highest average performance on criterion
Constant Ratio
Basis – choose so that selection ratio for groups = success ratio for groups
Select the best candidate but give a boost to minority group members’ scores so that selection probability = success probability
Constant Ratio
Adjust test scores for minority groups upwards by half the mean difference between groups
Leads to somewhat lower average performance on criterion
Cole/Darlington
Basis – If there is special value in selecting minority group members, then a minority score of Y on criterion is equal to a majority score of Y + k on criterion
Separate regression equations used for different groups and adjustment made
Leads to lower average performance on criterion
Cole/Darlington
If a value is placed on selection of minority group members, and intercept is lower for that group, then we consider minority test score X1 and majority test score X2 equal
X1 X2
k
Quota
Basis – idea that all groups should have equal outcomes
Selection based on different regression equations for each group
Produces lower average performance on criterion