View
41
Download
3
Category
Preview:
DESCRIPTION
Mladen Koljatic\nMonica Silva\nRichard P. Phelps\n - PowerPoint PPT Presentation
Citation preview
Overreach in High-Stakes Testing: The case of Chile’s National University
Admission Test (PSU)
9th ITC Conference, San Sebastian, SpainJuly 5, 2014
Mladen KoljaticMonica SilvaRichard P. Phelps
2
Topics of the presentation
1.Types of admission tests
2.The PSU context and problems
3.Lessons to be learned
4.The role of international testing associations
1. Types of admission tests
Aptitude Tests or Achievement Tests in the Context of Admission?
“ Where school inputs vary widely…academic achievement tests are likely to measure the opportunity to learn more than the ability to learn. [Developing] nations concerned about picking their future talent must consider the possibility that an aptitude test…may be more able to overcome the local differences in school quality” (Heyneman, 1987, p. 253)
SOURCE: Phelps, Standardized Testing Primer, 2007
Aptitude Tests or Achievement Tests in the Context of Admission?
“ Achievement tests are fairer to students because they measure accomplishment rather than ill-defined notions of aptitude; they can be used to improve performance; they are less vulnerable to charges of socioeconomic bias; they are more appropriate for schools, because they set clear curricular guidelines and clarify what is important for students to learn…the movement away from aptitude tests is an appropriate step for U.S. students, schools and universities” (Atkinson, 2001)
Purpose of Admission Tests
To predict college successwithout providing an unfair or unjustified advantage to any particular group of test-
takers
SOURCE: Phelps, Standardized Testing Primer, 2007
2. The PSU context and problems
School Context: 2 curriculums
Curricular Branch
Type of School
ScientificHumanistic (SH)
Technical-Professional (TP)
Municipal
Private-Subsidized
Private- Paid —
The context of Chile´s PSU
PSU focused on measuring the SH curriculum…
Students who attend the TP tracks the poorest in the nation over 90% wish to pursue tertiary education
social mobility highly related to university education in Chile
* Source: Mineduc, 2009
The context of Chile´s PSU
PSU test scores used Admission Financial aid Government subsidies to universities
very high-stakes associated to test scores
11
The context of Chile´s PSU Centralized admission system to all public funded
universities (+ some private universities) PSU is run by a department within the largest public
university (U. de Chile) Admission system requires: PSU test scores High-school GPA (HSGP) Class ranking No other criterion is considered for application to
higher education institutions
History of Chile´s PSU
From 1966-2003 : aptitude-type test (similar to SAT) From 2004- the present: PSU
Achievement tests: measure 100% of SH curriculum Mathematics Language Science (Chemistry, Physics and Biology) Social Sciences (History, Geography and Economics)
Change of tests aimed at evaluating outcomes of an educational reform
13
Project to Develop the PSU TESTS
Supported and promoted by the Ministry of Education and council of rectors of public universities (CRUCh)
Developed using public funds
Project led by an economist and a social psychologist with no training or expertise in the development of high-stakes tests
Multiplicity of Purposes for the PSU
Measure the implementation of a new curriculum Fairly measure mastery of one of the two national cu
rricula (SH) Incentivize high schools to implement the new curric
ulum Incentivize high school students to study more Predict overall success of students in universities Predict success across very different types of univers
ity programs Reduce socio-economic disparities in performance
15
“ Evidence of the test´s technical quality should be provided for each purpose“ (St.13.2)
as well as “adequate monitoring of potential negative consequences” (St.13.1)
Multiplicity of Purposes for the PSU
How the PSU Runs
Source: adapted from the Pearson Report (2013)
Council of Rectors of Chilean Universities "owners" of the PSU
designated by CRUCh as supervisors and official evaluators of the PSU PRESIDED BY CREATORS OF THE TEST
agency responsible for developing test items, test assembly, tests administration, test scoring, for CRUCh and associated universities
Main funder of the PSU since 2007.It holds the purse for CRUCh universities, since they are funded through public monies
CRUCH
CRUCH´S COMMITTEE OF
TECHNICALADVISORS FOR PSU
U. de Chile
Ministry of Education
Official Chilean Evaluations of the PSU
Eight evaluative reports from CRUCh between 2004-2010:
Pseudo-evaluations: overly optimistic reports aimed to construct a positive image of the test
Do not report any analyses for the TP group When reporting gaps, omit the TP data from the analyses
(without saying so) Deny the existence of a growing score gap between private
paid and municipal school groups, especially those attending the TP track
CRUCh denies access to data bases to independent researchers
19
International Evaluations of the PSU
ETS (2005) Pearson (2013)
20
Efforts to suppress information
2005: ETS Report not released, according to U. Chile because of secrecy clause imposed by ETS
2012: New audit. CRUCh intended: Auditors to report to expert team chosen by CRUCh …to curtail analyses for TP students... to control and edit final report
21
2007: U. de Chile sued to release ETS report
A group of faculty and students represented by a pro-transparency nonprofit organization go to court judge rules in favor of U. de Chile
Efforts to reveal information
22
Efforts to reveal information
ETS report is finally released in 2012 thanks to TRANSPARENCY LAWS
passed in the country
From 2007: 5 years of pressure by university student leaders, faculty and leaders of NGOs for new audit
2012: Funded by the Ministry of Education (maximum amount suggested by CRUCh´s CTA: US$400,000)
Pearson’s Final Report of the PSU (2013): Recommendations
PSU needs to be revised to advance equity for TP groups
reduction in contents or study alternatives for the equitable
assessment of CH and TP students PSU is below predictive standards of other
international tests belying claims from CRUCh’s experts as to the
predictive capacity of the PSU
24
Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS
Inadequate use of PSU cut scores to assign financial help and scholarships
Inadequate use of scores to pass judgment on the quality of schools
25
Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS
Faulty piloting of test items DIF analyses does not include factors as
socioeconomic status, high school curricular branch (S&H and TP)
Inadequate difficulty level of the PSU Math Test (detected in ETS’s 2005 audit)
Actual reporting of PSU Science score deemed “untenable”: Science PSU requires not one, but three scores
26
Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS
Inadequate level of analysis expected when equating a high-stakes assessment: misleading information regarding year-to-year equating and pilot calibrations. “THESE ACTIVITIES ARE NOT TAKING PLACE EVEN THOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDICATE THEY ARE.” (p.43) PSU equating is below international standards
ETC…
27
Pearson’s Evaluation of the PSU (2013)…
ComprehensivePearson met with stakeholders other than CRUChIncorporated some analyses for the TP group (at no extra cost)Specified new studies that needed (e.g.,, to guide reduction of contents)
However…Recommendations are not prioritized Highly technical report, not aimed at policy makers Did not look at effects on enrollment
Pearson’s Final Report of the PSU (2013):
Revealed how easy it is to mislead the public when it comes to technical matters
CRUCh deceived the public as to the quality of the tests, in terms of prediction and fairness
CRUCH does NOT reduce contents in the PSU, against Pearson's reccomendation
CRUCh's new commitee of technical advisors (CTA) is controlled by an associate of the PSU test-makers
29
Aftermath of Pearson’s Report...
Past Problems = Present ProblemsVested interestsNo real changes in the governance systemLack of transparencyNo accountability
Main problems remain unsolved…
30
Aftermath of Pearson’s ReportStudents demand an end to the PSU elimination of the test
Only HSGPA and class ranking as selection criteriaMoratorium not only to the PSU but to other educational tests as well …
31
3. Lessons to be learned
32
Changes in tests…
Some nations that are considering moving towards achievement tests might benefit from looking at the Chilean experience to avoid a proliferation of purposes, test corruption and abuse…
Changes entail risks…
-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test
-There is a need for a governance scheme that guarantees independent evaluations in order to protect applicants’ rights to be tested by good and fair tests
34
Data access is not only a problem in developing nations…
“The SAT item-level information is not frequently available to independent researchers who are not affiliated with ETS or the College Board…”
Source: Santelices & Wilson, 2010.
Governance Issues in Order to Prevent Test Abuse
• Test-developing agency should be professional and independent
• Account for funds received and quality of work• Report to external boards of experts that can dem
onstrate independence from the agency • Periodically audit tests for quality and fairness• Audit reports & data bases should be public
--> TRANSPARENCY AND ACCOUNTABILITY
4. The role of international testing associations
Changes in tests…
Question to ask:
How can international organizations such as ITC play a more active role in promoting good consulting practices by test experts in developing nations?
Is there a larger role for testing associations such as ITC in the promotion
of good practices worldwide?
•The ITC Guidelines on Adapting Tests•The ITC Guidelines on Test Use•The ITC Guidelines on Computer-Based and Internet-delivered Testing•The ITC Guidelines on Quality Control in Scoring, Test Analysis and Reporting of Test Scores…•The ITC Guidelines for Test Review?•The ITC Guidelines for International Consulting?
Overreach in High-Stakes Testing: The case of Chile’s National
University Admission Test (PSU)
Mladen KoljaticMonica Silva
Richard P. Phelps
9th ITC Conference San Sebastian, Spain
July 5, 2014
41
Predictive validities of the PSU(CRUCh vs Pearson estimates)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA
Predictive validities: ACT and PSU
SOURCE: ACT, Research Summary Services, 1997_1998; Pearson, Final Report Evaluation of the Chile PSU, January 2013
Predictive validities: SAT and PSU
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
44
Predictive validities: SAT and PSU(BUSINESS MAJORS)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
Predictive validities: SAT and PSU(Education)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
46
Percentage-point drop in enrollment of municipal school graduates attending two top universities
UC U. de Chile PAA (adm. 2003) 17 32 __________________________________ PSU (adm.2004) 17 29PSU (adm.2005) 16 25 PSU (adm.2006) 14 20 PSU (adm.2007) 13 --PSU (adm.2008) 11 --PSU (adm.2009) 11 --PSU (adm.2010) 12 --
Source: official institutional info
Drop in enrolled students from Municipal Schools, other CRUCh Universities
porcentaje matrïculados municipal
0
10
20
30
40
50
2003 2004 2005 2006 2007 2008
PUC
U CONCE
USACH
PUCV
Fuente: E. Simonsen, 2009
Estimated average 10% drop in enrollment from students from municipal schools at top CRUCh Universities in three years NO
measures taken…
Recommended