Embed Size (px)
DESCRIPTION
Citation preview
OECD Feasibility Study for an international Assessment of Higher Education Learning Outcomes (AHELO)
IMHE General Conference
18 September 2012
Karine Tremblay and Diane Lalancette, OECD
AHELO rationale
Feasibility study design and operationalisation
Progress to date
Emerging insights
National perspectives: the Mexican and Russian experiences
AHELO rationale
Despite huge progress in quality assurance, institutional quality remains largely unknown No perfect proxy Reputation race Rankings biased Satisfaction culturally sensitive Labour market outcomes problematic
Shared vision of OECD Education Ministers (2006) Shift from quantity to quality
So what?
Information vacuum filled by available informationLearning outcomes need to be taken into account Defining them Incorporating them in quality assurance Measuring them (AHELO)
AHELO rationale
Feasibility study design and operationalisation
Progress to date
Emerging insights
National perspectives: the Mexican and Russian experiences
Goals of the feasibility study
Assess whether we can measure what undergraduate students know and can do upon graduation …
… across diverse countries, languages, cultures and types of institutions
Test the science of the assessment
Test the practicality of implementation
Design parameters for the feasibility study
Artificial distinction between 4 strands of work To examine different types of learning outcomes and different testing
approaches Generic skills and 2 different disciplines to prove the concept A research strand exploring the issue of value-added measurement
A proof of concept approach Not the perfect (nor final) instruments Ensuring diversity of participants (types of institutions, countries,
languages, cultural backgrounds)
An improvement-driven approach Rich collection of contextual data (from students, faculties, leadership) To go beyond diagnosis tool towards analysis of “what works”
• •
ObserversBahreinBrazilSaudi ArabiaSingapore
•
•
• •
• • •
•
• •
•
• • •
•
•
• •
•
•
• •
A range of geographic, linguistic and cultural backgrounds involved
•
Generic SkillsColombiaEgyptFinlandKoreaKuwaitMexicoNorwaySlovak RepublicUnited States (CT, MO, PA)
EngineeringAbu DhabiAustraliaCanada (Ontario)Colombia EgyptJapanMexicoRussian Fed.Slovak Republic
EconomicsBelgium (Fl.)EgyptItalyMexicoNetherlandsRussian Fed.Slovak Republic
AHELO rationale
Feasibility study design and operationalisation
Progress to date
Emerging insights
National perspectives: the Mexican and Russian experiences
Work undertaken in 2 phases
Generic Skills
Framework
EconomicsFramework
EngineeringFramework
Project management,survey operations and
analyses of results
Contextual dimension surveys
Frameworks
Instrument development &
small-scale validation
Generic Skills
Instrument
EconomicsInstrument
EngineeringInstrument
Implementation
Phase 1 –Initial proof of
concept
Phase 2 – Scientific feasibility & proof of practicality
Jan 2010-June 2011
Mar 2011-Dec 2012 Where we are now
Data collection from February to June 2012 17 countries involved in 25 strand replications Data collected from over
23,000 students 4,900 faculties 270 institution coordinators
1,000 test sessions and 20,000 computers involved Scoring completed in June 2012 Analysis of results and findings underway Final report by December 2012
Quick facts on AHELO fieldwork
AHELO rationale
Feasibility study design and operationalisation
Progress to date
Emerging insights
National perspectives: the Mexican and Russian experiences
Cultural adaptation and translation followed established guidelines
Assessment frameworks have been validated in all strands
Students and faculty random sampling was conducted in the majority of participating institutions
International online testing generally ran smoothly in most countries
Scorer training was conducted in all participating countries
Independent quality monitoring of fieldwork activities indicated that most countries followed survey procedures
Emerging insights
What can we learn now from the data?
... answers in December 2012
Feasibility study report (Volume 1): December 2012
AHELO conference: 14-15 March 2013 in Utrecht
Feasibility study report (Volume 2): April 2013
Next steps
AHELO sponsors
