View
215
Download
0
Category
Tags:
Preview:
Citation preview
Assessment of Higher Education Learning Outcomes:a ground-breaking initiative to assess quality in higher education on an international scale
Diane LalancetteDirectorate for EducationOrganisation for Economic Cooperation and Development
UNICA Rectors’ Seminar 2010Tallinn
28 May 2010
2
What we know about HE quality…
Proxies of higher education quality exist, but none are perfect
• Rankings often biased towards input factors and research• Subjectivity of reputation factor• Cultural sensitivity of satisfaction factor• Labour market outcomes sensitive to conjuncture and local
economic conditions
So what?Learning outcomes as a promising direction
•Defining them (Tuning process in Bologna area)
•Incorporating them in quality assurance processes
•Measuring them → AHELO
The massification of participation in higher education has meant much more heterogeneous abilities and expectations of students than in the past
3
Objectives
What is AHELO?
Why undertake the study?
Why is AHELO important?
A ground-breaking initiative to assess what students know and can do upon graduationCreating measures that would be valid:
• For all cultures and languages;• And also for the diversity of HE institutions.
After decades of quantitative growth in HE, consensus on the need to ensure quality for all (Athens, 2006)… but information gap on learning outcomes
Carry out a feasibility study to provide a proof of concept (Tokyo, 2008)
• Employs a wide range of measures• Provides faculties, students and government agencies with a more balanced
assessment of HE quality – not just research-driven rankings!• No sacrifice of HEIs’ missions or autonomy in their subsequent efforts to
improve performance.
4
Multi-dimensional definition of quality
Addressing the needs of various users and uses
• “Bottom line” of performance• “Value-added” to assess the quality of services • Contextual data to reveal best practices and problems, and to identify teaching and
learning practices leading to greater outcomes
Both in discipline-related competencies …
• Easily interpretable in the context of departments and faculties ... • But require highly differentiated instruments
And in generic skills
• Less dependent on occupational and cultural contexts, applicable across HEIs … • But reflect cumulative learning outcomes and less relevant to the subject-matter
competencies that are familiar to HEIs, departments or faculties
5
The feasibility study at a glance
What?Not a pilot, but rather a research approach to provide a proof of concept and proof of practicality.
Why?The outcomes will be used to assist countries to decide on the next steps. No pre-determined conclusion.
When?Phase 1 cognitive labs: January 2010 to June 2011 Phase 2 implementation: late 2011-2012
Who?Target population of students who are near, but before, the end of their first 3-4 year degree.
To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible.
Goal?
How? OECD guiding international expert committeesContractors in charge of developing the assessment instruments.
6
Remarks on data collection
• No comparative data at the national level
• Institutions/departments are the units of analysis, hence measures and reporting at HEI/dept level
• Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)
7
AHELO: 4 strands of work
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
Discipline strand in Engineering
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
Discipline strand in Economics
With each assessment, a collection of
contextual information:
• to look beyond student performance: (e.g. institutional missions, selectivity, student characteristics and exposure to “good practices”, satisfaction).
• to make AHELO an effective tool to reveal best practices and to identify shared problems.
International pilot test of the US Collegiate Learning Assessment
(CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different
cultural, linguistic and institutional contexts.
Generic skills strand
+ contextual data + contextual data
+ contextual data
8
AHELO: 4 strands of work
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Engineering
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Economics
Several perspectives to explore the issue of value-
added (conceptually, psychometrics), building on recent OECD work at
school level.
Research-based “Value-added” or “Learning gain”
measurement strand International pilot test of the US Collegiate Learning Assessment
(CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different
cultural, linguistic and institutional contexts.
+ contextual data
Generic skills strand
9
AHELO tests of instruments
3 assessment instruments
AustraliaJapan
Sweden
Generic skills Engineering Economics
Contextual indicators and indirect proxies of qualityto analyse determinants of LO and their development
3 groups of countries
Colombia (tbc)FinlandKuwaitKoreaMexicoNorway
USA
Belgium (Fl.)Italy
MexicoNetherlands
Russian Federation
2 contextual instruments
10
Work to be undertaken in 2 phases
Generic Skills
Framework(started)
EconomicsFramework(underway)
EngineeringFramework(underway)
Project management,survey operations and
analyses of results
Contextual dimension surveys
Frameworks
Instrument development & small-scale
validation
Generic Skills
Instrument(started)
EconomicsInstrument(underway)
EngineeringInstrument(underway)
Implementation
Phase 1 -Initial proof of concept
Phase 2 -Scientific feasibility & proof of
practicality
11
Practical considerations
• Test of practicality of implementation: international standards for test administration and student participation rates within HEIs
• Assessments possibly computer-delivered or web-based
• Performance described through proficiency levels and “can-do” statements
• Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)
• Use existing instruments, respecting their integrity; or combine item pools of existing instruments in ways that cover frameworks; or develop new test material
12
Imagine quality education in 2017
If AHELO were one of the QA instruments available to HEIs… • HEIs could decide to participate… or not• HEIs could decide to use it for internal improvement only, or make results public
What AHELO could do… • Benchmark HEIs against their peers
to identify strengths and weaknesses • Assist HEIs’ with student recruitment
and students’ choices• Limit academic drift by showcasing
result’s within one’s mission• Enable researchers to investigate
teaching and learning processes at depth: What works and under which contexts/conditions
• Spur QA reflection
What AHELO would not do… • Rank countries and/or HEIs…• Provide a miracle solution to
accountability requirements: learning outcomes are only one aspect of the mission of HEIs (along with equity, regional mission, etc) There is no single criterion in HE quality space
• Replace QA
13
A study with great potential…
… Diagnosis is the basis of any improvement
… Shaping the future of higher education to address key challenges
Better information on student learning outcomes is the first step to improve teaching and learning for all:
Provide evidence for national and institutional policy and practiceEquip institutions with the method and tools to improve teaching
EquityBuild fairer higher education systems, promoting success for all
ResponsivenessBetter connect higher education and society
EffectivenessHelp students make informed choices to ensure success for all
ImpactFoster international transparency and mobility
… Scope for a few more participants
14
AHELO is managed by the OECD IMHE Programme
Institutional Management in Higher Education
15
16
Thank you
Thank you!
Please sendquestions or comments to
Diane.Lalancette@oecd.org
For more information, visitwww.oecd.org/edu/ahelo
Recommended