Dr Sarah Richardson Senior Research Fellow, Higher Education Australian Council for Educational...

Insights from the AHELO Feasibility Study – why, what and how

Dr Sarah RichardsonSenior Research Fellow, Higher Education

Australian Council for Educational Researchsarah.richardson@acer.edu.au

Eternal challengesHow do we measure quality in teaching and

learning?How effective are institutions at educating

students?What knowledge and skills do students

possess at the end of their degrees?How do students’ learning outcomes compare

to international benchmarks?

Final-year Bachelor degree students

To determine a robust approach to measuring learning outcomes in ways that are valid across cultures and languages, and across the diversity of institutional settings and missions

To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning

How do the learning outcomes of our students measure up against international benchmarks?

Benchmarking against like institutionsStudents – demographics, time usage,

performance, motivation, plansTeaching – curriculum structure, class sizes,

staff contact, feedback, industry experience Institutional characteristics – size, location,

funding, mission, student characteristics

To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning

AHELO: international participation – 17 countries (including Egypt, Kuwait and Abu Dhabi) and 12 languages (including

Arabic)

Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes?

Test fields: Economics, Engineering

23000 students

5000 faculty

Generic Skills Economics Civil Engineering

250 institutionsOnline

International agreementExpert Groups – Engineering and EconomicsDomain parameters / conceptual structureMeasurement priorities – real worldReview and map draft assessment materials Scoring guide – desired responsesStructure reporting and review resultsGeneric Skills – different approach used

Economics learning outcomessubject knowledge and understandingsubject knowledge and its application to real

world problemsability to make effective use of relevant data

and quantitative methodsability to communicate to specialists and non-

specialistsability to acquire independent learning skills

Assessment instrumentsCombine pre-existing / new materialsInternational collaboration / resourcesAdapted / created for assessment frameworkConstructed response and MCQsPilot & qualitative feedback Localisation – equivalency across languagesOnline delivery

Engineering

Engaging participantsCountries / institutions / faculty / studentsWhat’s in it for me?Marketing and awareness raisingCultural variationsSample / populationAuthority / persuasionIncentives and inducementsIntrinsic motivation

Analysis and reportingPsychometric analysis

How did the test items function?What was the spread of difficulty?Did difficulty vary according to student and

institution characteristics?Reporting – international, national and

institutional levels

Using results to enhance teaching and learningPerformance by student group

Which students need greatest support?Performance by competency

Which aspects of teaching need enhancing?Performance by country

What is missing in the curriculum?

Insights from AHELOMeasure learning outcomes internationally? International agreement on skills and knowledge? Assessment materials to suit diverse contexts? Maintain equivalence in translation? Engage institutions, faculty and students? Online delivery in different languages/alphabets? Useful benchmark reports without identification? AHELO Feasible?

Learning

Employment

Curricula

Faculty

Improve Quality

Selection Diagnostic

Progression Outcomes