Embed Size (px)
Citation preview
Evaluating the impact of research projects in tertiary learning and teaching: exploring the geography of change
Plenary Address, HERDSA Conference, Auckland
1:45 –2:30 pm, Tuesday 2nd July 2013
Kirsty Weir & Peter Coolbear (Ako Aotearoa)
Tilly Hinton (Office of Learning and Teaching & University of the Sunshine Coast
Plan of our presentation
• Why? – why are we doing this?
• Context – the impact landscape
• Practice - activities undertaken in Australia and New Zealand to better understand the impacts of funded learning and teaching research
• Reflections - what this means for researchers, institutions and funding agencies
Why Measure impact of research into teaching and learning?
• Moral purpose
• Addressing a value for money agenda - part of the funding landscape (PBRF, ERA, etc):
• Identifying ways to gain additional leverage from work already done
• Improving funding body processes
• Understanding the benefits that result from learning and teaching research
• Surfacing stories from project teams that could otherwise go unheard
Funding overview
• Ako Aotearoa funds evidence-based change projects with a high potential to benefit learners. There is only one, primary objective of Ako Aotearoa funding – to enhance educational outcomes for learners.
• The OLT funds “academics and professional staff to investigate, develop and implement innovations in learning and teaching. Grants facilitate scholarship and research into learning and teaching, and promote systemic change in the sector.”
What do we mean by impact?
OutcomesOutputsDeliverables
Research assessment
Journal impact factor
ValueTranslationTransfer
Valorisatio
n
Dissemination
EngagementCommunication
Buy-inUptakeOutreachEmbedding
Sustainability
Sustained change
Project logic
Evidence-based change
Evidence-based improvement
Deep change
Scalability
Learner benefit
What do we mean by impact?
“None of these terms is entirely satisfactory, but we lack a better vocabulary.”
Levin (2004, p. 2)
Louise CarterYear 2 Massey University Design student
• “…impact occurs when research, in any of its multiple forms, makes a difference to subsequent actions that people take or refrain from taking.” (Levin 2004)
• impact is determined when there is a “clear demonstration of learner benefit.” (Alkema 2012)
• “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (UK Research Excellence Framework definition)
• “impact is about people not publications” (Stilgoe 2011)
So we need to recognise and manage the risks here
• Adopting an overly transactional model
• Over simplification of the change process
• Assumptions about causality
• Assumptions about generalisability
The Ako Aotearoa Impact Evaluation Framework (IEF)
APPLIED IN NEW ZEALAND:• Conversations with project teams at 6,
12 and 24 months post project completion. Focusing on:• outputs• impact on practice• impact on learners• impact on project teams
• Evaluative conversations placed in the context of the original project rationale and goals
• Applied to all projects Ako funds (including retrospectively)
• Influence Stories written for 15 projects
APPLIED IN AUSTRALIA:• A pilot of the Ako approach all aspects
but for a random sample of 18 projects
• Conversations with project leaders/teams once only, focussing on:– Changes in practice– Student benefits– Impact on the project team– Future plans– Deliverables
• Applied to a random sample of projects across all grant types
• Influence Summaries written for 18 projects
Why post-project conversations?
• Conversations surface unrecognised benefits• Change processes take time, almost always beyond the final report
date• Relationships between projects and funding body are valuable• Student learning is central• Evidence-based culture• Meaning is made collaboratively• Opportunities for further benefits can be unearthed through the
conversation• An opportunity to reflect and be heard
Why stories or case studies?
• Case studies address the causality problem by providing context and identifying: Enablers Barriers Other change drivers
• As case studies develop, they tell us about the directions of travel and differing time-frames of change
• Projects are taken on their own terms, rather than applying metrics or indicators
“stories” “narratives” “case studies” “summaries”
What this means for researchers
• A research plan which moves to implementation of findings
• Understand climate of readiness for change, engagement, and transfer
• Willingness to tackle tough change agendas• Move from requesting to dollars to selling an
opportunity for change/benefit• Recognition from funders that change takes time and
is ongoing
What has funding made possible?
• Online graduate attributes system deeply embedded across an entire school
• Education students supported to develop resilience, a factor in retaining staff in the teaching profession
• Report cited as key evidence for a wage increase across the profession• Emergent academics have ongoing access to mentorship and peer support• Almost three quarters of a million dollars of industry and philanthropic
funds secured to further the work started by the grant• Special issues of journals that further explore themes of the grant• Publications from a grant cited fifty-eight times in other publications• Work integrated learning experience changed for all students in the
discipline after project negotiated a change in accreditation standards
What does this mean for learners?
Academic enhancement Learning environment and resources
Relationships Personal development0
1
2
3
4
5
6
7
8
Actual impact (evidence)Observed impact
15 Major NZ projects:benefits to learners
Ako Aotearoa National Projects – the running score sheet:
14 of 15 projects have had identifiable impact on practice and / or learners. In total these have had an:
• impact on the practice of 2,400 staff
• 50,000 students have been offered enhanced learning opportunities
What this means for institutions
• Processes for tracking and enabling impact• Understand, lever and strategically request funding
from funders• A longer term investment, projects only notionally
“finish” on submission of the final report • Build a culture focussed on impact• Aspects could be scaled into institutional funding
models
What this means for funding agencies
Ako Aotearoa• Government endorsement
• The Impact Evaluation Conversations are interventions in their own right
• Further leverage by Ako Aotearoa and project teams to enhance impact
Office for Learning and Teaching
The pilot during 2012-13 has been well-received in the
sector.
The model is not scalable directly into the Australian
context.
Work is underway to translate the strengths of the approach
into a scalable set of strategies
Ako Aotearoa has established rubrics for the assessment of future proposals
Rating Project integrity Achieving sustainable change
Excellent For a project to be considered ‘excellent’’, all aspects must be met• The project design and method
has no significant gaps or areas of weakness
• How learner outcomes will be achieved is well reasoned
• The project team has mana relevant to the proposed work
• Capability building is embedded in the project
• The focus of the project is highly valued in New Zealand and internationally
For a project to be considered ‘excellent’’, all aspects must be met• The organisation develops a clear
commitment to act on the outcomes of the work
• Relationships are established with key stakeholders
• The dissemination plan is well developed
• The project team is actively engaged in achieving sustained change for learners beyond the scope of the project and participating organisations
Mapping the complexity of impact
“Necessarily then the academic community ought move towards a critical map of impact networks via a cartography of impact(s).”
(Watermeyer, 2012)
Massey University engineering design students
Peter Coolbear
www.akoaotearoa.ac.nz
Tilly Hinton
[email protected]@matilda_marie
http://sayat.me/TillyHintonhttp://olt.gov.au/secondment-projects
