Welcome

Synthesis Report on Assessment and Feedback with Technology Enhancement

Introduction to the report

Lester GilbertGary Wills

Denise WhitelockVeronica Gale

Synthesis report on assessment and feedback with technology enhancement (SRAFTE)

10th November 2010

Project purpose

Consult the academic community on useful references Seminar series Survey Advisors Invited contributors

Prioritise evidence-based references Synthesise main points For readers:

Academics using technology enhancement for assessment and feedback Learning technologists Managers of academic departments

Kirkpatrick

Level 1Reaction: opinions on look and feel, enjoyment, value, etc

Level 2Learning: able to demonstrate new understanding or skill

Level 3Performance change: new understanding or skill leads to demonstrable improvements in outcomes or outputs

Level 4Impact: improvements in outcomes lead to demonstrable impact upon team, section, department, group, organisation, … Kirkpatrick, D. (1976). Evaluation of training. Training and Development Handbook: A Guide to Human Resource Development, 2, 18-11.

Evidence categories

1a – peer reviewed and experimental method 1b – peer reviewed and effect sizes 2 – peer reviewed and quantified evidence 3 – case studies & frameworks in peer-reviewed journals and conference proceedings 4 – case studies in non-peer reviewed reports and guidance from experienced practitioners

Count

10

7

12

49

40

Total 118

Structure of the report

Whitelock et al, (2006)e-Assessment: Case Studies of Effective and Innovative Practice

Goals

Design

Creation

Testing

Piloting

Delivery

Capturing student responses

Feedback to students

Evaluation

Feedback to stakeholders

Themes

Pedagogy first, an integral part of the learning framework Literature focused on goals and design rather than tools Technology as an enabler:

More effective way of delivering learning design Allows new ways of capturing and comparing for assessments e.g.,

portfolios, group work, peer assessment For tutors providing more detailed feedback and feed forward Prompt delivery of feedback learning gains, encourage self-regulation

and reflection, can save tutor time, human touch Can save HEI time and cost after initial set-up and subject to underlying

drivers Some challenges to negative assumptions about technology Investment needed in staff development and support, creation and

delivery processes and QA

Technology

Large classes Stable content Significant risk reduction Significant cost reduction Lower confines of Bloom’s taxonomy Geographical & temporal dispersion

Success indicators1

1 Gilbert and Gale (2007)

Example contents

Tutors: challenging assumptions about disadvantages of technology use Dermo (2009), “e-Assessment and the student learning experience: A survey of student

perceptions of e-assessment” Evidence category 1b Use of technology did not disadvantage older or female students, was not found to be stressful for

students to use and was trusted by students

Learning technologists: answer matching accuracy Jordan and Mitchell (2009), “e-Assessment for learning? The potential of short-answer free-text

questions with tailored feedback” Evidence category 1a Answer matching demonstrated to be of similar or greater accuracy than specialist human markers

Senior Managers: cost savings and institutional change Ruedel, Whitelock, and Mackenzie (2007), “Key factors for effective organisation of e-assessment” Evidence category 3, patterns of facilitating factors and organisational structures

Main conclusions

Dearth of experimentally valid empirical evidence (category 1a)

Many descriptive case studies (categories 3 & 4)

Need more evidence at Kirkpatrick levels 2, 3, and 4

It’s the pedagogy, not the technology, that makes the difference

Nothing very surprising

Any questions? Comments?

Thanks!