WORKPLACED–B

ASED E–A

SSESSMENT TECHNOLOGY

FOR COMPETENCY–B

ASED HIG

HER MULT

IPROFESSIO

NAL EDUCATIO

N

Participatory Design of a Learning Analytics enhancement E-portfolio: The Human Factor

Eelco Schreurs1, Atta Badii2, Annemarie Camp1, Inge Duimel-Peeters1, Ovidiu Serban2, Daniel Thiemert2, Jaime Costa2, Marieke van der Schaaf3, Jeroen Donkers1, Geraldine Clarebout1

1

2

3

INTRODUCTION

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

2

Workplace learning

Different learning opportunities for

each learner

Encountered cases

Feedback

Same competency level to be achieved

Importance of “good” assessment instruments

INTRODUCTION

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

3

Importance of “good” assessment instruments

Measure required competency level

Considering variety of learning

experiences

(Electronic) Portfolio

• Work completed• Feedback• Progress made• Reflections

=> Longitudinal and multi-faceted information (van Tartwijk, Driessen, van der Vleuten & Stokking, 2007; Sengler & Kanthan, 2012)

www.blendspace.com

INTRODUCTION

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

4

Succes factors

Mentor

Part of institutional assessment procedure

Flexible format

Threads

Availability of time

User friendeliness

(Driessen, van Tartwijk, van der Vleuten and Wass, 2007)

Learning analytics?

adaptivePersona

-lized

(Greller & Drachsler, 2012, p. 44)

LEARNING ANALYTICS WITHIN THE WATCHME PROJECT

Learning analytics as (personal) probabilistic learning models

Based on all information in electronic portfolio

Computerized processing of large amount of data

BUT: also human factor; initiated through interface

(Dron & Anderson, 2009)

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

5

RESEARCH QUESTION

Starting point:

Human-centred design approach usability of e-portfolio

Research Question

What are the needs and objectives of the different stakeholders (‘the human factor’) in relation to a learning analytics enhanced electronic portfolio assessment?

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

6

METHOD

Participants

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

7

Participatory meeting

Anaesthesiology Veterinary Medicine Teacher Education Team Members

1 2 1 3 14 (10 technical, 4 educational)

2 4 1 1 3 (technical)

3 7 (5 physicians, 2 trainees)

- - 9 (technical)

METHOD

DesignParticipatory design strategy (Berns, 2004; Könings, Brand-Gruwel & van Merriënboer, 2010)

Three participatory meetings between May - September 20141st participatory meeting

visualise the landscape in which the WATCHME-project and its stakeholders will exist.

2nd participatory meetingvisualise the landscape for each domain (veterinary medicine, anaesthesiology and teacher education)

3rd participatory meeting=> ideas and requirements for just-in-time feedback=> ideas and requirements for the visualization module=> ideas for building a student model

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

8

RESULTSFirst participatory meeting:

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

9

Technical stake-holders

Develop new FB modules

Integration of different systems

and models

Creation of cross-domain system

Managerial stake-holders

Comparedifferent groups

Hospitals: higher quality,

safety of care,

transparancy

Curriculum improve-

ment

RESULTSFirst participatory meeting:

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

10

Info from different sources / observers

Trainers / assessors

Easy and fast access

to info

360° insight

Trainees

Trans-parancy

Structured feedback conver-sations

Safeguards confidentia-

lity & anonymity

History of learner => focussed

FB

Just-in-time

feedbackLongitu-

dinal monitori

ng

Anywhere and anytime!Just-In-Time feedback

RESULTSSecond participatory meeting:1. Distinction needed between supervisors and assessors

• Supervisors: • Analyze group of students of same cohort • Development of individual student compared to group• Accessible through personal computer

• Assessors:• Quick overview on smartphone• Elaborate overviews on personal computer• Need for notification when assessment needed

2. Definition of different categories of JIT-feedback• reminders/notifications• content feedback• longitudinal feedback

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

11

RESULTSSecond participatory meeting:3. feedback in a quantitative (scores) and qualitative way (narrative feedback)

needed

4. Insight into the learning progress• Over longer period about students’ performance on a general level• Short term for performance for particular activities

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

12

RESULTS

Third participatory meeting1. Focus on JIT-feedback

• Preference for narrative feedback• Possibility to tag narrative feedback• For numeric feedback: comparison with peers or European standard• Draw attention on difference score assessor and self-assessment• Alert when they miss milestone or deadline

2. Rules for not well-performing students• Number of mini-clinical evaluation exercises (mini-CEX) is dropping.• Poor reflective skills: poor narratives, small sentences.• Pattern in recent assessments.• Poor performers seek supervisors who do not give feedback or only

positive feedback (feedback avoidance)

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

13

DISCUSSION AND CONCLUSION

• Participatory meetings allowed to grasp the requirements and expectations of the stakeholders

• System will need to provide personalized feedback

• System will need to be adaptive with respect to: • Device used• Professional domain• Role of the user

• Shortcomings:• Not all target groups participated => convenience sample• Data analysis => time pressure

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

14

ACKNOWLEDGMENTS

• European commission: grant agreement n°: 619349 : ‘WATCHME Workplace-based e-assessment technology for competency-based higher multi-professional learning

• Suzanne Schut (for acting as an interim coördinator)

E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5

15