E A R L I S I G 1 M A D R I D A U G 2 0 1 4

IMPROVING WORKPLACE-BASED ASSESSMENT AND FEEDBACKBY LEARNING ANALYTICS AND E-PORTFOLIO

MADRID 2014 SIG 1 CONFERENCE EARLI. AUGUST 27-29.

MARIEKE VAN DER SCHAAFUTRECHT UNIVERSITYTHE NETHERLANDS

www.project-watchme.eu

@Project_WatchMe

Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

EU-PROJECT WATCHME

PARTNERS

1. Utrecht University, NL (www.uu.nl) GENERAL LEAD

2. University Medical Centre Utrecht, NL (www.umcutrecht.nl)

3. Szent Istvan University, Hungary (www.sziu.hu)

4. University of Tartu, Estonia (www.ut.ee)

5. Universitätsmedizin Charité Berlin, Germany (www.charite.de)

6. University of California San Francisco, USA (www.ucsf.edu)

7. Maastricht University, NL (www.maastrichtuniversity.nl) TECHNICAL LEAD

8. Mateum, NL (www.mateum.nl)

9. University of Reading, UK (www.reading.ac.uk)

10. Jayway, Denmark (www.jayway.com)

11. NetRom, Rumania/NL (www.netrom.nl)

Educational partners

Technical partners

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

- Develop complex competences- Integrated in context- Demands long learning trajectories in workplace- Deliberate practice: feedback and reflection

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Feedback that is (Wiggins, 1997):- Timely- Specific- Understandable to receiver- Allow receiver to act on (refine, revice, practice,

retry)

That feeds into students’major feedback questions (Hattie & Timperley, 2007):

- Where am I going? (goals, feedup)- How am I going? (feedback)- Where to next? (feedforward)

WHAT IS NEEDED

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

“Well done!”“Pleasure to supervise!”“Reliable candidate”“Poor fund of content knowledge”“Needs lots of supervision”

UNFORTUNATELY, HOW MANY DAILY FEEDBACK PRACTICES LOOK LIKE

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

CHALLENGE: COMPETENCY BASED EDUCATION AND ASSESSMENT

World-wide wish to restructure professional’s training, focusing on the attainment of specific competencies and changing assessment practices. From 90s onwards.

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

FROM 90S ONWARDS: ELECTRONIC PORTFOLIOS

Portfolios contain selected evidence of performance and products accompanied by reflections.

• Reflective ‘log’ of trainee over longer time span

• Repository of evidence regarding performance etc

• Different instruments and methods

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

ELECTRONIC PORTFOLIOS MIXED SUCCESS

Underuse of potential data and not very well tailored to worksituations (not timely).

Drawbacks:

• Authentic, personal unique character of evidence, difficults scoring.

• Full standardization impossible, asks for interpretation of assessor (no golden standard or external criterion).

• Impacts reliability and validity of the assessment.

• Limited impact on learning

• Low motivation of students

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Example Digital portfolio, example veterinary education

Aggregation of data pointsMini-CEX (clinical evaluation exerc.)Multisource feedbackCase reportsKnowledge testsPersonal development plan

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

CRITICS TO USE OF CB-ASSESSMENT

Too much emphasis on accountability, and outside control by politics and society

Too behavioristic - leads to reductionist checkbox assessment approach

Too theoretical; it remains unclear what competencies are

Competencies cannot be measured validly

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Illustration of critics

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Medical expert

Communicator

Collaborator

Manager

Health advocateScholar

Professional

MOST CRITICS COME DOWN TO THE ANALYTIC APPROACH

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Medical expert

Communicator

Collaborator

Manager

Health advocateScholar

Professional

THE ANALYTIC APPROACH

With nursing staffWith family

With patients

With colleagues

With trainees

…

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Medical expert

Communicator

Collaborator

Manager

Health advocateScholar

Professional

THE ANALYTIC APPROACH

With nursing staffWith family

With patients

With colleagues

With trainees

…

Consultation

Breaking bad newsExplain medicationWith children

With elderly

…

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

WATCHME’S AIM

Improve efficiency and quality problems of workplace-based feedback and assessment by means of a mobile electronic portfolio system, that is enhanced with:

- student models that monitor the learners’ competency development.

- visualisation tools that inform learners, teachers and institutes just in time as well as on an analytical level.

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

ELECTRONIC PORTFOLIO

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

STUDENT IS OWNER

Trainee’s portfolio data remain personal property

They control to whom they share information to

Use for research purposes by means of informed consent

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

ENTRUSTABLE PROFESSIONAL ACTIVITY

Task based instead of construct based approach

Crucial question: would I entrust this candidate unsupervised with this task? (with my sick mother, animal or teach my daughter/son…)

An EPA is a task that an individual can be trusted to perform unsupervised, in a given professional context, once sufficient competence has been demonstrated. International Competency-Based Medical Education Collaborators, March 18, 2014

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Medical expert

Communicator

Collaborator

Manager

Health advocateScholar

Professional

THE SYNTHETIC FRAMEWORK APPROACH

EPA1

EPA2

EPA3

EPA4

EPA5

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

EXAMPLES ENTRUSTABLE PROFESSIONAL ACTIVITY (EPA)

Conducting patient hand-overs

Anesthetic management a patient

Conducting a normal, low risk delivery

Interviewing adolescents regarding high risk health behavior

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

SUPERVISION LEVELS OPERATIONALIZED

1. Acting is not allowed: more observation

2. Acting is allowed under direct supervision: supervision physically present; proactive supervision

3. Acting is allowed under indirect supervision: supervision not physically present, but quickly available if needed; reactive supervision

4. Acting is allowed “unsupervised”:under clinical oversight; distant/backstage supervision; post hoc report

5. Supervision may be provided to juniors (to junior residents, to medical students at level 2 or 3)

A formal acknowledgment happens at level 4

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Developing Entrustable Professional Activities

Title Keep it short; avoid words related to proficiency or skill. Ask yourself: Can a trainee be scheduled to do this? Can an entrustment decision for unsupervised practice for this EPA be made and documented?

Description To enhance universal clarity, include everything necessary to specify: What is included? What limitations apply? Limit the description to the actual activity. Avoid justifications why the EPA is important, or references to knowledge and skills.

Link with a competency framework

Which competency domains apply? Which sub-competencies apply? Include only the most relevant ones. These links serve to justify the EPA and to build observation and assessment methods.

Required know-ledge, skills and attitudes

Which KSA are necessary to execute the EPA? Formulate this in a way to set expectations. Refer to resources that reflect necessary or helpful standards (books, a skills course, etc.)

Information sources to assess progress

Consider observations, products, monitoring of knowledge and skill, multi-source feedback.

Planning level 4 (“unsupervised practice”)

Estimate when during training full entrustment to allow for unsupervised practice is expected to be reached. Acknowledge the flexible nature of this timing. Expectations of entrustment moments can shape an individual workplace curriculum.

Making a formal entrustment decision

How often must the EPA be executed proficiently for Level 4 (unsupervised practice). Who will judge this? Should external experts be involved? What does a formal entrustment look like (signed, documented, publicly announced)?

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

EXAMPLEEPA: "Investigation Ordering and Interpretation"  

To enable an entrustment decision the candidate:..

(Communication) Communicates investigations and results clearly to the patient.

(Advocacy) Assists patients who are unlikely able to navigate the health system in arranging the investigation

(Scholar) Doesn’t order unnecessary screening tests

(Scholar) Doesn’t order tests when the pre-test probability is minimal 

(Professional) Seeks assistance when s/he doesn’t know the answer.

(Professional,Manager) Ensures timely receipt and management of results

   Thanks to Daniel Ince-Cushman (McGill) 10-3-14.

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

COMBINING THE COMPETENCY-FRAMEWORK WITH ENTRUSTABLE PROFESSIONAL ACTIVITIES

Medical expert

Communicator

Collaborator

Manager

Health advocateScholar

Professional

++

+

+

+

+

+

++

++

+

+

+

+

++

++

+

++

++

+

++

+

++

EPA1 EPA2 EPA3 EPA4 EPA5

Assessment

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

EPA-approach used in Family Medicine, Kingston, Ontario

Thanks toJane Griffiths & Karen Schultz

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

GROWTH OF ENTRUSTABILITY OVER TIME

training deliberate professional practice

proficient

expert

competent

advanced

novice

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

CURVES OF ONE TRAINEE

training deliberate professional practice

EPA1

EPA4

EPA2

EPA3

EPA5

Compe-tence

thres-hold

Justified entrustment decisions

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

ANOTHER TRAINEE

training deliberate professional practice

EPA1

EPA4EPA2

EPA3

EPA5

Compe-tence

thres-hold

Justified entrustment decisions

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Example of formalized entrustment decision

From: EPA Handbook of Royal Australian & New Zealand College of Psychiatrists

The International Conference on Residency Education | La conférence internationale sur la formation des résidents

Mehta et al 2013Digital badge as EPA-STAR

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

COMBINE EPA APPROACH WITH LEARNING ANALYTICS

Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013).

Application of probabilistic student models that enable tailored visualised feedback based on multi sorted assessments

Bayesian models provide detailed model of trainees’ entrustability

BAYESIAN NETWORKS1) CONSTRUCTION OF GRAPH REPRESENTING QUALITATIVE INFLUENCES OF

SITUATION MODEL2) ASSIGNMENT OF PROBABILITY TABLES TO EACH NODE IN GRAPH

MULTI-ENTITY BAYESIAN NETWORKS (LASKEY, 2008)

Mfrag as basis concept

Mfrag: parameterized Bayesian network fragment that represents uncertain relationships among a small collection of related hypotheses

Can be instantiated and combined to form complex graphical probability models

Learns from incoming data

http://unbbayes.sourceforge.net/

University of Brasil & George Mason University (Costa & Laskey)

Tool for constructing and querying MEBN and other extended Bayesian network formats

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

MEBN ARCHITECTURE

ePASSdatabases

Temporary BNBackgroundMachine learning engine

REQUEST

BNConstructor

REPLY

BNReasonerstructure

MEBNFragment

Store

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

Architecture

database

Mobile device

Student Model Database

Learning Analytics Engine

-Online Learning

Just in time Feedback

Aggregated Visualisation

Portfolio system(ePASS)

WPB AssessmentGAME

EPA Matrix

ARCHITECTURE (ZOOMED OUT)

E A R L I S I G 1 M A D R I D A U G 2 0 1 4

1.Utrecht University, NL 2.University Medical Centre Utrecht, NL3.Szent Istvan University, Hungary4.University of Tartu, Estonia5.Universitätsmedizin Charité Berlin, Germany6.University of California San Francisco, USA 7.Maastricht University, NL8.Mateum, NL9.University of Reading, UK10.Jayway, Denmark 11.NetRom, Rumania/NL

THANK you for your attention!www.project-watchme.eu