Promoting)best)assessment)practice ... · EBMA Annual report 2016 – 2017 7 ! Review’of’activities’andachievements’in’2016D2017’ EBMA’participantsmeeting2017’

Promoting best assessment practice in medical education in Europe

Annual report academic year

2016 – 2017

EBMA Annual report 2016 – 2017 2

Contents

Introduction by the President .................................................................................. 3

EBMA Governance ................................................................................................. 4 EBMA membership ................................................................................................ 5

Review of activities and achievements in 2016-2017 .................................................. 7 EBMA participants meeting 2017 .......................................................................... 7

Update Workplace-based assessment working group ............................................... 7 European Knowledge Test ................................................................................... 9

EBMA International Progress Testing ..................................................................... 9 EBMA Scientific Committee ................................................................................ 10

EBMA Training Psychometrics ............................................................................. 11 EBMA Conferences .............................................................................................. 12

EBMA Annual Conference on Assessment ............................................................. 12 EBMA Annual Conference 2018 ........................................................................ 13

Conference ‘Looking ahead in progress testing’ .................................................... 14


Introduction by the President

Welcome to the annual report for this last academic year 2016 -‐ 2017. As you will read, the European Board of Medical Assessors goes from strength to strength with more activities and more partners. I particularly wish to acknowledge the huge contribution that Lesley Southgate has made to the development of the organisation and she has left EBMA in excellent shape.

It was great to see so many of you at the annual conference in Egmond and I look forward to working with you in the next year. Please contact us with any questions, suggestions, ideas and enjoy reading the report.

Introduction by the Director With pleasure we note that EBMA is growing as an assessment community. A community of expertise to promote good assessment. Most memorable was our annual conference. We heard many comments of appreciation of a get-‐together with people having a similar interest. EBMA will continue to support this community with our products, services and meetings. 2017 was a productive year where our foundation became legally embedded. We also had some changes in the Board membership. We thank Professor Lesley Southgate for all the years of leadership. She will remain on Board as our Past-‐President. We look forward working with you again in the near future.

Prof. Adrian Freeman

Prof. Cees van der Vleuten


EBMA Governance EBMA has now completed it’s formal process to become an independent Foundation. The Foundation is chartered in January 2017 in Maastricht, the Netherlands, registered at the Chambre of Commerce in the Netherlands. There are official by laws in Dutch and translated into English. There are several bodies within the foundation, which are the following:

• Board Representing and managing the Foundation

• Council of Audit & Governance Supervision of the Board’s management and day to day running of the Foundation. Gives assistance and advice to the board, on request and on own initiative.

• Council of Participants Sharing knowledge and professional skills. Participants could be appointed for the Foundation’s bodies or committees.

• Technical and Scientific committee(s) Mapping out the strategic policy regarding scientific and operational matters and informing the Board accordingly. Currently there are 3 active committees: European Knowledge Testing/International Progress Testing, Workplace-‐based assessment, Scientific committee.

The picture below presents the organizational and legislative structure of the foundation.

In 2017 the Board was constituted by the following persons: President – Prof. Lesley Southgate Director – Prof. Cees van der Vleuten Treasurer – Prof. Adrian Freeman


Board member – Prof. José Miguel Pêgo In November 2017, the following changes have been mad and the board is constituted as followed: President – Prof. Adrian Freeman Director: Prof. Cees van der Vleuten Immediate Past president – Prof. Dame Lesley Southgate Treasurer / Board member – Prof. José Miguel Pêgo The Council of Audit and Governance: Chair – Prof. Pauline McAvoy Member – Prof. Debbie Jaarsma The Council of Participants:

1. Belgium: Ghent University 2. Finland: University of Helsinki 3. Georgia: David Tvildiani Medical University 4. Netherlands: University Medical Center Groningen 5. Netherlands: Maastricht University 6. Poland: Jagiellonian University 7. Poland: Medical Universit of Lodz 8. Portugal: University of Minho 9. Spain: Institutio Fundación Practicum 10. UK: University of Exeter 11. UK: Plymouth University

EBMA membership

Individual membership In 2017, EBMA launched during the EBMA annual conference the possibility to become an individual (affiliated) member. This membership is for anyone who has a strong interest in medical education assessment and who is not associated with an organisation/institution which is an EBMA member. It allows individuals to sustain a relationship with EBMA as individual members will be part of a community of assessment professionals. In 2017, EBMA welcomed 8 individual members and is sure that more individuals will subscribe. The objectives of the EBMA Individual membership programme are:

1) To maximize the participation of individuals within the activities of EBMA. 2) To create a broader community of assessment specialist or persons interested in medical

education assessment within Europe. 3) To link individuals with (other) assessment specialist throughout the world and provide a

meeting place to share interest in medical education assessment.


The first meeting with all our individual members will be scheduled during the EBMA conference in Braga in November 2018.

Institutional membership A number of institutions have acknowledged the benefits to being a member of EBMA. Several institutional members make use of the EBMA European Knowledge Test and/or have implemented the EBMA International Progress Test. The main reason to become a member is the community aspect of EBMA where joint initiatives between participants are being executed and events organized. Institutional membership is also designed to assist schools and universities with developing, supporting and sustaining their assessment programme. Several visits of EBMA have been made to University Medical Centre Groningen, Medical University of Lodz and David Tvildiani Medical University to train faculty members and provide advice on topics such as workplace-‐based assessment, EBMA international progress testing and evidence-‐based assessment. Detailed information about EBMA membership can be found on our website: https://www.ebma.eu/joiningebma/


Review of activities and achievements in 2016-‐2017

EBMA participants meeting 2017 On Monday November 13th the annual meeting of the Council of Participants took place in Egmond aan Zee, the Netherlands. The meeting was limited to two hours and subjects included current and future EBMA conferences, ongoing projects, the future of EBMA and funding opportunities. Moreover, the meeting was discussed the needs of all participants on training in assessment, wishes with respect to new initiatives and the new organization of EBMA in the form as a Foundation.

Update Workplace-‐based assessment working group

Aims The aim of the Workplace-‐Based Assessment working group is to produce a WBA-‐course on line enabling workplace-‐based assessment (WBA) of medical performance, communication, professionalism and collaboration in authentic working settings by observation of medical consultations and actual behaviour of students / trainees within a team to provide narrative feedback for learning. Both assessment procedures and instruments have been developed for use in each European country.

Definition WBA WBA has been defined as a direct observation in authentic clinical environments of a specific element of interconnected competencies performed by a (learning) professional. WBA is typically brief, criterion-‐based and low-‐stakes prompting an immediate feedback encounter. WBA is an essential part of programmatic assessment, a systematic method to collect data across learner’s training from multiple domains of competence. WBA supports educators to tailor future instruction to assist with remediation or acceleration of learning. So, WBA assesses performance on the “does-‐level” of Miller’s pyramid. It has to be undertaken in the workplace to facilitate learning through constructive feedback. It should be undertaken frequently to increase reliability. WBA identifies


areas for development to help trainees and trainers. If desired, WBA contributes to evidence for summative judgements on the progress of the trainee.

The benefits of WBA within EBMA The blueprint of the EBMA WBA has been based on both CanMEDS and Good Medical Practice competency frameworks. WBA focuses on observable performance and is based on specific criteria. It encompasses integrated skills, knowledge, behaviour and attitudes (competencies) regarding medical performance, communication with patients, collaboration within a team and leadership. WBA on line samples across important workplace tasks or activities (including Entrustable professional activities) and provides descriptors / profiles to aid the assessor’s judgement and feedback.

EBMA WBA on line encourages trainee/trainer dialogue, reflection by trainee / trainer, and provides a personal trajectory of progress by a trainee to improve practice.

Products of the WBA working group – work in progress – We have developed a number of slides (PPT) for our clients to support development and implementation of WBA in their country. Moreover, we produce assessment instruments and manuals:

• Video assessment of clinical encounters (VAC). • The clinical reasoning and reflecting form (CRRF), helps trainee in the reflecting process, a

worthwhile add-‐on to the tool and the process of both the VAC and the Mini-‐CEX. • Mini-‐CEX: instrument for direct observation of clinical encounters • Multi Source Feedback (MSF) • E-‐Portfolio: for reflection, feedback, uploading video encounters etc.

The clinical reasoning and reflecting form (CRRF) In September 2017 we presented the Video Assessment of Clinical performance and the clinical reasoning and reflecting form (CRRF) at the AMEE conference in Helsinki. It is an instrument to assess patient related skills by videoing real life consultations. We showed an online video system (within EPASS) for submitting the consultations by the candidate, allowing assessors to review, score, provide feedback and make a decision about the candidate’s performance. The overall conclusion was that the instruments fulfills needs, participants were positive about the content of the instruments and about using it. It helps to give feedback in a proper and complete way. Remarks were on legal and on cultural matters. It is ready for use by partner institutions of EBMA.

In November 2017 we piloted the Mini-‐CEX instrument at the EBMA Conference in Egmond and we presented our Multi Source Feedback instrument as well. Developments will be made to improve content and clarity of scales used.

Future plans Next year a pilot is planned with non-‐GP’s and non-‐UK doctors for the EBMA WBA instruments and the working group will propose a market plan on how to use, introduce and spread the instruments.

Members of the WBA working group

Dr. Paul Ram (Maastricht University, NL) – Chair


Dr. Thomas Gale (Plymouth University, UK)

Agata Stalmach-‐Przygoda (Jagiellonian University, Poland)

Magda Szopa (Jagiellonian University, Poland)

Prof. Pauline McAvoy (London, UK)

Prof. Lesley Southgate (London, UK

European Knowledge Test

The European Knowledge Test is a useful assessment tool to detect strong points of a medical school’s undergraduate curricula by comparing the results of other students within the pool of participating institutions. The University of Helsinki is one of the European schools who use this EBMA test to find areas for further development and to give their students feedback within a European context. During the EBMA Conference, an Angoff procedure for the standard setting of the EKT has taken place, in order to calculate the impact of the recently included items on the test cut score.

Also in the recent EBMA Conference, the University of Helsinki medical school gave an oral presentation about the usefulness of the European Knowledge Test (EKT) as a benchmarking tool. By comparing the results of their students with the performance of the pool of participating institutions, the stakeholders at Helsinki were able to detect the strong points of their curriculum and the areas that need further development. The results have confirmed their earlier perceptions. But now they were able to have much more precise insights, as now they have access to the magnitude of the differences in performance levels between their students and the students from other European universities.

The magnitude of such differences (i.e. the effect sizes) are expressed in terms of Cohen’s d coefficients, which provide information in terms of how many standard deviations their students are above or below the pool of participating schools. Such coefficients are calculated for the total test scores but also for more than 40 subscores, divided in four axes: clusters (clinical sciences, basic sciences, behavioral sciences and statistics), traditional disciplines (such as Pediatrics and Internal Medicine) and system-‐based categories (e.g. respiratory system and musculoskeletal system).

A few weeks ago dr. Carlos Collares visited Helsinki for some workshops and work meetings with the staff and they unanimously agreed the EKT was a useful tool for quality assurance. One good development about the EKT is that the test is now also offered as an online computer-‐based test, allowing a shorter testing time for students. As a consequence of this implementation, test results can now be processed more quickly too.

EBMA International Progress Testing

There are some important updates regarding EBMA International Progress Test in the period covered by this report. Finland, represented by the University of Helsinki has joined EBMA International Progress Test initiative after a successful pilot application of the computerized adaptive version of


the test. David Tvildiani Medical University, in Tbilisi, Georgia, has also done a pilot application of the adaptive International Progress Test with their students.

Concurrently, under the auspices of the Educational Institute of the Faculty of Health, Medicine and Life Sciences, Maastricht University in the Netherlands and Suleiman Al Rajhi University in Saudi Arabia had the opportunity to apply a computerized adaptive progress test. This implementation has had supervision and support from Maastricht University staff, some of which are also part of EBMA. To our knowledge, it is the first computerized adaptive progress test used with both formative and summative purposes.

Meanwhile, in Mexico, Monterrey Tec, one of the most prestigious higher education institutions in Latin America, has completed three years of successful use of the computerized adaptive International Progress Test. We expect that EBMA and Monterrey Tec shall strengthen the relationship in the years to come.

In all institutions cited above there were great evaluations by students and staff as well as very high degrees of reliability for all academic years (0,90-‐0,96). The adaptive algorithm used in the test dynamically selects the difficulty of the items according to the level of performance demonstrated by the test taker. This allows the increase and the homogeneity of reliability estimates but also the reduction of test length by approximately 50%, thus also reducing student fatigue. More importantly, the adaptive approach to progress testing to can be construed as a better alignment of progress testing initiatives to modern learning theories, such as constructivism. The adaptive nature of the test can be construed a rough, imperfect, but yet useful approximation to Vygotsky’s zone of proximal development.

Conscious that one size does not fit all, EBMA is aimed to promote a greater level of flexibility and local customization of the International Progress Test content, including the possibility of using either the paper-‐based fixed version or the computerized adaptive version of the International Progress Test, the inclusion of translated items to the local language, as well as a greater integration between partners for item exchange and benchmarking initiatives. Our next goals also include scientific research, particularly the effects of adaptive progress testing on student motivation and learning. And last, but not least, EBMA is focused on expanding the community of participating schools, so that more students can benefit from this innovative assessment tool of, for and as learning, while continuing to provide an increasingly cost-‐effective solution to medical schools worldwide.

EBMA Scientific Committee The Board of EBMA has appointed Dr. Amir Sam to lead the Scientific Committee. Dr Amir Sam is an individual member of EBMA and a Consultant Endocrinologist at Hammersmith and Charing Cross Hospitals.. He completed his PhD in Neuroendocrinology as a Wellcome Trust Clinical Research Fellow at Imperial College London. Dr. Sam is now exploring with the Board possible topics for potential research activities.


EBMA Training Psychometrics

EBMA has noticed a growing demand for more practical, hands-‐on activities involving Psychometrics. Many people wanted not only to interpret evidence on validity and reliability, but also to carry out adequate analyses themselves.

The EBMA Course on Psychometrics applied to healthcare professions education has been created to fulfil a gap: a course that gives more than just the theoretical aspects, thereby enabling participants to carry out many analyses themselves and to interpret the findings.

Our first edition of the course in July 2017 had 25 participants. They came from different countries, had different ages and different levels of expertise. It was encouraging to see the more experienced students helping their colleagues. And it was wonderful to see participants from all age-‐groups able to run item response theory analyses in R, without any prior contact with R, nor any other psychometric software. We especially appreciated that our participants were able to understand the limitations of Psychometrics while getting a glimpse of the most recent, emerging, Psychometric techniques too.

Testimonial – Training participant 2017

Dr. Rachmad Swarwo Bekti, Faculty of Medicine Universitas Brawijaya, Indonesia

‘I recommend to everybody who is interested in psychometric application in medical education to join the training, especially those who still knew nothing on this subject. I learned in practice of using RStudio to do essentials psychometry analysis (promptly in IRT & EFA) and practicing G-theory analysis by using most affordable software. This training has also made me confident enough to provide interpretation of any assessment result. The introduction section and practical sessions would shift the participant from novice to advance beginner status.’


EBMA Conferences EBMA Annual Conference on Assessment About 150 assessment professionals from 25 different countries came together for the 2nd EBMA conference (14-‐15 November) in the Netherlands to share knowledge on the latest trends in assessment in medical education. The main subject was Transformative Assessment and participants attended the conference to explore opportunities for assessment developments, to look how assessment can be transformative and to find out about current issues and initiatives in assessment in medical education.

The keynote lecture by Prof. Shiphra Ginsburg considered the value of narrative feedback during assessment processes and the keynote of Prof. Janke Cohen-‐Schotanus addressed how assessment can be used to optimize student learning. There was time for discussion and reflection which led to a constructive atmosphere indicated by the participants. During both days a range of workshops, poster presentations and thematic presentations were given and at the end of each day a symposia and debate was organized around the theme transformative assessment. The results of a feedback survey completed by the participants showed that in general the conference was really focused on assessment, friendly, small-‐scaled and visited by professionals throughout the world with a similar interest. The conference was unique because of the specific topic of assessment and the networking opportunities with a limited group of people. With respect to the content, participants mentioned that this conference kept them up-‐to-‐date since assessment is moving towards different directions. It was useful for developing strategies, change management and to explore the fundamentals and principles of assessment


Points for improvement as a result of the survey are related to the set-‐up of the programme, which is more room for research papers about similar topics on assessment, a meet the expert session and the inclusion of a social programme. The organizing committee of 2018 has been formed and started her first activities. All results of the survey will be taken into account by organizing the next conference.

EBMA Annual Conference 2018 The EBMA Conference 2018 will take place at the University of Minho, Braga, Portugal on November 22, 23 and 24 2018 and the main subject is ‘Technology enabled assessment’.

Technology is an integrant part of our lives and its presence is becoming ever more embedded in the context of Medical Education. Whether using technology to enhance the experience of traditional learning methods or to the application of large scale education such as Massive Open Online Courses, technology has reshaped the landscape of Medical Education. Assessment has not been an exception to this change. There has been a progressive shift from traditional paper and pencil testing to electronic formats that have greatly improved the administration of tests and provided means of immediate feedback that were unthinkable before. Additionally, other forms of assessment have emerged that were impossible without technology. Simulation based assessment, computer adaptive testing or computer generated items are just some examples of how technology has reshaped older methods but there is a whole new field that is starting to be integrated in the way we assess and monitor student’s performance. Concepts such as the Internet-‐of-‐Things where smart devices are interlinked to monitor and enhance students’ experience or the use of Artificial Intelligence to improve the knowledge that we can gather from traditional data and the new devices are just some of the new avenues in assessment.

The conference will present an exciting program with two fantastic keynote speakers:

• Andre De Champlain (Medical Council, Canada), an expert in psychometrics and the use of automatic item generation

• Marco Antonio Carvalho-‐Filho (Groningen University, The Netherlands), who has developed an innovative program using e-‐learning and social networks in the context of emergency medicine education.


Conference ‘Looking ahead in progress testing’ EBMA organizes in collaboration with University of Exeter Medical School a one-‐day conference titled ‘Looking ahead in progress testing’ on June 22th 2018 in Exeter, UK.

This is an international conference with a line-‐up of leading expert speakers in the field of medical assessment.

The keynote speakers are:

• Professor Cees van der Vleuten, Professor of Education, Maastricht University, The Netherlands

• Professor Adrian Freeman, Professor of Medical Education, University of Exeter Medical School, UK

• Assistant Professor Carlos Collares, Maastricht University, The Netherlands

This conference is open to all and those experienced with Progress Testing (PT) can come and present academic findings; and those new to PT can learn more about it and present their academic questions.

The organizing committee of 2018 consist of: • Prof. José Miguel Pêgo, University of Minho,

Portugal (Chair) • Prof. Cees van der Vleuten, Maastricht University,

the Netherlands • Prof. Janusz Janczukowicz, Medical University of

Lodz, Poland • Dr. Thomas Gale, Plymouth University, UK • Annemarie Camp MSc, European Board of Medical

Assessors • Danielle Townsend MSc, European Board of

Medical Assessors In 2019 the EBMA annual conference is planned prior to the AMEE conference.


More information can be found here: http://medicine.exeter.ac.uk/progresstestingconference/

Promoting best assessment practice in medical education in Europe EBMA European Board of Medical Assessors Universiteitssingel 60 6229 ER Maastricht P.O. Box 616, 6200 MD Maastricht The Netherlands Phone: +31 43 38 85 733 E-‐mail: [email protected]

WWW.EBMA.EU

Documents

Promoting)best)assessment)practice ... · EBMA Annual report 2016 – 2017 7 ! Review’of’activities’andachievements’in’2016D2017’ EBMA’participantsmeeting2017’