View
216
Download
0
Tags:
Embed Size (px)
Citation preview
An overview of Assessment
Aim of the presentation
• Define and conceptualise assessment
• Consider the purposes of assessment
• Describe the key elements of a good assessment
• Describe two main types of assessment
Definitions of assessment
• ‘… the process of documenting, often times in measurable terms, knowledge, skills, attitudes and beliefs.’
• ‘The classification of someone or something with respect to the worth’
Other words that can mean “assessment”
• Appraisal
• Categorisation
• Evaluation
• Judgement and value judgement
• Adjudication
• Estimation
In medical educational terms we can think of assessment as the process by which knowledge, skills and behaviours may be tested and judgments made
about competence or performance.
Purposes of Assessment
• Measuring competence
• Diagnosing a student’s/trainee’s problems
• Measuring improvement
• Showing the effectiveness of the curriculum/quality assurance
• Introducing curriculum change
• Identifying effective teaching
• Self-evaluation• Ranking• Motivation for teachers
and learners• Progress testing
(developmental measures of improvement)
• Deriving income
The principles of assessment
What should be assessed?
The curriculum and assessment
The curriculum & the outcomes should define assessments
The ideal assessment fits the curriculum
• In real life there is often a less than perfect match
• The objectives are not fully and transparently defined
• Students will define theirown “hidden curriculum” and may miss elements that are tested
Assessing all elements of the curriculum
A simple model
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.
Knows
Shows how
Knows how
Does
Cognition
Behaviour
Knows
Shows how
Knows how
Does
The assessment pyramid
Knowledge: Written tests
Competence: OSCEs
Performance:Workplace
Knows
Shows how
Knows how
Does
The emerging assessment pyramid
Meta-cognition
Constructing a “good” test
U= Rx Vx Ex Cx A
U= utilityR= reliabilityV= validityE= educational impactC= costsA=acceptability
van der Vleuten (1996) The assessment of professional competence:developments,research and practical implications. Adv Health Sciences Education. 1 (1) 41-67
Reliability:If this test were administered again
would the results be the same?
• REPEATABILITY/REPRODUCABILITY– Would the assessors make the same
judgements?– Would a different set of questions result in a
significantly different score?– Were there other things that may have
influenced the result?
Validity: Does the test measure what it thinks it is measuring?
• Face validity– Does it look appropriate?
• Content validity– Does it only assess what you want to
assess?
• Criterion validity– Is it predictive of future performance?
Educational impact
Curriculum Assessment
StudentTeacher
From: Lambert Schuwirth
Issues that affect educational impact
• The content– What is taught and learned
• The format– Written, clinical
• Scheduling– When things happen
• Regulatory structure– Assessment rules & information
Cost
Balancing the cost/benefit
Acceptance
• Political issues and support within the faculty
• Acceptance by the students
• Perceptions of fairness
Models of assessment
Formative assessment
• Provides information to the candidates about their strengths and weaknesses
Summative Assessment
• A measure of an end point achievement
• Even summative tests can (and should) be formative
• Must be robust and defensible
Assessment guidelines to achieve a high utility index
Assessment guidelines
• Establish the purpose• Define what is to be tested• Blueprint to guide selection of items• Select most appropriate test method/format• Administration and scoring• Standard setting
Newble DI, Jolly B, Wakeford R, eds. The certification and recertification of doctors: issues inthe assessment of clinical competence. Cambridge: Cambridge University Press; 1994.
Using the guidelines at Sheffield• Establish the purpose
• Define what is to be tested
• Blueprint to guide selection of items
• Select most appropriate test method/format
• Administration and scoring
• Standard setting
• To graduate safe/competent junior doctors
• Outcome objectives for core clinical problems
• Assessments of knowledge & skills are blueprinted
• Range of tests used• Centralised assessment
management• Hofstee/borderline methods
Strategic management of assessment
Assessment Committee
Curriculum Committee
Phase 4Phase 3Phase 2Phase 1
Administrators/academics
Assessments used at Sheffield
• Formative– On-line weekly tests– Mini Cex
• Summative– Extended Matching Questions– Modified Matching Questions– Observed long cases– Objective Structured Clinical Examination– Professional behaviour– Assessments of Student Selected Components
Formative assessments
• Weekly on-line tests (EMQs)– Face-to-face feedback– On-line personal results and peer
performance
Formative assessments
• Mini Cex (Mini Clinical Evaluation Exercise)– USA: National Board of Medical Examiners– Performance testing in real practice– Multiple observations/multiple observers– Longitudinal assessment to assess
professional growth– Feedback is inherent in the assessment
method
Mini Cex
Excellent Satisfactory Borderline Unsatisfactory
History taking
Physical examination
Communication skills
Professionalism
Overall clinical competence
Summative assessments
• End of year tests of knowledge– Extended Matching Questions– Modified Essay Questions
• Tests of clinical competence– Observed long cases– OSCE
• Assessment of professional behaviour• SSCs – reports, presentations, essays,
posters, leaflets
Observed long case
• 2 Observed long cases during final clinical attachments
• Successful completion entry requirement for the OSCE
• A “mastery” test – may be repeated
OSCE
• Separated from knowledge test by 4 months• 12 stations
– History - 3 stations – Physical examination - 6 stations– Communication - 2 stations– X-ray interpretation – 1 station
Was: 15 stations 5 mins eachNow: 12 stations 10 mins each
• Checklist and global rating scoring
Professional behaviours
• Longitudinal assessment of professional behaviours to give reliability
• Assessed against outcome objectives and Good Medical Practice (GMC 2001)
• Borderline/unsatisfactory performance triggers interview
• Consistent poor performance may lead to review by Fitness to Practise committee and possible exclusion
Student Selected ComponentsSSCs
• 25% of the overall course
• Beyond the core curriculum (depth +/- breadth)
• Various modes of assessment
• Assesses generic skills, critical analysis, ethics, research skills, clinical understanding