EFFICIENCY AND EFFICIENCY AND EFFECTIVENESS IN EFFECTIVENESS IN

ASSESSMENTASSESSMENTStrategies for Streamlining AssessmentStrategies for Streamlining Assessment

Win HornbyWin HornbyTeaching FellowTeaching Fellow

The Robert Gordon UniversityThe Robert Gordon University

ObjectivesObjectives

To look at the changing environment within To look at the changing environment within which assessment now takes placewhich assessment now takes place

To look at a framework for evaluating To look at a framework for evaluating Efficiency and EffectivenessEfficiency and Effectiveness

To look at some empirical data based on a To look at some empirical data based on a survey of assessment practice in my own survey of assessment practice in my own universityuniversity

To propose a number of strategiesTo propose a number of strategies

@ Win Hornby 2004

The Importance of assessment?The Importance of assessment?

““Students can with difficulty Students can with difficulty escape from the effects of poor escape from the effects of poor

teaching.......teaching.......they cannot ( by definition if they they cannot ( by definition if they

wish to graduate) escape the wish to graduate) escape the effects of poor assessment”effects of poor assessment”

Boud (1995:35)Boud (1995:35)

@ Win Hornby 2004

Time spent on Assessment?Time spent on Assessment? RGU Annual Assessment Survey 2003RGU Annual Assessment Survey 2003 5-6 minutes per student-credit point per annum!5-6 minutes per student-credit point per annum! 75-90 minutes per student per annum taking a 75-90 minutes per student per annum taking a

standard 15 credit modulestandard 15 credit module Gross this number up, university of 10,000 Gross this number up, university of 10,000

students, 4 year programmes, 120 credits p.astudents, 4 year programmes, 120 credits p.a Collect in all assessments in one yearCollect in all assessments in one year Get one member of staff to mark it, working 9 till 5 Get one member of staff to mark it, working 9 till 5

365 days a year!365 days a year!

It would take It would take 35 years35 years to mark! to mark!

@ Win Hornby 2004

Aims of AssessmentAims of Assessment

Assessment has four main roles: Assessment has four main roles: formativeformative,, to provide support for future learning; to provide support for future learning; summativesummative,, to provide information about to provide information about

performance at the end of a course; performance at the end of a course; certificationcertification,, selecting by means of qualification selecting by means of qualification

andand evaluativeevaluative,, a means by which stakeholders can a means by which stakeholders can

judge the effectiveness of the system as a whole.judge the effectiveness of the system as a whole.

@ Win Hornby 2004

The Changing EnvironmentThe Changing Environment

Changing role of the stakeholdersChanging role of the stakeholders Changing resource baseChanging resource base Changing course architectureChanging course architecture

@ Win Hornby 2004

The Changing EnvironmentThe Changing Environment

How have we adapted to this changing How have we adapted to this changing environment?environment?

What changes in assessment practices?What changes in assessment practices? Research Evidence?Research Evidence?

@ Win Hornby 2004

Rust (2000)Rust (2000)“….although the prevailing orthodoxy in UK higher education is now to describe all courses in terms of learning outcomes, assessment systems have not changed.

Under current arrangements, rather than students having to satisfactorily demonstrate each outcome (which is surely what should logically be the case….) marking still tends to be more subjective with the aggregation of positive and negative aspects of the work resulting in many cases in fairly meaningless marks being awarded with 40% still being sufficient to pass”

Research EvidenceResearch Evidence

Based on a survey of assessment practices in UK Based on a survey of assessment practices in UK universities in 1999 by Brown and Glasner they universities in 1999 by Brown and Glasner they found:found:

90% of the assessment of a typical British Degree 90% of the assessment of a typical British Degree consists of examinations PLUS tutor marked consists of examinations PLUS tutor marked reports or essaysreports or essays

Recent evidence based on 2003 Survey of Recent evidence based on 2003 Survey of assessment practices in my own universityassessment practices in my own university

Shows the pattern changing, moving away from Shows the pattern changing, moving away from “typical model”“typical model”

More assessment done by coursework onlyMore assessment done by coursework only

@ Win Hornby 2004

RGU Survey of Assessment RGU Survey of Assessment Methods 2003Methods 2003

Exam Only

CW & Exam

CW Only

Whole University

3.1 43.6 49.7

Faculty of Design & Tech

0 50 50

Faculty of Health & Soc Care

6.6 29.5 55.7

Faculty of Mgt / ABS

1.8 56.4 40

The Trade OffThe Trade Off

EffectivenessEffectiveness– Encouraging Staff to experiment with alternative Encouraging Staff to experiment with alternative

assessment modesassessment modes

EfficiencyEfficiency– Resource pressuresResource pressures– Coping with the Changing EnvironmentCoping with the Changing Environment

@ Win Hornby 2004

Effectiveness DefinedEffectiveness Defined To what extent are the methods used To what extent are the methods used

educationally valid?educationally valid? To what extent are the assessment methods To what extent are the assessment methods

used closely linked with used closely linked with desired skills and desired skills and competencescompetences? ?

Are the assessment methods Are the assessment methods “constructively “constructively aligned”aligned” to the stated outcomes to use Biggs to the stated outcomes to use Biggs (1997) phrase? (1997) phrase?

Does the assessment method match the Does the assessment method match the task task and outcomes?and outcomes?@ Win Hornby 2004

Effectiveness DefinedEffectiveness Defined Is there Is there over-reliance on just one modeover-reliance on just one mode of of

assessment such as formal unseen assessment such as formal unseen examinations?examinations?

Are students Are students overloadedoverloaded thus encouraging thus encouraging coping strategies which lead to what coping strategies which lead to what Entwistle has described as Entwistle has described as “surface” as “surface” as opposed to “deepopposed to “deep” learning? (Entwistle 1981)” learning? (Entwistle 1981)

Do the various stakeholders Do the various stakeholders understand the understand the criteriacriteria employed in the assessment method employed in the assessment method and what they are designed to assess?and what they are designed to assess?

@ Win Hornby 2004

Are our criteria clear?Are our criteria clear?

RGU Assessment Survey ResultsRGU Assessment Survey Results 69% of modules always have criteria by 69% of modules always have criteria by

which students are assessedwhich students are assessed 20% of modules “more often than not” have 20% of modules “more often than not” have

criteriacriteria 7% occasionally use criteria7% occasionally use criteria 4% never have criteria!4% never have criteria!

@ Win Hornby 2004

Efficiency DefinedEfficiency Defined What are the What are the educational and opportunityeducational and opportunity costscosts

of each method in terms of staff time, resources of each method in terms of staff time, resources etc.?etc.?

What are the costs of systems to ensure What are the costs of systems to ensure fidelityfidelity in in the assessment method used?the assessment method used?

What are the What are the administrativeadministrative costscosts of different of different methods?methods?

What are the costs to ensure assessment is What are the costs to ensure assessment is reliable and free from biasreliable and free from bias??

What are the costs of complying with various What are the costs of complying with various stakeholder demands on transparencystakeholder demands on transparency??

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

“STAR”

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

“STAR”

“DOGS” Or

WOTS

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

ROLLS ROYCE

“STAR”

“DOGS” Or

WOTS

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

ROLLS ROYCE

“STAR”

“DOGS” Or

WOTS

?OLD DOUBLE

DECKER BUS?

@ Win Hornby 2004

Are we Effective and Efficient in our Are we Effective and Efficient in our Assessment?Assessment?

Critical Incident Accounts

Learning Logs/ Diaries

Self Assessment

Computer Aided Assessment

Peer Assessment

Work-Based Learning

Portfolios

Case StudiesLearning ContractsAssignmentsProjects/ Dissertations

Annotated BibliographiesBook “Reviews”

EDUCATIONAL EFFECTIVENESS

RESOURCEEFFICIENCY

High

Low

Low High

Unseen Examinations

@ Win Hornby 2004

Consequences of overassessmentConsequences of overassessment Poor FeedbackPoor Feedback “ “ When it comes to giving feedback some When it comes to giving feedback some

staff stop at nothing!”staff stop at nothing!” Late feedbackLate feedback ““Feedback is like fish it goes off after a Feedback is like fish it goes off after a

week!”week!” Formative Assessment sacrificedFormative Assessment sacrificed Students cut classes/tutorialsStudents cut classes/tutorials Students work strategicallyStudents work strategically Students look for “short cuts” including plagiarising Students look for “short cuts” including plagiarising

work and personatingwork and personating Little meaningful learning takes placeLittle meaningful learning takes place

@ Win Hornby 2004

Strategies for Streamlining Strategies for Streamlining AssessmentAssessment

1.1. Strategic Reduction of Summative Strategic Reduction of Summative assessmentassessment

2.2. Front-end loadingFront-end loading

3.3. In Class assessmentIn Class assessment

4.4. Self and Peer assessmentSelf and Peer assessment

5.5. Group AssessmentGroup Assessment

6.6. Automated Assessment and feedbackAutomated Assessment and feedback

@ Win Hornby 2004

Option 1. Strategic Reduction of Option 1. Strategic Reduction of Summative AssessmentSummative Assessment

Reduce the instances of assessment?Reduce the instances of assessment? Exemptions from exams on basis of coursework Exemptions from exams on basis of coursework

performance?performance? Assess learning outcomes once only?Assess learning outcomes once only? Combine assessments across modules?Combine assessments across modules? Abolish resit examinations and reassess Abolish resit examinations and reassess

differently?differently? Mechanisms for balancing types of assessment Mechanisms for balancing types of assessment

between modules?between modules? Timetabling assessments?Timetabling assessments?

@ Win Hornby 2004

Option 2. Front-end loadingOption 2. Front-end loading

Coursework briefing sessions?Coursework briefing sessions? Discuss, explain and “unpack” criteria?Discuss, explain and “unpack” criteria? Get students to “engage” with the criteria?Get students to “engage” with the criteria? Get students to assess previous cohort’s Get students to assess previous cohort’s

work?work? Allow students input into deciding the Allow students input into deciding the

criteria?criteria? Rust, Price and O’Donovan Study (2003)Rust, Price and O’Donovan Study (2003)

@ Win Hornby 2004

Option Option 3. In Class assessment3. In Class assessment Marked/graded in class by students?Marked/graded in class by students? Build in several of these for coursework?Build in several of these for coursework? Gow (quoted in Hornby (2003) First year Gow (quoted in Hornby (2003) First year

Engineering students @RGUEngineering students @RGU Reported:Reported: Improved attendance ratesImproved attendance rates Improved motivation to learnImproved motivation to learn Higher retention ratesHigher retention rates Higher pass rates in the final examination, Higher pass rates in the final examination,

hence fewer resits to markhence fewer resits to mark Improved second year performance because Improved second year performance because

underpinning first year knowledge betterunderpinning first year knowledge better@ Win Hornby 2004

Option 4. Self and Peer assessmentOption 4. Self and Peer assessment

Some scepticism that students can be Some scepticism that students can be trusted to do this for themselvestrusted to do this for themselves

Reliability of Assessment?Reliability of Assessment? Survey of Research Evidence?Survey of Research Evidence? Tutor/Student Correlations Low (r=0.21)Tutor/Student Correlations Low (r=0.21) High Variability between tutor ratings and High Variability between tutor ratings and

student ratingsstudent ratings Students over-rate themselves?Students over-rate themselves?

@ Win Hornby 2004

Self and Peer assessmentSelf and Peer assessment

Self Assessment of 48 honours year Accounting StudentsSelf Assessment of 48 honours year Accounting Students Using Grade Related CriteriaUsing Grade Related Criteria DimensionsDimensions

– PresentationPresentation– ResearchResearch– Knowledge and UnderstandingKnowledge and Understanding– AnalysisAnalysis– EvaluationEvaluation

““Unpacked” these with briefing and online discussion forumUnpacked” these with briefing and online discussion forum Results?Results? ““Tutors, Who Needs them?”Tutors, Who Needs them?”

@ Win Hornby 2004

Self and Peer assessmentSelf and Peer assessment

Using GRC and some “front end loading” we Using GRC and some “front end loading” we found:found:

High Correlation Tutor Students RatingsHigh Correlation Tutor Students Ratings High Agreement between students ratings High Agreement between students ratings

and tutor ratingsand tutor ratings No evidence of over-rating by studentsNo evidence of over-rating by students Evidence that students found the experience Evidence that students found the experience

educationally valuableeducationally valuable

@ Win Hornby 2004

@ Win Hornby 2004

@ Win Hornby 2004

Self and Peer assessmentSelf and Peer assessment

The Educational Value of Self AssessmentThe Educational Value of Self Assessment Students feedback indicated benefits from Students feedback indicated benefits from

the self assessment process but to the self assessment process but to paraphrase:paraphrase:

““Why was this not done at an earlier Why was this not done at an earlier stage in the degree programme! Why stage in the degree programme! Why wait until my final year to do this? We wait until my final year to do this? We should be doing this kind of thing from should be doing this kind of thing from Day1”Day1”

@ Win Hornby 2004

Option 5. Group AssessmentOption 5. Group Assessment

Potential to be very effective and efficientPotential to be very effective and efficient But is it reliable?But is it reliable? What about the “hassle” factor?What about the “hassle” factor? The free rider problem?The free rider problem?

@ Win Hornby 2004

Group AssessmentGroup Assessment

Barnes (quoted in Hornby (2003))Barnes (quoted in Hornby (2003)) Third Year Hospitality Management StudentsThird Year Hospitality Management Students Reports on using an evaluation instrument for both self Reports on using an evaluation instrument for both self

and peer assessmentand peer assessment Administered electronically via Question Mark PerceptionAdministered electronically via Question Mark Perception Assesses a number of different dimensions of individual Assesses a number of different dimensions of individual

and group work such as and group work such as Attendance, ideas generation, contribution to group Attendance, ideas generation, contribution to group

report, knowledge and skill acquisition, effortreport, knowledge and skill acquisition, effort Also an anonymous assessment of the work of others Also an anonymous assessment of the work of others

which was fed back to each studentwhich was fed back to each student Not only did student get a grade but also got a score on Not only did student get a grade but also got a score on

each of the identified dimensions and a reporteach of the identified dimensions and a report

@ Win Hornby 2004

Option 6. Automated Assessment Option 6. Automated Assessment and Feedbackand Feedback

Automated FeedbackAutomated Feedback Automated AssessmentAutomated Assessment

@ Win Hornby 2004

Automated FeedbackAutomated Feedback

Checklists and Statement BanksChecklists and Statement Banks RGU Case Studies in Streamlining AssessmentRGU Case Studies in Streamlining Assessment

– Checklists in Applied SciencesChecklists in Applied Sciences– Statement Banks in the Economics of Tax ModuleStatement Banks in the Economics of Tax Module– Grade related statements on each of the dimensionsGrade related statements on each of the dimensions– Cut and Paste + a bit of personalisationCut and Paste + a bit of personalisation– E-mail feedback in real-time E-mail feedback in real-time – ““Quick and dirty”Quick and dirty” v v “Clean and slow“Clean and slow”?”?

Whole Class feedback online via iNETWhole Class feedback online via iNET

@ Win Hornby 2004

Automated AssessmentAutomated Assessment

Cooper (quoted in Hornby (2003))Cooper (quoted in Hornby (2003)) Bsc (Hons) Sports Science students in the School of Bsc (Hons) Sports Science students in the School of

Health Sciences.Health Sciences. Self Assessment linked to video clips of particular Self Assessment linked to video clips of particular

movements of the body to identify key muscles and jointsmovements of the body to identify key muscles and joints Required to identify muscles producing the movements and Required to identify muscles producing the movements and

the type of muscle work involved.the type of muscle work involved. Stop/ Start/Replay facilityStop/ Start/Replay facility Multiple choice questions with feedback on each responseMultiple choice questions with feedback on each response iNET discussion forum to ask questions about the various iNET discussion forum to ask questions about the various

parts of the syllabus and self assessment assignmentsparts of the syllabus and self assessment assignments Transferability of the idea to Science and engineering Transferability of the idea to Science and engineering

areasareas

@ Win Hornby 2004

How will all this save me time?How will all this save me time?

Some strategies do not involve extra costs Some strategies do not involve extra costs (e.g. strategic reduction)(e.g. strategic reduction)

Some strategies do involve some initial set Some strategies do involve some initial set up costsup costs

View these like an investment appraisal View these like an investment appraisal projectproject

Initial capital outlayInitial capital outlay Reap benefits over timeReap benefits over time

@ Win Hornby 2004

ReferencesReferences

Quality Enhancement in Teaching, Learning Quality Enhancement in Teaching, Learning and Assessmentand Assessment

Assessment @RGUAssessment @RGU

http://www.rgu.ac.uk/celt/quality/page.cfm?pge=6201http://www.rgu.ac.uk/celt/quality/page.cfm?pge=6201

@ Win Hornby 2004