Digital Forms of Assessment

Assessment of Practical Performance Using Digital FormatsPaul NewhouseAssociate Professor - Director, CSaLT

A research project funded by a three year Australian Research Council (ARC) Linkage grant and the Curriculum Council of WA.

Conducted by a research team led by Paul Newhouse and John Williams from the …

Centre for Schooling and Learning Technologies (CSaLT) School of Education, Edith Cowan University

The Problem - generally Student performance in many areas is difficult to

capture using paper and pen in 3 hours.

Students need to be provided with assessment opportunities that are:

– authentic

– reliably and manageably judged

Good teaching must be encouraged and rewarded

All students need to be engaged with their study

In the developed world a very small proportion of tasks are done on paper.

RationaleWhat is assessed is critical because:

(1) Students tend to focus on, and be motivated by assessment.

(2) Teachers tend to ‘teach to the test’.

(3) We are accountable to our society for the outcomes of the use of resources in education.

“If tests designed to measure key learning in schools ignore some key areas because they are harder to measure and attention to those areas by teachers and schools is then reduced, then those responsible for the tests bear some responsibility for that” (McGaw, 2006, p. 3).

Performance assessments that are locally managed and involve multiple sources of evidence assist students in learning and teachers in teaching for higher order skills… Large scale examples … come from states such as Nebraska, Wyoming, Connecticut and New York, ...

The evidence from research on these and other systems indicate that through using performance assessments schools can focus instruction on higher order skills, provide a more accurate measure of what students know and can do, engage students more deeply in learning ... (Wood, Darling-Hammond, Neill and Roschewski, 2007) - Report to US Congress

Critical questions

Any performance assessment approach or strategy will not be perfect and requires compromises.

(1) What skills or knowledge are best demonstrated through practical performance?

(2) What are the critical components of that practical performance?

(3) What representations of performance could be used?

(4) What level of compromise in reliability, authentication and cost is acceptable in preference to NOT assessing the performance at all?

Learning from the Literature Completed a wide ranging literature review into

– Performance assessment

– Computer-based assessment

– Assessment

Created online database of papers + Endnote library

Created a wiki to begin analysis

Project e-scape (2004-2007) Technology Education Research Unit, University of London (Richard Kimbell & Tony Wheeler)

Assessment of student performance on a design task.

Learning from the UK

2007 Pilot Project … Considered the assessment options of:

– Electronic Portfolio

– Performance/Simulated Tasks Exam

– Production Exam

Worked one-on-one with a group of teachers to develop assessment instruments (no common tasks).

Student work collected from 7 AIT classes and 7 Engineering classes

Development of marking tools

Work marked using Scales of Achievement

Comparative pairs trial

2008-2010

ARC grant for three year investigation to involve FOUR courses.

How are digitally based representations of student work output on authentic tasks most effectively used to support highly reliable summative assessments of student performances for courses with a substantial practical component?

Proposed Research[1] Proof of Concept: The aim will be to trial each of the forms of

assessment in one course in at least four schools. This will include placing student work in an online repository and marking using the comparative pairs technique.

[2] Prototype: The aim will be to implement a prototype of each form of assessment in one course in at least four schools under ‘normal’ or typical conditions. This will include the use of direct capture of student work into the online repository in at least one course.

[3] Scalable Product: The aim will be to implement each of the forms of assessment in at least one course in a representative sample of schools.

Four courses: Applied Information Technology, Engineering Studies, Italian, and Physical Education Studies

Research TeamPaul Newhouse - coordination of the project & leader AIT team.John Williams - co-cordination & leader Engineering team.Dawn Penney (Uni of Tas) - leader Physical Education Studies team.Cher Ping Lim - leader Italian team.Jeremy Pagram - expertise in the use of digital devices.Andrew Jones - local leadership for PES team.Mark Hackling - oversight for the design of assessment instruments.Ron Oliver - quality assurance for research output. Russell Waugh - expertise in Rasch analysis.

Specialist Consultants - Professor Richard Kimbell (University of London) and TAG Learning. - Professor David Andrich and Dr. Stephen Humphry (UWA)An Advisory Group - representatives of school systems and Curric Council.Working Parties - researchers, Council personnel and participating teachers.Technical Support - ECU technical support + contracted support.Management and Research Assistants - project manager and two post-doctoral

scholars Martin Cooper and Alistair Campbell.Two Doctoral students

Key Features

(1) The representation in digital files of the performance of students doing practical work.

(2) The presentation of digital representations of student performance in an online repository so that they are easily accessible to markers.

(3) Assessing the digital representations of student performance using both analytic marking and the paired comparison judgement method with holistic and component criteria-based judgements.

Online Repository Storage of digital representations of student work for online

access.

Considered using MAPS as in pilot study– Difficulty accessing variety of file types

– Marking tools separate from access to student work

Decided to try accessing student work through folders/files on a server using Filemaker online to deliver through a browser

Database Driven Marking Tools Developed two prototypes in Filemaker online …

– Standards-Reference ‘Analytical’ marking tools

– Comparative pairs marking tools

Allows the viewing of digital work and criteria and recording of judgements both quantitatively and qualitatively

Markers can be anywhere in the world

Easy transfer to spreadsheets and statistic packages for analysis.

Determining Assessment Tasks For each course completed a situation analysis

– Student characteristics

– Course requirements

– The Performance

(1) What skills or knowledge are best demonstrated through practical performance?

(2) What are the critical components of that practical performance?

(3) Why can’t those components be demonstrated on paper?

– The Technologies

(1) What alternative representations other than paper could be used?

(2) What level of compromise is acceptable?

– The Teachers

(1) What are key characteristics of typical teachers?

(2) How do these characteristics relate to assessment in the course?

(3) How do these characteristics relate to the use of ICT in the course?

Determined appropriate form of assessment

Described task(s), marking criteria, and created student materials.

Courses & Assessment Tasks AIT

– Portfolio + Performance Tasks Exam

Engineering Studies– Extended Production Exam

Italian– Oral Exam with Portfolio

Phys Ed Studies– Performance Tasks and Response Exam

Capturing Performance AIT

– USB Flash Drives, templates in Word, Powerpoint etc.

Engineering Studies– eeePC and Filemaker Runtime template

Italian– Digital video camera with radio microphone

– Digital still camera and desktop computers

Phys Ed Studies– PTZ remote control cameras

– Combined video feeds with audio

– Filemaker runtime template

– USB Flash Drives

Physical Education Studies

PES Assessment Task

A challenge/problem was set for each sport– Rugby lineout

– Soccer corner

– Volleyball return serve

– Swimming medley

(1) Develop response

(2) Video skills routines

(3) Video ‘game’ (main and alternate strategies)

(4) Reflection & evaluation

Physical Education - video

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

PES First Exam

Challenge set for the sport– Rugby lineout

– Soccer corner

– Volleyball return serve

– Swimming medley

(1) Develop response

PES Final Exam

PES Marking 2

PES Comparative Pairs Marking

Engineering Studies

Engineering Assessment Task A series of specified activities which will take students from a design brief

to the construction of a model over a period of 3 hours.

Each activity was timed, so students had a specific time frame in which to complete each activity.

Each student had mini computer (ASUS eeePC) on which they will compile their portfolio within a pre-formed template already loaded onto their computer.

The Task Manager or Teacher coordinated the activity by showing a presentation which will be projected onto a screen for all the class to see. This was the mechanism for managing the set time students had on each activity, prompting the students to move onto the next page of their portfolio at the appropriate time.

The task involved the design and modelling of a solar water heater for a rural developing country context.

Engineering Studies 3 hour practical exam supported by eeePC (built-in mic), webcam,

USB drive and Filemaker database.

Marking the Portfolio

Italian

High quality radio microphone connected to a digital video camera.

Italian Assessment TaskPart A - Folio: Andiamo!

Collect a series of tasks and activities including map activity, retrieval chart and question answers, brainstorm, fact sheet, word processed reflection on practice talk and one minute voice recording evaluating final performance, to show development of ideas and preparation for presentation (Part B).

Part B - Presentation: Destinazione: Western Australia

Prepare a talk to be recorded on video/DVD and send to a school in Italy. Your talk will focus on your local area/WA destination, providing information such as, features of the area, recreational activities and cultural events. You should also consider the time of visit, and choose activities which are appropriate. Comparisons with the area you have chosen and region in Italy where your presentation will be shown should also be included. During your presentation you will be asked to respond to a question relating to the information you’ve prepared.

Applied Information Technology

AIT Assessment Task Digital Reflective Process Portfolio

(1) A digital product designed as a prototype of an information solution.

(2) The design process document for that digital product.

(3) Two other digital artefacts that illustrate skills in areas from a specified list.

Performance Tasks Exam

Completed using a computer workstation and included a set of reflective questions (on-screen) concerning the digital portfolio and a set of short performance tasks.

– Exam (Component 4) - Reflective Questions

– Exam (Component 5) - Performance Task(s)

Students completed a set of scripted tasks at a computer workstation over a short period of time (2 hours) under ‘exam’ conditions.

Produced a logo, graphs and brochure.

AIT - Pairs Marking Tool

Paul Newhouse AIT Pairs Marking

0

50

100

150

200

250

300

350

400

45 90 135 165 210 255 300 345 395 440 485 555

Total Time (mins)

Total Pairs

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Minutes/Pair

Total TimePairs Rate

Rasch Analysis of Analytical Marking

Exam

SI=0.85

Portfolio Product

SI=0.96

What now? Refine assessment tasks

Improve use of digital technologies (e.g. MAPS)

Further improve implementation of assessment tasks

Improve efficiency of comparative pairs marking (e.g. MAPS Pairs Engine)

Scale up in 2010 (more representative schools)

Report available …

http://csalt.education.ecu.edu.au/docs/assess.htm