Programme assessment design: How might technology help?

Dr Greg BenfieldOxford Centre for Staff and Learning Development

Oxford Centre for Staff and Learning Development Landscape: social software

©BBC http://news.bbc.co.uk/1/hi/technology/3830527.stm

An ‘”underworld” of digital communication among learners’(LEX, Creanor et al 2006)

Google and Wikipedia preferred information search & retrieval tools (LXP, Conole et al 2006)

“The concept of ‘time’ is changing – both in terms of expectation of information and results on demand. There is evidence of a fragmentation of the learning timetable”(LXP, Conole et al 2006)

Oxford Centre for Staff and Learning Development

Immersed in technology

“technologically adept and had integrated ICT into their lives (JISC 2007: 10)

“student respondents [are] immersed in technology ownership and use and impatient with instructors who don’t have adequate technical skills” (Salaway et al 2007: 5)

©BBC http://news.bbc.co.uk/1/hi/technology/3830527.stm

Oxford Centre for Staff and Learning Development

Assessment and feedback

Formative use of CAA stands out as a rare application of e-learning leading to measurablemeasurable impactimpact on student performance (Sharpe et al 2006)

Oxford Centre for Staff and Learning Development Scenarios

1. VLE supporting self directed study in financial accounting

2. Automated assessments in life sciences

3. Personal response systems supporting discussion of problems in engineering

Sharpe, R., Benfield, G., Roberts, G. & Francis, R. (2006) The undergraduate experience of blended e-learning: a review of UK literature and practice undertaken for the Higher Education Academy. At http://www.heacademy.ac.uk/4884.htm

Oxford Centre for Staff and Learning Development

Background

• 600 undergraduate students

• Needed efficient management of self-directed study

• Student diversity

• Student expectations of flexibility

Scenario 1: Newcastle Business School

Oxford Centre for Staff and Learning Development

Background

• 600 undergraduate students

• Needed efficient management of self-directed study

• Student diversity

• Student expectations of flexibility

An approach to the problem

• Online resources

• Weekly directed study tasks

• Answers on timed release

• Formative self- assessment quizzes

Scenario 1: Newcastle Business School

Oxford Centre for Staff and Learning Development Scenario 1: Newcastle Business School

conceptualisation

construction

dialogue

Oxford Centre for Staff and Learning Development

Survey responses from 200 students Most agreed Blackboard helped them study Most used it weekly High usage of content and quizzes, less of

discussions

Students valued Access to learner materials Greater independence Time saving More convenient

What the students said

Oxford Centre for Staff and Learning Development

Background

• 1st year introductory lecture based course

• Lecture notes on web since 2002

• High failure rate

• Poor turnaround time for assignments

• Inconsistency in quality of feedback and marking

Scenario 2: Life Sciences at Dundee

Oxford Centre for Staff and Learning Development

Background

• 1st year introductory lecture based course

• Lecture notes on web since 2002

• High failure rate

• Poor turnaround time for assignments

• Inconsistency in quality of feedback and marking

An approach to the problem

• Computer-aided assessment (CAA)

Scenario 2: Life Sciences at Dundee

Oxford Centre for Staff and Learning Development

• No negative feedback regarding the use of CAA

• 99% of those who responded said the online tutorials improved their understanding of the topics covered

• 93% felt that the marks awarded were appropriate

• “The ability to re-do tests and assignments again once you have already done it is good. It enables you to continue learning the more you try it.”

• The overall module pass rate rose from 73% to 93%.

Evaluation

Oxford Centre for Staff and Learning Development

Background

• Weak conceptual understanding

• Insufficient interaction and discussion in class

• Low student motivation

An approach to the problem

• Personal response system in class

• Question and answer with discussion/defence of answers.

Scenario 3: Engineering at Strathclyde

Oxford Centre for Staff and Learning Development

Evaluation

74% of students agreed that in comparison with traditional lecture classes, the teaching methods used here were more effective in helping them to acquire an understanding of fundamental engineering concepts.

95% agreed that I am more actively involved in PRS than traditional classes

In focus groups students attributed their understanding to being active in class having time to think, reflect and discuss discussion with other students in peer groups.

Scenario 3: Engineering at Strathclyde

Oxford Centre for Staff and Learning Development Extended, technology-

enhanced assignments

http://www.brookes.ac.uk/schools/education/arts/diaries/home.html!

http://blogs.warwick.ac.uk/groups/en-all

http://core.mwbrookes.org.uk/

E.g. student journals using blogs

Oxford Centre for Staff and Learning Development

…. or e-portfolios

http://elgg.net/

Oxford Centre for Staff and Learning Development Wikis

Podcasts

Student research journals (Geoverse)

Collaborative student artifacts

Oxford Centre for Staff and Learning Development References

Biggs, J. (2003). Teaching for Quality Learning at University Second Edition. Maidenhead, Open University Press.

Browne, T. and Jenkins, M. (2003). 'VLE surveys: a longitudinal perspective between March 2001 and March 2003 for Higher Education in the United Kingdom.' UCISA. online http://www.ucisa.ac.uk/groups/tlig/vle/vle2003.pdf, accessed 12 November 2003

Catley, P (2005). ‘One Lecturer's Experience of Blending E-learning with Traditional Teaching or How to Improve Retention and Progression by Engaging Students’. Brookes eJournal of Learning and Teaching, 1(2) online at http://www.brookes.ac.uk/publications/bejlt/volume1issue2/academic/catley05_1.html!

Collis, B. & Moonen, J. (2005). An On-Going Journey: Technology as a Learning Workbench. [Online] at http://bettycollisjefmoonen.nl/Book-Learning-Workbench-V2.pdf

Conole, G., De Laat, M., Dillon, T. and Darby, J. (2006, November 2006). "JISC LXP: Student Experiences of Technologies Draft Final Report." November 2006.[Online] Retrieved 20 Nov, 2006, from http://www.jisc.ac.uk/media/documents/lxp_project_final_report_nov_06.pdf.

Creanor, L., Trinder, K., Gowan, D. and Howells, C. (2006, August 2006). "LEX: The Learner Experience of e-Learning Final Project Report August 2006." [Online] Retrieved 2 November, 2006, from http://www.jisc.ac.uk/uploaded_documents/LEX%20Final%20Report_August06.pdf

DfES (2005) 'Harnessing Technology: Transforming learning and children's services'. Online at http://www.dfes.gov.uk/publications/e-strategy/

HEFCE (2005). HEFCE strategy for e-learning, online at http://www.hefce.ac.uk/pubs/hefce/2005/05_12/

JISC (2003). 'Virtual and Managed Learning Environments.' Joint Information Systems Committee. online at http://www.jisc.ac.uk/index.cfm?name=issue_vle_mle , accessed 25 August 2005.

JISC (2004). Effective Practice with e-Learning: A good practice guide in designing for e-Learning. Bristol, JISC. Online at http://www.jisc.ac.uk/uploaded_documents/jisc%20effective%20practice3.pdf

Oxford Centre for Staff and Learning Development References cont’d

JISC (2005). Innovative Practice with e-Learning. Bristol, JISC. Online at http://www.jisc.ac.uk/uploaded_documents/publication_txt.pdf

JISC. (2007). "Student Expectations Study: Key findings from online research and discussion evenings held in June 2007 for the Joint Information Systems Committee." [Online] Retrieved 10 September, 2007, from http://www.jisc.ac.uk/media/documents/publications/studentexpectations.pdf

Laurillard, D. (1993). Rethinking University Teaching-A framework for the effective use of educational technology. New York, Routledge.

Laurillard, D. (2002). Rethinking University Teaching-a conversational framework for the effective use of educational technology. London, RoutledgeFarmer.

Mayes, T and de Freitas, S. (2004) Review of e-learning theories, frameworks and models. JISC. Online at http://www.jisc.ac.uk/uploaded_documents/Stage%202%20Learning%20Models%20(Version%201).pdf

Sharpe, R, Benfield, G, Roberts, G and Francis, R (2006). "The undergraduate experience of blended e-learning: a review of UK literature and practice undertaken for the Higher Education Academy." Retrieved 3 October, 2006, from http://www.heacademy.ac.uk/research/Sharpe_Benfield_Roberts_Francis.pdf

Salaway, G., Caruso, J. B. and Nelson, M. R. (2007). "The ECAR Study of Undergraduate Students and Information Technology, 2007." [Online] Retrieved 9 October, 2007, from http://www.educause.edu/ir/library/pdf/ers0706/rs/ERS0706w.pdf