Where technology enables knowledge The National ICT Research Centre 1 m-learning – Evaluating the Effectiveness and the Cost John Traxler National ICT

1

Where technology enables knowledge

The National ICT Research Centre

m-learning – Evaluating the Effectiveness and the Cost

John Traxler

National ICT Research Centre

2Where technology enables knowledge

The National ICT Research Centre 2

Other presentationsDeveloper’s sessions:• Geoff Stead + Jo Colley Monday 12:00

Combined session on content and technology

Research Papers:• John Traxler Monday 14:30

Evaluating effectiveness and cost

• Alice Mitchell + Kris Popat Monday 16:00

The potential of games

• Jill Attewell + Carol Savill-Smith Tuesday 14:45

Focus on learners and learning



Understanding Cost is Important• Profitability, Return on Investment

• for universities, colleges, trainers, schools

• The transition from print-based learning• perceived economies of scale, possible large up-front costs,

increased risks

• globalisation; competition; industrialisation

• increased training and staff development

• learning delivered by tools and teams

• quality, accountability, visibility

• changes in working practices



Outline of Talk

• Predicting the Costs of Software• Predicting the Costs of Educational

Multimedia• Predicting the Costs of m-learning• Matching the Costs of m-learning to the

Benefits• The Real Difficulties



Software Cost-Estimation

• Many different approaches, most dependent on local technology, history, environment; modest successes

• Attempts to calculate effort sometime before implementation and delivery

• Often based on some measure of program size e.g. KLOC or Function Points



Basic COCOMO

• Basic version: effort depends of program size

• E = ab(KLOC) exp(bb)

• D= cb(KLOC) exp(db)

• N, number of people, =E/D



Project ab bb cb db

Type

Organic 2.4 1.05 2.5 0.38

Semi 3.0 1.12 2.5 0.35detached

Embedded 3.6 1.20 2.5 0.32

Basic COCOMO



Intermediate COCOMO• Intermediate version : effort depends

on program size and “cost-drivers”, each on 6-point scale from “very low” to “extra high“– product attributes

• reliability complexity etc

– hardware attributes• performance memory etc

– personnel attributes• experience capability etc

– project attributes• tools methods

etc



Educational Software Costs - “rules of thumb”

• Nothing useful will happen in the first three to six months after you decide to go with[courseware]. It doesn’t make any difference whether you go with a vendor or start producing your own in-house (Lee & Zemke 1987)

• The first course produced by a new [courseware] development group will be a collection of mistakes. Throw it away. (Lee and Zemke 1987)

• Where team members are not used to working together or are geographically apart, add 10-15% [to the total effort] (Casey et al 1988)

• Analysis and design comprise 50% of total effort (Casey et al. 1988)• Even a skilled Instructional Design author will revise plus or minus

half the material after first or second draft (and then 20-25% in the third draft) (Casey et al. 1988) -cited by Marshall



Some Industry Figures



Multimedia Educational Software Cost-Estimation - Marshall

• COCOMO used on 14 projects in 1990’s• mainly 1 hour learner time (cf. KLOC)• 4 potential cost-drivers (24 sub-heads)

– course difficulty– development environment– subject expertise– interactivity

» Ian Marshall, Abertay University



Multimedia Educational Software Cost-Estimation - Marshall

• Significant cost-drivers so far– development environment

• instructional design method (+ 4 more)

– course difficulty• number of objectives

• level of objectives

• existing course material



Gagne’s Media-Mix





Laurillard’s Model or Framework

teacher’s world

delivery

discussion

adaptationreflection

descriptions(theory)

actions(practice) interaction



print, lecture, Web pages

teacher’s world

theory

practice

delivery



seminar, conferencing

teacher’s world

discussion

theory

practice



laboratories, fieldwork

teacher’s world

theory

practice

interaction

reflection?



teaching package

teacher’s world

delivery

discussion ?

adaptation?

theory

practice



20

VLE courses

workshops

theory

practice

course, noticeboard

group folder,assignments,chat, forum, e-mail

set tasks

agree goals, do tasks

feedback, experience

chat

discusstasks

apply theory



21

m-learning courses

exercises

theory

practice

iPAQ

mPortal, SMS

develop exercises

iPAQ

SMS, mPortal

mPortal

discusstasks

apply theory



Practical Strategies

• Course Resource Appraisal Model– implemented as Excel spreadsheet

– Open University course resource planning and management tool

– based on Laurillard’s work

– looks at student workload and author workload across all media options

• Media Advisor– also based on Laurillard’s work but simplified

– developed at UNL by Martin Oliver & Grainne Connole

– aimed at individual lecturers to be used iteratively

– in public domain, on CD





Media Advisor



Media Advisor

26


The National ICT Research Centre

m-learning in an Imperfect World

Some of the Real and Hidden Costs



Constraints to m-learning• Hands-on - computing, music-making, workshop

• In vivo - medicine/dentistry/nursing/veterinary, field trips (perhaps)

• Interpersonal - interview skills, presentations

• Social - team-work (perhaps), business, marketing

• Expressive - ballet, dance

• Using Tools and Machines - engineering

• Laboratory Use - science (perhaps)• ( and exams, assessments, vivas etc)



Costs to m-learning Students

• 75% of (undergraduate) students own PC, 29% on internet, nearly 100% have ‘mobiles, almost none have PDAs

• time seen as main cost by (undergraduate) students (but is this location-dependent?)



Costs to Students

• Online learning(!) material was often printed– Text of practicals 33%

– Discussion of practicals 31%

– Web pages 45%

– Conference messages 54%» PLUM Report No. 122

• Would m-learning students want to do the same or can they read from PDAs?



Students’ Preferences

“Which of these learning tools do you prefer to use?”

– books 67%

– lectures 36%

– videos 36%

– computers 19%• Campaign for Learning, 1996 - predates PDAs!

Why these preferences?

What will be preferences of m-learning students?

– audio-tapes 11%

– internet7%

– none/no pref. 3%

– none/don’t want to learn 3%



Student Use of PC-based CMC

• Limited active participation– participate : lurk

» 40: 60

» 30: 70

» 10: 90

• varied strategies to improve these– based on postings to “teaching online” mailbase, 1999

– see Gilly Salmon’s work @OU

• Can m-learning turn lurkers into participants?



Costs to m-learning Students

• Forms of disadvantage• dyslexia; visual impairment

• not confident with IT or English

• less affluent

• “wrong” cognitive or learner style

• New forms of disadvantage• bandwidth poverty

• interface poverty (wrt PCs)



Thanks for your time!

My email [email protected]

Samples are at www.ctad.co.uk/m-learning

m-learning project is at www.m-learning.org

Documents

Where technology enables knowledge The National ICT Research Centre 1 m-learning – Evaluating the Effectiveness and the Cost John Traxler National ICT