ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne1 Benchmarking in e-learning: an overview Professor Paul Bacsich Matic Media

1ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne

Benchmarking in e-learning:an overview

Professor Paul BacsichMatic Media Ltd

andMiddlesex University, UK


The Menu

UK e-learning – why listen?Benchmarking overviewPick & Mix systemMBS case studyConclusions


Myself

Consultant to several UK agencies & universitiesAdjunct professor at Middlesex University

– Global Campus and School of Computing ScienceOpen University for 25 yearsOne of the former Directors of UK eUniversities,

which aimed to be a global provider of e-learningCurrent work includes developing a global

benchmarking methodology for e-learning– Already piloted at Manchester Business School– Presented to EU conference in Brussels,

and in Colombia, ALT-C, HEA and Sydney Uni


Not in my talk!

Costs of e-learning (Activity Based Costing)Competitor analysis of e-learning providersWhat went wrong with UK eUniversities?

– Two volumes of reports (35 chapters, over 2500 pages) and research overview available

– More soon


Why listen to UK?UK has many years experience of quality

management in universities, via various organisations– Latest is Quality Assurance Agency for Higher Ed

This has guidelines for quality in e-learningUK has substantial experience in distance

learning and e-learning including global deliveryBenchmarking is part of Higher Ed policy for

e-learning (HE Academy, under way now)UK-Australian collaboration/co-funding on a

number of issues including e-Framework


The main UK agenciesQAA – for qualityJISC – for support of ICT in universities

– UKERNA to run the JANET high-speed network across all UK

Higher Education Academy (HEA) – for pedagogy

Some smaller agencies:– Leadership Foundation for HE (LFHE)– Observatory for Borderless HE (OBHE)


Quality Assurance Agency UK

Covers all four UK home nations“Code of practice for the assurance of academic

quality and standards in higher education”See www.qaa.ac.uk/academicinfrastructure/

codeOfPractice/ Not much on pedagogy – this is left to the

discretion of the academicOnly 1 private uni in UK, over 100 public ones


QAA in e-learning

“Collaborative provision and flexible and distributed learning (including e-learning)”

September 2004

BUT– Some feel it says too little, others do not want to be

restricted– It was too late – 4 years?– No international comparisons (whereas research has)


Pedagogy

Higher Education Academy“works with universities and colleges,

discipline groups, individual staff and organisations to help them deliver the best possible learning experience for all students”

Runs Subject Centres for each subjectAdvising on e-learning since early 2005

– Slowish progress


JISC and JANET

Joint Information Systems Committee– ICT for universities and colleges (not schools)– England, Scotland, Wales, N Ireland

JANET is the UK National Academic and Research Network (JANET)

JISC funds JANET via UKERNA company


In UK, universities compete- and now in e-learning

Universities want to judge how well they are doing in e-learning

Funding agencies and public want to knowBut universities don’t want to tell if they are

doing badly!And universities (like people) are not good

at judging themselves


Benchmarking

Like Activity Based Costing (ABC), it has been around for many years

Unlike ABC, but like BPR, quality, excellence, etc; no one is now sure what it means…


Back to Basics (Xerox)

a process of self-evaluation and self-improvement through the systematic and collaborative comparison of practice [process]and performance [metrics, KPIs]with competitors [or comparators]in order to identify own strengths and weaknesses,and learn how to adapt and improveas conditions change.


Benchmarking (in Universities)

There are several reports that will tell you how to do benchmarking in general– Higher Education Academy (UK)– Learning and Skills Development Agency (UK)– Department of Education Training and Youth

Affairs (Australia)/Sydney Uni


Benchmarking in e-Learning

My surveys and proposals– http://www.alt.ac.uk/altc2005/timetable/files/527/Benchm

ark_overview.doc

E-Learning Maturity Model (NZ) – MarshallNUTN/Hezel emerging work (Jan 2006?)Work by OECD and OBHENational Learning Network (UK) – collegesSo far unpublished work (Sydney/OU and ACODE)

There are few published reports re approaches


Best Practice in e-Learning

There are a few reports (US):– APQC/SHEEO Study 1998 (US)– IHEP “Quality on the Line” 2000 (US)

And several projects (EU):– BENVIC– SEEQUEL– Swiss Virtual Campus @ Lugano: MINE

(adapting the IHEP work for EU)– E-xcellence (EADTU and others)


Benchmarking e-learning

A global “synthesis” incorporating what work has been done

elsewhere


Focus of my work

Focussed purely on e-learningBut not to any particular style (e.g. DL)Oriented to institutions past the “a few projects”

stageSuitable for desk research as well as “invasive”

studiesSuitable for single- and multi-institution studiesStarted work in Jan 2005, already piloted at

Manchester Business School against 12 competitors world-wide


Processes or Outputs?

Outputs: measure first (can be done by desk research)

Processes: later (best done in clubs or invasive studies)

Inputs: not of so much interest to students; but of course of great interest to funders


Metrics or Bureaucratic

Use a 6-point scale– 5 from Likert plus 1 more for “excellence”

Backed up by metrics where possibleAlso contextualised by narrativeSome issues of judging “best practice”;

judging “better practice” is easier– e.g. VLE convergence

Some criteria are rather “criteria bundles”


Other Decisions

Explicit (otherwise you are not trying)Independent or collaborativeInternal or externalHorizontal

– focus on processes across whole institution– but can look at individual projects, missions

and departments to get “range of scores”


How Many Benchmarks?

It is like ABC: how many activities?Answer: Not 5, not 500Better answer: Well under 100

– Composite some criteria together– Remove any not specific to e-learning– Be careful about any which are not provably

critical success factors– Institutions may wish to add specific ones to

monitor their objectives and KPIs.


How Many do Others Have?

LSDA (UK) has 14 – but collegesIHEP (US) has 24 – but oldAPQC/SHEEO (US) had 14 – but olderEMM (NZ) has 43 – but some are being

merged and some are outside core e-learning area

OECD has many but several are “taxonomic” not critical success factors


Pick and Mix System

Based on survey of “best of breed” ideas6-point scale (Likert + excellence)Backed up by narrative and metrics18 core criteria (e-learning specific)Can easily add more in same vein for local needsOutput and student-oriented aspects coveredFocussed on critical success factorsMethodology-agnosticRequires no long training course to understand

– But must know and be undogmatic about e-learning


“Adoption phase” (Rogers)

1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority

taking it up4. Early majority adopted; late majority taking

it up5. All taken up except laggards, who are now

taking it up (or retiring or leaving)6. First wave embedded, second wave under

way (e.g. m-learning after e-learning)


“Training”

1. No systematic training for e-learning2. Some systematic training, e.g. in some projects

and departments3. U-wide training programme but little monitoring of

attendance or encouragement to go4. U-wide training programme, monitored and

incentivised5. All staff trained in VLE use, training appropriate to

job type – and retrained when needed6. Staff increasingly keep themselves up to date in a

“just in time, just for me” fashion except in situations of discontinuous change


“Accessibility”1. e-learning material and services is not accessible2. Much e-learning material and most services conform to

minimum standards of accessibility3. Almost all e-learning material and services conform to

minimum standards of accessibility4. All e-learning material and services conform to at least

minimum standards of accessibility, much to higher standards

5. e-learning material and services are accessible, and key components validated by external agencies

6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study

Too aspirational, too international, too regulated?


Case StudyJan-Apr 2005

Manchester Business School within Manchester U

(done by Matic Media Ltd)


Methodology

Externally-focussed (internal going on now)Looked at 12 “comparator” business schools

(2 UK, 10 non-UK) – no time to discussFocus on speedy desk research (Web+DB)Focus on criteria susceptible to that

– plus “narratives of good practice”

Aim: to learn lessons for MBS


A few MBS conclusions

Numeric (not so interesting) – “taxonomic”Tabular (see next slide)Lots of case study narrative (but structured)Top-level conclusions include:

– Saturation wireless networks universal– e-Portfolios used in “sandstone-level”– Alumni get same IT systems as students


Criterion LevelFactor 1 2 3 4 5 6

Adoption phase (Rogers)

Leuven? Illinois UBCVirginia

Babson?

VLE stage LeuvenVirginia

IllinoisMichigan

BabsonUBC

Tools use UBC BabsonIUPUILeuven

e-Learning Strategy

Virginia Penn State

Organisation Virginia IllinoisTwente

BabsonUBCIUPUIPenn State

Technical support to academics

Penn State Babson


Work in progressPresentation at UK HEA Town MeetingDiscussions with EU projects on “quality” and “excellence” Implications of report on UKeU Committee for Academic

Quality (in e-Learning)Keynote at EFQEL conferenceWorkshop and Presentation at Online Educa Berlin

(Nov/Dec 2005)Detailed comparison of methodologies (NB costing)See how this can be taken into account for UK HEA

strategy for benchmarking e-learning– 12+60 HEIs to be involved in 2006


Thank you for listeningAny questions?

Professor Paul Bacsich

[email protected] www.cs.mdx.ac.uk/staff/profiles/

p_bacsich.html

Documents

ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne1 Benchmarking in e-learning: an overview Professor Paul Bacsich Matic Media