View
218
Download
0
Category
Preview:
Citation preview
1ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Benchmarking in e-learning:an overview
Professor Paul BacsichMatic Media Ltd
andMiddlesex University, UK
2ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
The Menu
UK e-learning – why listen?Benchmarking overviewPick & Mix systemMBS case studyConclusions
3ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Myself
Consultant to several UK agencies & universitiesAdjunct professor at Middlesex University
– Global Campus and School of Computing ScienceOpen University for 25 yearsOne of the former Directors of UK eUniversities,
which aimed to be a global provider of e-learningCurrent work includes developing a global
benchmarking methodology for e-learning– Already piloted at Manchester Business School– Presented to EU conference in Brussels,
and in Colombia, ALT-C, HEA and Sydney Uni
4ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Not in my talk!
Costs of e-learning (Activity Based Costing)Competitor analysis of e-learning providersWhat went wrong with UK eUniversities?
– Two volumes of reports (35 chapters, over 2500 pages) and research overview available
– More soon
5ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Why listen to UK?UK has many years experience of quality
management in universities, via various organisations– Latest is Quality Assurance Agency for Higher Ed
This has guidelines for quality in e-learningUK has substantial experience in distance
learning and e-learning including global deliveryBenchmarking is part of Higher Ed policy for
e-learning (HE Academy, under way now)UK-Australian collaboration/co-funding on a
number of issues including e-Framework
6ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
The main UK agenciesQAA – for qualityJISC – for support of ICT in universities
– UKERNA to run the JANET high-speed network across all UK
Higher Education Academy (HEA) – for pedagogy
Some smaller agencies:– Leadership Foundation for HE (LFHE)– Observatory for Borderless HE (OBHE)
7ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Quality Assurance Agency UK
Covers all four UK home nations“Code of practice for the assurance of academic
quality and standards in higher education”See www.qaa.ac.uk/academicinfrastructure/
codeOfPractice/ Not much on pedagogy – this is left to the
discretion of the academicOnly 1 private uni in UK, over 100 public ones
8ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
QAA in e-learning
“Collaborative provision and flexible and distributed learning (including e-learning)”
September 2004
BUT– Some feel it says too little, others do not want to be
restricted– It was too late – 4 years?– No international comparisons (whereas research has)
9ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Pedagogy
Higher Education Academy“works with universities and colleges,
discipline groups, individual staff and organisations to help them deliver the best possible learning experience for all students”
Runs Subject Centres for each subjectAdvising on e-learning since early 2005
– Slowish progress
10ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
JISC and JANET
Joint Information Systems Committee– ICT for universities and colleges (not schools)– England, Scotland, Wales, N Ireland
JANET is the UK National Academic and Research Network (JANET)
JISC funds JANET via UKERNA company
11ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
In UK, universities compete- and now in e-learning
Universities want to judge how well they are doing in e-learning
Funding agencies and public want to knowBut universities don’t want to tell if they are
doing badly!And universities (like people) are not good
at judging themselves
12ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Benchmarking
Like Activity Based Costing (ABC), it has been around for many years
Unlike ABC, but like BPR, quality, excellence, etc; no one is now sure what it means…
13ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Back to Basics (Xerox)
a process of self-evaluation and self-improvement through the systematic and collaborative comparison of practice [process]and performance [metrics, KPIs]with competitors [or comparators]in order to identify own strengths and weaknesses,and learn how to adapt and improveas conditions change.
14ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Benchmarking (in Universities)
There are several reports that will tell you how to do benchmarking in general– Higher Education Academy (UK)– Learning and Skills Development Agency (UK)– Department of Education Training and Youth
Affairs (Australia)/Sydney Uni
15ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Benchmarking in e-Learning
My surveys and proposals– http://www.alt.ac.uk/altc2005/timetable/files/527/Benchm
ark_overview.doc
E-Learning Maturity Model (NZ) – MarshallNUTN/Hezel emerging work (Jan 2006?)Work by OECD and OBHENational Learning Network (UK) – collegesSo far unpublished work (Sydney/OU and ACODE)
There are few published reports re approaches
16ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Best Practice in e-Learning
There are a few reports (US):– APQC/SHEEO Study 1998 (US)– IHEP “Quality on the Line” 2000 (US)
And several projects (EU):– BENVIC– SEEQUEL– Swiss Virtual Campus @ Lugano: MINE
(adapting the IHEP work for EU)– E-xcellence (EADTU and others)
17ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Benchmarking e-learning
A global “synthesis” incorporating what work has been done
elsewhere
18ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Focus of my work
Focussed purely on e-learningBut not to any particular style (e.g. DL)Oriented to institutions past the “a few projects”
stageSuitable for desk research as well as “invasive”
studiesSuitable for single- and multi-institution studiesStarted work in Jan 2005, already piloted at
Manchester Business School against 12 competitors world-wide
19ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Processes or Outputs?
Outputs: measure first (can be done by desk research)
Processes: later (best done in clubs or invasive studies)
Inputs: not of so much interest to students; but of course of great interest to funders
20ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Metrics or Bureaucratic
Use a 6-point scale– 5 from Likert plus 1 more for “excellence”
Backed up by metrics where possibleAlso contextualised by narrativeSome issues of judging “best practice”;
judging “better practice” is easier– e.g. VLE convergence
Some criteria are rather “criteria bundles”
21ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Other Decisions
Explicit (otherwise you are not trying)Independent or collaborativeInternal or externalHorizontal
– focus on processes across whole institution– but can look at individual projects, missions
and departments to get “range of scores”
22ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
How Many Benchmarks?
It is like ABC: how many activities?Answer: Not 5, not 500Better answer: Well under 100
– Composite some criteria together– Remove any not specific to e-learning– Be careful about any which are not provably
critical success factors– Institutions may wish to add specific ones to
monitor their objectives and KPIs.
23ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
How Many do Others Have?
LSDA (UK) has 14 – but collegesIHEP (US) has 24 – but oldAPQC/SHEEO (US) had 14 – but olderEMM (NZ) has 43 – but some are being
merged and some are outside core e-learning area
OECD has many but several are “taxonomic” not critical success factors
24ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Pick and Mix System
Based on survey of “best of breed” ideas6-point scale (Likert + excellence)Backed up by narrative and metrics18 core criteria (e-learning specific)Can easily add more in same vein for local needsOutput and student-oriented aspects coveredFocussed on critical success factorsMethodology-agnosticRequires no long training course to understand
– But must know and be undogmatic about e-learning
25ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
“Adoption phase” (Rogers)
1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority
taking it up4. Early majority adopted; late majority taking
it up5. All taken up except laggards, who are now
taking it up (or retiring or leaving)6. First wave embedded, second wave under
way (e.g. m-learning after e-learning)
26ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
“Training”
1. No systematic training for e-learning2. Some systematic training, e.g. in some projects
and departments3. U-wide training programme but little monitoring of
attendance or encouragement to go4. U-wide training programme, monitored and
incentivised5. All staff trained in VLE use, training appropriate to
job type – and retrained when needed6. Staff increasingly keep themselves up to date in a
“just in time, just for me” fashion except in situations of discontinuous change
27ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
“Accessibility”1. e-learning material and services is not accessible2. Much e-learning material and most services conform to
minimum standards of accessibility3. Almost all e-learning material and services conform to
minimum standards of accessibility4. All e-learning material and services conform to at least
minimum standards of accessibility, much to higher standards
5. e-learning material and services are accessible, and key components validated by external agencies
6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study
Too aspirational, too international, too regulated?
28ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Case StudyJan-Apr 2005
Manchester Business School within Manchester U
(done by Matic Media Ltd)
29ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Methodology
Externally-focussed (internal going on now)Looked at 12 “comparator” business schools
(2 UK, 10 non-UK) – no time to discussFocus on speedy desk research (Web+DB)Focus on criteria susceptible to that
– plus “narratives of good practice”
Aim: to learn lessons for MBS
30ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
A few MBS conclusions
Numeric (not so interesting) – “taxonomic”Tabular (see next slide)Lots of case study narrative (but structured)Top-level conclusions include:
– Saturation wireless networks universal– e-Portfolios used in “sandstone-level”– Alumni get same IT systems as students
31ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Criterion LevelFactor 1 2 3 4 5 6
Adoption phase (Rogers)
Leuven? Illinois UBCVirginia
Babson?
VLE stage LeuvenVirginia
IllinoisMichigan
BabsonUBC
Tools use UBC BabsonIUPUILeuven
e-Learning Strategy
Virginia Penn State
Organisation Virginia IllinoisTwente
BabsonUBCIUPUIPenn State
Technical support to academics
Penn State Babson
32ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Work in progressPresentation at UK HEA Town MeetingDiscussions with EU projects on “quality” and “excellence” Implications of report on UKeU Committee for Academic
Quality (in e-Learning)Keynote at EFQEL conferenceWorkshop and Presentation at Online Educa Berlin
(Nov/Dec 2005)Detailed comparison of methodologies (NB costing)See how this can be taken into account for UK HEA
strategy for benchmarking e-learning– 12+60 HEIs to be involved in 2006
33ACODE 39: Quality in Teaching and Learning, 17-18 November 2005, Melbourne
Thank you for listeningAny questions?
Professor Paul Bacsich
bacsich@matic-media.co.uk www.cs.mdx.ac.uk/staff/profiles/
p_bacsich.html
Recommended