Dr Jill Fresen and Lesley Boyd Quality Management in E-learning: Towards an integrated approach Quality Management in E-learning: Towards an integrated

Dr Jill Fresen and Lesley Boyd

Quality Management in E-

learning:

Towards an integrated approach

Quality Management in E-

learning:

Towards an integrated approach

Dept of Telematic Learning and Education Innovation,University of Pretoria

IndependentQuality Assurance Consultant

“Quality assurance and online delivery are hot topics in universities, yet until recently discussions of each have had little to do with each other.”

“Quality assurance and online delivery are hot topics in universities, yet until recently discussions of each have had little to do with each other.”

Reid (2003)Academic Exchange Quarterly, 7(1), 17-23.

Reid (2003)Academic Exchange Quarterly, 7(1), 17-23.

ProblemProblem

AgendaAgendaBackground of QA in e-learningBackground of QA in e-learning

Methodology of project at UPMethodology of project at UP

Towards integrationTowards integration

Lessons learntLessons learnt

Where to next?Where to next?






Contact EducationContact Education

face-to-face

Distance EducationDistance Education

paper-based

TerminologyTerminology

E-learningE-learning

Web-supported learning

Web-based learning

Internet-based learning

Online learning

Technology-enhanced learning

Web-based learning

Internet-based learning

Online learning

Technology-enhanced learning

Other terms:Other terms:

E-Learning @ UPE-Learning @ UP

Web-support Multi-media (on CD-Rom) Video production Video conferencing Interactive Television Graphic services Photographic services

Web-support Multi-media (on CD-Rom) Video production Video conferencing Interactive Television Graphic services Photographic services

project managerinstructional designereducation consultantprogrammergraphic artistlibrary consultant

project managerinstructional designereducation consultantprogrammergraphic artistlibrary consultant

lecturersstudentslecturersstudents

Practitioners

Stakeholders Clients

Role playersRole players

managementgovernmentfunderscommunity

managementgovernmentfunderscommunity

Who are our clients?Who are our clients?

Lecturers• instructional design• educational support• training• technical support

Lecturers• instructional design• educational support• training• technical support

Students• end users• training• technical support

Students• end users• training• technical support

Case StudiesCase Studies

University of Southern Queensland Distance Education Centre (DEC) - first ISO9001 accreditation

University of Bangor in Wales - tools for judging pedagogical quality

Frameworks, benchmarks and best practice indicators: critical success factors for e-learning

University of Southern Queensland Distance Education Centre (DEC) - first ISO9001 accreditation

University of Bangor in Wales - tools for judging pedagogical quality

Frameworks, benchmarks and best practice indicators: critical success factors for e-learning



Building blocks for the futureBuilding blocks for the future



Methodology of the UP project


Apply standard QA theory to e-learning applications, by implementing a Quality Management System

Investigate what factors promote quality e-learning

Implement systematic evaluation and customer feedback systems












QMS Approach - 1QMS Approach - 1

Pro-active evaluation and improvement rather than external accountability Ownership and commitment from all practitioners in the E-education Unit

Pro-active evaluation and improvement rather than external accountability Ownership and commitment from all practitioners in the E-education Unit

QMS Approach – 2QMS Approach – 2

formal

online

process based

consensus driven

ISO9000 cognisant

formal

online

process based

consensus driven

ISO9000 cognisant

Documented, auditable, visibleDocumented, auditable, visible

Minimise paperworkMinimise paperwork

Understand the entire ID process from beginning to end and how awareness of this helps people to work together effectively.

Understand the entire ID process from beginning to end and how awareness of this helps people to work together effectively.

Based on consensus ownership not prescriptionBased on consensus ownership not prescription

Recognise useful principles behind ISO9000Recognise useful principles behind ISO9000

QMS – TimescalesQMS – Timescales

2001 TLEI management request a formal quality approach to be implementedNov 2001 – May 2002 Training Workshops for all management and staff May 2002 – Sept 2003 Production of procedure documentation & development of online QMSSept 2003 – loading of procedure documents into online QMSOct 2003 Launch of ‘live’ QMS2003 – 2005 ‘Death valley’ of implementation2006 Self evaluation of system – recognised as benchmark for services provided to clients

2001 TLEI management request a formal quality approach to be implementedNov 2001 – May 2002 Training Workshops for all management and staff May 2002 – Sept 2003 Production of procedure documentation & development of online QMSSept 2003 – loading of procedure documents into online QMSOct 2003 Launch of ‘live’ QMS2003 – 2005 ‘Death valley’ of implementation2006 Self evaluation of system – recognised as benchmark for services provided to clients

QMS – ObjectivesQMS – ObjectivesTo provide a defined framework for all role players to work together consistently along the entire project timeline.To enable everyone, including new staff, to understand ‘the way things are done around here’.To identify together areas for improvement.To provide an integrated and simple method to access and use supporting documentation eg checklists, forms, templates.To ensure that the right tools are available to allow for comprehensive checks and to minimise errors.To try and catch any errors as soon as possible before it’s too late or too expensive to fix them.To evaluate completed projects and help to assess their impact on teaching and learning at UP.To learn lessons which may help to improve future projects.To share more with each other about ways of doing things.To demonstrate to any external stakeholders (e.g. auditors or UP management) that TLEI has a formal quality management system in place to control e-education projects.

To provide a defined framework for all role players to work together consistently along the entire project timeline.To enable everyone, including new staff, to understand ‘the way things are done around here’.To identify together areas for improvement.To provide an integrated and simple method to access and use supporting documentation eg checklists, forms, templates.To ensure that the right tools are available to allow for comprehensive checks and to minimise errors.To try and catch any errors as soon as possible before it’s too late or too expensive to fix them.To evaluate completed projects and help to assess their impact on teaching and learning at UP.To learn lessons which may help to improve future projects.To share more with each other about ways of doing things.To demonstrate to any external stakeholders (e.g. auditors or UP management) that TLEI has a formal quality management system in place to control e-education projects.

QMS - MethodologyQMS - Methodology

QA training workshopsFormalised intuitive documentation such as the Instructional Design Toolkit and the Project TimelineQMS Triangle

QA training workshopsFormalised intuitive documentation such as the Instructional Design Toolkit and the Project TimelineQMS Triangle

Ons slakkieOns slakkie

Quality Management System

Quality Management System

DODO

Detailed work instructionsDetailed work instructions

Standards, checklists, forms

Standards, checklists, forms

ProceduresProceduresProcessesProcesses

PLANPLANStrategyStrategy

PoliciesPoliciesObjectivesObjectives

CONTROL

CONTROLRecordsRecords

Measurements and Targets

Measurements and Targets

ACTACT

Feed

back L

oop

Feed

back L

oop

Boyd, 2001 – adapted from Waller, Allen & Burns, 1993

ISO 9000 modelISO 9000 model

Value adding activitiesValue adding activities

Information flowInformation flow SABS

(2000)SABS (2000)

Project TimelineProject Timeline

Based on traditional ADDIE model of instructional designIllustrates the major PROCESS

Based on traditional ADDIE model of instructional designIllustrates the major PROCESS

Project TimelineProject Timeline

Analysis

ProceduresProcedures

Each step documented into a PROCEDURE using:

• task teams (jigsaw and pizza)• a procedure template• a fully worked example• a set of self evaluation questions

Each step documented into a PROCEDURE using:

• task teams (jigsaw and pizza)• a procedure template• a fully worked example• a set of self evaluation questions

ProceduresProcedures

Rapid prototyping and cross- referencing between task teams

All procedures then uploaded into the live QMS with a document control master list

Live system launched and left to sink or swim!

Rapid prototyping and cross- referencing between task teams

All procedures then uploaded into the live QMS with a document control master list

Live system launched and left to sink or swim!









AssumptionsAssumptions

Technology infrastructure• reliability, accessibility, availability• accuracy, stability of student data

Commitment, motivation of clients Computer literacy of clients Information literacy of clients Positive attitude of lecturers Consultation and support to lecturers Sound instructional design practice Sound teaching and learning practice Commitment to cont. improvement

Technology infrastructure• reliability, accessibility, availability• accuracy, stability of student data

Commitment, motivation of clients Computer literacy of clients Information literacy of clients Positive attitude of lecturers Consultation and support to lecturers Sound instructional design practice Sound teaching and learning practice Commitment to cont. improvement

Exogenous factorsExogenous factors

Quality of the institutional LMS

Stability of national telecoms infrastructure

Class size

Workload of lecturers and students

Recognition and incentives for lecturers

Quality of the institutional LMS

Stability of national telecoms infrastructure

Class size

Workload of lecturers and students

Recognition and incentives for lecturers

Critical Success Factors

Critical Success Factors

Six studies from literature Six categories:

• Institutional Factors• Technology Factors • Lecturer Factors• Student Factors • Instructional Design Factors• Pedagogical Factors

Six studies from literature Six categories:

• Institutional Factors• Technology Factors • Lecturer Factors• Student Factors • Instructional Design Factors• Pedagogical Factors









Four levelsFour levels

ReactionLearningBehaviou

rResults

ReactionLearningBehaviou

rResults

Customer satisfaction

Kirkpatrick, D. 1998

Return on investment

Client satisfaction

Client satisfaction

Student feedback survey – online at end of each semesterStudent feedback survey – online at end of each semester

Lecturer interviews – pilot group of 22 active usersLecturer interviews – pilot group of 22 active users

Student feedback survey

Student feedback survey

July 2003 baseline data Overall Experience survey: 4 650 respondentsResponse rate 27.4 %

• Technical adequacy• Educational support• Affective domain• Online communication• Perceived learning

July 2003 baseline data Overall Experience survey: 4 650 respondentsResponse rate 27.4 %

• Technical adequacy• Educational support• Affective domain• Online communication• Perceived learning

Student frustration - 2003

Student frustration - 2003

Frustration index:

83% of students experience moderate to high levels of frustration

83% of students experience moderate to high levels of frustration

0

20

40

60

80

Low Moderate High

17%17%

73%73%

10%10%

Frustration index factors

Frustration index factors

insufficient computers available insufficient printers available extent of technical difficulties insufficient support from student CD inadequate student training in WebCT an impersonal learning experience slow response from classmates feelings of annoyance and/or stress

insufficient computers available insufficient printers available extent of technical difficulties insufficient support from student CD inadequate student training in WebCT an impersonal learning experience slow response from classmates feelings of annoyance and/or stress

Student satisfaction - 2003

Student satisfaction - 2003

Satisfaction index:

87% of students experience moderate to high levels of satisfaction

87% of students experience moderate to high levels of satisfaction

0

1020

3040

50

Low Moderate High

13%13%

44%44% 43%43%

Satisfaction index factors

Satisfaction index factors

feeling comfortable communicating via online tools feeling more freedom to express oneself than in a traditional classrooomlearning from the contributions of other studentspromoting one’s ability to work as a team or group memberincreased ability to plan one’s own work enriching learning environment

feeling comfortable communicating via online tools feeling more freedom to express oneself than in a traditional classrooomlearning from the contributions of other studentspromoting one’s ability to work as a team or group memberincreased ability to plan one’s own work enriching learning environment

Longitudinal measuresLongitudinal measuresMeasures of student satisfaction, 2003 – 2005Measures of student satisfaction, 2003 – 2005

2003 2004 2005

I have access to my own computer (either at home, at work or in the residence)

80.5% 78.6% 73.1%

I find it difficult to find a computer on campus when necessary

85.1% 50.3% 36.5%

I find it difficult to find a printer on campus when necessary

73.3% 45.3% 42.2%

My technical difficulties were solved within 24 hours 43.7% 79.3% 76.1%

I felt comfortable communicating via online tools 72.9% 80.0% 82.8%

Web-supported communication helped me to express myself more than I would have in a traditional classroom

79.1% 57.9% 60.9%

I found web-supported learning to be an enriching learning experience

81.9% 77.7% 78.7%

I experienced feelings of annoyance or stress during this web-supported learning experience

80.0% 37.9% 32.5%

I found the opportunities for ‘anywhere, anytime’ learning on the web to be convenient

73.8% 87.0% 88.3%

Lecturer feedback survey

Lecturer feedback survey Pilot interviews: 22 lecturers Pilot interviews: 22 lecturers

Benefits• useful for organisation and admin• communication and interaction• excellent support and service

Benefits• useful for organisation and admin• communication and interaction• excellent support and service Problems• library / copyright issues• human element is difficult

Problems• library / copyright issues• human element is difficult

Lessons learnt• training for staff and students• e-learning adds value

Lessons learnt• training for staff and students• e-learning adds value






Lessons: RisksLessons: Risks

formal online process based consensus driven ISO9000 cognisant

formal online process based consensus driven ISO9000 cognisant

Work load pressures for development of online system

Once created, it must be maintained and kept to up to date for internal functionality and external visibility

Work load pressures for development of online system

Once created, it must be maintained and kept to up to date for internal functionality and external visibility

Resistance to ideas of process; associated with commerce and inappropriate to HE?

Resistance to ideas of process; associated with commerce and inappropriate to HE?Inability to reach consensus amongst role players or to drive this forward

Expectation that consultant should be providing ‘answers’

Tendency to overlook the value of the system – that it is only a document management tool

Inability to reach consensus amongst role players or to drive this forward

Expectation that consultant should be providing ‘answers’

Tendency to overlook the value of the system – that it is only a document management tool

Automatic dismissal and perceptions of bureaucracy as soon as ISO9000 term is mentioned!

Automatic dismissal and perceptions of bureaucracy as soon as ISO9000 term is mentioned!

Resistance to generation of documentation

Time taken versus perceived benefits

Expectation that consultant should be providing best practice procedures

Inappropriate expectations; eg that the system will guarantee quality improvements

Resistance to generation of documentation

Time taken versus perceived benefits

Expectation that consultant should be providing best practice procedures

Inappropriate expectations; eg that the system will guarantee quality improvements

Lessons: BenefitsLessons: Benefits

Gartner hype cycle:

Source: http://news.bbc.co.uk/1/hi/technology/3577746.stm#graphic

Lessons: BenefitsLessons: Benefits1. Consensus that the QMS is on the slope of

enlightenment

2. Self evaluation and improvement of each step in the process; consensus on ‘the way we do things around here’

3. Rationalisation of pre-existing documents, checklists and forms

4. The QMS is a repository for the most up to date versions of all resources for planning, developing & evaluating projects

5. System informs new staff on UP procedures and enables them to feel self sufficient more quickly

1. Consensus that the QMS is on the slope of enlightenment

2. Self evaluation and improvement of each step in the process; consensus on ‘the way we do things around here’

3. Rationalisation of pre-existing documents, checklists and forms

4. The QMS is a repository for the most up to date versions of all resources for planning, developing & evaluating projects

5. System informs new staff on UP procedures and enables them to feel self sufficient more quickly

Lessons learntLessons learnt1. Fundamental ID model (process)2. Importance of Analysis and Evaluation3. Timely training in the basics of QA

theory and practice4. Doubts about the need for a

formalised QMS5. Time required for reflection on

practice6. Roles and responsibilities of all role

players7. Manage lecturers’ unrealistic

expectations8. Understand that the system will go

through a change management cycle

1. Fundamental ID model (process)2. Importance of Analysis and Evaluation3. Timely training in the basics of QA

theory and practice4. Doubts about the need for a

formalised QMS5. Time required for reflection on

practice6. Roles and responsibilities of all role

players7. Manage lecturers’ unrealistic

expectations8. Understand that the system will go

through a change management cycle






Conceptual frameworkConceptual framework

Applies process theory, critical success factors, and four levels of evaluation into an integrated framework

Applies process theory, critical success factors, and four levels of evaluation into an integrated framework

Conceptual frameworkConceptual framework

inputs process

outputs

measures

Lecturer satisfaction

Student satisfaction

ProductsProducts

web-supported courses &

multimedia products

feedback loop

ID factors

Pedagogical factors

Instructional Design Process

Instructional Design Process

Institutional factors

Technology factors

Lecturer factors

Student factors







Use the QMS and the conceptual framework to produce measures about effectiveness, efficiency and contribution of e-learning to teaching and learning

Use measures to prioritise improvements

Implement summative evaluation as a joint exercise between all role players

Integrate evaluation of e-learning with institutional programme review

Use the QMS and the conceptual framework to produce measures about effectiveness, efficiency and contribution of e-learning to teaching and learning

Use measures to prioritise improvements

Implement summative evaluation as a joint exercise between all role players

Integrate evaluation of e-learning with institutional programme review

Further readingFurther reading

Fresen (2005) Quality assurance practice in web-supported learning in higher education. PhD thesis. Available at http://upetd.up.ac.za/UPeTD.htm

Fresen & Boyd (2005) Caught in the web of quality. International Journal of Educational Development, 25(3), 317-331.

Fresen (in press)A taxonomy of factors to promote quality web-supported learning. International Journal on E-learning. To appear in 2006.

Fresen (2005) Quality assurance practice in web-supported learning in higher education. PhD thesis. Available at http://upetd.up.ac.za/UPeTD.htm

Fresen & Boyd (2005) Caught in the web of quality. International Journal of Educational Development, 25(3), 317-331.

Fresen (in press)A taxonomy of factors to promote quality web-supported learning. International Journal on E-learning. To appear in 2006.


Quality in e-learning means learning transformed and enhanced with the integration of technology where ‘e’

means:

empoweredenabledengaged

enlightenedemotional

enthusiastic

Quality in e-learning means learning transformed and enhanced with the integration of technology where ‘e’

means:

empoweredenabledengaged

enlightenedemotional

enthusiastic

Dr Ulf-Daniel Ehlers: Presentation at E-learning in Post-Secondary Education Conference, Alberta, Canada June 2005

Sharing experiencesSharing experiences

Jill Fresen: [email protected]

Lesley Boyd: [email protected]

Online QMS:http://www.up.ac.za/telematic/quality/quality.htm

Jill Fresen: [email protected]

Lesley Boyd: [email protected]

Online QMS:http://www.up.ac.za/telematic/quality/quality.htm

Documents

Dr Jill Fresen and Lesley Boyd Quality Management in E-learning: Towards an integrated approach Quality Management in E-learning: Towards an integrated