Quality Assurance Strategies:Quality Assurance Strategies:Evaluating Learning ResourcesEvaluating Learning Resources

Quality Assurance Strategies:Quality Assurance Strategies:Evaluating Learning ResourcesEvaluating Learning Resources

Presented byPresented by::Karin Lundgren-CayrolKarin Lundgren-CayrolIn collaboration with:In collaboration with:

Gilbert Paquette and Suzanne LapointeGilbert Paquette and Suzanne Lapointe for CICE & GTN-Norméticfor CICE & GTN-Normétic

I2LOR WORKSHOP ON REPOSITORIESI2LOR WORKSHOP ON REPOSITORIESMontreal, November 8 2006Montreal, November 8 2006

OverviewOverviewOverviewOverview

• Goal and Objectives• Basic Concepts

– Learning Ressources (LR)– LR Repository Types– Quality Assurance Strategy– Quality Dimensions

• Integration/Development Processes

• LR Life-Cycle and Quality Assurance Strategies

• Conclusion

Goals and ObjectivesGoals and ObjectivesGoals and ObjectivesGoals and Objectives

Provide a set of Quality Assurance Provide a set of Quality Assurance Strategies to ensure Strategies to ensure LOR LOR

sustainabilitysustainabilityObjectivesObjectives• Establish quality assurance strategies• Provide pertinent evaluation criteria• Provide guidelines and principles

• Repository Managers & Developers• LO designers • LO users

What’s a Learning What’s a Learning Resource ?Resource ?

What’s a Learning What’s a Learning Resource ?Resource ?

Video

Learning Design

CD-ROM

Software

Electronic book

SCORM Object

ImageAny entity, digital or non-digital,

which is intended to be, or may be

used for the purpose of learning,

education or training 

Granularity

Quality Assurance Strategy?Quality Assurance Strategy?Quality Assurance Strategy?Quality Assurance Strategy?

List of StakeholdersRepository Type

User ProfileList of Needs

Metadata ProfileIntegrating a

Repository

Organisational and Technical

DecisionsIdentify Quality

Assurance Strategies

1.1. Design and ProduceDesign and Produce2.2. Integrate into LRIntegrate into LR3.3. Reuse and ReviseReuse and Revise

1.1. Design and ProduceDesign and Produce2.2. Integrate into LRIntegrate into LR3.3. Reuse and ReviseReuse and Revise

ISO 9000: 2000 Quality Assurance is the planned and Quality Assurance is the planned and

systematicsystematic activities put in place activities put in place to ensure quality requirements to ensure quality requirements

for a product or service will be fulfilled.for a product or service will be fulfilled.Document all processes. Document all processes.

Quality DimensionsQuality DimensionsQuality DimensionsQuality Dimensions

• Instructional QualityInstructional QualityContent clarity and conciseness, instructional Content clarity and conciseness, instructional strategies aligned to the learning objectives, strategies aligned to the learning objectives, appropriate media according to target appropriate media according to target audience, etc…audience, etc…

• Media and Ergonomical QualityMedia and Ergonomical QualityUser-friendliness, motivating, visually User-friendliness, motivating, visually attractive, built-in accessibility features, etc..attractive, built-in accessibility features, etc..

• Technical QualityTechnical QualityTechnical interoperability and robustness, Technical interoperability and robustness, metadata schema and tagging procedures, metadata schema and tagging procedures, conformance to standardsconformance to standards

IdentifyQuality

AssuranceStrategies

LO Design and Production

Deposit andindexing

C

CReuse and Annotate

C

Pedagogical and ergonimical quality LO

I/P

I/PInteroperable findable LO (Technical

Quality)

I/P

I/P

Demands forRevision/New LR

I/P

TracesAnnotation

Recommendations

I/P

Integratingor developing a

LOR

Expressgoals

identify actors

Identifyneeds

Specify types of LOs

C

C

C

C

Grid of Needs/User

I/P

List type of users and

stakeholders

I/P

I/P

Goal andobjectives

I/P

Blueprintof the LORI/P

Choose MetadataSchema

C

MetadataProfile

P

I/P

I/P

List of types of LOs

I/P

I/P

I/P

R

S

Ensure Management

Support

R

Compatible Technical and organisational Infrastructure

R

Policy on Intellectual

propertyDMR

RP

Process preceeding theProcess preceeding theintegration/development of integration/development of a Learning Object Repositorya Learning Object Repository

The LR Life-cycle and The LR Life-cycle and how to identifyhow to identify

Quality Assurance Quality Assurance StrategiesStrategies

Strategies at the Organisational Strategies at the Organisational LevelLevel

Strategies at the Organisational Strategies at the Organisational LevelLevel

Integrating a LR

Stakeholdersupport

InstitutionalSupport

TechnologicalInfrastructure

Policy on copyrights

DefineDefineGoal of LRGoal of LR

IdentifyIdentifyActorsActors

SpecifySpecifyMetadata Metadata schemaschema

IdentifyIdentifyNeedsNeeds

SpecifySpecifyResource Resource

TypesTypes

Goal

Stakeholder list

ResourceTypes

& ExpertiseLISTof User

Profile

• Repository TypeRepository Type• StakeholdersStakeholders• User ProfileUser Profile• List of NeedsList of Needs• List of Resource TypesList of Resource Types• Metadata ProfileMetadata Profile

LR Life-Cycle and LR Life-Cycle and Identifying Quality Strategies Identifying Quality Strategies

LR Life-Cycle and LR Life-Cycle and Identifying Quality Strategies Identifying Quality Strategies

Design/Design/ProductionProduction

Deposit andDeposit andmetadata referencingmetadata referencing

Adaptation/Adaptation/Reuse/AssignmentReuse/Assignment

Life-cycle Life-cycle of a LRof a LR

Before inclusion

During contribution

After inclusion

Pedagogical/Media Pedagogical/Media QualityQuality Technical QualityTechnical Quality

All dimensionsAll dimensions

REVISIONREVISION

LR Repository Types and Main LR Repository Types and Main CharactersiticsCharactersitics

LR Repository Types and Main LR Repository Types and Main CharactersiticsCharactersitics

• Private RepositoriesPrivate Repositories – Quality = personal usefulness of the LOQuality = personal usefulness of the LO– Metadata schema – Recommendation systems (social/collaborative

tagging/folksomy)

• Community repositoryCommunity repository– Quality = Member ship and peer reviewed LO’sQuality = Member ship and peer reviewed LO’s– Domain specific interest, practice or discipline– Protects the consumer trough membership– May use domain ontology

• Public RepositoryPublic Repository – Quality = Annotation and Right’s ManagementQuality = Annotation and Right’s Management– Metadata validation system and automatic capturing– Propose evaluation criteria for different resources– Provide evaluation, recommendation and annotation

systems

SummarySummarySummarySummary

Graphical designer: Julie Aubin, 2006

Too Much is OverkillToo Much is OverkillToo Little Just KillsToo Little Just Kills

Too Much is OverkillToo Much is OverkillToo Little Just KillsToo Little Just Kills

Karin Lundgren-CayrolKarin Lundgren-CayrolSuzanne LapointeSuzanne LapointeGilbert PaquetteGilbert Paquette

CICE, TÉLUQ - UQAMCICE, TÉLUQ - UQAM

Give me a

chance!

During Contribution During Contribution During Contribution During Contribution

Strategies here could be referring to all types of quality dimensions:- Pedagogical and Ergonomical Quality by

enforcing Peer Review - Controlled contribution

But if Strategies has been applied at the design/production stage, strategies here normally only need to refer to- The quality of the metadata - Interoperability of the Learning Resource and

its Metadata Record

During ContributionDuring ContributionDuring ContributionDuring Contribution

Multi-actor expertise for metadata – Peer Review Strategies:

• Involving Content Expert / Author /ID Expert• Title, LR language, key words, description, type,

version, contributors, intended end user, learning context

• Text mining algorithms can help – Metadata Quality Strategies

• Library technician• Version, classification, rights, relations metadata• Overall respect of the standard used

– Interoperability • Computer Technicians• Format, size, required conditions, Installation

Remarks, meta-metadata, record language

General Principle: Reduce form-filling: use wizards, Reduce form-filling: use wizards,

smart automatic or semi-automatic smart automatic or semi-automatic computer agentscomputer agents

During ContributionDuring ContributionDuring ContributionDuring Contribution

• Demand Membership for contributors: – Gives responsability and motivation; – Make contributors visible by ‘Business Card’

portfolio, chat groups etc.– Suggest domain specific Community of Practices

• Make sure that the author provides the following infos:– Degree of Pedagogical Reusability (RightsRights)– Content quality indications (Method,checklist,

evaluation used etc)– Interface quality (idem)– How it might be an efficient learning or teaching

tool

Learning Design as Agregated Learning Design as Agregated ObjectsObjects

Learning Design as Agregated Learning Design as Agregated ObjectsObjects

BasicResources

ToolsDocuments

MediaElements

Actors

Units ofLearning

OperationsScenarios Processes

Before InclusionBefore InclusionBefore InclusionBefore Inclusion

Strategies that refers to Instructional Strategies that refers to Instructional Design of the Learning Resource and Design of the Learning Resource and aims at: aims at: - Pedagogical QualityPedagogical Quality- Ergonomical QualityErgonomical Quality

Demands specilized ID methods if LR Demands specilized ID methods if LR is a Unit of Learning compliant with :is a Unit of Learning compliant with :- IMS LD IMS LD - SCORM SCORM - Other like WEBCT, BlackBoard etc..Other like WEBCT, BlackBoard etc..

Some Guidance PrinciplesSome Guidance PrinciplesSome Guidance PrinciplesSome Guidance Principles

• Design and Production Strategies– Use a solid and adapted ID method– Support Collaborative design, that is let

specialists apply their expertise– Clearly identify knowledge and user

competencies – Favor pedagogical strategies putting the

learner in the center – Apply evaluation criteria during

development and implement at least one learner/peer evaluation cycle

– Be informed about Access4All production principles

Instuctional Engineering Instuctional Engineering MethodMethod

Instuctional Engineering Instuctional Engineering MethodMethod

Knowledge Modeling 210 Knowledge modeling principles212 Knowledge model214 Target competencies310 Learning units content 410 Learning instruments content 610 Knowledge and competency management

Instructional Modeling

220 Instructional principles222 Learning events network224 Learning units properties320 Instructional scenarios322 Learning activities properties420 Learning instruments properties620 Actors and group management

Materials Modeling

230 Media principles 330 Development infrastructure 430 Learning materials list 432 Learning materials models 434 Media elements 436 Source documents630 Learning system / resource management

Delivery Modeling

240 Delivery principles242 Cost-benefit analysis340 Delivery planning440 Delivery models442 Actors and user’s materials444 Tools and telecommunication446 Services and delivery locations540 Assessment planning640 Maintenance / quality management

Problem definition100 Training system102 Training objectives

104 Target Learners106 Actual situation

108 Reference documents

MISAMISA

• Content Expert• Instructional Designer• Media Specialist• Delivery Specialist• Project Leader

Collaborative Design Collaborative Design Collaborative Design Collaborative Design

Note:Note: Interactive Objetsare Software and

answers to software quality

criteria

Basic principle:Basic principle:Build or

integrate objects that you

can quality certify

The The LORI LORI Evaluation CriteriaEvaluation CriteriaThe The LORI LORI Evaluation CriteriaEvaluation Criteria

• Content QualityVeracity, accuracy, balanced presentation of ideas, and appropriate level of detail

• Goal AlignmentAlignment among learning goals, activities, assessments, and learner characteristics

• Feedback and AdaptabilityAdaptive content or feedback driven by differential learner input or learner modeling

• MotivationAbility to motivate, and stimulate the interest or curiosity of, an identified population of learners

• Visual DesignDesign of visual and auditory information for enhanced learning and efficient mental processing

• Usability Ease of navigation, predictability of the user interface, and the quality of UI help features

• Accessibility Support for learners with disabilities

• ReusabilityAbility to port between different courses or learning contexts without modification

• Standard conformanceAdherence to international standards and specifications

Nesbit, Belfer & Vargo, 2003

LORI instrumentLORI instrumentLORI instrumentLORI instrument

Adding:

General quality or the rate (X/45) or % ?

Other criteria ?

Accessibility4AllAccessibility4AllAccessibility4AllAccessibility4All

The TILE PROJECT : http://barrierfree.ca/tile/index.htm

Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0

After Inclusion in the After Inclusion in the RepositoryRepository

After Inclusion in the After Inclusion in the RepositoryRepository

Refers to strategies that provide:Refers to strategies that provide:• Retrieval quality

– Free text search– Browsing by Classification Schemes or

Typologies– Ontological search– Federated search

• Easy display of information– Descriptive resumé– Full meta data record– How many used it– What do they think

• Peer reviews and evaluations from actual users/reusers.

– Simple Annotation systems– Social tagging/bookmarking and Folksomy – Collaborative filtering techniques– Recommendation algorithms

After Inclusion StrategiesAfter Inclusion StrategiesAfter Inclusion StrategiesAfter Inclusion Strategies

•List of new and innovative high quality LRs

•Suggestions for: – Revisions to authors/designers

for improvements– Renewal of resource– New resource

MERLOTMERLOTMERLOTMERLOT

Metadata displayMetadata displayMetadata displayMetadata display

PALOMA WEBPALOMA WEBPALOMA WEBPALOMA WEB

Federated search displays Source Repository

EDNA OnlineEDNA OnlineEDNA OnlineEDNA Online

SEARCH ARIADNE DISPLAYSEARCH ARIADNE DISPLAYSEARCH ARIADNE DISPLAYSEARCH ARIADNE DISPLAY

FORMIST/deLicio.usFORMIST/deLicio.usFORMIST/deLicio.usFORMIST/deLicio.us

Social Tagging Information

SMETESMETESMETESMETETechnical

information

Rightsinformation

Reviews information