#oersymposium2014 S5 P1 Paul Kawachi

Open Education Network http :// www . open - ed. net

The TIPS Quality Assurance Framework

for creating OER : Validation

-------------------------------- ------ ------------------------------------

O E ROPEN EDUCATIONAL RESOURCESopen educational resources

Commonwealth Educational Media Centre for Asia


The TIPS Quality Assurance Framework

for creating OER : Validation

-------------------------------- ------ ------------------------------------

Paul Kawachi FRSA

kawachi @ open-ed.net

http://www.open-ed.net / oer-quality/ validation.ppt


overview

TIPS Framework

http://cemca.org.in/ckfinder/userfiles/files/

OERQ_TIPS_978-81-88770-07-6.pdf

Version 1.0


overview

TIPS Framework four layers

of ideas

T

I

P

S


overview

TIPS Framework


overview

TIPS Framework

from among

205 QA criteria

the most comprehensive

compilation to date

65 items

indicated by 60+

global experts

http://www.open-ed.net/oer-quality/criteria.pdf


overview

for teachers in class


overview

TIPS Framework

65 items

are here subjected to

delphi-style referral

for

content validation


definitions :

an open educational resource (OER)

is defined as a digital self-contained unit

of self-assessable teaching with

an explicit measurable learning objective,

having an open-licence clearly attached

to allow adapting, and generally being

free-of-cost to reuse

UNESCO - CoL 2011 ; Kawachi 2013

introduction


• wave analysis to confirm coverage - Leslie 1972

• content validity ratio analysis - Lawshe 1975

• multiple surveys and online discussions

methods


wave analysis

three waves of responses from individual OER experts


the three waves

each constructed using the survey responses

from ten individual OER experts

are visually similar

increasing confidence that a sufficient number

of data are collected, and CVR analysis can proceed

wave analysis


wave analysis

similar to fourth wave in gold from anon OER groups


wave analysis

fairly similar to waves 5 & 6 from individual teachers


age

age distribution for each set of responses


age distribution :

there was no clear normal distribution

there are tendencies for the

- Set-1 OER experts to be older

- Set-2 OER group participants to be younger

and

- Set-3 teachers also to be relatively young

age


geographic distribution of all responses

geographic

CC BY SA blank world map image by anon from WikiMedia http://commons.wikimedia.org/wiki/File:World_map_blank_black_lines_4500px_monochrome.png


survey population characteristics :

gender

Set-1 : individual OER experts

Set-2 : group OER experts

Set-3 : individual teachers

N

35

14

21

male

21

7

0

38

female

13

7

11

31


content validity ratio :

survey asks content experts to rank each criterion as

validation

• Essential

• Not-essential but useful

• Not necessary



min av CVR for a criterion to be retained at p < 0.05

validation



av CVR E for all items

= CV Index

for instrument

~ 0.80 for validity

at p < 0.05

validation

Set-1 Set-2 Set-3

retaining 24 criteria gives overall CVI = 0.80 0.79 0.79



validation

Set-1 Set-2 Set-3

( 9th May 2014 ) N=32 N=13 N=19

retaining 21 criteria : overall CVI = 0.80 0.79 0.79

( 5th June 2014 ) N=35 N=14 N=21


( 27th May 2014 ) N=35 N=13 N=19




5th June 2014

these 38 criteria retained are

all those described in the

shorter version

TIPS Framework

validated at

CVI > 0.80 at p < 0.05

validation

http://www.open-ed.net/oer-quality/tips.pdf



moreover

the original 65 criteria

of version 1.0

TIPS Framework

has CVR E+U = 0.94

by the 35 OER experts

indicating original was valid

validation


suggestions :

six criteria indicated at highest CVR E by OER experts

validation

• Don't use difficult or complex language, and

do check the readability to ensure it is appropriate to age / level

• Make sure that the knowledge and skills you want the student to learn are

up-to-date, accurate and reliable. Consider asking a subject-matter expert for advice

• Be sure the open licence is clearly visible

• Ensure your OER is easy to access and engage

• Present your material in a clear, concise, and coherent way, taking care with sound quality

• Use open formats for delivery of OER to enable maximum reuse and re-mix.


Quality Assurance Guidelines

for

Open Educational Resources

version 2.0

TIPS Framework

revised criteria with

some highlighted as

more important

and rubric added for

Quality Improvement

results

version 2.0

38 criteria


literature references :

CoL (2011). UNESCO-CoL Guidelines for Open Educational Resources (OER)

in Higher Education : OER Workshop, 25 May, Dar-es-Salaam. Retrieved April 25, 2014, from

http://oerworkshop.weebly.com/uploads/4/1/3/4/4134458/2011.04.22.oer_guidelines_for_higher_education.v2.pdf

Kawachi, P. (2013a). Quality assurance guidelines for open educational resources : TIPS framework, version 1.0.

New Delhi, India : Commonwealth Educational Media Centre for Asia (CEMCA). Retrieved March 7, 2014, from

http://cemca.org.in/ckfinder/userfiles/files/OERQ_TIPS_978-81-88770-07-6.pdf

Kawachi, P. (2013b). Quality not static. OER Quality, blog posting 30th January. Retrieved March 7, 2014, from

http://oerquality.wordpress.com/2013/01/30/quality-not-static/

Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28 (4), 563-575.

Retrieved February 21, 2014, from http://www.bwgriffin.com/gsu/courses/edur9131/content/Lawshe_content_valdity.pdf

Leslie, L.L. (1972). Are high response rates essential to valid surveys ? Social Science Research, 1, 323-334.

references


acknowledgements :

the author thanks Sanjaya Mishra and the

Commonwealth Educational Media Centre for Asia (CEMCA),

New Delhi, and the Commonwealth of Learning (CoL),

Vancouver, for their support

the author also thanks Ramesh Sharma for technical

assistance in reaching the survey population

thank you


You can download these slides freely from the website

http://www.open-ed.net / oer-quality/ validation.ppt

______________________________________________

or by email to me at

kawachi @ open-ed.net