STUDENT SATISFACTION WITH STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL WEB-BASED EDUCATIONAL

RESOURCES FOR COMPUTER & RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching INFORMATION SCIENCE: Teaching

with Merlot Collectionwith Merlot Collection

Emrah Orhun, Ph.D.Emrah Orhun, Ph.D.

Troy State University MontgomeryTroy State University Montgomery

Alabama, USAAlabama, USA

OutlineOutline

IntroductionIntroduction Learning and Teaching Materials in The Learning and Teaching Materials in The

Merlot CollectionMerlot Collection Students’ Satisfaction with Merlot Materials Students’ Satisfaction with Merlot Materials

in CIS Classesin CIS Classes ConclusionsConclusions

IntroductionIntroduction

Difficulties in locating relevant and Difficulties in locating relevant and affordable materials of high qualityaffordable materials of high quality – World Lecture Hall at the University of Texas World Lecture Hall at the University of Texas

» 83 categories to browse and search 83 categories to browse and search utilities to locate materialsutilities to locate materials

– WebCT (2002) WebCT (2002) » 1,250 publisher course content packs in more 1,250 publisher course content packs in more

than 50 subject areas than 50 subject areas » customizable materials around which to build customizable materials around which to build

courses combined with the functionality of WebCT's courses combined with the functionality of WebCT's course management software. course management software.

Quality issue Quality issue

World Lecture HallWorld Lecture Hall materials materials – courses offered at accredited colleges and courses offered at accredited colleges and

universities around the world universities around the world WebCT packsWebCT packs

– editorial standards of the publisher and editorial standards of the publisher and author of the textauthor of the text

MERLOT (Multimedia Educational Resource MERLOT (Multimedia Educational Resource for Learning and Online Teaching)for Learning and Online Teaching)

Web-based open resource designed Web-based open resource designed primarily for faculty and students in higher primarily for faculty and students in higher education education

emphasizes the quality issue emphasizes the quality issue

MERLOT repositoryMERLOT repository

– Over 8,000Over 8,000 free learning materials and free learning materials and support materials that can be integrated support materials that can be integrated within the context of a larger course within the context of a larger course

Searching and Browsing in MERLOTSearching and Browsing in MERLOT

Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category

Figure 2. Search results for TECHtionary

Support resourcesSupport resources

– sample assignments sample assignments

– evaluations of the learning materialsevaluations of the learning materials

»discipline panel peer reviews discipline panel peer reviews

»individual user reviews individual user reviews

Peer Review ProcessPeer Review Process

Discipline communities Discipline communities Biology, business, chemistry, engineering, Biology, business, chemistry, engineering,

health sciences, history, information health sciences, history, information technology, mathematics, music, physics, technology, mathematics, music, physics, psychology, teacher education, and world psychology, teacher education, and world languages. languages.

Discipline teams Discipline teams (“MERLOT Editorial Boards”) (“MERLOT Editorial Boards”)

– selected by the MERLOT partner organizations selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. in accordance with the MERLOT guidelines.

– develop evaluation standards, develop evaluation standards, – conduct peer reviews of learning materials conduct peer reviews of learning materials

within the discipline, and within the discipline, and – make connections with their professional make connections with their professional

organizationsorganizations

Peer review processPeer review process

Evaluated by at least two team members who Evaluated by at least two team members who have relevant expertise. have relevant expertise.

Review process stagesReview process stages– triagetriage review to identify worthy candidates review to identify worthy candidates – review of candidates review of candidates – composite review report sent back to the authors composite review report sent back to the authors

for feedback and permission to post the review on for feedback and permission to post the review on MERLOT. MERLOT.

– approved review is posted to the MERLOT Web approved review is posted to the MERLOT Web site by the discipline leader site by the discipline leader

The evaluation criteria: The evaluation criteria: –quality of content, quality of content,

–potential effectiveness as a potential effectiveness as a teaching tool, teaching tool,

–and ease of use. and ease of use.

Each dimension evaluated separately Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is highest), and an overall rating is computed computed

A review must average three stars in A review must average three stars in order to be posted to the site. order to be posted to the site.

Figure 3(a). Peer Review of TECHtionary (Part a)

Figure 3(b). Peer Review of TECHtionary (Part b)

Figure 3(c). Peer Review of TECHtionary (Part c)

Figure 4. Member Comments for TECHtionary

Current state of MERLOTCurrent state of MERLOT

Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002)members, and peer reviews (May 28, 2002)

   TotalTotal AddedAdded

during last during last 365 days365 days

AddedAdded

during last during last 30 days30 days

MaterialsMaterials 68636863 1991 1991 229229

MembersMembers 86138613 4142 4142 600600

Peer Peer ReviewsReviews

479479 361361 2020

Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviews members, and peer reviews

((February 27, 2003February 27, 2003))

TotalTotal AddedAdded

during last during last 365 days365 days

AddedAdded

during last during last 30 days30 days

MaterialsMaterials 83318331 23382338 284284

MembersMembers 1263712637 55275527 712712

Peer ReviewsPeer Reviews 778778 8686 22

Member Member CommentsComments

17901790 305305 1818

AssignmentsAssignments 322322 5959 11

Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviewsmembers, and peer reviews

for IT Categoryfor IT Category

TotalTotal AddedAdded

during last 365 during last 365 daysdays

AddedAdded

during last 30 during last 30 daysdays

MaterialsMaterials 559559 274274 2121

MembersMembers 524524 259259 3030

Peer ReviewsPeer Reviews 3636 33 00

Member Member CommentsComments

218218 9999 66

AssignmentsAssignments 1111 99 00

May 25, 2002May 25, 2002 September September 26, 200226, 2002

February 27, February 27, 20032003

MaterialsMaterials 68636863 73957395 83318331

% Peer % Peer ReviewsReviews

7.07.0 8.58.5 9.39.3

% Member % Member CommentsComments

21.221.2 20.820.8 21.421.4

% % AssignmentsAssignments

3.43.4 3.83.8 3.93.9

Ratio of Peer Reviews, Member Comments, and Ratio of Peer Reviews, Member Comments, and Assignments to MaterialsAssignments to Materials

ProblemProblem Percentage of reviewed materials lowPercentage of reviewed materials low

Potential SolutionPotential Solution Peer review by community of learnersPeer review by community of learners

CauseCause Editorial peer review process too Editorial peer review process too

slowslow

STUDENTS’ SATISFACTION WITH STUDENTS’ SATISFACTION WITH MERLOT MATERIALS IN CIS CLASSESMERLOT MATERIALS IN CIS CLASSES

Goal of StudyGoal of Study

To find out to what extent MERLOT To find out to what extent MERLOT provides relevant and quality materials for provides relevant and quality materials for the CIS coursesthe CIS courses..

QuestionsQuestions

Is the quality and quantity of learning materials in Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific students’ knowledge/learning needs in specific courses?courses?

Is the MERLOT collection ‘usable’ by the Is the MERLOT collection ‘usable’ by the students?students?

Does the MERLOT collection induce the students Does the MERLOT collection induce the students to become members of learning communities?to become members of learning communities?

Can peer review by students substitute for peer Can peer review by students substitute for peer review by editorial boards?review by editorial boards?

Questions re-formulatedQuestions re-formulated

How do students rate their satisfaction with How do students rate their satisfaction with MERLOT collection?MERLOT collection?

How do students rate the learning materials How do students rate the learning materials as reviewers? Do students’ ratings of the as reviewers? Do students’ ratings of the learning materials differ from the ratings learning materials differ from the ratings given by Discipline Panel members?given by Discipline Panel members?

Teaching with the MERLOT Teaching with the MERLOT Collection in CIS CoursesCollection in CIS Courses

Summer-Fall 2002 semesters in the following Summer-Fall 2002 semesters in the following CIS classes at Troy State University CIS classes at Troy State University Montgomery:Montgomery: – CIS 5547 CARCIS 5547 CAR Applied Systems AnalysisApplied Systems Analysis– CIS 5547 CBRCIS 5547 CBR Applied Systems Analysis Applied Systems Analysis – CIS 3332 CIS 3332 Software Engineering ISoftware Engineering I– CIS 3357 CIS 3357 Logical Structures of Logical Structures of

Computer DesignComputer Design

Overall, 41 students participated in the evaluation. Overall, 41 students participated in the evaluation. A total of 252 evaluations were made. A total of 252 evaluations were made.

 

Evaluation of web-based learning materials   1. 1.  Visit MERLOT (www.merlot.org) site and register as a member.2. 2.       For each chapter covered in the course:3. a.      Find a relevant learning material in the IT subcategory.4. b.      Review the material using the guidelines provided by MERLOT.

In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT.

5. c.      Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all.

6. d.      Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment.

Student Assignment

ChapterChapter TopicTopic TitleTitle URLURL Learning Learning outcomeoutcome

Satisfaction Satisfaction with MERLOT with MERLOT

collection for this collection for this task 5= very good task 5= very good

1=very poor1=very poor

11 Players in the Players in the System GameSystem Game

Enterprise Enterprise Resource Resource Planning Planning SupersiteSupersite

http://www.http://www.erpsupersiteerpsupersite.com.com

Provides Provides analysis on analysis on ERP trendsERP trends

33

22 Information Information System Building System Building BlocksBlocks

Interface Interface Design for Design for Educational Educational MultimediaMultimedia

http://www.http://www.cdlcdl..eduedu..rachelrachel/PDF//PDF/intdesmmintdesmm..pdfpdf

Describes the Describes the principle of principle of interface interface designdesign

44

33 Information Information System System DevelopmentDevelopment

Risk analysis Risk analysis and and management management for Projects for Projects (RAMP)(RAMP)

http://www.http://www.rampriskramprisk.com/.com/frontindexfrontindex..htmhtm

Comprehensive Comprehensive outline of outline of RAMPRAMP

  55

Example of Relevant MERLOT Material

ChapterChapter TopicTopic TitleTitle URLURL Learning Learning outcomeoutcome

11 Players in the Players in the System GameSystem Game

New New responsibilities responsibilities Fall Heavy on IT Fall Heavy on IT WorkersWorkers

www.www.informationweekinformationweek.com.com

Talks about the Talks about the new job new job responsibilities responsibilities of current and of current and upcoming IT upcoming IT employees.employees.

22 Information Information System Building System Building BlocksBlocks

TP Middleware TP Middleware keeping up with keeping up with Mainframe Mainframe RevivalRevival

www.www.adtmagadtmag.com.com

Explains the Explains the functionality of functionality of the product TP the product TP middleware on middleware on mainframe mainframe systems.systems.

33 Information Information System System DevelopmentDevelopment

Tips for a Tips for a successful successful Software Software Distribution Distribution Project.Project.

www.www.computerworldcomputerworld.com.com

Shares other Shares other companies companies success stories of success stories of in relation to in relation to Software Software Distribution Distribution Project.Project.

Example of New MERLOT material posted

How do students rate their satisfaction with How do students rate their satisfaction with MERLOT collection?MERLOT collection?

COURSE

Number of

Students

Number of

Topics

Number of

Evaluations

Average sdev

CIS3357 Logical Structures of Computer Design 8 7 49 3.8 1.3

CIS5547 (A+B) Applied Systems Analysis 22 14 165 3.8 0.8

CIS3332 Software Engineering I 11 4 38 3.8 1.0

Students’ mean satisfaction ratings for MERLOT collection

Ch

Topic

NAverag

e sdev Min MaxPercent rated 4-5

1 Binary Numbers & Codes 8 4.7 1.4 3 5 87.5

2 Digital Circuits 8 4.6 1.5 3 5 75.0

3 Combinational Circuits 8 4.3 1.9 1 5 62.5

4 Sequential Logic Systems 8 4.2 1.9 1 5 62.5

5 Registers & Counters 8 3.8 2.1 1 5 50.0

6 Memory & Programmable Logic 5 3.2 1.5 1 5 40.0

7 Register Transfer & Computer Operations 4 3.8 1.9 1 5 75.0

CIS3357 Students’ satisfaction with MERLOT collection

Ch

Topic

NAverag

e sdev Min MaxPercent rated 4-5

1 Process 13 4.2 0.8 1 5 76.9

2 Project Management 12 4.2 0.4 2 5 91.7

3 Requirements Analysis I 11 3.8 1.2 1 5 72.7

4 Requirements Analysis II 10 3.6 1.2 1 5 70.0

5 Software Architecture 13 4.2 0.8 1 5 76.9

6 Detailed Design 12 4.2 0.4 2 5 91.7

CIS3332 Students’ satisfaction with MERLOT collection

Ch

Topic

NAvera

ge sdev Min MaxPercent rated

4-5

1 Players in the System Game 22 3.6 0.8 2 5 59.1

2 Information System Building Blocks 20 3.7 0.8 2 5 55.0

3 Information System Development 19 3.6 1.2 1 5 63.2

4 Project Management 10 4.1 0.7 3 5 80.0

5 Systems Analysis 12 3.8 0.6 3 5 75.0

6 Requirements Discovery 12 3.9 0.8 3 5 66.7

11

Application Architecture and Modeling

10 3.9 0.7 3 5 70.0

CIS5547 Students’ satisfaction with MERLOT collection

A Object-Oriented Analysis and Modeling 9 3.7 0.7 3 5 55.6

B Object-Oriented Design and Modeling 9 3.9 0.8 3 5 66.7

13

Output Design and Prototyping 9 4.0 0.5 3 5 88.9

14

Input Design and Prototyping 9 4.2 0.4 4 5 100.0

15

User Interface Design 9 3.9 0.8 3 5 66.7

16

Systems Construction and Implementation

7 4.1 0.9 3 5 71.4

17

Systems Operations and Support 8 4.1 0.8 3 5 75.0

CIS5547 Students’ satisfaction with MERLOT collection (contd.)

How different are students’ How different are students’ evaluations from Editorial Peer evaluations from Editorial Peer

Reviews?Reviews?

TypeType ****** ******** **********

TutorialTutorial File System DesFile System Design for an NFS ign for an NFS File Server AppFile Server Applianceliance

  

Cascading StyleCascading Style Sheets, Level1 Sheets, Level1

Electronic ComElectronic Commerce Tutorial merce Tutorial

  

FTP TutorialFTP Tutorial

Learning Learning Microsoft Word Microsoft Word 20002000

  

Web-Database Web-Database IntegrationIntegration

  

Mulder's Mulder's Cascading Style Cascading Style Sheets TutorialSheets Tutorial

  

Sharky's Sharky's Netscape Netscape Frames TutorialFrames Tutorial

  

Tje ABCs of Tje ABCs of Building a Web Building a Web SiteSite

Data Data CommunicationCommunicationss

  

ADSLADSL

  

Computer Computer Knowledge Knowledge Virus Tutorial Virus Tutorial

Reference Reference MaterialMaterial

Study Guides Study Guides and Strategiesand Strategies

Tips for Using Tips for Using BlackboardBlackboard

AnimationAnimation TECHtionaryTECHtionary Warriors of the Warriors of the netnet

Distribution of materials in IT

Material Material TypeType

****** ******** **********

TutorialTutorial      

ADSLADSL

Reference Reference MaterialMaterial

Web-DatabaseWeb-Database Integration Integration

AnimationAnimation TECHtionaryTECHtionary   

Distribution of materials according to type and rating Distribution of materials according to type and rating (excluding materials with no user comments with 3 (excluding materials with no user comments with 3 categories)categories)

Sample of Materials from IT Category

Sample of StudentsSample of Students

CIS3332 SOFTWARE ENGINEERING (16)CIS3332 SOFTWARE ENGINEERING (16) CIS5547 SYSTEMS ANALYSIS (27)CIS5547 SYSTEMS ANALYSIS (27)

CaseCase Editorial Editorial Board Board RatingRating

Member Member RatingRating

Students’ Students’ RatingRating

(mean)(mean)TECHtionaryTECHtionary 3.03.0 ********** 3.753.75

Web-DatabaseWeb-Database Integration Integration 3.673.67 ****** 3.823.82

Students’ Ratings

MeanMean Std. Dev.Std. Dev. Std. ErrorStd. Error VarianceVariance Coef.Var.Coef.Var. CountCount

3.7543.754 .421.421 .086.086 .177.177 11.21811.218 2424

MinimumMinimum MaximumMaximum RangeRange SumSum Sum Sum SquaredSquared

# Missing# Missing

2.52.5 4.44.4 1.91.9 90.190.1 342.33342.33 00

Students’ ratings for TECHtionary

Editorial Board Rating: 3.00Editorial Board Rating: 3.00

MeanMean Std. Dev.Std. Dev. Std. ErrorStd. Error VarianceVariance Coef. Var.Coef. Var. CountCount

3.8293.829 .545.545 .119.119 .297.297 14.23814.238 2121

MinimumMinimum MaximumMaximum RangeRange SumSum Sum Sum SquaredSquared

# Missing# Missing

2.82.8 4.94.9 2.12.1 80.480.4 313.76313.76 00

Students’ ratings for Web-Database Integration

Editorial Board Rating: 3.67Editorial Board Rating: 3.67

Is the mean student rating different from the rating Is the mean student rating different from the rating given by the Editorial Board?given by the Editorial Board?

TECHtionary– Different (one-group t-value = 8.773, p=0.0001)

Web-Database Integration– Insufficient evidence (one-group t-value = 1.333,

p=0.1975)

ConclusionsConclusions

Students are satisfied with the Students are satisfied with the quality and quality and quantity of learning materials they found in quantity of learning materials they found in the IT category.the IT category.

Current data suggest that students rate Current data suggest that students rate materials differently than editorial boards.materials differently than editorial boards.

The present limited study does not provide The present limited study does not provide evidence for the notion that peer review by evidence for the notion that peer review by students can substitute for peer review by students can substitute for peer review by editorial boards.editorial boards.