Evaluating the Impact of Open Educational Resources: A ... · an open psychology textbook. ... (see Table 1). Final exam grades dropped the ... anatomy, computer science, social sciences,

portal: Libraries and the Academy, Vol. 17, No. 4 (2017), pp. 685–693. Copyright © 2017 by Johns Hopkins University Press, Baltimore, MD 21218.

FEATURE: REPORTS FROM THE FIELD

Evaluating the Impact of Open Educational Resources: A Case StudyYoung Mi Choi and Cathy Carpenter

abstract: This paper presents a case study of the replacement of a textbook with free to use materials in a class on Human Factors and Ergonomics. Data on exam grades, course grades, and student opinions over a total of five semesters are reported, before and after the implementation. The results show that class performance remained similar before and after. The use of free resources has not appeared to negatively affect learning outcomes. Advantages and challenges to the development and use of free materials are discussed.

Introduction

The cost of college textbooks is an increasing and significant expense. Students are often forced to consider the cost of books when deciding when or what classes to take. An alternative to expensive textbooks is the development and use of open

educational resources (OER). This paper presents a case study of using free materials to substitute for a textbook in a class on Human Factors and Ergonomics. The effectiveness of the new materials is evaluated based on data from exam grades, course grades, and student opinions collected both before and after the substitution. The results show that class performance is similar both when using the textbook and after replacing it with open educational resources. The use of free materials did not appear to impair learning outcomes. Benefits and difficulties in finding, collecting, and using free materials are discussed.

Background

The cost of textbooks, particularly in the United States, is an increasing burden for col-lege students. Between 1986 and 2004, textbook prices nearly tripled.1 Today, such costs reach $1,200 per year for an average student.2 The cost of books can affect the decision whether to take a course. In addition, 65 percent of students report choosing not to This

mss

. is pe

er rev

iewed

, cop

y edit

ed, a

nd ac

cepte

d for

publi

catio

n, po

rtal 1

7.4.

Evaluating the Impact of Open Educational Resources686

purchase a required textbook because of the cost. Almost all the students who made this decision (94 percent) were concerned that it would impact their grade. Ideally, students should not have to worry about cost preventing them from taking a needed course or impacting their ability to perform and learn from the class after they enroll.

Open textbooks are an alternative to costly textbooks that can be made freely avail-able.3 Open textbooks are part of the larger movement of open educational resources (OER). The OER movement includes any teaching, learning, and research resources that are in the public domain or have been made available under a license that allows them to be freely used or repurposed by others. Open educational resources can include full courses, course materials, textbooks, videos, tests, software, tools, or other techniques that are used to support access to knowledge.4

While open educational resources are becoming increasingly available, there are questions related to student perception and effectiveness of the materials. There are difficulties related to finding and properly using OER.5 The availability of OER can be limited because of the work required to develop the resources, often not included in an educator’s official list of responsibilities.6

There are benefits to instructors in the development of good OER. In theory, open educational resources will provide higher quality and more depth than materials that could be developed by any single instructor. OER materials often benefit from the “wisdom of the crowd” in which the collective knowledge and experience of a group of experts contribute greater diversity and depth to the learning materials than any single person could. The use of OER also can allow the instructor to spend less time on developing class resources and more on supporting student learning.7

So far, it is not known how a shift to OER will impact student learning or satisfac-tion. Little research on this question is avail-able, though some results indicate that such a change has been positive. In one case study, Houston Community College in Texas adopted an open psychology textbook. One of three class sections used the open textbook, while the other two sections employed the original textbook. Comparison of the class outcome metrics indicated that student performance on exams, class grades, and retention rates were higher for the class that used the open textbook

than for classes that used non-OER materials.8 This paper presents a case study of the development and implementation of free

course materials. A textbook transformation grant from Affordable Learning Georgia (ALG) supported the development of the materials.9 ALG is an initiative of the Univer-sity System of Georgia (USG) to provide grant-supported opportunities for USG faculty, libraries, and institutions to convert their use of expensive textbooks and learning ma-terials into lower cost options. The goals of the textbook transformation grants include:

1. Pilot different approaches in USG courses for textbook transformation including adoption, adaptation, and creation of open educational resources (OER) and/or

There are benefits to instruc-tors in the development of good OER. In theory, open education-al resources will provide higher quality and more depth than materials that could be devel-oped by any single instructor.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.

Young Mi Choi and Cathy Carpenter 687

identification and adoption of materials already available in GALILEO (Georgia Library Learning Online) and USG libraries.10

2. Provide support to faculty, libraries, and their institutions to implement these approaches.

3. Lower the cost of college for students and contribute to their retention, progres-sion, and graduation.

Method

The authors of this article, an assistant professor in the Georgia Tech College of Design and the head librarian in the Georgia Tech Architecture Library, collected new class materials for a course in Human Factors and Ergonomics over the fall 2014 semester. The Human Factors course was chosen due to the availability of funding to identify new materials, the relative expense of available textbooks, and the observation that many students attempted to take the class without purchasing the textbook. The new materials were selected to include the key topics and learning areas that were covered by the original course textbook. This work also involved updating lectures, activities, and other course materials; however, the updates were minor.

The course first used the new collection of materials during the spring 2015 semester. Several measurements were utilized to determine outcomes with the new materials. The class grades on the midterm exam, final exam, and final average were used because the pool of questions from which the exams were generated was unchanged. Student opinions were gathered through the end-of-semester Course Instructor Opinion Survey (CIOS). Three questions from this survey were examined:

• How prepared were to you take the subject?• How much did you learn?• How effective was the course overall?

Finally, after the course, students were asked to send comments about their experience in the course and their opinion of the free materials. Only one section of the class is of-fered in each semester, so data from each of these sources were compared with previous semesters.

Results

The course offered in the spring 2014 and fall 2014 semesters utilized a non-free textbook. The no-cost materials were introduced in the spring 2015 semester. There have been significant differences in midterm exam grades in each semester both before and after the introduction of the no-cost materials (see Table 1). Final exam grades dropped the semester the no-cost materials were introduced but have since remained on par with grades from before (see Table 2). There were no significant differences in final course grades from the fall 2014 semester through spring 2016 (see Table 3). Figure 1 shows a historical plot of the class averages for the midterm exam, final exam, and final course grades.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


Table 1.Midterm exam grades from previous semesters

Spring 2014 Fall 2014 Spring 2015 Fall 2015 Spring 2016

Mean 85.20 80.38 82.69 78.17 76.67Standard error 0.92 1.89 0.96 1.30 1.15Median 87.00 82.00 85.00 79.00 78.00Mode 91.00 88.00 86.00 89.00 80.00Standard deviation 8.36 10.67 7.88 9.54 8.46Variance 69.81 113.79 62.16 91.01 71.58Range 44.00 46.00 35.00 35.00 38.00Count (N) 82 32 67 54 54Confidence (95%) 1.84 3.85 1.92 2.60 2.31

Table 2.Final exam grades from previous semesters



This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


Table 3.Final course grades from previous semesters



Figure 1. Plots of average midterm, final exam, and final course grades

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


Figure 2 shows results from the end-of-semester Course Instructor Opinion Survey (CIOS). Student opinion of the amount learned in the class and the overall effective-ness of the class remained relatively consistent since the start of using free materials in spring 2015.

Discussion

The introductory course on Human Factors and Ergonomics is a required course in the School of Industrial Design at the Georgia Institute of Technology (Georgia Tech) in At-lanta for either a major or minor in industrial design. Free resources appropriate for the course were difficult to find because few open resources were available on this subject. Human Factors and Ergonomics itself is drawn from multiple disciplines, such as psy-chology, mechanics, anatomy, computer science, social sciences, and research methods. Individual chapters or articles focusing on each of the component topics were sought to fully cover the objectives of the class. The resulting collection of resources was a mix of completely free resources and free materials only available to Georgia Tech students.

Many of the resources treated the individual topics in depth. They often provided extra information necessary for their subject of focus but not directly relevant to a learn-ing objective for Human Factors and Ergonomics. A related challenge was the level of detail within the resources, some of which quickly became more technical than needed. The topics and level of detail in some of the resources were appropriate, however. For example, in introductory level open material on the subject of psychology, the majority of the text was also relevant to human factors. The focus of the text may have been dif-ferent, but 60 to 70 percent of the material covered was applicable. On other subjects, such as mechanics or biology, available open text materials were as much as 95 percent

Figure 2. Course Instructor Opinion Survey (CIOS) results from previous semesters

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


not relevant. Assigned reading for students had to be carefully selected to ensure a level appropriate for an introductory class. Even so, the context was not uniform. A reading might include references to terms or concepts explained in parts of a text not included for the Human Factors and Ergonomics course. A reading might also include terms and concepts that were not important for the class but could not be easily left out short of rewriting the entire passage.

The task of selecting appropriate materials might have been easier in a more ad-vanced class in Human Factors and Ergonomics. The initial goal was to use fully free materials and allow this collection to be shared with others. The lack of appropriate level introductory material ultimately prevented this result. The final collection included resources from textbooks and articles that are not free to everyone but are accessible for free to students through the Georgia Tech library. This helped to give the final materials a more consistent level of difficulty. It was, however, a collection of separate, unsynthe-sized materials. Students could not always tell just from the readings which parts were directly relevant to the course. For the students, this can generate confusion and eventual apathy toward the materials, especially in an introductory class.

Because the readings did not form a coherent whole, some students had challenges in actually reading them. At times, several separate readings were needed to provide coverage of a class topic. Readings were posted weekly, with some weeks requiring a high volume of reading to prepare for each class session. Some weeks required reading a single book chapter covering 30 or so pages, while the assignments for other weeks comprised more than one chapter (or parts of a book chapter) with additional articles, which might amount to 100 pages of reading. The relevance of all the material might not become clear until the lecture or activities of the correspond-ing class sessions. Because of this, many students admitted to often skipping the readings and relying only on studying class notes and reviewing slides.

Even with the challenges, the overall result was positive. Based on the course grades, there were no significant differences in student performance in the spring 2015 semester, when the free materials were first used. Average final exam grades dropped that semester. This could indicate that the materials used for the second half of the semester require some updating, but it may have been due to factors other than the reading materials.

The CIOS scores also remained stable; students reported a high overall level of sat-isfaction with the class and amount learned compared to previous semesters. The results were nearly identical to the previous semester and much improved from spring 2014. This improvement was encouraging because the size of the class in spring semesters has historically been much larger than in the fall. Student opinion scores in the larger classes tend to be lower than those in smaller classes because there is less time for individual coaching. Despite the drop in final exam scores, students’ feedback indicated that they are learning and feel the class is effective.

Based on the course grades, there were no significant differences in student per-formance in the spring 2015 semester, when the free materials were first used.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


Overall student reaction to the free materials has also been positive. Most students said they appreciated the cost savings. All of the reading materials were provided in electronic format. Many students also enjoyed the flexibility to access and read the mate-rials from anywhere on a variety of devices. This delivery method did present problems to some students. Several reported eyestrain and fatigue from reading long passages on computer screens, tablets, or phones.

Some of the student comments highlighted interesting attitudes. Several students indicated disappointment with other courses where a textbook purchase was required

but did not seem necessary because they never actually had to read it. Some students commented that they did not read the free materials either; instead, they relied on notes taken during lectures or activities. This could indicate that some students are conditioned to rely on themselves as a money-saving strategy (even if a book would be helpful) or believe there is a good chance that a book will not be used (even if it is required). This highlights the importance of ensuring that assigned

materials (free or not) are well integrated to encourage close reading and deeper under-standing that can be augmented through other class materials and activities.

Some students reported that the free materials from different sources gave them more “perspective” than they would get from a textbook. The class lectures and topics covered were not changed between the course that used a purchased textbook and the one that used free materials. Though the overall content was similar, the different voices in the

collection of free materials may have encouraged some students to explore different topics more than they might have done with a single textbook. Keeping in mind that some students may not read the materials anyway, little or no additional exploring of topics may have actually happened. Still, if the open educational resources motivated some students to seek additional information, it would be encouraging, especially for a subject like Human Factors and Ergonomics, where the abil-

ity to research, locate, and use a variety of different kinds of information is important.

Conclusion

These data should prove helpful to those considering implementing OER for other courses. As a single class, it can only be considered a case study. Results in other kinds of courses may be different. At worst, carefully chosen free materials can be as effective as non-free textbooks. In addition to the savings realized by students and the decrease of associated financial worries, free materials from a variety of sources can present dif-ferent voices on a subject and may encourage students to explore topics further. Digital formats offer students ease and flexibility, though they can lead to eyestrain and other physical challenges for some. Other challenges include the need to further refine materi-

Overall student reaction to the free materials has also been positive. Most students said they appre-ciated the cost savings.

Some students reported that the free materials from different sources gave them more “perspective” than they would get from a textbook.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.


als to help make them more coherent and to integrate the materials more deeply with other class activities to encourage reading.

Young Mi Choi is an assistant professor in the College of Design at the Georgia Institute of Technology in Atlanta; she may be reached by e-mail at: [email protected].

Cathy Carpenter is head of the Architecture Library in the College of Design at the Georgia Institute of Technology in Atlanta; she may be reached by e-mail at: [email protected].

Notes

1. United States Government Accountability Office, “College Textbooks: Enhanced Offerings Appear to Drive Recent Price Increases,” 2005, GAO-05-806, http://www.gao.gov/assets/250/247332.pdf.

2. Ethan Senack, “Fixing the Broken Textbook Market,” U.S. PIRG (Public Interest Research Group) Education Fund and the Student PIRGs, 2014, http://www.uspirg.org/sites/pirg/files/reports/NATIONAL%20Fixing%20Broken%20Textbooks%20Report1.pdf.

3. Stephen Downes, “Models for Sustainable Open Educational Resources,” Interdisciplinary Journal of E-Learning and Learning Objects 3, 1 (2007): 29–44; Ahrash N. Bissell, “Permission Granted: Open Licensing for Educational Resources,” Open Learning 24, 1 (2009): 97–106; Susan D’Antoni, “Open Educational Resources: Reviewing Initiatives and Issues,” Open Learning 24, 1 (2009): 3–10.

4. William and Flora Hewlett Foundation, “Open Educational Resources,” 2016, http://www.hewlett.org/programs/education/open-educational-resources.

5. Isabelle Brent, Graham Gibbs, and Anna Gruszczynska, “Obstacles to Creating and Finding Open Educational Resources: The Case of Research Methods in the Social Sciences,” Journal of Interactive Media in Education 1 (2012), doi:http://doi.org/10.5334/2012-05.

6. Brenda Gourley and Andy Lane, “Re-Invigorating Openness at the Open University: The Role of Open Educational Resources,” Open Learning 24, 1 (2009): 57–65.

7. Ibid. 8. John Hilton and Carol Laman, “One College’s Use of an Open Psychology Textbook,” Open

Learning 27, 3 (2012): 265–72. 9. Affordable Learning Georgia, “Textbook Transformation Grants,” 2017, http://www.

affordablelearninggeorgia.org/about/textbook_transformation_grants.10. University System of Georgia, “About GALILEO [Georgia Library Learning Online], 2017,

http://about.galileo.usg.edu/.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.

This m

ss. is

peer

review

ed, c

opy e

dited

, and

acce

pted f

or pu

blica

tion,

porta

l 17.4

.

Documents

Evaluating the Impact of Open Educational Resources: A ... · an open psychology textbook. ... (see Table 1). Final exam grades dropped the ... anatomy, computer science, social sciences,