1    

 

 

 

 

Evaluation  of  the  Pilot  of  the  Blackboard  Learning  Management  System  Implemented  in  Fall  2013  

 

Mary  Piontek,  Ph.D  -­‐  Indiana  University  Anastasia  Morrone,  Ph.D  –  Indiana  University  

 

December  18,  2013  

 

Prepared  by  

 

Center  for  Evaluation  &  Education  Policy  CEEP  

University  Information  Technology  Services  UITS  

 

 

 

 

 

 

 

2    

 

Evaluation  of  the  Pilot  of  the  Blackboard  Learning  Management  System  Implemented  in  Fall  2013  

Executive  Summary  December  18,  2013  

Overview  

The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the Blackboard Learning Management System Pilot (Blackboard LMS) Program being implemented during the 2013-2014 academic year across IU campuses. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. Of the 25 faculty members who piloted the Blackboard LMS in their courses during the fall 2013 semester across Indiana University campuses, 14 responded to the survey (56% = 14/25). Of the 895 students who were enrolled in a course that piloted the Blackboard LMS during the fall 2013 semester across Indiana University campuses, 197 responded to the survey (22% = 197/895). Faculty and student respondents represented the IU Bloomington, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast, and IUPU Columbus campuses. Moderate  Satisfaction  with  Blackboard  LMS  by  Faculty  and  Student  Respondents  

The faculty respondents were asked to rate their level of satisfaction with the features of Blackboard LMS. Not all features were equally used across the courses; none of the faculty respondents conducted online chat sessions (Collaborations > Chat) or connected and encouraged students to connect with Blackboard users and groups within or outside of the course (Blackboard Global Learning Network). All of the faculty respondents (100%) used the features of uploading, organizing, and sharing course materials (Content Collection > Course Content) and posting course announcements (Announcement). The highest rated feature on a scale of 1 to 4 (not at all satisfied to highly satisfied) was creating a course calendar with a mean of 3.67; however it should be noted that only 23% of respondents (3) used this feature. Features used by at least 50% of respondents had means in the moderately satisfied range (mean range 2.71-3.56). Similarly, student respondents were asked to rate the utility (i.e., usefulness) of the features of Blackboard LMS on a scale of 1 to 4 (not at all useful to highly useful). The features were rated in the slightly to moderately useful range (1.91 to 3.04). The highest rated feature was My Grades (for viewing a list of the graded items in the course and the grades received) with a mean of 3.04 (moderately useful rating). The lowest rated feature was Journal (for keeping a learning journal shared with the instructor) with a mean of 1.91; however it should be noted that only 30% of respondents indicated that they had used this feature. Faculty and student respondents were also asked to rate their level of agreement with a variety of statements about Blackboard LMS on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement by faculty was Blackboard enabled me to do what I wanted for my course(s) with a mean of 4.00 (agree rating). The lowest rated statement by faculty was Blackboard increased my effectiveness as a teacher with a mean of 3.23 (neither agree nor disagree rating).

3    

The highest rated statement by student respondents was Blackboard helps me to complete course assignments with a mean of 3.26. The lowest rated statement by students was Blackboard helps me to study for exams/tests with a mean of 2.56. The statement with the highest percentage of strongly disagree and disagree ratings from students was Blackboard helps me to study for exams/tests (49% of respondents to the question). When asked what they most liked about Blackboard LMS faculty respondents highlighted the ease of posting assignments and announcements (by the faculty members) and posting to discussion boards (by the students); the ease of grading students’ posts and assignments (by the faculty members); the announcements and email functions for communicating with students; the smart phone/mobile accessibility; publisher (test banks) compatibility; ease of creating quizzes/test and using the Grade Center for calculating final course grades; ease of uploading videos and files; the ability to customize the appearance and organization of the course site; and Blackboard’s overall uncluttered, attractive appearance. Faculty respondents disliked that students had difficulties in navigating the system; web browser incompatibility (e.g. Internet Explorer); the additional steps necessary in the Grade Book feature to review grades for an individual student; the cumbersome nature of Turnitin for grading; difficulty forwarding emails in Blackboard to the IU system; and the lack of integration with Adobe Connect. Features or tools in Blackboard LMS that allowed faculty to design their courses and/or teach in a new way included the ability to create online tests that students could complete in a computer lab or at remote locations, grading analytics that allowed for customized student feedback, using GoogleDocs spreadsheets for project management, and using Wikis. When asked what they most liked about Blackboard LMS student respondents noted the overall appearance and layout of the interface; the notification symbol and To-do lists on the dashboard; the calendar with due dates of assignments; the ease of use of the discussion boards and blogs; the ability to preview assignment submissions; and the ease of accessing and tracking grades. Student respondents disliked being unable to tell if an assignment was correctly submitted; the use of multiple tabs that are not intuitive; difficulties in accessing quizzes/tests; technical difficulties of the site loading very slowly, freezing/crashing, messages/emails being lost, and not fully opening in certain web browsers; and having to use two LMS systems for their courses (Blackboard and Oncourse).   Faculty  and  Student  Preferences  for  LMS  Systems  Differ  

Sixty-four percent (9) of faculty respondents prefer Blackboard over Oncourse LMS, 36% (5) prefer Oncourse over Blackboard LMS, and zero percent have no preference. Conversely, nine percent of student respondents (17/183) prefer Blackboard over Oncourse LMS, 75% (138/183) prefer Oncourse over Blackboard LMS, and 15% percent (28/183) have no preference.

4    

Evaluation  of  the  Pilot  of  the  Blackboard  Learning  Management  System  Implemented  in  Fall  2013  

Summary  of  Survey  December  18,  2013  

 Overview  

The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the Blackboard Learning Management System Pilot (Blackboard LMS) Program being implemented during the 2013-2014 academic year across IU campuses (IU Bloomington, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast, and IUPU Columbus). This aggregate report for UITS summarizes the findings of the evaluation activities conducted between August and December 2013. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. As part of the evaluation, staff members from CEEP and UITS collected data from faculty members (total n=25) who used Blackboard LMS in their courses during the fall 2013 semester and from students (total n=895) enrolled in those courses. The evaluation data collection included online surveys of faculty members and students aimed at capturing stakeholders’ experiences with Blackboard LMS and their perceptions of its quality and utility. Dr. Mary Piontek, Senior Research Associate at CEEP, oversaw the design and implementation of the evaluation; created the materials for the Institutional Review Board (IRB) application and submitted the IRB application; conducted the faculty and student survey collection activities; completed the analyses of the survey data; and created the preliminary version of this report on behalf of CEEP. Dr. Anastasia Morrone, Associate Vice President for Learning Technologies and Associate Professor of Educational Psychology, participated in the design and development of the survey instruments and also contributed to the final version of this report.  

 

 

 

 

 

 

   

5    

Summary  of  Faculty  Survey  

In November 2013 the 25 faculty members who piloted the Blackboard LMS in their courses during the fall 2013 semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via email to their university email address by Dr. Mary Piontek at CEEP. The faculty members were invited to voluntarily participate in the survey data collection on November 11th, with follow-up emails sent on December 2nd and December 11th to all non-respondents. The survey was closed at 5pm (EDT) on December 15, 2013. Fourteen faculty members responded to the survey (56% = 14/25). The survey included closed-ended and open-ended questions related to the faculty members’ experiences with Blackboard LMS and their perceptions of its quality and utility. The survey took approximately 20 minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 18 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix A, Faculty Survey: Blackboard Pilot Fall 2013 Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals. Nine respondents identified themselves as female and four as male; one respondent did not identify gender. Table 1 provides the years of experience as a faculty member/instructor in higher education, with all respondents having at least two or more years of experience. Table 2 provides the campus affiliation of the respondents, with IUPUI and IU Bloomington having the most respondents. Table  1:  Years  as  Faculty  Member/Instructor  in  Higher  Education  Years  as  a  Faculty  Member/Instructor   Total  N   %  1 year or less 0 0% 2-5 years 1 7% 6-10 years 6 43% 11-20 years 2 14% 21-30 years 4 29% More than 30 years 1 7% Total 14 100%  Table  2:  Campus  Affiliation  Campus  Affiliation   Total  N   %  IU Bloomington 3 21% IUPUI 4 29% IU East 1 7% IU Kokomo 1 7% IU Northwest 1 7% IU South Bend 0 0% IU Southeast 3 21% IUPU Columbus 1 7% Total 14 100%

During the fall 2013 semester Blackboard LMS was scheduled to be piloted in 25 courses; respondents to the faculty survey represented 12 of those courses. Those courses included the academic areas of business, chemistry, computer science/computer information technology, English, history, nursing,

6    

philosophy, political science, psychology, social work, and sociology [Note: the list of the courses that the faculty respondents taught are not included in this report to provide anonymity to respondents]. Sixty-four percent (9) of the courses were structured as face-to-face format (see Table 3). Thirty-six percent (5) of respondents described themselves as being very comfortable in using different types of technology (see Table 4). Respondents used a variety of technology devices to interact with Blackboard LMS including most often laptop/netbook computer (92%), a tablet (54%), and mobile phone with web access (46%) (see Table 5). Table  3:  Course  Format  Course  Format   Total  N   %  Face-to-face 9 64% In a hybrid format using a blend of face-to-face and online interaction 2 14% Online with face-to-face interaction only for exams 0 0% Only online with no face-to-face interaction 3 21% Other 0 0% Total 14 100% Table  4:  Level  of  Comfort  with  Technology  Level of Comfort in Using Different Types of Technology Total N % Very Uncomfortable 3 21% Somewhat Uncomfortable 1 7% Somewhat Comfortable 5 36% Very Comfortable 5 36% Total 14 100% Table  5:  Devices  Used  to  Interact  with  Blackboard  LMS  Mobile Devices(s) Used to Interact with Blackboard Total N %

N/13  Mobile phone without web access 1 8% Mobile phone with web access 6 46% Portable media player without web access (e.g., mp3 player) 0 0% Portable media player with web access (e.g., iPod Touch) 0 0% Ebook reader (e.g., Kindle) 1 8% Tablet (e.g., iPad) 8 54% Laptop/Netbook computer 12 92% Other device 3 23% The faculty respondents were also asked to rate their level of satisfaction with the features of Blackboard LMS (see Table 6). Not all features were equally used across the courses; none of the faculty respondents conducted online chat sessions (Collaborations > Chat) or connected and encouraged students to connect with Blackboard users and groups within or outside of the course (Blackboard Global Learning Network). All of the faculty respondents (100%) used the features of uploading, organizing, and sharing course materials (Content Collection > Course Content) and posting course announcements (Announcement). The highest rated feature on a scale of 1 to 4 (not at all satisfied to highly satisfied) was creating a course calendar (Course Calendar) with a mean of 3.67; however it should be noted that only 23% of respondents (3) used this feature. Features used by at least 50% of respondents had means in the moderately satisfied range (mean range 2.71-3.56).  

7    

Table  6:  Level  of  Satisfaction  with  Blackboard  LMS  Tasks  Please  rate  your  level  of  satisfaction  with  the  Blackboard  tools  and  features  designed  to  support  the  following  teaching  and  course  management  tasks:  

Did  Not  Use  

Not  at  all  Satisfied  

1  

Slightly  Satisfied  

2  

Moderately  Satisfied  

3  

Highly  Satisfied  

4  

Total  N  

Mean  

Creating and publishing the course syllabus (Content)

4 (31%)

0 (0%)

1 (8%)

2 (15%)

6

(46%)

13 3.56

Creating a course calendar (Course Calendar)

10 (77%)

0 (0%)

0 (0%)

1 (8%)

2

(15%)

13 3.67

Posting course announcements (Announcements)

0 (0%)

0 (0%)

3 (21%)

2 (14%)

9

(64%)

14 3.43

Uploading, organizing, and sharing course files (Content Collection > Course Content)

0 (0%)

0 (0%)

2 (14%)

5 (36%)

7

(50%)

14 3.36

Posting audio/video lectures or other multimedia (Content Collection > Course Content; Content Area > Video)

7 (50%)

1 (7%)

1 (7%)

2 (14%)

3

(21%)

14 3.00

Creating course web pages (Content Area > Blank Page)

5 (36%)

0 (0%)

1 (7%)

3 (21%)

5

(36%)

14 3.44

Organizing course content, activities, and assessments into a series of modules or lessons (Content Area, Learning Modules)

6 (46%)

0 (0%)

1 (8%)

2

(15%)

4

(31%)

13 3.43

Posting assignments (Content > Assignment)

1 (7%)

0 (0%)

4 (29%)

1

(7%)

8

(57%)

14 3.31

Assigning individual and collaborative writing tasks (Journals, Wikis, Blogs)

10 (72%)

2 (14%)

0 (0%)

1

(7%)

1

(7%)

14 2.25

Assigning peer reviews on student work (Self and Peer Assessment, Wikis, Blogs)

13 (93%)

1 (7%)

0 (0%)

0 (0%)

0 (0%)

14 1.00

Using Turnitin originality checking on assignments (Turnitin Direct Assignment)

12 (86%)

0 (0%)

0 (0%)

2 (14%)

0 (0%)

14 3.00

   

8    

Table  6:  Level  of  Satisfaction  with  Blackboard  LMS  Tasks  (continued)  Please  rate  your  level  of  satisfaction  with  the  Blackboard  tools  and  features  designed  to  support  the  following  teaching  and  course  management  tasks:  

Did  Not  Use  

Not  at  all  Satisfied  

1  

Slightly  Satisfied  

2  

Moderately  Satisfied  

3  

Highly  Satisfied  

4  

Total  N  

Mean  

Creating and administering online quizzes, tests, and/or surveys (Tests, Surveys, and Pools)

5 (36%)

2 (14%)

0 (0%)

4 (29%)

3

(21%)

14 2.89

Facilitating graded and ungraded discussions (Discussions)

8 (57%)

0 (0%)

1 (7%)

1 (7%)

4

(29%)

14 3.50

Giving feedback on and/or grading student submissions (Grade Center > Needs Grading)

4 (31%)

2 (15%)

1 (8%)

2 (15%)

4

(31%)

13 2.89

Creating and using rubrics to grade student work (Rubrics, Grade Center)

10 (72%)

2 (14%)

0 (0%)

1 (7%)

1 (7%)

14 2.25

Setting up and using the gradebook to enter and track student grades (Grade Center)

1 (7%)

4 (29%)

0 (0%)

3 (21%)

6 (43%)

14 2.85

Monitoring course activity and student progress (Course Reports, Performance Dashboard, Retention Center)

4 (29%)

0 (0%)

2 (14%)

4 (29%)

4 (29%)

14 3.20

Sending and receiving messages to and from students and groups (Course Messages, Send Email)

3 (21%)

3 (21%)

1 (7%)

0 (0%)

7 (50%)

14 3.00

Creating and managing groups for group assignments, group discussions, and/or group projects (Groups)

9 (64%)

1 (7%)

0 (0%)

1 (7%)

3 (21%)

14 3.20

Conducting online chat sessions (Collaborations > Chat)

14 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

14 N/A

Keeping track of your course tasks (Calendar, To Do, Needs Attention)

10 (71%)

1 (7%)

3 (21%)

0 (0%)

0 (0%)

14 1.75

Importing or exporting course content (Packages and Utilities)

7 (50%)

1 (7%)

3 (21%)

0 (0%)

3 (21%)

14 2.71

   

9    

Table  6:  Level  of  Satisfaction  with  Blackboard  LMS  Tasks  (continued)  Please  rate  your  level  of  satisfaction  with  the  Blackboard  tools  and  features  designed  to  support  the  following  teaching  and  course  management  tasks:  

Did  Not  Use  

Not  at  all  Satisfied  

1  

Slightly  Satisfied  

2  

Moderately  Satisfied  

3  

Highly  Satisfied  

4  

Total  N  

Mean  

Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud, Piazza, etc. (Web Link)

13 (93%)

0 (0%)

0 (0%)

1 (7%)

0 (0%)

14 3.00

Customizing the navigation, look, and feel of your course (Quick Setup Guide, Teaching Style)

4 (29%)

1 (7%)

0 (0%)

4 (29%)

5 (35%)

14 3.30

Connecting or encouraging students to connect with Blackboard users and groups within or outside of your course (Blackboard Global Learning Network)

13 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

13 N/A

 Fifty percent of faculty respondents (7) found Blackboard LMS very easy to use; 43% (6) found it highly useful for their teaching and 31% (4) found the online documentation to be highly useful (see Tables 7 and 8). Table  7:  Overall  Ease  of  Use  of  Blackboard  LMS  Please  rate  the  overall  ease  of  use  of  Blackboard.    

Difficult  to  Use  

Slightly  Easy  to  Use  

Moderately  Easy  to  Use  

Very  Easy  to  Use  

Total  N  

Mean  

Overall ease of use of Blackboard 2 (14%)

2 (14%)

3 (21%)

7 (50%)

14 3.07

Table  8:  Overall  Utility  of  Blackboard  LMS  for  Teaching  Please  rate  the  overall  usefulness  of  Blackboard  for  your  teaching.  

Not  at  all  Useful  

Slightly  Useful  

Moderately  Useful  

Highly  Useful  

Total  N  

Mean  

Overall usefulness of Blackboard for your teaching

0 (0%)

3 (21%)

5 (36%)

6 (43%)

14 3.21

Overall usefulness of Blackboard’s online documentation

0 (0%)

4 (31%)

5 (38%)

4 (31%)

13 3.00

Respondents were also asked to rate their level of agreement with a variety of statements about Blackboard LMS (see Table 9). All of the statements were rated by faculty respondents with a mean of 3.23 or above on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement was Blackboard enabled me to do what I wanted for my course(s) with a mean of 4.00 (agree rating). The lowest rated statement was Blackboard increased my effectiveness as a teacher with a mean of 3.23 (neither agree nor disagree rating).

10    

Sixty-four percent (9) of faculty respondents prefer Blackboard over Oncourse LMS, 36% (5) prefer Oncourse over Blackboard LMS, and 0% percent have no preference. When asked what they most liked about Blackboard LMS faculty respondents highlighted the ease of posting assignments and announcements (by the faculty members) and posting to discussion boards (by the students); the ease of grading students’ posts and assignments (by the faculty members); the announcements and email functions for communicating with students; the smart phone/mobile accessibility; publisher (test banks) compatibility; ease of creating quizzes/test and using the Grade Center for calculating final course grades; ease of uploading videos and files; the ability to customize the appearance and organization of the course site; and Blackboard’s overall uncluttered, attractive appearance. Features or tools in Blackboard LMS that allowed faculty to design their courses and/or teach in a new way included the ability to create online tests that students can complete in a computer lab or at remote locations, grading analytics that allowed for customized student feedback, using GoogleDocs spreadsheets for project management, and using Wikis. When asked what they liked least about Blackboard LMS faculty respondents noted students’ difficulties in navigating the system; web browser incompatibility (e.g. Internet Explorer); the additional steps necessary in the Grade Book feature to review grades for an individual student; the cumbersome nature of Turnitin for grading; difficulty forwarding emails in Blackboard to the IU system; and lack of integration with Adobe Connect. Table  9:  Level  of  Agreement  with  Statements  about  Blackboard  LMS  Please  rate  your  level  of  agreement  with  the  following  statements  about  Blackboard.  

Strongly  Disagree  

1  

Disagree  2  

Neither  Agree  nor  Disagree  

3  

Agree  4  

Strongly  Agree  5  

Not  Applicable  

Total  N  

Mean  

Blackboard enabled me to do what I wanted for my course(s).

0 (0%)

3 (23%)

1 (8%)

2 (15%)

7 (54%)

0 (0%)

13 4.00

Blackboard was easy for my students to learn how to use.

2 (15%)

1 (8%)

3 (23%)

4 (31%)

3 (23%)

0 (0%)

13 3.38

Blackboard increased my efficiency as a teacher.

2 (15%)

0 (0%)

5 (39%)

2 (15%)

4 (31%)

0 (0%)

13 3.46

Blackboard increased my effectiveness as a teacher.

2 (15%)

1 (8%)

5 (39%)

2 (15%)

3 (23%)

0 (0%)

13 3.23

Blackboard was a valuable aid to me in my teaching.

2 (15%)

1 (8%)

2 (15%)

3 (23%)

5 (39%)

0 (0%)

13 3.62

Using Blackboard has been beneficial to my students’ overall learning.

1 (8%)

3 (23%)

3 (23%)

3 (23%)

3 (23%)

0 (0%)

13 3.31

11    

Summary  of  Student  Survey   In November 2013 the 895 students who were enrolled in a course that piloted the Blackboard LMS during the fall 2013 semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via email to their university email address by Dr. Piontek. The students were invited to voluntarily participate in the survey data collection on November 11th, with follow-up emails sent on December 2nd and December 13th to all non-respondents. The survey was closed at 5pm (EDT) on December 15, 2013. One hundred ninety-seven students responded to the survey (22% = 197/895). The survey included closed-ended and open-ended questions related to the students’ experience with Blackboard LMS and their perceptions of its quality and utility. The survey took approximately 20 minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 18 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix B, Student Survey: Blackboard Pilot Fall 2013 Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals.  One hundred twenty-three (63%, 123/195) respondents identified themselves as female and seventy-one (36%, 71/195) as male. One respondent did not identify gender. The age of respondents ranged from 18 to 62 years of age with the greatest percentage (69%, 135/195) in the 18 to 25 range. Female student respondents ranged from 18 to 62 years with the largest percentage at age 19 (22%); male student respondents ranged from 18 to 49 years with the largest percentage at age 22 (17%). Thirty-four percent (66/197) of student respondents are seniors (see Table 10). Fifty-five percent of respondents are affiliated with the IUPUI campus (see Table 11). During the fall 2013 semester Blackboard LMS was scheduled to be piloted in 25 courses; respondents to the student survey represented 22 courses. Those courses included the academic areas of business, chemistry, computer science/computer information technology, engineering, English, history, liberal arts, philosophy, political science, sociology, and social work (see Table 12). Fifty-eight percent (113) of the respondents were in face-to-face courses (see Table 13). Table  10:  Academic  Level  Academic  Level   Total  N   %  First-year undergraduate (Freshman) 29 15% Second-year undergraduate (Sophomore) 53 27% Third-year undergraduate (Junior) 30 15% Four or more years undergraduate (Senior) 66 34% Masters student (MA, MS, MBA, MFA, MSW, MPA, etc.) 12 6% Doctoral Student (EdD, PhD, etc.) 7 4% Total 197 100%  

12    

Table  11:  Campus  Affiliation  Campus  Affiliation   Total  

N  %  

IU Bloomington 36 18% IUPUI 107 55% IU East 21 11% IU Kokomo 1 1% IU Northwest 2 1% IU South Bend 2 1% IU Southeast 22 11% IUPU Columbus 5 3% IUPUI 107 55% Total 196 100% Table  12:  Name  of  Enrolled  Course      Course  Name   Total  

N  %  

BUSE D530, Business Policy & Strategy, SE, Section 18733 5 3% BUSE D530, Business Policy & Strategy, SE, Section 19665 5 3% CHEM C106, Principles of Chemistry II, IN, Section 21206 42 24% COAS FY104, First Year Seminar (Social Problems), SE, Section 20061 1 1% CSCI A110, Intro to Computers & Computing, EA, Section 14146 11 6% CSCI A110, Intro to Computers & Computing, EA, Section 14147 8 5% ENG W130, Principles of Composition, SB, Section 17727 0 0% ENG W131, Elementary Composition 1, IN, Section 21666 4 2% ENG W131, Elementary Composition 1, IN, Section 21667 4 2% ENG W131, Elementary Composition 1, NW, Section 15475 2 1% ENG W131, Elementary Composition 1, NW, Section 16274 0 0% HIST H114, History of Western Civiliz 2, KO, Section 27617 1 1% HPER P423, Financial Principles in Sport, IN, Section 26598 3 2% HPER P426, Sales Management in Sport, IN, Section 25687 5 3% MET 32000, Applied Thermodynamics, IN, Section 31948 2 1% OLS 37800, Labor/Management Relations, IN, Section 22530 16 9% OLS 37800, Labor/Management Relations, IN, Section 31668 19 11% PHIL P240, Bus & Morality Ethics in Cntxt, SE, Section 34609 5 3% PSY B358, Intro to I/O Psychology, CO, Section 33745 4 2% SOC S163, Social Problems, SE, Section 19276 1 1% SOC S315, Work and Occupations, SE, Section 31827 2 1% SOC S416, The Family, SE, Section 30089 4 2% SPAN S305, Masterpieces of Spanish Lit 1, SB, Section 30304 0 0% SWK S102, Diversity in a Pluralistic Soc, BL, Section 13980 6 3% SWK S221 Human Growth & Development 5 3% Other 21 12% Total 176 100%  

13    

Table  13:  Course  Format  Course  Format   Total  N   %  Face-to-face 113 58% In a hybrid format using a blend of face-to-face and online interaction 41 21% Online with face-to-face interaction only for exams 4 2% Only online with no face-to-face interaction 33 17% Other 4 2% Total 195 100% Thirty-two percent (61/193) of respondents described themselves as being very comfortable in using different types of technology and 17% (33/193) described themselves as being very uncomfortable (see Table 14). Respondents used a variety of technology devices to interact with Blackboard LMS including most often a laptop/netbook computer (93%) and mobile phone with web access (77%) (see Table 15). Most students (70%) averaged fewer than five hours per week using Blackboard for their courses (see Table 16). Table  14:  Level  of  Comfort  with  Technology  Level  of  Comfort  in  Using  Different  Types  of  Technology  

Female   Male   Total  N   %  

Very Uncomfortable 18 15 33 17% Somewhat Uncomfortable 7 7 14 7% Somewhat Comfortable 56 28 85 44% Very Comfortable 40 21 61 32% Total 121 71 193 100% Table  15:  Devices  Used  to  Interact  with  Blackboard  LMS  Mobile  Devices(s)  Used  to  Interact  with  Blackboard   Total  N   %  

(N/193)  Mobile phone without web access 26 13% Mobile phone with web access 148 77% Portable media player without web access (e.g., mp3 player) 21 11% Portable media player with web access (e.g., iPod Touch) 35 18% Ebook reader (e.g., Kindle) 23 12% Tablet (e.g., iPad) 73 38% Laptop/Netbook computer 179 93% Other device 4 2% Table  16:  Average  Number  of  Hours  Per  Week  Using  Blackboard  LMS  Average  Number  of  Hours  Using  Blackboard  for  the  Course  During  the  Fall    2013  Semester  

Total  N   %  

Never 4 2% Fewer than 5 hours 136 70% 5-10 hours 42 22% 11-15 hours 6 3% 16-20 hours 5 3% More than 20 hours per week 0 0% Total 193 100% Fifteen percent of student respondents (28/184) found Blackboard LMS very easy to use, and 11% found the online documentation highly useful (see Tables 17 and 18).

14    

 

15    

Table  17:  Overall  Ease  of  Use  of  Blackboard  LMS  Please  rate  the  overall  ease  of  use  of  Blackboard.    

Difficult  to  Use  

Slightly  Easy  to  Use  

Moderately  Easy  to  Use  

Very  Easy  to  Use  

Total  N  

Mean  

Overall ease of use of Blackboard 45 (24%)

38 (21%)

73 (40%)

28 (15%)

184 2.46

Table  18:  Overall  Utility  of  Blackboard  LMS  for  Teaching  Please  rate  the  overall  usefulness  of  Blackboard  for  your  teaching.  

Not  at  all  Useful  

Slightly  Useful  

Moderately  Useful  

Highly  Useful  

Total  N  

Mean  

Overall usefulness of Blackboard’s online documentation

37 (20%)

60 (32%)

68 (37%)

20 (11%)

185 2.38

The student respondents were asked to rate the utility (i.e., usefulness) of the features of Blackboard LMS on a scale of 1 to 4 (not at all useful to highly useful) (see Table 19). The features were rated in the slightly to moderately useful range (1.91 to 3.04); not all features were equally used by the student respondents. The highest rated feature was My Grades (for viewing a list of the graded items in the course and the grades you received) with a mean of 3.04 (moderately useful rating). The lowest rated feature was Journal (for keeping a learning journal shared with your instructor) with a mean of 1.91; however it should be noted that only 30% of respondents indicated that they had used this feature. Student respondents were also asked to rate their level of agreement with a variety of statements about Blackboard LMS (see Table 20). The features were rated in the neither agree nor disagree range (2.56-3.26) on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement was Blackboard helps me to complete course assignments with a mean of 3.26 (neither agree nor disagree rating). The lowest rated statement was Blackboard helps me to study for exams/tests with a mean of 2.56. The statement with the highest percentage of strongly disagree and disagree ratings was Blackboard helps me to study for exams/tests (49% of respondents to the question). Nine percent of student respondents (17/183) prefer Blackboard over Oncourse LMS, 75% (138/183) prefer Oncourse over Blackboard LMS, and 15% (28/183) have no preference. When asked what they most liked about Blackboard LMS student respondents noted the overall appearance and layout of the interface; the notification symbol and To-do lists on the dashboard; the calendar with due dates of assignments; the ease of use of the discussion boards and blogs; the ability to preview assignment submissions; and the ease of accessing and tracking grades. Student respondents disliked being unable to tell if an assignment was correctly submitted; the use of multiple tabs that are not intuitive; difficulties in accessing quizzes/tests; technical difficulties of the site loading very slowly, freezing/crashing, messages/emails being lost, and not fully opening in certain web browsers; and having to use two LMS systems for their courses (Blackboard and Oncourse).    

16    

Table  19:  Utility  of  Blackboard  LMS  Features  Please  rate  the  usefulness  of  the  following  features  of  Blackboard  in  contributing  to  your  learning  in  this  course.  

Did  Not  Use  This  Feature  

Not  at  all  

Useful  1  

Slightly  Useful  

2  

Moderately  Useful  

3  

Highly  Useful  

4  

Total  N  

Mean  

Announcements (for reading announcements and other timely news and information posted by your instructor or department)

10 (5%)

16 (9%)

54 (30%)

52 (29%)

50 (27%)

182 2.79

Assignments (for submitting individual or group assignments)

39 (22%)

22 (12%)

35 (19%)

39 (22%)

46 (25%)

181 2.77

Blog and Wikis (for individual and group writing tasks assigned by your instructor)

78 (43%)

38 (21%)

26 (14%)

27 (15%)

12 (7%)

181 2.13

Calendar (for managing your personal calendar and viewing course events and due dates)

88 (48%)

35 (19%)

23 (13%)

20 (11%)

16 (9%)

182 2.18

Chat (for live text messaging with classmates and other Blackboard users)

114 (63%)

29 (16%)

12 (6%)

14 (8%)

12 (6%)

181 2.13

Content (for viewing course materials and completing activities organized into lessons or modules)

15 (8%)

21 (12%)

35 (19%)

60 (33%)

50 (28%)

181 2.84

Course Messages (for sending and receiving messages to and from your instructor and other students)

41 (23%)

42 (23%)

31 (17%)

46 (25%)

21 (12%)

181 2.33

Discussions/Discussion Board (for participating in online discussions with the entire class or in small groups)

67 (37%)

30 (17%)

24 (13%)

31 (17%)

29 (16%)

181 2.52

Groups (for collaborating with a specific group of students on assignments, discussions, blogs, wikis, or projects)

74 (41%)

40 (22%)

28 (15%)

29 (16%)

11 (6%)

182 2.10

Journal (for keeping a learning journal shared with your instructor)

128 (70%)

27 (15%)

10 (5%)

12 (7%)

5 (3%)

182 1.91

Content > My Content (for storing personal files related to your course work)

105 (58%)

28 (15%)

14 (8%)

23 (13%)

11 (6%)

181 2.22

My Grades (for viewing a list of the graded items in the course and the grades you received)

15 (8%)

18 (10%)

24 (13%)

56 (31%)

67 (37%)

180 3.04

Quizzes/Tests (for taking and receiving feedback on online quizzes, tests, and self-assessments)

55 (30%)

21 (12%)

24 (13%)

35 (19%)

47 (26%)

182 2.85

Roster (for viewing a list of the other people in the course)

79 (43%)

30 (17%)

30 (17%)

22 (12%)

20 (11%)

181 2.31

Rubrics (for understanding how your work will be or was graded)

67 (37%)

25 (14%)

28 (15%)

33 (18%)

29 (16%)

182 2.57

 

17    

Table  19:  Utility  of  Blackboard  LMS  Features  (continued)  Please  rate  the  usefulness  of  the  following  features  of  Blackboard  in  contributing  to  your  learning  in  this  course.  

Did  Not  Use  This  Feature  

Not  at  all  

Useful  1  

Slightly  Useful  

2  

Moderately  Useful  

3  

Highly  Useful  

4  

Total  N  

Mean  

Send Email (for sending messages to the external email account of other course members)

76 (42%)

38 (21%)

24 (13%)

17 (9%)

26 (15%)

181 2.30

Surveys (for taking online surveys)

98 (54%)

28 (15%)

24 (13%)

19 (10%)

13 (7%)

182 2.20

Tasks (for completing a list of tasks prepared by the instructor)

61 (34%)

27 (15%)

33 (18%)

37 (20%)

23 (13%)

181 2.47

Table  20:  Level  of  Agreement  with  Statements  about  Blackboard  LMS  Please  rate  your  level  of  agreement  with  the  following  statements  about  Blackboard.  

Strongly  Disagree  

Disagree   Neither  Agree  nor  Disagree  

Agree   Strongly  Agree  

Not  Applicable  

Total  N  

Mean  

Blackboard helps me to learn the course materials/content.

37 (21%)

31 (17%)

44 (25%)

43 (24%)

13 (7%)

11 (6%)

179 2.79

Blackboard helps me to study for exams/tests.

42 (23%)

46 (26%)

37 (21%)

32 (18%)

12 (7%)

11 (6%)

180 2.56

Blackboard helps me to complete course assignments.

29 (16%)

22 (12%)

29 (16%)

52 (29%)

36 (20%)

11 (6%)

179 3.26

Blackboard helps me to take quizzes/exams.

33 (18%)

25 (14%)

35 (20%)

37 (21%)

28 (16%)

19 (11%)

177 3.01

Blackboard helps me to make efficient use of my time in the course.

41 (23%)

32 (18%)

39 (22%)

32 (18%)

28 (15%)

8 (4%)

180 2.85

Blackboard helps me to be in control of my own learning in the course.

37 (21%)

30 (16%)

47 (26%)

37 (21%)

19 (10%)

10 (6%)

180 2.83

Blackboard helps me to communicate with my professor.

45 (25%)

39 (22%)

36 (20%)

35 (19%)

14 (8%)

11 (6%)

180 2.61

Blackboard expands access to learning materials/resources available to me (e.g., print, audio, video, etc.).

31 (17%)

26 (15%)

40 (22%)

47 (26%)

25 (14%)

10 (6%)

179 3.05

Blackboard was beneficial to my overall learning in the course.

44 (24%)

32 (18%)

40 (22%)

39 (22%)

21 (12%)

4 (2%)

180 2.78

 

   

18    

Appendix  A  Faculty  Survey:  Blackboard  Pilot  Fall  2013  Semester  

 Introduction  and  Consent   In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the Blackboard Learning Management System (Blackboard) Pilot Program being conducted during the fall 2013 semester. You have been invited to participate in this survey because you are teaching one or more courses in Blackboard this semester. Participation in this survey is completely voluntary. Should you decide to participate, your responses will be kept confidential and will be reported only in aggregate in published reports. Individual comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately 20 minutes to complete. Data collection will close at 5pm (EST) on Day, Month, Date, 2013. You may terminate your participation in the survey at any time. If you consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey. I consent to participate in this survey. (required check box) Part  I:    Course/IU  Context     Q1: For how many years have you been an instructor/faculty member in higher education? Scale: 1 year or less, 2-5 years, 6-10 years, 11-20 years, 21-30 years, more than 30 years Q2: What is your gender? (open text box) Q3: At which campus are you a faculty member? (Choose your PRIMARY academic appointment)

IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast

Q4: In which course during fall 2013 did you use Blackboard? (If you used Blackboard in multiple

courses, please choose one for the purpose of this survey.) Please note: Although we ask you to identify the course in which you piloted Blackboard, this survey is not an evaluation of the course. We will use this information to report aggregate response rates across the courses and to create categories of types (disciplinary areas) and levels of courses for reporting purposes. (drop down list of Blackboard pilot courses)

19    

Q5: The course was delivered (Choose one BEST answer) Scale: primarily face-to-face; using a blend of face-to-face and online interaction; online with face-to-face interaction only for exams; exclusively online with no face-to-face interaction; other (please describe)

Part  II:    Use  of  Technology     Q6: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q7: Which device(s) do you currently use to interact with Blackboard? (Choose all that apply)

mobile phone; portable media player (e.g., iPod Touch); ebook reader (e.g., Kindle); tablet (e.g., iPad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe)

Part  III:    Feedback  on  Blackboard     Q8: Based on your experience during the fall 2013 pilot, please rate your level of satisfaction with the

Blackboard tools and features designed to support the following teaching and course management tasks: Scale: did not use, not at all satisfied, slightly satisfied, moderately satisfied, highly satisfied

• Creating and publishing the course syllabus (Content) • Creating a course calendar (Course Calendar) • Posting course announcements (Announcements) • Uploading, organizing, and sharing course files (Content Collection > Course Content) • Posting audio/video lectures or other multimedia (Content Collection > Course Content;

Content Area > Video) • Creating course web pages (Content Area > Blank Page) • Organizing course content, activities, and assessments into a series of modules or lessons

(Content Area, Learning Modules) • Posting assignments (Content > Assignment) • Assigning individual and collaborative writing tasks (Journals, Wikis, Blogs) • Assigning peer reviews on student work (Self and Peer Assessment, Wikis, Blogs) • Using Turnitin originality checking on assignments. (Turnitin Direct Assignment) • Creating and administering online quizzes, tests, and/or surveys (Tests, Surveys, and

Pools) • Facilitating graded and ungraded discussions (Discussions) • Giving feedback on and/or grading student submissions (Grade Center > Needs Grading) • Creating and using rubrics to grade student work (Rubrics, Grade Center) • Setting up and using the gradebook to enter and track student grades (Grade Center) • Monitoring course activity and student progress (Course Reports, Performance

Dashboard, Retention Center) • Sending and receiving messages to and from students and groups (Course Messages,

20    

Send Email) • Creating and managing groups for group assignments, group discussions, and/or group

projects (Groups) • Conducting online chat sessions (Collaborations > Chat) • Keeping track of your course tasks (Calendar, To Do, Needs Attention) • Importing or exporting course content (Packages and Utilities) • Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud,

Piazza, etc. (Web Link) • Customizing the navigation, look, and feel of your course (Quick Setup Guide, Teaching

Style) • Connecting or encouraging students to connect with Blackboard users and groups within

or outside of your course (Blackboard Global Learning Network) • Other

Q9: Please rate the overall ease of use of Blackboard. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q10: Please rate the overall usefulness of Blackboard for your teaching. Scale: not at all useful, slightly useful, moderately useful, highly useful Q11: Please rate the overall Usefulness of Blackboard’s online documentation. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q12: What did you like MOST about Blackboard? Why? (open text box) Q13: What did you like LEAST about Blackboard? Why? (open text box) Q14: Which, if any, features/tools in Blackboard allowed you to design your course and/or teach in a

new way? (open text box)

Q15 Please rate your level of agreement with the following statements about Blackboard:

1. Blackboard enabled me to do what I wanted for my course(s). 2. Blackboard was easy for my students to use. 3. Blackboard increased my efficiency as a teacher. 4. Blackboard increased my effectiveness as a teacher. 5. Blackboard was a valuable aid to me in my teaching. 6. Using Blackboard has been beneficial to my students’ overall learning.

21    

Q16 Overall, compared to Indiana University’s current LMS (Oncourse), please select the choice that best describes your preference for Oncourse versus Blackboard:

• I prefer Blackboard over Oncourse. • I prefer Oncourse over Blackboard. • I have no preference.

Q17 Additional Comments

Is there anything else you would like to tell us about your experience using Blackboard this semester? (open text box)

22    

Appendix  B  Student  Survey:  Blackboard  Pilot  Fall  2013  Semester  

 Introduction  and  Consent   In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the Blackboard Learning Management System (Blackboard) Pilot Program being conducted during the Fall 2013 semester. You have been invited to participate in this survey because you are enrolled in one or more courses being taught in Blackboard this semester. Participation in this survey is completely voluntary. Should you decide to participate your responses will be kept confidential and will be reported only in aggregate in published reports. Comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately 15 minutes to complete. Data collection will close at 5pm (EST) on Day, Month, Date, 2013. You may terminate your participation in the survey at any time.

Pre-Question 1: If you are 18 years of age or older as of today’s date and consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey.

If you are NOT 18 years of age or older as of today’s date or do not consent, please close the link to the survey now.

I acknowledge that I am at least 18 years of age or older as of today’s date and consent to participate in this survey (will be a required check box) Part  I:    Student  Demographics/Institutional  Context     Q1: What is your current academic level?

Scale: first-year undergraduate (freshman), second-year undergraduate (sophomore), third-year undergraduate (junior), four or more years undergraduate (senior), master’s degree student (MA, MS, MBA, MFA, etc.), doctoral student (EdD, PhD, etc.)

Q2: At which campus are you enrolled as a student? (Choose ONE)

IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast

Q3: What is your age? (open text box) Q4: What is your gender? (open text box)

23    

Q5: Which course were you enrolled in during fall 2013 that used Blackboard? (If you were enrolled in more than one course that used Blackboard, please choose one for the purpose of this survey) Please note: Although we ask you to identify the course in which you were enrolled, this survey is not an evaluation of the course. We will use this information to report aggregate response rates within each course and to create categories of types (disciplinary areas) and levels of courses for reporting purposes.

(drop down list of Blackboard pilot courses)

Q6: That course was delivered (Choose one BEST answer) primarily face-to-face; using a blend of face-to-face and online interaction; online with face-to-face interaction only for exams; exclusively online with no face-to-face interaction; other (please describe)

Part  II:    Use  of  Technology     General Level of Experience/Use: Q7: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q8: What type(s) of networked device(s) do you currently use on a regular basis?

mobile phone; portable media player (e.g., iPod Touch); ebook reader (e.g., Kindle); tablet (e.g., iPad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe)

Q9: On average, how many hours per week have you been spending in Blackboard for this course? Scale: none, fewer than 5 hours, 5-10 hours, 11-15 hours, 16-20 hours, more than 20 hours Part  III:    Feedback  on  Blackboard       Q10: Please rate the usefulness of the following tools and features of Blackboard in contributing to

your learning in this course. Scale: did not use this feature, not at all useful, slightly useful, moderately useful, highly useful

Note: Some of these tools and features may have a different name in your course.

• Announcements (for reading announcements and other timely news and information posted by your instructor or department)

• Assignments (for submitting individual or group assignments) • Blog and Wikis (for individual and group writing tasks assigned by your instructor) • Calendar (for managing your personal calendar and viewing course events and due dates) • Chat (for live text messaging with classmates and other Blackboard users)

24    

• Content (for viewing course materials and completing activities organized into lessons or modules)

• Course Messages (for sending and receiving messages to and from your instructor and other students)

• Discussions/Discussion Board (for participating in online discussions with the entire class or in small groups)

• Groups (for collaborating with a specific group of students on assignments, discussions, blogs, wikis, or projects)

• Journal (for keeping a learning journal shared with your instructor) • Content > My Content (for storing personal files related to your course work) • My Grades (for viewing a list of the graded items in the course and the grades you

received) • Quizzes/Tests (for taking and receiving feedback on online quizzes, tests, and self-

assessments) • Roster (for viewing a list of the other people in the course) • Rubrics (for understanding how your work will be or was graded) • Send Email (for sending messages to the external email account of other course

members). • Surveys (for taking online surveys) • Tasks (for completing a list of tasks prepared by the instructor)

Q11: Please rate the overall ease of use of Blackboard. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q12: Please rate the overall usefulness of Blackboard’s online documentation for students. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q13: What did you like MOST about Blackboard? Why? (open text box) Q14: What did you like LEAST about Blackboard? Why? (open text box) Q15: Perceived impact on learning

Scale: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree, not applicable

Blackboard helps me to learn the course materials/content. Blackboard helps me to study for exams/tests. Blackboard helps me to complete course assignments. Blackboard helps me to take quizzes/exams. Blackboard helps me to make efficient use of my time in the course. Blackboard helps me to be in control of my own learning in the course. Blackboard helps me to communicate with my professor.

25    

Blackboard expands access to learning materials/resources available to me (e.g., print, audio, video, etc.). Blackboard was beneficial to my overall learning in the course.

Q16 Overall, compared to Indiana University’s current LMS (Oncourse), please select the choice that

best describes your preference for Oncourse versus Blackboard:

• I prefer Blackboard over Oncourse. • I prefer Oncourse over Blackboard. • I have no preference.

Q17 Additional Comments

Is there anything else you would like to tell us about your experience using Blackboard this semester? (open text box)