1    

 

 

 

 

Evaluation  of  the  Pilot  of  the  Desire2Learn  Learning  Management  System  Implemented  in  Fall  2013  

 

Mary  Piontek,  Ph.D  -­‐  Indiana  University  Anastasia  Morrone,  Ph.D  –  Indiana  University  

 

December  18,  2013  

 

Prepared  by  

 

Center  for  Evaluation  &  Education  Policy  CEEP  

University  Information  Technology  Services  UITS  

   

2    

Evaluation  of  the  Pilot  of  the  Desire2Learn  Learning  Management  System    Implemented  in  Fall  2013  

Executive  Summary  December  18,  2013  

Overview  

The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the Desire2Learn Learning Management System Pilot (Desire2Learn LMS) Program being implemented during the 2013-2014 academic year across IU campuses. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. Of the 17 faculty members who piloted the Desire2Learn LMS in their courses during the fall 2013 semester, six responded to the survey (35% = 6/17). Of the 154 students who were enrolled in a course that piloted the Desire2Learn LMS during the fall 2013 semester, twenty-four responded to the survey (16% = 24/154). Faculty and student respondents represented the IU Bloomington, IUPUI, IU East, IU Kokomo, and IUPU Columbus campuses. Slight  to  Moderate  Satisfaction  with  Desire2Learn  by  Faculty  and  Student  Respondents    

All six faculty respondents used Desire2Learn LMS to post course announcements (News) and upload, organize, and share course files (Manage Files, Content); and were moderately satisfied with those features (mean ratings of 2.83 and 3.17 respectively on a scale of 1-4, from not at all satisfied to highly satisfied). Features used by at least 50% of respondents (i.e., three or more respondents) had means that spanned from slightly satisfied to moderately satisfied range (mean range 1.50-3.67). Features not used by any of the faculty respondents included the features of assigning individual or collaborative portfolio or web projects to students (ePortfolio); creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades); conducting online chat sessions (Chat); importing or exporting course content (Import/Export/Copy Components); and integrating an external learning tool or platform with the course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools). Similarly, student respondents were asked to rate the utility (i.e., usefulness) of the features of Desire2Learn LMS on a scale of 1 to 4 (not at all useful to highly useful). The highest rated feature was Grades with a mean of 3.44 (moderately useful rating). The lowest rated feature was ePortfolio with a mean of 2.00 (slightly useful rating); however, it should be noted only 29% of respondents (7) rated this item. Faculty and student respondents were also asked to rate their level of agreement with a variety of statements about Desire2Learn LMS on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement by faculty respondents was Desire2Learn enabled me to do what I wanted for my course(s) with a mean of 3.83 (agree rating); the lowest rated statement was Desire2Learn was a valuable aid to me in my teaching with a mean of 2.80 (neither agree nor disagree rating). For student respondents, the highest rated statement was Desire2Learn helps me to complete course assignments with a mean of 3.37 (agree rating). The statement with the highest percentage of strongly disagree and disagree ratings from student respondents was Desire2Learn helps me to study for exams/tests (30%); the statement with the lowest percentage of strongly disagree and disagree ratings was Desire2Learn was beneficial to my overall learning in the course (18% of respondents to the question).

3    

When asked what they most liked about Desire2Learn LMS faculty respondents noted the ability to randomize assessment questions; the overall interface’s look and feel; and the drag and drop capability. Faculty respondents disliked the difficulty in navigating the initial interface (Set Up feature) and between students’ assignments and gradebook feature; and difficulty in accessing students’ coursework submissions (Dropbox feature). No features or tools in Desire2Learn LMS were described as allowing faculty to design their courses and/or teach in a new way. Student respondents most liked about Desire2Learn the appearance of the interface; receiving confirmations of assignment submissions and updates on course grades; and ease of access to supplemental videos for a course. Student respondents disliked the difficulty in learning how to use the site’s features; confusion in navigating discussion board postings and in locating course content and assignments; and the lack of interface/linkage between the Desire2Learn email system and the IU email system.   Similar  Faculty  and  Student  Preferences  for  LMS  Systems  

Thirty-three percent (2) of faculty respondents prefer Desire2Learn over Oncourse LMS, 50% (3) prefer Oncourse over Desire2Learn LMS, and 17% (1) have no preference. Twenty-six percent (6/23) of student respondents prefer Desire2Learn over Oncourse LMS, 57% (13/23) prefer Oncourse over Desire2Learn LMS, and 17% (4/23) have no preference.    

4    

Evaluation  of  the  Pilot  of  the  Desire2Learn  Learning  Management  System    Implemented  in  Fall  2013  

Summary  of  Survey  December  18,  2013  

Overview    The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the Desire2Learn Learning Management System Pilot (Desire2Learn LMS) Program being implemented during the 2013-2014 academic year across IU campuses (IU Bloomington, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast, and IUPU Columbus). This aggregate report for UITS summarizes the findings of the evaluation activities conducted between August and December 2013. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. As part of the evaluation, staff members from CEEP and UITS collected data from faculty members (total n=17) who used Desire2Learn LMS in their courses during the fall 2013 semester and from students (total n=154) enrolled in those courses. The evaluation data collection included online surveys of faculty members and students aimed at capturing stakeholders’ experiences with Desire2Learn LMS and their perceptions of its quality and utility. Dr. Mary Piontek, Senior Research Associate at CEEP, oversaw the design and implementation of the evaluation; created the materials for the Institutional Review Board (IRB) application and submitted the IRB application; conducted the faculty and student survey collection activities; completed the analyses of the survey data; and created the preliminary version of this report on behalf of CEEP. Dr. Anastasia Morrone, Associate Vice President for Learning Technologies and Associate Professor of Educational Psychology, participated in the design and development of the survey instruments and also contributed to the final version of this report.    

5    

Summary  of  Faculty  Survey  

In November 2013 the 17 faculty members who piloted the Desire2Learn LMS in their courses during the fall 2013 semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via email to their university email address by Dr. Mary Piontek at CEEP. The faculty members were invited to voluntarily participate in the survey data collection on November 11th, with follow-up emails sent on December 2nd and December 11th to all non-respondents. The survey was closed at 5pm (EDT) on December 15, 2013. Six faculty members responded to the survey (35% = 6/17). The survey included closed-ended and open-ended questions related to the faculty members’ experiences with Desire2Learn LMS and their perceptions of its quality and utility. The survey took approximately 20 minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 18 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix A, Faculty Survey: Desire2Learn Pilot Fall 2013 Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals. Three respondents identified themselves as female and one as male; one respondent did not identify the gender. Table 1 provides the years of experience as a faculty member/instructor in higher education, with all respondents having no more than 10 years of experience. Table 2 provides the campus affiliation of the respondents. Table  1:  Years  as  Faculty  Member/Instructor  in  Higher  Education  Years  as  a  Faculty  Member/Instructor   Total  N   %  1 year or less 2 33% 2-5 years 1 17% 6-10 years 3 50% 11-20 years 0 0% 21-30 years 0 0% More than 30 years 0 0% Total 6 100% Table  2:  Campus  Affiliation  Campus  Affiliation   Total  N   %  IU Bloomington 1 17% IUPUI 1 17% IU East 2 33% IU Kokomo 1 17% IU Northwest 0 0% IU South Bend 0 0% IU Southeast 0 0% IUPU Columbus 1 17% Total 6 100%

During the fall 2013 semester Desire2Learn LMS was scheduled to be piloted in nine courses; respondents to the faculty survey represented six courses. Those courses included the academic areas of chemistry, nursing, political science, and psychology [Note: the list of the courses that the faculty

6    

respondents taught is not included in this report to provide anonymity to respondents]. Thirty-three percent of the courses were structured as face-to-face or only online with no face-to-face interaction (see Table 3). All respondents (100%) described themselves as being very comfortable in using different types of technology (see Table 4). Respondents used a variety of technology devices to interact with Desire2Learn LMS including most often a laptop/netbook computer (100%), and a tablet (67%) (see Table 5). Table  3:  Course  Format  Course  Format   Total  N   %  Face-to-face 2 33% In a hybrid format using a blend of face-to-face and online interaction 1 17% Online with face-to-face interaction only for exams 0 0% Only online with no face-to-face interaction 2 33% Other 1 17% Total 6 100% Table  4:  Level  of  Comfort  with  Technology  Level  of  Comfort  in  Using  Different  Types  of  Technology   Total  N   %  Very Uncomfortable 0 0% Somewhat Uncomfortable 0 0% Somewhat Comfortable 0 0% Very Comfortable 6 100% Total 6 100% Table  5:  Devices  Used  to  Interact  with  Desire2Learn  LMS  Mobile  Devices(s)  Used  to  Interact  with  Desire2Learn   Total  N   %  

N/22  Mobile phone without web access 0 0% Mobile phone with web access 2 33% Portable media player without web access (e.g., mp3 player) 0 0% Portable media player with web access (e.g., iPod Touch) 1 17% Ebook reader (e.g., Kindle) 0 0% Tablet (e.g., iPad) 4 67% Laptop/Netbook computer 6 100% Other device 0 0% The faculty respondents were also asked to rate their level of satisfaction with the features of Desire2Learn LMS (see Table 6). Not all features were equally used across the courses. All six respondents posted course announcements (News); and uploaded, organized, and shared course files (Manage Files, Content), with mean ratings of 2.83 and 3.17 respectively. Features used by at least 50% of respondents (i.e., three or more respondents) had means in the slightly satisfied to highly satisfied range (mean range 1.50-3.67). No respondents (0%) used the features of assigning individual or collaborative portfolio or web projects to students (ePortfolio); creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades); conducting online chat sessions (Chat); importing or exporting course content (Import/Export/Copy Components); and integrating an external learning tool or platform with the course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools).

7    

Table  6:  Level  of  Satisfaction  with  Desire2Learn  LMS  Tasks  Please  rate  your  level  of  satisfaction  with  the    Desire2Learn  tools  and  features  designed  to  support  the  following  teaching  and  course  management  tasks:  

Did  Not  Use  

Not  at  all  Satisfied  

1  

Slightly  Satisfied  

2  

Moderately  Satisfied  

3  

Highly  Satisfied  

4  

Total  N  

Mean  

Creating and publishing the course syllabus (Content)

2 (33%)

0 (0%)

0 (0%)

3 (50%)

1 (17%)

6 3.25

Creating a course calendar (Calendar)

4 (67%)

0 (0%)

1 (17%)

1 (17%)

0 (0%)

6 2.50

Posting course announcements (News)

0 (0%)

1 (0%)

2 (33%)

0 (10%)

3 (50%)

6 2.83

Uploading, organizing, and sharing course files (Manage Files, Content)

0 (0%)

0 (0%)

2 (33%)

1 (17%)

3 (50%)

6 3.17

Posting audio/video lectures or other multimedia (Content)

5 (83%)

0 (0%)

0 (0%)

0 (0%)

1 (17%)

6 4.00

Creating course web pages (Content > Modules > Create a File)

4 (67%)

1 (17%)

0 (0%)

0 (0%)

1 (17%)

6 2.50

Organizing course content, activities, and assessments into a series of modules or lessons (Course Builder, Content)

1 (17%)

0 (0%)

3 (50%)

0 (0%)

2 (33%)

6 2.80

Posting assignments (Dropbox)

3 (50%)

0 (0%)

0 (0%)

1 (17%)

2 (33%)

6 3.67

Using Turnitin originality checking on assignments (Dropbox)

5 (83%)

0 (0%)

0 (0%)

0 (0%)

1 (17%)

6 4.00

Creating and administering online quizzes, tests, and/or surveys (Quizzes, Self-Assessments, Surveys)

6 (83%)

0 (0%)

0 (0%)

1 (17%)

0 (0%)

6 3.00

Assigning individual or collaborative portfolio or web projects to students (ePortfolio)

6 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

6 N/A

Facilitating graded and ungraded discussions (Discussions)

1 (17%)

1 (17%)

1 (17%)

1 (17%)

2 (33%)

6 2.80

Giving feedback on and/or grading student submissions (Dropbox, Discussions, Grades)

1 (17%)

1 (17%)

1 (17%)

0 (0%)

3 (50%)

6 3.00

Giving audio or video feedback (Dropbox, Discussions, Grades)

5 (83%)

0 (0%)

0 (0%)

0 (0%)

1 (17%)

6 4.00

   

8    

Table  6:  Level  of  Satisfaction  with  Desire2Learn  LMS  Tasks  (continued)  Please  rate  your  level  of  satisfaction  with  the    Desire2Learn  tools  and  features  designed  to  support  the  following  teaching  and  course  management  tasks:  

Did  Not  Use  

Not  at  all  Satisfied  

1  

Slightly  Satisfied  

2  

Moderately  Satisfied  

3  

Highly  Satisfied  

4  

Total  N  

Mean  

Creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades)

6 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

6 N/A

Setting up and using the gradebook to enter and track student grades (Grades)

1 (17%)

2 (33%)

2 (033)

0 (0%)

1 (17%)

6 2.00

Monitoring course activity and student progress (View User Progress)

2 (33%)

0 (0%)

2 (33%)

0 (0%)

2 (33%)

6 3.00

Creating course learning outcomes and objectives and aligning course activities with them (Competencies)

3 (50%)

1 (17%)

0 (0%)

0 (0%)

2 (33%)

6 3.00

Sending and receiving messages to and from students and groups (Email, Pager)

2 (33%)

3 (50%)

0 (0%)

1 (17%)

0 (0%)

6 1.50

Creating and managing groups for group assignments, group discussions, and/or group projects (Groups)

3 (50%)

2 (33%)

0 (0%)

0 (0%)

1 (17%)

6 2.00

Conducting online chat sessions (Chat)

6 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

6 N/A

Keeping track of your course tasks (Calendar, Upcoming Events, Notifications)

3 (50%)

2 (33%)

0 (0%)

0 (0%)

1 (17%)

6 2.00

Importing or exporting course content (Import/Export/Copy Components)

6 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

6 N/A

Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools)

6 (100%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

6 N/A

Customizing the navigation, look, and feel of your course (Homepages, Navigation and Themes)

3 (50%)

1 (17%)

0 (0%)

1 (17%)

1 (17%)

6 2.67

   

9    

Thirty-three percent of faculty respondents (2) found Desire2Learn LMS very easy to use, highly useful for their teaching, and useful for online documentation (see Tables 7 and 8). Table  7:  Overall  Ease  of  Use  of  Desire2Learn  LMS  Please  rate  the  overall  ease  of  use  of  Desire2Learn.  

Difficult  to  Use  

Slightly  Easy  to  Use  

Moderately  Easy  to  Use  

Very  Easy  to  Use  

Total  N  

Mean  

Overall ease of use of Desire2Learn 2 (33%)

2 (33%)

0 (0%)

3 (33%)

6 2.33

Table  8:  Overall  Utility  of  Desire2Learn  LMS  for  Teaching  Please  rate  the  overall  usefulness  of  Desire2Learn  for  your  teaching.  

Not  at  all  Useful  

Slightly  Useful  

Moderately  Useful  

Highly  Useful  

Total  N  

Mean  

Overall usefulness of Desire2Learn for your teaching

1 (17%)

1 (17%)

2 (33%)

2 (33%)

6 2.83

Overall usefulness of Desire2Learn’s online documentation

1 (17%)

3 (50%)

0 (0%)

2 (33%)

6 2.50

 

Respondents were also asked to rate their level of agreement with a variety of statements about Desire2Learn LMS (see Table 9). All of the statements were rated 2.80 or above on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement was Desire2Learn enabled me to do what I wanted for my course(s) with a mean of 3.83 (agree rating). The lowest rated statement was Desire2Learn was a valuable aid to me in my teaching with a mean of 2.80 (neither agree nor disagree rating). Thirty-three percent (2) of faculty respondents prefer Desire2Learn over Oncourse LMS, 50% (3) prefer Oncourse over Desire2Learn LMS, and 17% (1) have no preference. When asked what they most liked about Desire2Learn LMS faculty respondents noted the ability to randomize assessment questions; the overall interface’s look and feel; and the drag and drop capability. When asked what they liked least about Desire2Learn LMS faculty respondents noted difficulty in navigating the initial interface (Set Up feature) and between students’ assignments and gradebook feature; and difficulty in accessing students’ coursework submissions (Dropbox feature). No features or tools in Desire2Learn LMS were noted that allowed faculty to design their courses and/or teach in a new way.  

10    

Table  9:  Level  of  Agreement  with  Statements  about  Desire2Learn  LMS  Please  rate  your  level  of  agreement  with  the  following  statements  about  Desire2Learn.  

Strongly  Disagree  

1  

Disagree  2  

Neither  Agree  nor  Disagree  

3  

Agree  4  

Strongly  Agree  5  

Not  Applicable  

Total  N  

Mean  

Desire2Learn enabled me to do what I wanted for my course(s).

0 (0%)

1 (17%)

1 (17%)

2 (33%)

2 (33%)

0 (0%)

6 3.83

Desire2Learn was easy for my students to learn how to use.

1 (17%)

2 (33%)

1 (17%)

0 (0%)

2 (33%)

0 (0%)

6 3.00

Desire2Learn increased my efficiency as a teacher.

2 33%)

1 (17%)

1 (17%)

0 (0%)

2 (33%)

0 (0%)

6 2.83

Desire2Learn increased my effectiveness as a teacher.

2 (33%)

1 (17%)

1 (17%)

0 (0%)

2 (33%)

0 (0%)

6 2.83

Desire2Learn was a valuable aid to me in my teaching.

1 (20%)

1 (20%)

2 (40%)

0 (0%)

1 (20%)

0 (0%)

5 2.80

Using Desire2Learn has been beneficial to my students’ overall learning.

1 (17%)

1 (17%)

2 (33%)

0 (0%)

2 (33%)

0 (0%)

6 3.17

11    

Summary  of  Student  Survey  

In November 2013 the 154 students who were enrolled in a course that piloted the Desire2Learn LMS during the fall 2013 semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via email to their university email address by Dr. Piontek. The students were invited to voluntarily participate in the survey data collection on November 11th, with follow-up emails sent on December 2nd and December 13th to all non-respondents. The survey was closed at 5pm (EDT) on December 15, 2013. Twenty-four students responded to the survey (16% = 24/154). The survey included closed-ended and open-ended questions related to the students’ experience with Desire2Learn LMS and their perceptions of its quality and utility. The survey took approximately 20 minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 18 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix B, Student Survey: Desire2Learn Pilot Fall 2013 Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals.  Eighteen (75%) respondents identified themselves as female and six (25%) as male. The age of students ranged from 18 to 59 years of age with the greatest percentage (63%, 15/24) in the 18 to 25 range. Thirty-three percent (8) of student respondents are second-year undergraduates (sophomores) (see Table 10). Fifty-eight percent of respondents are affiliated with the IU East campus (see Table 11). During the fall 2013 semester Desire2Learn LMS was scheduled to be piloted in nine courses; respondents to the student survey represented eight courses (see Table 12). Fifty-four percent (13) of the respondents were in courses with a face-to-face format (see Table 13). Table  10:  Academic  Level  Academic  Level   Total  N   %  First-year undergraduate (Freshman) 2 8% Second-year undergraduate (Sophomore) 8 33% Third-year undergraduate (Junior) 2 8% Four or more years undergraduate (Senior) 6 25% Masters student (MA, MS, MBA, MFA, MSW, MPA, etc.) 6 25% Doctoral Student (EdD, PhD, etc.) 0 0% Total 24 100% Table  11:  Campus  Affiliation  Campus  Affiliation   Total  N   %  IU Bloomington 3 13% IUPUI 5 21% IU East 14 58% IU Kokomo 1 4% IU Northwest 0 0% IU South Bend 0 0% IU Southeast 0 0% IUPU Columbus 1 4% Total 24 100%

   

12    

Table  12:  Name  of  Enrolled  Course      Course  Name   Total  

N  %  

CHEM C105, Principles of Chemistry I, EA, Section 14141 11 48% CHEM C105, Principles of Chemistry I, EA, Section 15017 1 4% EDUC W435, Technology Leadership IN K-12, CO, Section 32098 0 0% INFO B535, Clinical Information Systems, IN, Section 25570 2 9% NURS I631, Clinical Information Systems, IN, Section 23364 2 9% NURS T617, Evaluation in Nursing, IN, Section 24063 1 4% PHSL P215, Basic Human Physiology, EA, Section 14694 2 9% POLS Y490, Pol Principle and Social Justice, BL, Section 6852 3 13% PSY P495, Practicum in Psychology, KO, Section 27702 1 4% Total 23 100% Table  13:  Course  Format  Course  Format   Total  N   %  Face-to-face 13 54% In a hybrid format using a blend of face-to-face and online interaction 4 17% Online with face-to-face interaction only for exams 0 0% Only online with no face-to-face interaction 7 29% Other 0 0% Total 24 100% Thirty-eight percent (9) of respondents described themselves as being very comfortable in using different types of technology and 25% (6) described themselves as being very uncomfortable (see Table 14). Respondents used a variety of technology devices to interact with Desire2Learn LMS including most often a laptop/netbook computer (96%) and mobile phone with web access (75%) (see Table 15). Most students (46%) averaged fewer than five hours per week using Desire2Learn for their courses (see Table 16). Forty-three percent of student respondents (10) found Desire2Learn LMS moderately easy to use and 48% moderately useful (11) for online documentation (see Tables 17 and 18). Table  14:  Level  of  Comfort  with  Technology  Level  of  Comfort  in  Using  Different  Types  of  Technology   Total  N   %  Very Uncomfortable 6 25% Somewhat Uncomfortable 2 8% Somewhat Comfortable 7 29% Very Comfortable 9 38% Total 24 100%

   

13    

Table  15:  Devices  Used  to  Interact  with  Desire2Learn  LMS  Mobile  Devices(s)  Used  to  Interact  with  Desire2Learn   Total  N   %  

(N/129)  Mobile phone without web access 2 8% Mobile phone with web access 18 75% Portable media player without web access (e.g., mp3 player) 2 8% Portable media player with web access (e.g., iPod Touch) 4 17% Ebook reader (e.g., Kindle) 5 21% Tablet (e.g., iPad) 5 21% Laptop/Netbook computer 23 96% Other device 0 0% Table  16:  Average  Number  of  Hours  Per  Week  Using  Desire2Learn  LMS  Average  Number  of  Hours  Using  Desire2Learn  for  the  Course  During  the  fall  2013  Semester  

Total  N   %  

Never 6 25% Fewer than 5 hours 11 46% 5-10 hours 4 17% 11-15 hours 1 4% 16-20 hours 2 8% More than 20 hours per week 0 0% Total 24 100% Table  17:  Overall  Ease  of  Use  of  Desire2Learn  LMS  Please rate the overall ease of use of Desire2Learn.

Difficult to Use

Slightly Easy to

Use

Moderately Easy to

Use

Very Easy to

Use

Total N

Mean

Overall ease of use of Desire2Learn 4 (17%)

2 (9%)

10 (43%)

7 (31%)

23 2.87

Table  18:  Overall  Utility  of  Desire2Learn  LMS  for  Learning  Please  rate  the  overall  usefulness  of  Desire2Learn  for  your  learning.  

Not  at  all  Useful  

Slightly  Useful  

Moderately  Useful  

Highly  Useful  

Total  N  

Mean  

Overall usefulness of Desire2Learn’s online documentation

5 (22%)

2 (8%)

11 (48%)

5 (22%)

23 2.70

The student respondents were asked to rate the utility (i.e., usefulness) of the features of Desire2Learn LMS on a scale of 1 to 4 (not at all useful to highly useful) (see Table 19). The highest rated feature was Grades (e.g., potential points for an assignment, your results on a quiz/assignment, overall course grade, etc.) with a mean of 3.44. The lowest rated feature was ePortfolio (for personal or collaborative web sites about your accomplishments or other topics) with a mean of 2.00 (however, it should be noted only 29% of respondents rated this item). The most frequently used feature was Content (for viewing course materials and completing activities organized into lessons or modules) with 87% of respondents (21) rating the item. The least frequently used feature was Blog (for creating a web log with public or private posts and optionally receiving comments from other users) with only 17% of respondents (3) rating the item. Student respondents were also asked to rate their level of agreement with a variety of statements about Desire2Learn LMS (see Table 20). All of the statements had a mean rating of 3.00 or above on a scale of 1 to 5 (strongly disagree to strongly agree). The highest rated statement was Desire2Learn helps me to complete course assignments with a mean of 3.37. The statement with the highest percentage of strongly

14    

disagree and disagree ratings was Desire2Learn helps me to study for exams/tests (30%). The statement with the lowest percentage of strongly disagree and disagree ratings was Desire2Learn was beneficial to my overall learning in the course (18% of respondents to the question). Twenty-six percent (6/23) of student respondents prefer Desire2Learn over Oncourse LMS, 57% (13/23) prefer Oncourse over Desire2Learn LMS, and 17% (4/23) have no preference. When asked what they most liked about Desire2Learn LMS student respondents noted the appearance of the overall interface; receiving confirmations of assignment submissions and updates on course grades; and ease of access to supplemental videos for the course. When asked what they liked least about Desire2Learn LMS respondents noted difficulty in learning how to use the site’s features; confusion in navigating discussion board postings and in locating course content and assignments; and the lack of interface/linkage between the Desire2Learn email system and the IU email system.   Table  19:  Utility  of  Desire2Learn  LMS  Features  Please  rate  the  usefulness  of  the  following  features  of  Desire2Learn  in  contributing  to  your  learning  in  this  course.  

Did  Not  Use  This  Feature  

Not  at  all  

Useful  1  

Slightly  Useful  

2  

Moderately  Useful  

3  

Highly  Useful  

4  

Total  N  

Mean  

Blog (for creating a web log with public or private posts and optionally receiving comments from other users)

20 (83%)

1 (4%)

0 (0%)

0 (0%)

3 (13%)

24 3.25

Calendar (for viewing course events and due dates)

11 (46%)

1 (4%)

4 (17%)

2 (8%)

6 (25%)

24 3.00

Chat/Pager (for live text messaging with classmates and other Desire2Learn users)

17 (71%)

2 (8%)

0 (0%)

2 (8%)

3 (13%)

24 2.86

Checklist (for completing a list of tasks prepared by the instructor)

7 (29%)

1 (4%)

4 (17%)

7 (29%)

5 (21%)

24 2.94

Classlist (for viewing a list of the other people in the course)

9 (38%)

3 (13%)

1 (4%)

9 (38%)

2 (8%)

24 2.67

Content (for viewing course materials and completing activities organized into lessons or modules)

3 (13%)

1 (4%)

4 (16%)

5 (21%)

11 (46%)

24 3.24

Discussion (for participating in online discussions with the entire class or in small groups)

10 (41%)

4 (17%)

4 (17%)

1 (4%)

5 (21%)

24 2.50

Dropbox (for submitting individual or group assignments)

11 (46%)

2 (8%)

3 (13%)

2 (8%)

6 (25%)

24 2.92

Email (for sending and receiving messages to and from your instructor and other students)

4 (17%)

4 (17%)

3 (13%)

4 (17%)

9 (38%)

24 2.90

ePortfolio (for personal or collaborative web sites about your accomplishments or other topics)

17 (71%)

3 (13%)

2 (8%)

1 (4%)

1 (4%)

24 2.00

Grades (for viewing a list of the graded items in the course and the grades you received)

6 (25%)

2 (8%)

1 (4%)

2 (8%)

13 (54%)

24 3.44

15    

Table  19:  Utility  of  Desire2Learn  LMS  Features  (continued)  Please  rate  the  usefulness  of  the  following  features  of  Desire2Learn  in  contributing  to  your  learning  in  this  course.  

Did  Not  Use  This  Feature  

Not  at  all  

Useful  1  

Slightly  Useful  

2  

Moderately  Useful  

3  

Highly  Useful  

4  

Total  N  

Mean  

Groups (for collaborating with a specific group of students on assignments, discussions, or projects)

14 (58%)

3 (13%)

2 (8%)

1 (4%)

4 (17%)

24 2.60

Locker (for storing personal files or files belonging to your group)

19 (79%)

2 (8%)

1 (4%)

0 (0%)

2 (8%)

24 2.40

News (read announcements and other timely news and information posted by your instructor)

9 (38%)

2 (8%)

3 (13%)

5 (21%)

5 (21%)

24 2.87

Quizzes/ Self-assessments (for taking and receiving feedback on online quizzes, tests, and self-assessments)

8 (33%)

2 (8%)

2 (8%)

5 (21%)

7 (29%)

24 3.06

Rubrics (for understanding how your work will be or was graded)

11 (46%)

2 (8%)

4 (17%)

2 (8%)

5 (21%)

24 2.77

Surveys (for taking online surveys)

14 (58%)

2 (8%)

3 (13%)

2 (8%)

3 (13%)

24 2.60

User Progress (for tracking your own academic progress in the course)

10 (42%)

2 (8%)

0 (0%)

5 (21%)

7 (29%)

24 3.21

Table  20:  Level  of  Agreement  with  Statements  about  Desire2Learn  LMS  Please  rate  your  level  of  agreement  with  the  following  statements  about  Desire2Learn.  

Strongly  Disagree  

1  

Disagree  2  

Neither  Agree  nor  Disagree  

3  

Agree  4  

Strongly  Agree  5  

Not  Applicable  

Total  N  

Mean  

Desire2Learn helps me to learn the course materials/content.

3 (13%)

2 (9%)

5 (22%)

8 (35%)

2 (9%)

3 (13%)

23 3.20

Desire2Learn helps me to study for exams/tests.

4 (17%)

3 (13%)

6 (26%)

3 (13%)

4 (17%)

3 (13%)

23   3.00

Desire2Learn helps me to complete course assignments.

3 (13%)

2 (9%)

4 (17%)

5 (22%)

5 (22%)

4 (17%)

23   3.37

Desire2Learn helps me to take quizzes/exams.

4 (17%)

1 (4%)

5 (22%)

4 (17%)

6 (26%)

3 (13%)

23   3.35

Desire2Learn helps me to make efficient use of my time in the course.

3 (13%)

2 (9%)

8 (35%)

3 (13%)

4 (17%)

3 (13%)

23   3.15

Desire2Learn helps me to be in control of my own learning in the course.

3 (13%)

3 (13%)

8 (35%)

3 (13%)

3 (13%)

3 (13%)

23   3.00

 

16    

Table  20:  Level  of  Agreement  with  Statements  about  Desire2Learn  LMS  (continued)  Please  rate  your  level  of  agreement  with  the  following  statements  about  Desire2Learn.  

Strongly  Disagree  

1  

Disagree  2  

Neither  Agree  nor  Disagree  

3  

Agree  4  

Strongly  Agree  5  

Not  Applicable  

Total  N  

Mean  

Desire2Learn helps me to communicate with my professor.

3 (13%)

2 (9%)

6 (26%)

5 (22%)

5 (22%)

2 (9%)

23   3.33

Desire2Learn expands access to learning materials/resources available to me (e.g., print, audio, video, etc.).

2 (9%)

3 (13%)

8 (35%)

3 (13%)

4 (17%)

3 (13%)

23   3.20

Desire2Learn was beneficial to my overall learning in the course.

2 (9%)

2 (9%)

8 (35%)

6 (26%)

2 (9%)

3 (13%)

23   3.20

 

   

17    

Appendix  A  Faculty  Survey:  Desire2Learn  Pilot  Fall  2013  Semester  

Introduction  and  Consent  

In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the Desire2Learn Learning Management System (Desire2Learn) Pilot Program being conducted during the fall 2013 semester. You have been invited to participate in this survey because you are teaching one or more courses in Desire2Learn this semester. Participation in this survey is completely voluntary. Should you decide to participate your responses will be kept confidential and will be reported only in aggregate in published reports. Comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately 20 minutes to complete. Data collection will close at 5pm (EST) on Day, Month, Date, 2013. You may terminate your participation in the survey at any time. Pre-question 1: If you consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey. I consent to participate in this survey. (required check box) Part  I:    Course/IU  Context    

Q1: For how many years have you been an instructor/faculty member in higher education? Scale: 1 year or less, 2-5 years, 6-10 years, 11-20 years, 21-30 years, more than 30 years Q2: What is your gender? (open text box) Q3: At which campus are you a faculty member? (Choose your PRIMARY academic appointment)

IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast

Q4: In which course during Fall 2013 did you use Desire2Learn? (If you used Desire2Learn in

multiple courses, please choose one for the purpose of this survey.) Although we ask you to identify the course in which you piloted Desire2Learn, this survey is not an evaluation of the course. We will use this information to report aggregate response rates across the courses and to create categories of types (disciplinary areas) and levels of courses for reporting purposes. (drop down list of Desire2Learn pilot courses)

18    

Q5: The course was delivered (Choose one BEST answer) Scale: primarily face-to-face; using a blend of face-to-face and online interaction; online with face-to-face interaction only for exams; exclusively online with no face-to-face interaction; other (please describe)

Part  II:    Use  of  Technology    

Q6: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q7: Which device(s) do you currently use to interact with Desire2Learn? (Choose all that apply)

mobile phone; portable media player (e.g., iPod Touch); ebook reader (e.g., Kindle); tablet (e.g., iPad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe)

Part  III:    Feedback  on  Desire2Learn    

Q8: Based on your experience during the fall 2013 pilot, please rate your level of satisfaction with the Desire2Learn tools and features designed to support the following teaching and course management tasks: Scale: did not use, not at all satisfied, slightly satisfied, moderately satisfied, highly satisfied

• Creating and publishing the course syllabus (Content) • Creating a course calendar (Calendar) • Posting course announcements (News) • Uploading, organizing, and sharing course files (Manage Files, Content) • Posting audio/video lectures or other multimedia (Content) • Creating course web pages (Content > Modules > Create a File) • Organizing course content, activities, and assessments into a series of modules or lessons

(Course Builder, Content) • Posting assignments (Dropbox) • Using Turnitin originality checking on assignments (Dropbox) • Creating and administering online quizzes, tests, and/or surveys (Quizzes, Self-

Assessments, Surveys) • Assigning individual or collaborative portfolio or web projects to students (ePortfolio) • Facilitating graded and ungraded discussions (Discussions) • Giving feedback on and/or grading student submissions (Dropbox, Discussions, Grades) • Giving audio or video feedback (Dropbox, Discussions, Grades) • Creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions,

Quizzes, Grades) • Setting up and using the gradebook to enter and track student grades (Grades) • Monitoring course activity and student progress (View User Progress) • Creating course learning outcomes and objectives and aligning course activities with

them (Competencies) • Sending and receiving messages to and from students and groups (Email, Pager)

19    

• Creating and managing groups for group assignments, group discussions, and/or group projects (Groups)

• Conducting online chat sessions (Chat) • Keeping track of your course tasks (Calendar, Upcoming Events, Notifications) • Importing or exporting course content (Import/Export/Copy Components) • Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud,

Piazza, etc. (External Learning Tools) • Customizing the navigation, look, and feel of your course (Homepages, Navigation and

Themes) • Other

Q9: Please rate the overall ease of use of Desire2Learn. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q10: Please rate the overall usefulness of Desire2Learn for your teaching. Scale: not at all useful, slightly useful, moderately useful, highly useful Q11: Please rate the overall usefulness of Desire2Learn’s online documentation. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q12: What did you like MOST about Desire2Learn? Why? (open text box) Q13: What did you like LEAST about Desire2Learn? Why? (open text box) Q14: Which, if any, features/tools in Desire2Learn allowed you to design your course and/or teach in a

new way? (open text box)

Q15 Please rate your level of agreement with the following statements about Desire2Learn:

Scale: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree, not applicable

1. Desire2Learn enabled me to do what I wanted for my course(s). 2. Desire2Learn was easy for my students to use. 3. Desire2Learn increased my efficiency as a teacher. 4. Desire2Learn increased my effectiveness as a teacher. 5. Desire2Learn was a valuable aid to me in my teaching. 6. Using Desire2Learn has been beneficial to my students’ overall learning.

20    

Q16 Overall, compared to Indiana University’s current LMS (Oncourse), please select the choice that best describes your preference for Oncourse versus Desire2Learn:

• I prefer Desire2Learn over Oncourse. • I prefer Oncourse over Desire2Learn. • I have no preference.

Q17 Additional Comments

Is there anything else you would like to tell us about your experience using Desire2Learn this semester? (open text box)

   

21    

Appendix  B  Student  Survey:  Desire2Learn  Pilot  Fall  2013  Semester  

 Introduction  and  Consent  

In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the Desire2Learn Learning Management System (Desire2Learn) Pilot Program being conducted during the fall 2013 semester. You have been invited to participate in this survey because you are enrolled in one or more courses being taught in Desire2Learn this semester. Participation in this survey is completely voluntary. Should you decide to participate your responses will be kept confidential and will be reported only in aggregate in published reports. Comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately 15 minutes to complete. Data collection will close at 5pm (EST) on Day, Month, Date, 2013. You may terminate your participation in the survey at any time.

Pre-Question 1: If you are 18 years of age or older as of today’s date and consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey.

If you are NOT 18 years of age or older as of today’s date or do not consent, please close the link to the survey now.

I acknowledge that I am at least 18 years of age or older as of today’s date and consent to participate in this survey (will be a required check box) Part  I:    Student  Demographics/Institutional  Context    

Q1: What is your current academic level? Scale: first-year undergraduate (freshman), second-year undergraduate (sophomore), third-year undergraduate (junior), four or more years undergraduate (senior), master’s degree student (MA, MS, MBA, MFA, etc.), doctoral student (EdD, PhD, etc.)

Q2: At which campus are you enrolled as a student? (Choose ONE)

IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast

Q3: What is your age? (open text box) Q4: What is your gender? (open text box)

22    

Q5: Which course were you enrolled in during fall 2013 that used Desire2Learn? (If you were enrolled in more than one course that used Desire2Learn, please choose one for the purpose of this survey) Although we ask you to identify the course in which you were enrolled, this survey is not an evaluation of the course. We will use this information to report aggregate response rates within each course and to create categories of types (disciplinary areas) and levels of courses for reporting purposes. (drop down list of Desire2Learn pilot courses)

Q6: That course was delivered (Choose one BEST answer) primarily face-to-face; using a blend of face-to-face and online interaction; online with face-to-face interaction only for exams; exclusively online with no face-to-face interaction; other (please describe)

Part  II:    Use  of  Technology    

General Level of Experience/Use: Q7: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q8: What type(s) of networked device(s) do you currently use on a regular basis?

mobile phone; portable media player (e.g., iPod Touch); ebook reader (e.g., Kindle); tablet (e.g., iPad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe)

Q9: On average, how many hours per week have you been spending in Desire2Learn for this course? Scale: none, fewer than 5 hours, 5-10 hours, 11-15 hours, 16-20 hours, more than 20 hours Part  III:    Feedback  on  Desire2Learn      

Q10: Please rate the Usefulness of the following features of Desire2Learn in contributing to your learning in this course.

Scale: did not use this feature, not at all useful, slightly useful, moderately useful, highly useful

• Blog (for creating a web log with public or private posts and optionally receiving comments from other users)

• Calendar (for viewing course events and due dates) • Chat/Pager (for live text messaging with classmates and other Desire2Learn users) • Checklist (for completing a list of tasks prepared by the instructor) • Classlist (for viewing a list of the other people in the course) • Content (for viewing course materials and completing activities organized into lessons or

modules) • Discussion (for participating in online discussions with the entire class or in small

groups)

23    

• Dropbox (for submitting individual or group assignments) • Email (for sending and receiving messages to and from your instructor and other

students) • ePortfolio (for personal or collaborative web sites about your accomplishments or other

topics) • Grades (for viewing a list of the graded items in the course and the grades you received) • Groups (for collaborating with a specific group of students on assignments, discussions,

or projects) • Locker (for storing personal files or files belonging to your group) • News (read announcements and other timely news and information posted by your

instructor) • Quizzes/ Self-assessments (for taking and receiving feedback on online quizzes, tests,

and self-assessments) • Rubrics (for understanding how your work will be or was graded) • Surveys (for taking online surveys) • User Progress (for tracking your own academic progress in the course)

Q11: Please rate the overall ease of use of Desire2Learn. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q12: Please rate the overall usefulness of Desire2Learn’s online documentation for students. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q13: What did you like MOST about Desire2Learn? Why? (open text box) Q14: What did you like LEAST about Desire2Learn? Why? (open text box) Q15: Perceived impact on learning

Scale: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree, not applicable

Desire2Learn helps me to learn the course materials/content. Desire2Learn helps me to study for exams/tests. Desire2Learn helps me to complete course assignments. Desire2Learn helps me to take quizzes/exams. Desire2Learn helps me to make efficient use of my time in the course. Desire2Learn helps me to be in control of my own learning in the course. Desire2Learn helps me to communicate with my professor. Desire2Learn expands access to learning materials/resources available to me (e.g., print, audio, video, etc.). Desire2Learn was beneficial to my overall learning in the course.

24    

Q16 Overall, compared to Indiana University’s current LMS (Oncourse), please select the choice that best describes your preference for Oncourse versus Desire2Learn:

• I prefer Desire2Learn over Oncourse. • I prefer Oncourse over Desire2Learn. • I have no preference.

Q17 Additional Comments

Is there anything else you would like to tell us about your experience using Desire2Learn this semester? (open text box)