Pathfinder E-Learning Evaluation: Qualitative and Quantitative Evidence Ruslan Ramanau (Research Fellow in E-Learning, OCSLD)

Pathfinder E-Learning Evaluation:

Qualitative and Quantitative Evidence

Ruslan Ramanau (Research Fellow in E-Learning, OCSLD)

April 4, 2008 Ruslan Ramanau 22

Aims and Key Research Question

Pathfinder evaluation aimed to explore learner experiences of learning technologies for interpersonal interaction and collaboration at Oxford Brookes

The two key areas of investigation were: student experiences of e-learning as part of their curriculum informal social uses of software to support their Universities

studies Key research question: How do students experience

social uses of technology in different learning contexts?


Approach to Research Design

Combined-methods approach was used where extensive qualitative and quantitative data collection was carried out

Series of seven case studies to evaluate the nature and the impact of the e-learning innovation on learner experiences

Survey on patterns of online media use in their relation to views on learner choice and independence in studies as well as strategies for self – and context regulation

Qualitative data provided rich contextual data on learner experiences, while the survey data aimed at discerning patterns in student perceptions of e-learning


List of Case Studies

Geoverse undergraduate research journal in Geography

Use of the discussion boards to support distance students in Contextual Theology

Use of Wikis to support learning Japanese grammar Collaborative work on Technology and Sustainability

on buildings case study uploaded onto Wiki Group work on research project design in Research

Methods for Human Biosciences Collaborative group work on Environmental Processes

and Change to be presented in class Use of Facebook to support Business in Context

undergraduate module


Research Methods Qualitative work: a series of semi-structured face-to-face

interviews with students on mainly their experiences of group and collaborative work and the role of learning technologies to support these activities

Interview schedules were designed in consultation with the case study leaders and used course syllabi and course descriptions as the departure point

The questionnaire data aimed to investigate emerging patterns in learner uses of online media and how they are related to other aspects of learning, such as perceptions of teaching effectiveness and self-regulatory techniques

Qualitative data focused on how learning contexts at the course level shape learner experiences and quantitative data helped to look how in general use online and other media in their studies and lives


Questionnaire Instrument and Sampling

The questionnaire consisted of four sections: demographic profile of the respondents; patterns in accessing online resources and choice in forms of interpersonal contact; patterns in online media use; views on learner choice, learning community and self-regulation in learning

The first three sections were designed by the research team, drawing on the previous research in the UK, the fourth section used validated instruments, such as the CEQ (Course Experience Questionnaire) and MSLQ (Motivated Strategies for Learning Questionnaire)

Proportional sampling was used for the study, i.e. school affiliation was viewed as one of the crucial determinants of learning experience

Therefore the number of participants from each school was roughly proportional to the number of full-time undergraduate students in a specific school to the overall number of undergraduate students at Oxford Brookes (based on the 2006 figures available on the Intranet)


Questionnaire: Online and Paper Versions

The questionnaire was administered both online via Brookes Virtual and before and after face-to-face lectures in November 2007

The online survey helped to reach to students who frequently use Brookes Virtual or online resources in general

Paper survey distribution helped to reach out to all other participants SAs (Student Ambassadors) were employed to survey students

from the faculties under represented in the Pathfinder (such as Arts and the Humanities)

The total number of respondents exceeded 1200 and each school met or exceeded the target figures in terms of numbers of participants

To increase the response rate a prize draw with prizes worth from £35 to £50 in Amazon vouchers was conducted once the questionnaires were completed


Approach to Data Analysis

Analysis of descriptive statistics: means, standard deviations and variance for each of the items

First- and second-order factor analysis for items in section C

Univariate and multivariate tests of variance and covariance on first-order factor-based scales as the dependent variables and demographic variables as the predictors or the covariates

Correlations between factor-based scores in section C and CEQ and MSLQ scores in section D


Survey Descriptive Statistics A total of 1181 participants completed the questionnaire – 400

filled in the online version and 781 a paper version The age of the participants was in the range between 17 to 64

years of age and the average age was 21.7 years of age 37.3 percent of the sample were male and 62.7 were female 87.8 percent were UK residents and 12.2 – international students 3.2 percent of the sample (38 students) were employed full-time,

i.e. 30 hours per week or more; 22.5 percent were employed between 10 and 30 hours per week and 15.1 percent were employed less than 10 hours a week. 59.2 percent of students were not employed

46.5 percent of the sample were first-year students, 25.3 percent were in their second-year of study and 27.9 were in their third year of studies at Oxford Brookes

91.2 percent of students did not declare any disability and 8.8 percent did


Descriptive Statistics by School

Online Print Age 1st year %

2nd year %

3rd year%

Est. % of students

Total

A & H 25 125 21.1 36.3 21.9 41.1 10.7 150

BE 15 94 19.8 84.1 6.5 9.3 9.5 109

LS 33 59 22.1 32.6 28.3 39.1 19.0 92

BS 82 134 20.6 35.6 39.4 25.0 9.3 216

HSC 105 92 26.7 77.9 9.2 12.8 12.0 197

SSL 76 46 21.7 30.3 31.1 38.5 9.5 122

Tech 64 68 21.7 72.1 12.4 15.5 11.9 132

WIE 0 162 22.8 5.7 47.1 46.5 10.7 162


Preferences in Use of the Web

Home was the most popular study location: 79.8 of students preferred to study from home; 11.0 preferred a college or a university study centre; 6.7 named the library and 2.4 pooled computer room as their preferred location of study

57.2 percent of students chose their home laptop as the preferred device to get online; 28.0 chose to use their desktop PCs to get online; 7.3 percent of the sample preferred to get online wirelessly on campus and 6.9 used pooled computer rooms


Preferences in Forms of Contact

Students used a combination of forms of contact other students at Brookes or friends, but not their their tutors

In terms of the choice of forms of contact with tutors e-mail was the most popular choice with 82.4 percent of the sample, face-to-face was second chosen by 52.1 percent of respondents and Brookes Virtual with 10.3 percent – third

Contact with other students: face-to-face came top with 75.0 percent; mobile – second with 73 percent and e-mail third with 71.6 percent

Contact with friends: 88.4 percent face-to-face; 88.1 mobile and 57.5 e-mail

Social networking sites came a close fourth for interaction with friends and other students - 53.3 preferred them to interact with other students and 43.3 to contact their friends


Results of Factor Analysis

To explore patterns in online use a PCI (principal component analysis) with varimax rotation was employed

After examining the Scree Plot and eigenvalues a five-factor solution was deemed appropriate

Five first-order factor based scales were identified Web for multimedia creation, editing and file sharing (items

C15, C17, C18 and C21) Use of Web 2.0 services (C24 and C25) Web for fora, blog and website contribution (C8, C9 and

C10, C11) Web for games (C19, C20) Web as a learning resource(C2, C4, C5 and C6)


School Differences on Patterns of Use

Both univariate and multivariate tests of difference showed that school affiliation was a strong predictor of online media usage, as ANOVA tests for all the five factor-based scales were significant at the .001 level

Health and Social Care and Social Sciences and Law students tended to make more use of the Web as a learning resource than other schools

Technology students reported more frequent blog and fora and website contributions

Technology and Business students reported more frequent use of games and Web 2.0 services compared to other schools

MANCOVA tests showed that even given the effects of mode of administration the differences all the five dimensions were at the statistically significant level


Other Group Differences

In addition to school affiliation, significant differences were observed both across different modes of questionnaire administration and the two gender groups on the five dimensions

Gender differences were statistically significant, but at the .05 level and no differences for the use of the Web for resources across the two groups were reported

When variance on the scores across levels of study and schools were taken into account gender differences were also highly statistically significant at the .001 level


Differences by Year Of Study

In contrast to all other demographic variables year of study did not appear to be a predictor of student scores on the five factor-based scales

The only exception was the use of the Web as a learning resources with students in their 1st year using it less often and 3rd years more often than 2nd year students


Age Differences Investigating differences across age groups helped to test the

hypotheses of Oblinger and other theorists who hypothesized that students under 25 years of age use the Web more proactively than older students

All students were grouped into three age bands: 154 students of 17-19 years of age (teenagers), 406 students of 20-25 years of age 406 (early and mid-20s) and 579 students of 25 years of age and older (“mature students”)

The differences on all the five dimensions were at high level of statistical significance (at least .01)

Mature students tended to use the Web as a learning resource more often; students aged 20 to 25 tended to contribute to Web sites and blogs more often and used the multimedia resources far more often than mature students

17 to 19 year olds tended to use play games more often and 17 to 19 year olds and 20-25 year olds were more likely to use Web 2.0 tools

When the impact of gender, school and year of study were used as the covariates the differences the differences across age groups were even more salient


Further Analysis A Second-order factor analysis suggested a two-factor solution

with the use of Web as learning resource being the first dimensions and four other first-order scales – the second one

A bivariate correlation analysis was conducted on the means across five first-order factor-based scales and means for the help-seeking and learning community scales

Both perceptions of learning community and the use of peers in learning were positively associated with the use of the Web for multimedia creation and use and blog and website posting and the use of Web as a learning resource

The use of Web for games was only linked to the use of peers in studies and the use of Web 2.0 tools was not associated with either peer help-seeking or views on learning community


Qualitative Data

Students across all seven case studies took part in the study

17 interviews and one focus group session have been conducted

The interview data and the data from the focus group session has been transcribed and analysed

The data was coded and using the inductive approach to data analysis a number of themes was developed


Interview Themes - Timing Most students were vary of the time constraints and

struggled to find the time for group activities Most groups tended to work much harder in the last few

days of collaborative tasks (sometimes having to stay late or get up very early)

The groups where a regular pattern of work was maintained and sufficient contact was established by personal or electronic communication tended to express more satisfaction with the results of group work

Students noted that the time for forming groups and making sense of tasks was not sufficient and suggested that the lack of clarity in the end product created additional difficulties


Interview Themes: Leaders and Followers

Most students reported having one or two clear leaders in group work who either did a significant share of work or also kept the work and contact between group members going

Students referred to personality clashes and misunderstandings in group work that were rooted in differences between students, but they mainly managed to overcome the challenges of collaborative work

Virtually every responded suggested that not everyone was pulling their weight in group work, although on most case study projects they received the same mark


Interview Themes: Use of Learning Technologies

Quantitative data pointed to a variety of technologies that today’s students use in their lives and qualitative data supported this conclusion

Most students thought that face-to-face was a more efficient means of communication, but a modern university is seen as a site of various methods of contact and their choice depended on learner preferences and familiarity with e-learning tools in the group and their individual circumstances

Typically brief conversations and texts were used to organise and coordinate face-to-face meetings

Facebook and Messenger programmes were used for remote interaction when face-to-face was not an option

Use of other tools, such as Wikis or video and audio tools were delegated to one or two members of the group

Brookes Virtual was seen as the University technology and it was used for collaboration, but technical difficulties and varying levels of proficiency caused difficulties in its integration into student work


Conclusions

This and other data collected for Pathfinder and presented here can be useful baseline information on ways in which learning technologies are used in Oxford Brookes and traditional universities in general

It helped to point to different patterns of online use and establish dimensions on which today’s Net generation differs from older students

Qualitative data suggested a complex, context-dependant nature of learning technology use and the choice in the use of technology which is determined as much by student personalities and the ways in which study fits into their lives as pure availability or proficiency with a certain tool

More analysis will be conducted shortly on differences across groups of students on items in section D and cluster analysis on items in section C


Comments and Questions

Many thanks to the Pathfinder project leaders, the case study leaders, students involved in collecting the data and participating in interviews!

Comments and questions are more than welcome!

Documents

Pathfinder E-Learning Evaluation: Qualitative and Quantitative Evidence Ruslan Ramanau (Research Fellow in E-Learning, OCSLD)