Towards Learning Object Recommendations based on Teachers’ ICT Competence Profiles

Stylianos Sergis1,2, Panagiotis Zervas1,2, Demetrios G Sampson1,2 1Department of Digital Systems

University of Piraeus Piraeus, Greece

2Information Technologies Institute Centre for Research and Technology Hellas

Thessaloniki, Greece e-mail: steliossergis@iti.gr, pzervas@iti.gr, sampson@iti.gr

Abstract— Recommender Systems (RS) have been investigated in the Technology enhanced Learning (TeL) field for facilitating, among others, Learning Objects (LOs) selection and retrieval. However, most of the existing approaches focus on the learners' perspective and do not take into consideration teachers' profile. Moreover, the systems that do target teachers, do not explicitly exploit their ICT competence profiles. This can lead to recommending LOs that are beyond the teachers' current ability to use in their teaching practice. In this paper, we aim to tackle this problem and propose, as a first step, a set of mapping rules for aligning teachers' ICT competences and LO metadata elements. Moreover, a preliminary simulated evaluation is described, the results of which indicate that the mapping schema can provide robust identification of appropriate LOs based on both the users' ICT competences and the overall ratings of the educational resources.

Keywords – Recommender Systems; Teachers’ ICT Competence Profile; Learning Object Recommendations

I. INTRODUCTION

Recommender Systems (RS) are software tools providing suggestions to users for potentially useful items [1]. They are widely used in many areas, including Technology Enhanced Learning (TeL) [2]. Within this context, RS are mainly divided in three categories in terms of the method applied, i.e. content-based recommenders (CB), collaborative filtering (CF) recommenders and hybrid recommenders (H) [3]. Other methods, such as rule-based (RB) recommendations [4] are less popular.

One of the key applications of RS in TeL is the selection of Learning Objects (LO) [5]. More specifically, from the standpoint of teachers, the ever-increasing amount of educational digital resources being made available each day, requires efficient ways in order to filter the most useful [6]. Moreover, this usefulness can differ between users in terms of their unique profiles [5]. Therefore, deployed RS should strive to go beyond using merely rating similarities for LO recommendation, and build on user profile data to accommodate their unique preferences [7].

Under this light, this paper proposes a first step towards an approach to LO recommendation for teachers based on their specific ICT competence profiles. To this end, we present and preliminary evaluate a set of mapping rules that connect the teachers' ICT competences to the LO metadata

elements with the aim of identifying and exclude those that are beyond the teachers' current ability to use.

The remainder of this paper is structured as follows. First, a literature review is performed in the field of RS for TeL, in order to identify systems that focus specifically on teachers and, moreover, utilize their profile data as a means of providing recommendations. Section III defines the specific aim of this paper and describes the outline of our proposed solution deriving from the previous analysis. In sections IV and V a preliminary evaluation experiment is described and its results are presented. Finally, potential future work is discussed.

II. RECOMMENDER SYSTEMS IN TEL: A TEACHERS’ PERSPECTIVE

In order to identify the existing approaches to learning object recommendations for teachers, a literature review was performed. The process was aided from the works of [2] and [3], who have provided a very thorough overview of the RS for TeL landscape. The resulting list included 59 papers. Most of these systems concentrated solely on the standpoint of learners and addressed the specific needs of this user group. While this is reasonable, being that learners are the cornerstone of all learning interventions, it neglects the fact that, at least in formal contexts, the teachers also need assistance, based on the profile, for selecting LOs for their course design.

Under this light, a further content analysis of the initial pool of results indicated that 22 approaches consider teachers in their recommendation processes. This list is presented in Table 1. Related information is also presented, which includes recommendation method and consideration (or not) of user profiling data, in terms of user knowledge, interests etc. The latter is an important issue, as described in [8], where it is stated that recommender systems should "include, among others, the improved modeling of users and items, and incorporation of the contextual information". Therefore, it is important to identify the level of consideration of such data in the existing approaches of LO recommendations for teachers.

As the table depicts, most of these approaches focus on user-to-user or item-to-item rating similarity measures for providing recommendations, while few utilize user profiling

for recommendations, in terms of user knowledge, interests etc.

More specifically, [9] propose a recommendation method which includes the consideration of users' profile in terms of Information Technology experience. The level of granularity of the metric, however, is not detailed.

TABLE I. TEL RECOMMENDER SYSTEMS FOCUSING ON TEACHERS

RS Method Profile RS Method Profile

1 [9] H 12 [20] CB

2 [10] RB 13 [21] CF

3 [11] H 14 [22] H

4 [12] CF 15 [23] RB

5 [13] CF 16 [24] H

6 [14] CF 17 [25] H

7 [15] CB 18 [26] CF

8 [16] H 19 [27] CF

9 [17] CF 20 [28] CF

10 [18] RB 21 [29] CF

11 [19] RB 22 [30] H

Bozo et al. [11] incorporate the teacher profile in their proposal, but build it on mostly demographic data. Schoefegger et al. [17] apply heuristics to determine user work-related knowledge levels. This approach is interesting, but is based on knowledge topics that are profession-specific and, therefore, would require specific ontologies to capture teachers' actions. Similar to that, [28] propose a method for recommending colleagues and resources within the context of work, based (among others) on users' experiences of resource use, but do not provide specific details on the manner that this is captured. Schirru et al. [25] attempt to create recommendations based on users' interests. This is, also, an interesting approach, but does not seem to take into consideration the users' background knowledge. Sielis et al. [23] include user profiling in their approach, but do not provide details on the exact attributes included. Similarly, [30] propose a recommender system for pedagogical patterns, but do not offer information on the attributes of the teachers' profiles. Overall, none of the presented approaches explicitly describes the issue of capturing and exploiting teachers' ICT competences based on a solid modeling framework.

Under this light, it becomes clear that, currently, teachers’ ICT competences are not explicitly utilized for the delivery of LO recommendations. Bearing in mind that two significant factors affecting ICT uptake from teachers are their ICT competences [31] and their reported need for support in using them [32], this constitutes a research challenge for further investigation. More specifically, it would be useful to adopt an approach to LO recommendations that focus not only on returning meaningful educational resources based on query or similar ratings criteria, but also to filter such results and promote the most appropriate ones based on the teachers' ICT

competence profile and their overall ratings. By doing that, the returned resources would be both meaningful to, and appropriate for, the teacher to incorporate in their courses, without the need for further support.

Therefore, the aim of this paper is to describe and evaluate (in a preliminary manner) a set of mapping rules linking the ICT competences of the teachers to the LO metadata elements. This will enable the pre-filtering of LOs that are not suited for the current teacher, both in terms of ICT competence mismatch and in terms of low overall ratings, and provide information to the recommender system to either hide or outrank these LOs.

III. TEACHERS’ ICT COMPETENCE PROFILES AND LEARNING OBJECT

METADATA

A. Teachers’ ICT competence profile

In our work, the teachers’ ICT competence profile is built on the UNESCO ICT Competency Framework for Teachers (CFT)1 which comprises 6 categories of ICT competences spanning 3 proficiency levels. Each competence category is further divided in sub-competences that are differentiated according to the proficiency levels and address diverse areas of teachers' school work. The reasons for selecting the CFT over alternative frameworks have been discussed in [33].

B. Learning Object Metadata Model

The LOM model adopted was the IEEE LOM AP created in the context of the European project “Open Discovery Space” (ODS)2. The reason for adopting the ODS LOM AP was that this European initiative focuses on teachers and attempts (among other goals) to utilize their ICT competences for different types of recommendations. Therefore, useful data could be created to assist us in the evaluation of our proposed recommender system. The full LOM AP of the ODS project is described in an ODS project deliverable3.

C. Mapping Rules

Towards implementing the proposed solution to LO recommendations, there was a need to create mapping rules that could connect certain ICT competences to specific LOM elements’ values. A detailed content analysis of all CFT ICT competences was performed and compared against the ODS LOM AP. The resulting mapping schema was classified in three aggregation levels (AL), namely the educational resources, the lesson plans and the educational scenarios. The rationale for this classification is that each level should utilize different mapping rules. More specifically,

1 http://unesdoc.unesco.org/images/0021/002134/213475E.pdf 2 http://www.opendiscoveryspace.eu/ 3 D4.2: Open Discovery Space LOM Application Profile and Curriculum Based Vocabularies

educational resources cannot be linked to metadata elements related to, for example, “educational objectives” in their own regard. Similarly, “technical format” information is appropriate only for first aggregation level LOs. The mapping schema is presented in Tables 2 and 3.

TABLE II. AGGREGATION LEVEL 1 MAPPING RULES

Competence Code

[4.1] Technical Format [5.2] Educational Learning Resource

Type

TL3c application/ppt Lecture, Presentation

TL4b

application/msword, application/pdf, application/ogg, application/rtf, application/uue, text/plain, text/richtext

Glossaries, Textbook, Guide

TL4c application/ppt, video/*, audio/*, image/*

Presentation , Lecture

TL4d model/vrml , application/postscript

Application, Tool

TL4e text/html Website

TL4h

application/x-pn-realmedia, application/binary, application/ java, application/ x- shockwave-flash

Application, Drill and Practice, Demonstration, Tool, Educational Game

TL4j application/msexcel Application, Other,

Tool

TL4k text/html Application, Blog,

Broadcast, Social Media, Website, Wiki

KD4a

text/html, application/java, application/binary, application/base64, application/x-shockwave-flash, application/msexcel, application/vrml, text/xml

Application, Demonstration, Drill and Practice, Educational Game, Role Play, Reference, Simulation, Tool

KD4d - Application,

Assessment, Tool

KC4a

text/html, application/java, application/binary, application/base64, application/x-shockwave-flash, application/msexcel, application/vrml, text/xml

Application, Blog, Social Media, Tool, Wiki, Other

KC5a

application/java, application/binary, application/base64

Application, Educational Game, Experiment, Simulation, Role Play, Tool, Website

* The vocabularies for these rules were based on LOM and were adopted by the ODS project

As the table depicts, the mapping algorithms were built based on the LOM elements that meet each aggregation level's focus. For educational resources (AL 1), technical format is the dominant element for rule creation, since the level focuses on standalone tools that require a certain level of competence to handle.

Table 3 presents the mapping rules for ALs 2 and 3.

TABLE III. AGGREGATION LEVEL 2 AND 3 MAPPING RULES

Competence Code

[9.2] Taxonomy Path for [9.1] Teaching Approach

TL3b Behaviourist: Programmed instruction, Drill and practice Cognitivist: Direct Instruction Constructivist:Cognitive Apprenticeship

KD3c Cognitivist: Problem-based

KD3d

Cognitivist: Collaborative Learning, Problem-based, Inquiry Learning, Reciprocal teaching Constructivist: Socratic instruction, Communities of practice

KD3e Cognitivist: Collaborative Learning, Problem-based, Reciprocal teaching Constructivist: Socratic instruction KD3f

KC2c Cognitivist: Collaborative Learning, Inquiry based, Problem based Constructivist: Communities of practice

KC2d Cognitivist: Collaborative Learning Constructivist: Communities of practice, Cognitive apprenticeship, Socratic instruction

KC2e Constructivist: Cognitive apprenticeship, Action Research

KC3a Constructivist: Action research

KC3b

Cognitivist: Collaborative Learning, Inquiry Learning, Problem-based Constructivist: Communities of practice, Design-based learning

KC3c

Cognitivist: Collaborative Learning, Inquiry Learning, Problem-based Constructivist: Cognitive apprenticeship, Communities of practice, Design-based learning

KC3d

Constructivist: Cognitive apprenticeship, Experiential learning, Design-based learning

KC3e

Constructivist: Cognitive apprenticeship, Action Research

KC5a Cognitivist: Collaborative Learning Constructivist: Communities of practice, Socratic Instruction, Experiential Learning

* The vocabularies utilized in these mapping rules were created by the ODS project

For ALs 2 and 3, the teaching approach is the driving element. Higher level ICT competences are exploited at these levels in order to identify lesson plans and educational scenarios built on specific teaching approaches that the teacher is not familiar with or does not feel competent in delivering.

The rules in all ALs were created with a dual purpose. First, to allow its incorporating recommender system to identify inappropriate LOs and, therefore, to narrow down the list of recommendations at an early stage. Secondly, to use the mapping rules along with the teachers' ratings to identify LOs that, despite appearing "appropriate" in ICT terms, would not be potentially useful, as decided by the teachers themselves.

Naturally, there is a need to evaluate the mapping rules' classification accuracy. The first step towards addressing this goal is described in the next section.

IV. PRELIMINARY EVALUATION

A preliminary evaluation of the proposed mapping schema involved the design and execution of a simulated experiment. Its focus was to measure the classification accuracy of the mapping algorithms in terms of filtering

learning objects, which would not meet the teachers’s ICT competences and would have a very low overall rating, based on the precision, recall and F1 metrics [34]. For the purposes of the experiment, a partly synthetic dataset was created, fully aligned to the ODS LOM AP. It included the following data types: Teachers' ICT Competence profiles, preferred languages

and discipline domains. A total of 115 teachers’ data were harvested from the ODS database. The discipline domains were randomly assigned to the teachers. Each teacher profile was then represented by a vector of profiling criteria, namely the elements of their ICT profile and their linguistic and discipline domain data.

LO Metadata. Learning Objects that spanned all three aggregation levels were harvested and multiplied from the ODS database. Each LO was represented by a vector V={m1, m2, ..., mn} of metadata elements, based on its aggregation level. Full metadata records were used. The evaluation process included 1000 educational resources, 500 lesson plans and 500 educational scenarios.

Rating data. This type of data refers to the teachers' previous rating activities. The ratings were generated based on the existing data in the ODS database. More specifically, user- and item- similarity measures were utilized in order to extend the limited, available data.

View data. In the same vein, LO view data refer to the fact that a teacher had browsed a particular LO. The rationale for this data type's use is that teachers that would not be competent in using a learning object, would, probably, not rate it low for this reason alone. It is more likely that this particular teacher, after having searched for a specific type of LO and retrieved a set of results, would just ignore those that they deemed difficult, without providing rating feedback. Therefore, assuming that the teacher browses the system to identify appropriate LOs, “view” data, in combination with ratings, could assist (at least, in our simulated experiment) in the identification of LOs that were not deemed useful by the teachers.

The actual evaluation process was designed as follows. For each LO aggregation level, the corresponding mapping rules were applied in order to identify those LOs that were not appropriate in terms of the user’s ICT profile. Moreover, the LOs that were deemed appropriate were re-filtered in order to identify those that had received very low overall ratings. This was necessary in order to promote only LOs that would be deemed useful by the teachers themselves.

In order to verify the system’s decisions on the LOs’ appropriateness, as aforementioned, the rating and viewing data were utilized. More specifically, if a LO was flagged as ‘appropriate’ by the system, but the rating/viewing histories determined that the user had not found it useful then this would be a false classification decision. In the same vein, ‘inappropriate’ LOs with positive past feedback would, also,

have been classified falsely. Of course, not all teachers had rated/viewed all items. To accommodate such situations, user- and item- similarity matrices were utilized, created using the Jaccard coefficient, which has been proven to be efficient in applications within the TeL field [34].

The results of this preliminary experiment are presented in the following section.

V. PRELIMINARY RESULTS

A. Results Presentation

As aforementioned, the evaluation tests involved three separate executions for each AL. Moreover, since the part of evaluation of the mapping's classification accuracy built on collaborative filtering techniques, each execution run was repeated three times with an increasing number of teacher users (N). This was performed in order to evaluate the algorithm's robustness in terms of variant data availability. Table 5 presents the results of the evaluation tests.

TABLE IV. PRELIMINARY CLASSIFICATION ACCURACY RESULTS

Aggregation Level Precision Recall F1 N

Educational Resources 0.64 0.81 0.71

40 Lesson Plans 0.64 0.91 0.75

Educational Scenarios 0.64 0.92 0.76

Precision Recall F1

Educational Resources 0.65 0.81 0.72

80 Lesson Plans 0.7 0.9 0.79

Educational Scenarios 0.71 0.9 0.79

Precision Recall F1

Educational Resources 0.66 0.84 0.74

115 Lesson Plans 0.74 0.91 0.82

Educational Scenarios 0.78 0.93 0.85

B. Results Discussion

As Table 4 depicts, the mapping algorithms achieved a high level of F1 measure for all aggregation levels. More specifically, as the teachers' number increases, the algorithm can provide more precise decisions, since the users' histories are used in the classification process. The facts that (a) even for small N sizes, the F1 measure is relatively high, and (b) that the partly synthetic dataset was built based on frequency distributions of real data, indicate that the proposed algorithms can provide a solid measure for identifying LOs that are either inappropriate or potentially useless for teachers based on their ICT competence profiles. Of course, as aforementioned, this simulated evaluation is only a preliminary endeavor that will be complemented by a user-based experiment.

VI. CONCLUSIONS AND FUTURE WORK

In this paper, a literature review was performed in the field of RS in TeL, which highlighted a significant problem, namely, the lack of consideration of teachers’ ICT competences when recommending LOs. This approach rises as an important research challenge, considering that

teachers' ICT competences affect their level of ICT uptake. The first step towards addressing this goal was the

development of a set of mapping rules linking ICT competences and LOM elements. This tool can highlight LOs that the teacher would not be able to use in a comfortable manner and, either rank them low or hide them completely.

Preliminary work in the crucial aspect of evaluation involved a simulated experiment based on a partly synthetic dataset. The results showed high levels of classification precision and recall even for small N sizes.

Future work in this agenda includes (a) a real user evaluation that will enhance the robustness of the mapping rules and (b) the design and implementation of a hybrid recommender system that, based on the proposed mappings, will provide LO recommendations to teachers.

ACKNOWLEDGMENT

The work presented in this paper has been partly supported by the Open Discovery Space Project that is funded by the European Commission's CIP-ICT Policy Support Programme (Project Number: 297229).

REFERENCES

[1] F. Ricci, L. Rokach, and B. Shapira, "Introduction to recommender systems handbook," in Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P.B. Kantor, Eds. US: Springer, 2011, pp. 1-35.

[2] K. Verbert, N. Manouselis, X. Ochoa, M. Wolpers, H. Drachsler, I. Bosnic, and E. Duval, "Context-Aware Recommender Systems for Learning: A Survey and Future Challenges," IEEE Transactions on Learning Technologies, vol. 5, no. 4, pp. 318-335, 2012.

[3] N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, "Survey and Analysis of TEL Recommender Systems," in Recommender Systems for Learning, N. Manouselis, H. Draschler, K. Verbert, and E. Duval, Eds. New York: Springer, 2013, pp. 37-61.

[4] V.A. Zaldivar, D. Burgos, and A. Pardo, "Meta-Rule Based Recommender Systems for Educational Applications," In Educational Recommender Systems and Technologies: Practices and Challenges, O. Santos, and J. Boticario, Eds. PA: Hershey, 2012, pp. 211-231.

[5] N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, "TEL as a Recommendation Context," In Recommender Systems for Learning, N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, Eds. New York: Springer, 2013, pp. 21-36.

[6] N. Manouselis, H. Drachsler, R. Vuorikari, H. Hummel, and R. Koper, "Recommender systems in technology enhanced learning," in Recommender systems handbook, F. Ricci, L. Rokach, B. Shapira, and P.B. Kantor, Eds. US: Springer, 2011, pp. 387-415.

[7] K. Verbert, N. Manouselis, H. Drachsler, and E. Duval, "Dataset-Driven Research to Support Learning and Knowledge Analytics," Educational Technology & Society, vol. 15, no. 3, pp. 133–148, 2012.

[8] G. Adomavicius, and A. Tuzhilin, "Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions," Knowledge and Data Engineering, IEEE Transactions on, vol. 17, no. 6, pp. 734-749, 2005.

[9] A. Zapata, V.H., Menéndez, M.E. Prieto, and C. Romero, "A framework for recommendation in learning object repositories: An example of application in civil engineering," Advances in Engineering Software, vol. 56, pp. 1-14, 2013.

[10] C. Limongelli, A. Miola, F. Sciarrone, and M. Temperini, "Supporting Teachers to Retrieve and Select Learning Objects for Personalized Courses in the Moodle_LS Environment," In 12th International Conference Advanced Learning Technologies (ICALT), 2012, pp. 518-520.

[11] J. Bozo, R. Alarcón, and S. Iribarra, "Recommending Learning Objects According to a Teachers’ Contex Model". In Sustaining TEL: From Innovation to Learning and Practice, M. Wolpers, P.A. Kirschner, M. Scheffel, S. Lindstaedt, and V. Dimitrova, Eds. Berlin Heidelberg: Springer, 2010, pp. 470-475.

[12] N. Manouselis, R. Vuorikari, and F. Van Assche, "Simulated analysis of MAUT collaborative filtering for learning object recommendation, " in Proceedings of the 1st Workshop on Social Information Retrieval for Technology Enhanced Learning, 2007, pp. 27-35.

[13] A. Walker, M.M. Recker, K. Lawless, and D. Wiley, "Collaborative information filtering: A review and an educational application," International Journal of Artificial Intelligence in Education, vol. 14, no. 1, pp. 3-28, 2004.

[14] S. Rafaeli, M. Barak, Y. Dan-Gur, and E. Toch, "QSIA–a Web-based environment for learning, assessing and knowledge sharing in communities," Computers & Education, vol. 43, no. 3, pp. 273-289, 2004.

[15] B.E. Shelton, J. Duffin, Y. Wang, and J. Ball, "Linking open course wares and open education resources: creating an effective search and recommendation system," Procedia Computer Science, vol. 1, no. 2, pp. 2865-2870, 2010.

[16] P. Brusilovsky, L.N. Cassel, L.M. Delcambre, E.A. Fox, R. Furuta, D. D. Garcia, ... and M. Yudelson, "Social navigation for educational digital libraries," Procedia Computer Science, vol. 1, no.2, pp. 2889-2897, 2010.

[17] K. Schoefegger, P. Seitlinger, and T. Ley, "Towards a user model for personalized recommendations in work-integrated learning: A report on an experimental study with a collaborative tagging system," Procedia Computer Science, vol. 1, no. 2, pp. 2829-2838, 2010.

[18] Y. Wang, and K. Sumiya, "Semantic ranking of lecture slides based on conceptual relationship and presentational structure," Procedia Computer Science, vol. 1, no. 2, pp. 2801-2810, 2010.

[19] V.A. Zaldivar, and D. Burgos, "Meta-Mender: A meta-rule based recommendation system for educational applications," Procedia Computer Science, vol. 1, no. 2, pp. 2877-2882, 2010.

[20] H. Avancini, and U. Straccia, "User recommendation for collaborative and personalised digital archives," International Journal of Web Based Communities, vol. 1, no.2, pp. 163-175, 2005.

[21] S. El Helou, C. Salzmann, and D. Gillet, "The 3A Personalized, Contextual and Relation-based Recommender System," Journal of Universal Computer Science, vol. 16, no.16, pp. 2179-219, 2010.

[22] K.H. Tsai, T.K. Chiu, M.C. Lee, and T.I. Wang, "A learning objects recommendation model based on the preference and ontological approaches," in Sixth International Conference on Advanced Learning Technologies, 2006, pp. 36-40.

[23] G. Sielis, C. Mettouris, A. Tzanavari, and G. Papadopoulos, "Context-Aware Recommendations using Topic Maps Technology for the Enhancement of the Creativity Process," in Educational Recommender Systems and Technologies: Practices and Challenges, O. Santos, and J. Boticatio, Eds. Hershey, PA: IGI-Global, 2012, pp. 43-66.

[24] M. García-Valdez, and B. Parra, "A Hybrid Recommender System Architecture for Learning Objects," In Evolutionary Design of Intelligent Systems in Modeling, Simulation and Control, O. Castillo, and W. Pedrycz, Eds. Berlin Heidelberg: Springer, 2009, pp. 205-211).

[25] R. Schirru, S. Baumann, M. Memmel, and A. Dengel, "Extraction of Contextualized User Interest Profiles in Social Sharing Platforms," Journal of Universal Computer Science, vol. 16, no.16, pp. 2196-2213, 2010.

[26] H. Drachsler, D. Pecceu, T. Arts, E. Hutten, L. Rutledge, P. Van Rosmalen, ... and R. Koper, "ReMashed–recommendations for mash-up personal learning environments," In Learning in the synergy of multiple disciplines, U. Cress, V. Dimitrova, and M. Specht, Eds., Berlin Heidelberg: Springer, 2009, pp. 788-793.

[27] S. Fazeli, H. Drachsler, F. Brouns, and P. Sloep, "A trust-based social recommender for teachers," in Proceedings of the 2nd Workshop on

Recommender Systems for Technology Enhanced Learning, 2012, pp. 49-60.

[28] M. Aehnelt, M. Ebert, G. Beham, S. Lindstaedt, and A. Paschen, "A socio-technical approach towards supporting intra-organizational collaboration," In Times of Convergence. Technologies Across Learning Contexts, P. Dillenbourg, and M. Specht, Eds. Berlin Heidelberg: Springer, 2008, pp. 33-38.

[29] J. Dron, R. Mitchell, P. Siviter, and C. Boyne, "CoFIND—an experiment in N-dimensional collaborative filtering," Journal of Network and Computer Applications, vol. 23, no. 2, pp. 131-142, 2000.

[30] C. Cobos, O. Rodriguez, J. Rivera, J. Betancourt, M. Mendoza, E. Leon, E. Herrera-Viedma, "A hybrid system of pedagogical pattern recommendations based on singular value decomposition and variable data attributes," Information Processing & Management, vol. 49, no.3, pp. 607-625, 2013.

[31] G. Sang, M. Valcke, J. van Braak, and J. Tondeur, "Student teachers’ thinking processes and ICT integration: Predictors of prospective teaching behaviors with educational technology," Computers & Education, vol. 54, no. 1, pp. 103-112, 2010.

[32] S. Prestridge, "The beliefs behind the teacher that influences their ICT practices, " Computers & Education, vol. 58, no. 1, pp. 449-458, 2012.

[33] P. Zervas, K. Chatzistavrianos, and D. Sampson, "Modelling ICT Teachers’ Competences Profile", in ICT in Education in Global Context: Emerging Trends 2013-2014, R. Huang, Kinshuk, and J.K. Price, Eds. Springer, 2014, in press.

[34] K. Verbert, H. Drachsler, N. Manouselis, M. Wolpers, R. Vuorikari, and E. Duval, "Dataset-driven research for improving recommender systems for learning," In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, 2011, pp. 44-53.