1

Practical guidelines for designing and evaluating educationally oriented recommendations

Olga C. Santos, Jesus G. Boticario

Abstract

There is a need for designing educationally oriented recommendations that deal with educational

goals as well as learners’ preferences and context in a personalised way. They have to be both based

on educators’ experience and perceived as adequate by learners. This paper compiles practical

guidelines to produce personalised recommendations that are meant to foster active learning in

online courses. These guidelines integrate three different methodologies: i) user centred design as

defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational systems, and iii)

the layered evaluation of adaptation features. To illustrate guidelines actual utility, generality and

flexibility, the paper describes their applicability to design educational recommendations in two

different e-learning settings, which in total involved 125 educators and 595 learners. These

applications show benefits for learners and educators. Following this approach, we are targeting to

cope with one of the main challenges in current massive open online courses, which are expected to

provide personalised education to an increasing number of students without the continuous

involvement of educators in supporting learners during their course interactions.

Keywords

intelligent tutoring systems; interactive learning environments; lifelong learning; teaching/learning

strategies.

1. Introduction

An increasing and urgent demand for personalised content delivery and intelligent feedback on a

massive scale is coming up in online learning courses which are to cope with an increasing number

of students. This is particularly critical in nowadays Massive Open Online Courses (MOOCs)

(Sonwalkar, 2013). MOOCs are an emerging type of online courses aimed at large-scale

participation and open access via the web (Masters, 2011). Supporting this large number of learners

requires immediate responses to learners’ needs and thus significant tutoring resources, which can

make their deployment not feasible. In this context, readily available recommendations can provide

timely responses to support students’ needs and as a consequence reducing the educators’ workload

involved in assisting them throughout the course (Shaw et al., 2013). In particular, these

recommendations can offer a personalised guidance, which highlights potential useful contents

which are the result of frequent actions in online courses. These refer to course material, teacher

generated contents and students generated contents. Additionally, in this context any action,

whether passive –such as reading a given content– or active –such as contributing new materials,

providing comments or solving an exercise– which can be done in relation to any object (file, forum

message, etc.) in the learning management system that supports the given online course can be

considered as a candidate to becoming a useful recommendation for a learner in a specific course

situation. In actuality, learners may find themselves involved in an interactive environment which

offers a wide range of actions to take, but many times they do not have a clear view of which of

Accepted manuscript with copyright transferred to Elsevier.

© 2015. This manuscript version is made available under the CC-BY-NC-ND 4.0 license

http://creativecommons.org/licenses/by-nc-nd/4.0/.

Some changes may have been introduced in the final published version.

Computers & Education, Volume 81, February 2015, Pages 354-374.

(http://dx.doi.org/10.1016/j.compedu.2014.10.008)

The final publication is available at Science Direct:

http://www.sciencedirect.com/science/article/pii/S0360131514002280

2

them are more appropriate for their own learning. Here, it can be taken advantage of the well-

grounded research on recommender systems. In particular, recommender systems can be used to

guide users in a personalised way to useful objects in a large space of possible options (Burke,

2002) reducing the existing information overload. This is framed in the so-called personalisation

task of adaptive navigation support in educational scenarios (Brusilovsky and Peylo, 2003).

When recommendations are designed in educational scenarios, they should involve learners in

the learning process, and thus, suggest carrying out actions that foster their learning performance

(i.e., ensuring the accomplishment of given educational goals). It is noticeable that the most

common approach followed by educational recommender systems mainly focuses on pointing

learners to “read relevant resources” –as a mere information retrieval issue (Drachsler et al., 2015) –

and not on taking advantage of available recommendation opportunities that require actual

involvement of learners through other “potential actions” that can be done in the course space, as

suggested in early approaches (Zaïane, 2002).

The design of recommendations in general has not received much attention in related literature

so far. In fact, it has been neglected in the field of recommender systems. By and large, the focus

has been put on evaluating the performance of the recommendation algorithms (i.e. the analysis of

an algorithm's runtime in practice) in terms of information retrieval measures, such as accuracy,

recall, precision and so on (Konstan and Riedl, 2012). In this sense, there have been some efforts to

identify descriptions of domain-independent tasks in recommender systems, with the goal to help

distinguish among different evaluation measures (Herlocker et al., 2004).

The closest effort that we are aware of related to recommendations design in educational

scenarios is the repertory grid from the personal construct theory proposed by Kelly (1955), which

has been used by (Hsu et al., 2010) to develop reading material recommendations from domain

knowledge elicited from multiple experts. Here, recommendations are provided to the system by the

educators. Additionally, Brito et al. (2012) have proposed an architecture-centred solution for

designing educational recommender systems in a systematic manner. However, we have not find in

the literature approaches that address in educational scenarios the design and evaluation of

educationally oriented recommendations.

Within the educational arena, the spectrum of recommendation opportunities cannot be

considered just as an information retrieval issue. Here eliciting and using educators’ background on

attending learning needs may be crucial when catering for the learner’s needs in a given situation.

However, educators are not provided with guidelines that help them to designing and evaluating

educationally oriented recommendations that result from their experience in attending learners and

which may support adaptive navigation paths within online courses. Actually, as it will be discussed

later on, there is a gap found between literature demands (recommendations should focus on the

learning needs and foster active learning) and literature outcomes (educational recommender

systems actually deliver mainly learning contents).

To deal with this issue, there are three different methodologies that can be considered: i) user

centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational

systems, and iii) the layered evaluation of adaptation features.

Bearing all this in mind, in order to support the process of developing educationally oriented

recommendations, this paper presents a set of design and evaluation practical guidelines for three

specific iterations of the recommendation design and evaluation cycle (i.e., proof of concept,

elicitation of recommendations and delivery of recommendations). Resulting recommendations are

to be delivered to learners through a semantic educational recommender system -SERS (Santos and

Boticario, 2011a), which is in line with the service-oriented approach of the third generation of

learning management systems (Dagger et al., 2007) where external educational web based services

can interoperate with the learning management systems (Muñoz-Merino et al., 2009). SERS rely on

i) a recommendation model, ii) an open standard-based service-oriented architecture, and iii) a

usable and accessible graphical user interface to deliver the recommendations. The proposed

guidelines implement the recommendation cycle and focus on identifying recommendation

opportunities that come out from studying the teaching and learning issues that characterise the

3

educational domain. Thus, they require the involvement of educators and learners in that

identification process. To evaluate the benefits of applying these guidelines, we report their usage in

two very different educational contexts, which consider different approaches and have different

requirements. These contexts involve two different learning scenarios implemented in two different

learning management systems. In particular, this paper describes how those guidelines have been

applied in these two scenarios, involving a total of 125 educators and 595 learners.

Accordingly, this paper is structured as follows. First, we comment on related works that can

guide the design and evaluation of educational recommendations. Here the focus is on the users’

involvement in the design process and formative evaluation of adaptation features. Afterwards, we

present the set of practical guidelines, which are meant to support educators in designing and

evaluating user centred recommendations for educational scenarios. Next, we report on the

application of these guidelines in two very different contexts: 1) the DtP course in dotLRN learning

management system, and 2) the EBIFE course in Willow free-text adaptive computer assisted

assessment system. After discussing the approach and results provided, we conclude summarising

the main issues involved, and introduce current work, which extends the features considered in

these contexts to support educators in eliciting recommendations that account for affective issues.

2. Related works

The utility of recommender systems for the educational domain has been largely acknowledged

over the last fifteen years as a way to provide personalised support to learners while carrying out

learning tasks in web-based learning environments (Drachsler et al., 2015). Research has shown that

recommendations to be provided in the educational domain are different from those in other

domains (e.g., e-commerce, e-entertainment). In fact, there are a number of distinctive issues when

educational recommendations are compared with recommendations for consumers, mainly in terms

of goals, user features and recommendation conditions (Draschler et al., 2009a). Therefore,

recommender systems should not be transferred from commercial to educational contexts on a one-

to-one basis, but rather need adaptations in order to facilitate learning (Buder and Schwind, 2012).

In that respect, there are long-running challenges derived from the peculiarities of the educational

domain (Konstan and Riedl, 2012).

When recommendations are designed for educational scenarios a distinctive factor is, for

instance, that they should not be guided just by the learners’ preferences (Tang and McCalla, 2009).

Considering only users’ preferences as the bases for providing recommendations is typically done

in non-educational recommenders (Kluver et al., 2012). However, a personalisation support in

educational settings has to deal with diverse learning styles and other psycho-educational aspects of

the learning process (Bates and Leary, 2001; Blochl et al., 2003), as well as the cognitive state of

the learner (Drachsler et al., 2009a). In this context, the benefit of providing recommendations to

learners is to be related to improvements on their performance in the course, through a more

effective, efficient and satisfactory learning (Drachsler et al., 2009b). In other words, all these

conditions affect the design (knowledge modelling), development (techniques, algorithms and

architectures) and evaluation (in real world e-learning scenarios) of recommender systems in

education (Santos and Boticario, 2012).

In order to identify key issues to be considered when designing educational recommendations,

an extensive review of related literature covering 50 recommender systems has been carried out

elsewhere (Santos and Boticario, 2013). This review shows that despite the first approaches

remarked on recommending several types of actions (e.g., accessing a course notes module, posting

a message on the forum, doing a test, trying a simulation) and resources (Zaïane, 2002), most

research have neglected this possibility and systems have mainly focused on recommending some

specific item of content. In fact, it shows that there are very few examples of educational

recommender systems that foster users’ active actions (e.g., providing a contribution). However,

current educational approaches acknowledge the benefits of learners’ active involvement in her

learning process (Lord et al., 2012) and it is noticeable that fostering interaction can promote

collaboration with like-minded learners (Wang, 2007) and improve learning (Webb et al., 2004).

4

As to the involvement of users, user centred design methodologies can be followed to develop

systems that suit users’ needs (Gulliksen et al., 2003). Thus, to cater for the learner in terms of

suitable educational recommendations according to their needs, it is suggested to incorporate user

centred design methodologies in the recommendations development process covering both their

design and evaluation. However, none of the 50 systems reviewed (Santos and Boticario, 2013)

reported the application of a methodology that involves users in the design process to find out

relevant recommendations opportunities for their educational scenarios. Furthermore, regarding the

evaluation, an extended review with a total of 59 systems showed that there were only 7 works

which evaluated the effect on the learning performance of the recommendations delivered (Santos

et al., 2014a).

It has also been suggested that for context-rich domains (like the educational one), end-users

and stakeholders should be provided with tools for expressing recommendations that are of interest

(Adomavicius et al., 2011). In this way, educators need to be provided with some mechanism that

allows them to experiment designing recommendations to be delivered to their learners. This might

help them to cope with the wide variety of potential recommendation opportunities that exist in the

learning environments (Bieliková et al., 2014) and which have not yet been sufficiently explored.

Thus, interactions between actors (learners, educators, etc.), artefacts and environment make up a

process from where to understand the learning issues involved, evaluate the educational result and

support the design of effective technology (Gassner et al., 2003).

In this sense, involving domain experts in the recommendations generation process can produce

more accurate recommendations (Shen and Shen, 2004; Al-Hamad et al., 2008) as these can

reproduce educators’ decision-making behaviours (Hsu et al., 2010). Educators with wide

experience in on-line teaching have a comprehensive view of the difficulties encountered by

learners. Thus, they can put these difficulties in perspective as regards to the seriousness and

frequency of the issue for the learners (Hanover Research Council, 2009). Moreover, learners can

also be involved in the recommendation process to design and evaluate educational resources (Ruiz-

Iniesta et al., 2012) or to adapt parameters and recommendation algorithms (Farzan and Brusilovsky,

2006). In fact, the learner involvement in the recommendation process can have benefits related to

satisfaction and trust (Buder and Schwind, 2012). Therefore, to cope with aforementioned issues it

is sensible to consider that both learners (i.e., users) and educators (i.e., designers) have to be

involved in the recommendations development process.

From the above findings follows both that users should be involved from the beginning in an

educationally oriented recommendation elicitation process and that to this, they have to be

supported. However, despite knowing that learning is a personalised and evolving process that is to

be focused on the learner and regardless the benefits of applying user centred design to the

development of adaptive learning systems (Gena, 2006), user centred design is usually neglected.

Unfortunately, when developing adaptive learning systems, users are generally consulted (if at all)

towards the end of the development cycle (Harrigan, et al., 2009), forgetting that the design process

should be focused on the learner and not on the system (Mao et al., 2005). In fact, specific user

centred design methodologies are needed when the user's goals involve learning and teaching

(Gamboa Rodriguez et al., 2001).

In this respect, ISO 9241-210 ‘Ergonomics of human-system interaction - Part 210: Human-

centred design for interactive systems’ (ISO, 2010) is the international standard that sets the basis

for many user centred design methodologies. It is generic and can be applied to any interactive

system or product. This standard describes four principles of human-centred design: 1) active

involvement of users (or those who speak for them), 2) appropriate allocation of function (making

sure human skill is used properly), 3) iteration of design solutions (therefore allowing time in

project planning), and 4) multi-disciplinary design (but beware overly large design teams). Here,

user centred design is described as an iterative cycle, having as input the design plan that compiles

the underlying needs and requirements. Although ISO 9241-210 does not specify any methods, a

wide variety of usability methods that can be used to support user centred design are outlined in the

technical report ISO/TR 16982:2002 ‘Ergonomics of human-system interaction—Usability methods

5

supporting human-centred design’ (ISO, 2002). As the offer is wide, several initiatives have

researched into the most appropriate methods for user centred design such as the UsabiliyNet

European project (Bevan, 2003), the IDEO method cards (IDEO, 2003) and the Usability Body of

Knowledge (UXPA, 2012). Anyway, existing initiatives provide hints to select the most appropriate

methods to apply, but their selection has to be done taking into account the particularities of the

design environment, the context of use and the stage of the design process. Moreover, according to

the UsabilityNet project there are some conditions that should be taken into account in this decision:

i) limited time and/or resources to apply the methods, ii) availability of direct access to users, and

iii) limited skills and expertise of the people in charge of applying the methods.

As to how user centred design can be used to guide the production of educationally oriented

recommendations, first it has to be pointed out that user centred design relies on an iterative

development cycle that involves the user throughout the process. It leads to the definition of a set of

user requirements, and then guides the development of systems with built-in capabilities to provide

a good user experience. Actually, some usability methods have already been used to develop

recommender systems in non-educational domains (Zins et al., 2004). In the educational domain,

the e-learning life cycle has been proposed to support learner-centred adaptive educational scenarios.

It consists of four consecutive phases: design, publication, usage and auditing (Van Rosmalen et al.,

2004). In particular, to place the learner as the centre of this e-learning cycle, accessibility and

usability issues are to be taken into account throughout all the cycle phases (Martin et al., 2007).

So as to guide the development process, this user centred design iterative cycle calls for

formative evaluations (which address issues during the development or improvement of a system)

aimed at ensuring that results truly meet the user requirements identified during the design (Gena

and Weibelzahl, 2007). Moreover, recommender systems are interactive systems that offer an

adaptive output (i.e., personalised recommendation). As recommendations are to adapt its response

to the users’ needs, this adaptive support has also to be considered in the formative evaluation

(Mulwa et al., 2011). Literature has identified difficulties in evaluating adaptive systems (Van

Velsen et al., 2008). To overcome these difficulties, the adaptation process can be decomposed into

its constituents -called layers-, and each of these layers evaluated separately where necessary and

feasible (Paramythis et al., 2001). This approach can be used to evaluate the advantages of the

adaptation provided (Karagiannidis and Sampson, 2000) and guide the development process

(Paramythis et al., 2001). In this way, more can be learnt about what causes success or failure in the

adaptive response. The purpose here is to figure out why, and under what conditions, a particular

type of adaptation can be applied to achieve a specific goal.

The most up to date layered evaluation framework is the one proposed by Paramythis et al.

(2010). This work is a revised version of a previous combination carried out on three previous

frameworks (Weibelzahl and Lauer, 2001; Paramythis et al., 2001; Brusilovsky et al., 2004). This

revision defines five layers, corresponding to the main stages of adaptation. The layers are domain

independent and their relevance and application depends on the nature of the system. The layers

identified in the framework are the following: 1) Collection of input data: assembles the user

interaction data along with any other data available related to the interaction context; 2)

Interpretation of the collected data: provides meaning for the system to the raw input data

previously collected; 3) Modelling of the current state of the world: derives new knowledge about

the user, the interaction context, etc. and introduces that knowledge in the dynamic models of the

system; 4) Deciding upon adaptation: given a particular state of the world, as expressed in the

models maintained by the system, identifies the necessity of an adaptation and selects the

appropriate one; and 5) Applying (or instantiating) adaptation: introduces the adaptation in the user-

system interaction, on the basis of the related decisions. Moreover, after this piece-wise evaluation,

the framework also considers the evaluation of the adaptation as a whole, which is meant to get the

big picture. In this case, the application domain has to be taken into account to formulate and select

the appropriate evaluation metrics and methods. Up to now, the layered evaluation method has not

been applied to recommender systems, but it has been suggested as a powerful technique in

identifying areas of recommender systems that require more focused future work (Pu et al., 2012).

6

Nevertheless, there exist several open issues towards the standardization of the layered evaluation

frameworks applied to recommender systems (Manouselis et al., 2014).

Furthermore, besides the evaluation of the adaptation mechanism, in recommender systems it is

also necessary to conduct empirical evaluations that consider the entire process of how the user

experience comes about in the recommendation cycle (Knijnenburg et al., 2012). This means that

sumative evaluations (which are conducted after the system’s development and its purpose is to

provide information on the system’s ability to do what it was designed to do) are to be carried out.

From the review reported in this section follows three key issues, namely, (1) there is a need to

consider educational issues in recommender systems in education, (2) users should be involved in

designing educationally oriented recommendations, and (3) there is a lack of having practical

guidelines that help users in such design and corresponding formative evaluation. In this paper we

provide educators with guidelines to help them in designing and evaluating personalised

recommendations for their learners, which consider their learning needs, preferences and

educational context. To this, we argue that there is educators’ tacit knowledge obtained over years

of experience in supporting learners during their learning within online learning environments that

can be obtained following those guidelines. In particular, the proposed guidelines should integrate

the aforementioned methodologies that have come out in this study of related work, namely: 1) the

user centred design defined by ISO-9241-210, involving both educators and learners in the process,

2) the e-learning life cycle of personalised educational systems, and 3) the layered evaluation

approach to guide the formative evaluation of the adaptation features design. The guidelines should

also support empirical evaluations of the user experience along the recommendation process.

3. Practical guidelines

The practical guidelines that we propose have been identified from previous experience in several

research projects on technology enhanced learning and inclusion, namely aLFanet: IST-2001-33288

(Boticario and Santos, 2007), ADAPTAPlan: TIN2005-08945-C06-01 (Boticario and Santos, 2008),

CISVI: TSI-020301-2008-21 (Santos et al., 2010) and EU4ALL: IST-2006-034778 (Boticario et al.,

2012). They combine three methodological approaches: 1) user centred design following the

standard ISO 9241-210, 2) the four phases of the e-learning life cycle for developing personalised

educational systems, and 3) the layered evaluation approach that is required to formatively evaluate

the design of adaptive features for these systems.

The user centred design methodology that can support educators in identifying educationally

oriented recommendation opportunities in online courses has been defined elsewhere (Santos and

Boticario, 2011b) and is called TORMES. TORMES stands for Tutor-Oriented Recommendations

Modelling for Educational Systems. Its goal is to support educators in identifying recommendation

opportunities in learning environments that have an educational purpose, and which are perceived

as adequate by learners, both in content and time of delivery. It combines user centred design

methods and data mining analysis. Data mining techniques are used to extract information from

learners’ interactions and, from this, discover usage patterns. TORMES drives the recommendations

design in three consecutive steps: 1) elicitation of educationally sound recommendations validated

by users (i.e. educators and learners) with a collaborative review, 2) acquisition and validation of

the learners’ features to select the appropriate recommendations for the current context, and 3)

analysis of the recommendations provided and evaluation of their impact on the user. TORMES

follows the four user centred design activities defined by ISO 9241-210 in an iterative manner: 1)

Understanding and specifying the context of use: identifying the people who will use the system,

what they will use it for, and under what conditions they will use it; 2) Specifying the user

requirements: identifying any requirements or user goals that must be met for the system to be

successful, considering the variety of different viewpoints and individuality; 3) Producing design

solutions to meet user requirements, which can be made in stages to encourage creativity, from an

initial rough concept to a final full-blown design; and 4) Carrying out user-based evaluation of the

design against the requirements.

7

As discussed in Section 2, since recommendations are to enrich the adaptive support in

technology enhanced learning scenarios, they have to be managed along the e-learning life cycle of

adaptive educational systems. In previous research (i.e., the aLFanet project), it was concluded that

in order to provide a personalised learning experience, it is desirable that the actors involved in the

learning process (i.e. learners and educators) are supported during the e-learning life cycle, which

covers the following phases (Van Rosmalen et al. 2004): Design: deals with the preparation in

advance of the learning experience; Publication: manages the administration of the environment

where the learning experience is to be carried out; Use: focuses on the usage of the e-learning

environment services by learners and educators; and Auditing: provides feedback to the course

author on the learners’ experiences. The application of the e-learning life cycle in other projects

such as ADAPTAPlan and EU4ALL, showed that the recommendation process can flow along the

four phases of the e-learning life cycle (Santos, 2009) as follows: 1) the design phase covers the

generation of semantic educationally oriented recommendations described in terms of the

recommendation model; 2) the publication phase involves loading the recommendations generated

in the previous phase so that they can be instantiated through the e-learning services available in the

given e-learning environment; 3) the use phase delivers recommendations whose semantic

description matches the current runtime context, and monitors the interactions of the learners within

the e-learning environment; and 4) the auditing phase provides feedback on the recommendations

design by analysing the results on their usage over the course experience.

Since the stages of the e-learning life cycle (i.e. design, publication, usage and auditing) are to

be considered in the design of the recommendations, they should be integrated within the activities

of the user centred design interaction cycle as defined by ISO 9241-210 and thus, considered

explicitly in the user centred design cycle of the TORMES methodology. To cope with this, we

propose to split the design and formative evaluation activities of the ISO standard into two sub-

activities. In this way, the user centred design activity ‘Producing design solutions to meet user

requirements’ is broken down into two sub-activities, which correspond to the design and

publication phases of the e-learning life cycle. The rationale for this is to explicitly consider the

mapping of the recommendations needs elicited into the recommendation model proposed, and its

publication in the e-learning environment to be ready for the next activity (i.e. the formative

evaluation). These two sub-activities are defined as follows: 1) Modelling: application of the

recommendation model to semantically characterise the recommendations in terms of a semantic

recommendation model, and 2) Publication: instantiation of recommendations described with the

model into the learning environment that is going to be used to deliver the recommendations. This

recommendation model allows bridging the gap between recommendations' description provided by

the educator and the recommender logic, which is in charge of delivering recommendations in the

running course. These recommendations can be defined along the dimensions of “6 Ws and an H”

(Santos et al., 2014b): i) What (i.e., the type) is to be recommended, that is, the action to be done on

the object of the e-learning service (for instance, to post a message in the forum); ii) How and

Where (i.e,. the content) to inform the learner about the recommendation, which in a multimodal

enriched environment, should describe the modality and way in which the recommendation has to

be delivered to the learner; iii) When and to Who (i.e., the runtime information) the recommendation

is produced, which depends on defining the learner features, interaction agent capabilities and

course context that trigger the recommendation. It describes both the restrictions that may limit

recommendation delivery as well as the applicability conditions that trigger the recommendations;

iv) Why (i.e., the justification) a recommendation has been produced, providing the rationale behind

the action suggested; and v) Which (i.e., the recommendation features) additional semantic

information characterise the recommendations themselves (e.g., relevance, category). In turn, the

ISO activity ‘Evaluating designs against requirements’ is also broken down into another two sub-

activities, which correspond to the use and auditing phases of the e-learning life cycle. In this case,

the rationale behind is to explicitly separate the participation of the learners to produce the

interactions from the analysis of these interactions. These two sub-activities are defined as follows:

1) Usage: provide support to the learners when interacting within the course space by delivering

8

personalised recommendations, and 2) Feedback: carry out analysis of these interactions to

evaluate the recommendations and provide feedback to the design.

Another issue to be considered is the formative evaluation of the recommendations. As

discussed in Section 2, layered evaluation approaches are appropriate for this. Thus, to cope with

the evaluation of the design of the system adaptation features, the five typical layers of the layered

evaluation approach can also be mapped into the user centred design cycle of TORMES. In this way,

the layered evaluation approach is integrated with the activities identified in ISO 9241-210 as

follows: first (layer 1), the evaluation of the data collected is produced during the activity

‘Feedback’, as it is there where the analysis of the learners’ interactions with the recommendations

takes place. Second (layer 2), the evaluation of the interpretation of the data collected is done in

the activity ‘Understanding and specify the context of use’ of the following iteration, as the

interpreted data are used in this activity to get more insight into the context that complements the

information gathered from the learners with typical usability methods. Third (layer 3), the

evaluation of the modelling of recommendations with respect to the current state of the world in terms of the recommendation model, which is based on the data collected and interpreted, is done

in the activity Modelling of the following iteration. Fourth (layer 4), the evaluation of the strategy

selected to deliver the recommendations design upon adaptation is done in the activity

‘Publication’, when the recommendations are implemented and arranged in order to be ultimately

delivered to the learners. Fifth (layer 5) and last, the evaluation of the application of the

adaptation decisions (i.e. the delivery of the recommendations) is done in the activity ‘Usage’,

where the recommendations were instantiated and delivered to the learners in the environment.

The combination of the above three methodological approaches (i.e., user centred design as

defined in TORMES, e-learning life cycle and layered evaluation) results in the practical guidelines

compiled in Table 1. In order to drive the users along the development process, they cover three

typical iterations of the recommendation design and evaluation cycle: 1) proof of concept to

evaluate recommendations’ perception by the users, 2) elicitation of educational recommendations

derived from the practical experience of educators, and 3) delivery of the recommendations in a

large scale study. These interactions address different goals, require different input and produce

different output. They also specify the methods to consider in each of the activities, as well as the

expected outcomes for each of them. Although they suggest methods to use, the whole range of

usability methods1

are still applicable if needed, so as to keep the required flexibility and

adaptability to meet given particularities.

Iteration 1: Proof of

concept

Iteration 2: Elicitation of

recommendations

Iteration 3: Delivery of

recommendations

Iteration

Goal

Guide educators in

understanding the needs for

recommendations in e-

learning scenarios and

demonstrate the value of

extending the adaptive

navigation support in learning

environments with

recommendations.

Produce diverse educationally

oriented recommendations for a

given e-learning scenario and

focus on the perception of the

recommendations previous to

their delivery to final users in the

learning environment.

Offer learners the

recommendations elicited to find

out how the user experience

comes about in the

recommendation process and

understand their behaviour in

order to decide if

recommendations need to be

redesigned (i.e. formatively

evaluate them). Thus, it

constitutes an empirically study

of the recommendations

behaviour.

Iteration

Input

Design plan. Either the output from iteration

‘Proof of concept’ (if available)

or the design plan.

A set of recommendations

modelled and validated, usually

as a result of the iteration

‘Elicitation of educational

1 An exhaustive list of available user centred design methods can be consulted in the UsabilityNet website, the outcome

of the same name European project that provides user centred design resources to practitioners

9

Iteration 1: Proof of

concept

Iteration 2: Elicitation of

recommendations

Iteration 3: Delivery of

recommendations recommendations’.

Context

of use

Methods: meetings with

stakeholders

Outcomes: context specified

Evaluation layer: n/a

Methods: individual interviews,

questionnaires

Outcomes: redefined/adjusted

context of use and info to

produce scenarios

Evaluation layer: interpretation

of data collected in proof of

concept or externally (layer 2)

Methods: individual interviews

Outcomes: revised context of use

Evaluation layer: interpretation

of data in previous feedback

activity iteration (layer 2)

UC

D a

ctiv

itie

s +

e-l

earn

ing

lif

e c

ycl

e p

ha

ses

com

bin

ed

User

requi-

rements

Methods: brainstorming, user

observational studies, Wizard

of Oz

Outcomes: adaptation

requirements

Evaluation layer: n/a

Methods: scenario based

approach

Outcomes: scenarios of use with

educational sound

recommendations proposed in

them

Evaluation layer: n/a

Methods: focus group, interview

Outcomes: revised scenarios and

recommendations

Evaluation layer: n/a

Model-

ling of the

design

solution

Methods: modelling process

Outcomes: sample

recommendations semantically

modelled

Evaluation layer: modelling

the current state of the world

regarding recommendations

elicited and described in terms

of the model (layer 3)

Methods: focus group and card

sorting, modelling process

Outcomes: revised list of

modelled recommendations and

adjustments to the semantic

recommendation model

Evaluation layer: modelling the

current state of the world

regarding recommendations

elicited and described in terms of

the model (layer 3)

Methods: modelling process

Outcomes: revised modelling for

recommendations

Evaluation layer: modelling the

current state of the world

regarding recommendations

elicited and described in terms of

the model (layer 3)

Publi-

cation of

the design

solution

Methods: instantiation of

recommendations, pilot study

Outcomes: sample

recommendations

contextualised in the

environment

Evaluation layer: deciding

upon adaptations by checking

the applicability conditions

(layer 4)

Methods: instantiation of

recommendations, pilot study

Outcomes: technically

validation of the

recommendations

Evaluation layer: deciding

upon adaptations by checking

the applicability conditions

(layer 4)

Methods: instantiation of

recommendations, pilot study

Outcomes: educational

recommendations contextualised

in a large scale setting

Evaluation layer: deciding upon

adaptations by checking the

applicability conditions (layer 4)

Usage to

gather

evalua-

tion data

Methods: paper prototype,

storyboard, Wizard of Oz, user

observational studies

Outcomes: interaction data

from users on sample

recommendations

Evaluation layer: applying

adaptation decisions observing

system logs (layer 5)

Methods: functional prototype,

Wizard of Oz, card sorting

Outcomes: recommendations

rating and classification by users

Evaluation layer: applying

adaptation decisions by

analysing the value of

recommendations delivery

considered by the users (layer 5)

Methods: functional prototype,

observation studies

Outcomes: interaction data from

learners on educational

recommendations

Evaluation layer: applying

adaptation decisions observing

system logs (layer 5)

Feedback

from

evalua-

ting

design

requirem

ents

Methods: questionnaires,

interviews, data log analysis

Outcomes: users feedback on

the sample recommendations

Evaluation layer: collection

of data from users’ interaction

with recommendations and

feedback (layer 1)

Methods: descriptive statistics

Outcomes: feedback on

recommendations to identify the

most relevant for the given

context

Evaluation layer: collection of

data from users’ interaction with

recommendations and feedback

(layer 1)

Methods: data log, interviews,

questionnaires, significant testing

Outcomes: recommendations

feedback on empirical validation

showing recommendations that

need to be redesigned

Evaluation layer: collection of

data (layer 1)

10

Iteration 1: Proof of

concept

Iteration 2: Elicitation of

recommendations

Iteration 3: Delivery of

recommendations

Iteration

Output

A set of sample

recommendations that reflect

identified educational needs,

which made up a first mock-

up that can be shown to users

(educators/learners) and

tested. They translate

researcher ideas.

A set of validated

recommendations ready to be

delivered in the learning

management system and thus to

be formatively evaluated in a

large scale evaluation.

The identification of those

recommendations that need to be

redesigned because they did not

meet the educational objectives

proposed.

Table 1 Practical guidelines for the three iterations defined of the recommendation design and evaluation cycle

So as to clarify the issues involved in Table 1, next there is a more detailed description of the

activities considered in each of the three iterations. At the end of this section, these iterations are

summarised in Figure 1. In Section 5, we comment on the educational contexts where these

practical guidelines have been applied to cope with these iterations. More details on those contexts

are provided in some related works that report the educational scenarios where TORMES has been

applied for these three iterations: ‘Proof of Concept’ in DtP-dotLRN context (Santos and Boticario,

2010), ‘Elicitation of Educational Recommendations’ in DtP-dotLRN context (Santos and Boticario,

2013) and in EBIFE-Willow context (Pascual-Nieto et al., 2011; Santos et al., 2014a), and

‘Delivery of Recommendations’ in EBIFE-Willow context (Santos et al., 2014a). However, these

works do not include the combined methodological approach which results from integrating user

centred design, e-learning life cycle and layered evaluation within the practical guidelines that are

presented in this paper. The added value of this paper lies on both identifying and compiling

guidelines for designing learner centred educationally oriented recommendations and describing

how these guidelines can be applied to follow the methodologies that cover the recommendation

design and evaluation cycle.

The following subsections (3.1, 3.2 and 3.3) focus on describing the issues involved in Table 1

from a conceptual viewpoint, following the iterations and activities depicted afterwards in Figure 1

(Section 3.4). The examples provided in Section 4 are expected to clarify those conceptual issues

that might not be understood without an application context.

3.1 Iteration ‘Proof of Concept’

The objective of the iteration for the Proof of Concept is to come up with a preliminary research

idea as soon as possible. Thus, simple (and readily applicable) user centred design methods are the

most appropriate. Regarding the activity Context of use (Ctx1), when a proof of concept is carried

out, it is assumed that there are no reference systems available wherein potential recommendation

needs were previously identified (see related work in Section 2), and thereby there is no common

ground that can be used to get feedback from the users. As a consequence, the starting point to

define the context of use should be to review related approaches (mainly research ideas) from the

literature and previous experiences (e.g. researchers’ own experience as well as that of relevant

stakeholders). Assumptions resulting from the context of use should be validated. To this,

researchers can share their thoughts and get feedback from people who will use the system on two

key issues i) for what it will be used, and ii) under what conditions. For this, a meeting with

stakeholders can be of value, as it is a strategic way to collect information about the purpose of the

system and the overall context of use.

The activity User requirements (Req1) should focus on identifying the adaptation requirements

for the system within the context of use specified in the previous activity. The methods to be

applied for the requirement specification should allow users to come up with creative ideas on what

adaptation features are required. There are some methods that can provide valuable information,

such as a) brainstorming, which can be used to generate ideas for a given problem in a creative way,

b) user observational studies, which can be carried out with learners to see how they currently

interact in the environment where the recommender system is planned, and c) the Wizard of Oz

11

(Dahlbäck et al., 1993), which can be of practical use to clarify the logic behind as it enables

unimplemented technology to be evaluated by using a human to simulate the response of a system.

In the activity Modelling (Mod1) several recommendations can be proposed based on the

outcomes from the previous activity (e.g., the analysis of the results obtained from the user’s

interactions and the outcomes of the brainstorming) and modelled in terms of the semantic

recommendation model. Accordingly, the corresponding evaluation layer that has to be addressed is

the evaluation of the modelling of the state of the world regarding the recommendation needs

identified (third layer).

In the activity Publication (Pub1) a set of sample recommendations can be contextualised and

instantiated in the environment so that these recommendations can be tested with learners in the

following activity. When appropriate (i.e., adaptation capabilities are already provided in the

learning environment), the decision mechanism of the adaptation should be evaluated (fourth layer).

This can be done, for instance, with a pilot study to test the delivery of the recommendations after

they have been instantiated in the environment.

The activity Usage (Us1) deals with learners’ interactions with recommendations. If there is no

adaptation logic available, methods like paper prototypes, storyboards or the Wizard of Oz are quite

useful to present samples of recommendations to the learners and allow them to interact with the

recommendations and get feedback. In this case, as the adaptation effects may not be available,

when presenting the recommendations to the learner, those effects should be highlighted to her, so

that she can picture them and give her opinion. In turn, if the prototype is functional, user

observational studies can be carried out. In the latter, where the recommendations are offered in a

running prototype, the application of the adaptation decisions should be evaluated (fifth layer).

The activity Feedback (Fdb1) analyses the interactions done by learners. Explicit feedback

from the users can be gathered through questionnaires or interviews. Moreover, data log analysis

(e.g., through data mining) can be used to get knowledge from the interactions. If data from

interactions are collected and analysed, the first layer (i.e. collection of input data) approach should

be applied here to evaluate the feedback gathered. These collected data can be compared with the

results from the observational study carried out in the activity User requirements to analyse the

impact of adding the given recommendations. Furthermore, past experiences previous to the user

centred design approach can be also analysed and compared here within the design being tested.

An application of the user centred design approach as defined by TORMES for the iteration

“Proof of Concept” (Iteration 1 in Table 1) in the DtP-dotLRN context is reported elsewhere

(Santos and Boticario, 2010). The main outcomes of the application of the practical guidelines

proposed in this paper (which combine user centred design, e-learning life cycle and layer

evaluation methodological approaches) are summarised in Section 4.

3.2 Iteration ‘’Elicitation of educational recommendations’

As more factual and objective information about the effects of design decisions is expected in this

iteration, the methods suggested here require more resources (time and participants) than in the

previous iteration.

The current knowledge about the context of use (either from the previous iteration or from the

design plan) can be enriched in the activity Context of use (Ctx2). To this, individual interviews

with educators interested in using the recommendations in their courses as well as questionnaires

for a larger sample of educators can be carried out. The interview script should include relevant

questions that can be used in the next activity (see below) to obtain meaningful educational

scenarios from the educator responses. These interviews should focus on obtaining information that

reflects the type of problems encountered by learners, their goals and also positive experiences.

Specific emphasis should be made to get particularities of the learners’ profile or the course context

in order to gather information to support the system adaptation features. Data mining outcomes

from previous experiences of the educators with their learners can be provided in the interview to

support the educators’ argumentation (e.g., identifying association rules for actions carried out by

successful learners; see Section 4). If these data collected from previous experiences are discussed

12

here, then their interpretation is to be validated as defined by the second layer from the layered

evaluation approach (i.e., interpretation of the collected data).

The goal of the activity User requirements (Req2) is to give instruments to extract knowledge

from the educators on what the requirements are for the recommendations within the context of use.

Scenario-based methods (Rosson and Carroll, 2001) can be used. These scenarios consist in

involving the user in writing stories (i.e., scenarios) about the problems taking place in relevant

situations that come to their mind. Scenarios can be produced with the information obtained from

the interviews of the previous activity. Two types of scenarios are to be produced. The first one,

called problem scenario, should specify how educators carry out their tasks in the given context and

the problems identified in them, but they should not address what system features are to be used.

They are expected to cover a wide range of situations and diverse adaptation contexts and include

problematic issues that will test the system concept. The solution scenario, in turn, has to replace

those issues identified in the problem scenario with potential recommendations that can avoid them

and which are characterised in terms of the semantic recommendation model. To illustrate how

these scenarios are defined, some examples are provided in Section 4 (in particular, in Table 3).

In the activity Modelling (Mod2), a focus group can be used to involve several educators in

validating the recommendations elicited from the scenarios obtained in the previous activity and

refining the modelling done to them. In order to prepare the participants for the focus group aimed

at discussing the recommendations produced during the previous task and make them aware of the

recommendations list produced, they can be asked to individually rate the recommendations

obtained and categorise them with a card sorting method (Spencer, 2009). The purpose here is to

verify if the structure in which the educators expect the recommendations to be classified fits with

the classification proposed in the semantic recommendation model. To carry out the card sorting

activity, each of the recommendations defined should be written on a small card. Participants are

then requested to sort these cards into clusters according to their own educational criteria for

classification. If possible, an open card sorting (i.e. without a predefined set of categories) is

preferred, as this will help to depict the educators’ mental model without any bias. Categories can

be obtained with a hierarchical cluster analysis (Dong et al., 2001). After the focus group discussion,

a revised list of educational sound recommendations properly modelled is to be obtained. As a result

of the above tasks, the recommendation model may be readjusted (e.g., new attributes might be

identified to characterise the recommendations). Here, the evaluation of the modelling of the state

of the world regarding the recommendation needs identified (third layer) takes place.

In the activity Publication (Pub2), recommendations are to be instantiated to support their

eventual delivery to the learners. As in this iteration the focus is on the elicitation process, it has to

be checked if the information used to model the recommendations can be obtained from the

learning environment. Once the information involved in the modelling of the recommendations is

available, the recommendations have to be instantiated. When appropriate (i.e., the system has

adaptive capabilities), the decision mechanism of the adaptation should be evaluated (fourth layer),

for instance with a pilot study to test the delivery of the recommendations after they have been

instantiated in the learning environment.

The activity Usage (Us2) deals with the interactions with the recommendations designed.

Educators and learners are requested to interact with these recommendations in order to rate their

utility and classify them (with a closed card sorting) to find out what is their perception about them.

Participants can be given access to a running system where some sample recommendations are

offered, so they can get an idea of the meaning of a running recommendation. The running

prototype can be a functional system or a Wizard of Oz. In the former instance, the application of the

adaptation decisions should be evaluated (fifth layer).

In the activity Feedback (Fdb2), the results from the previous sub-activity are to be collected

and analysed with descriptive statistics. If interactions are gathered from the system and mined, the

first layer of the layered evaluation approach has to be applied (i.e. collection of input data). As a

result, a validated set of educationally oriented recommendations to be applied in the scenarios

elicited is obtained. These recommendations have been mapped into the model, and have been

13

validated from the users’ point of view. If the evaluation results are satisfactory, they are ready to

be delivered in the e-learning environment.

The results of applying user centred design as defined by TORMES for the iteration “Elicitation

of educational recommendations” (Iteration 2 in Table 1) are reported elsewhere for the two

different contexts, namely DtP-dotLRN (Santos and Boticario, 2013) and EBIFE-Willow (Pascual-

Nieto et al., 2011, Santos et al., 2014a). In this paper we rather focus on summarising in Section 4

the main outcomes of applying the practical guidelines compiled in Table 1 (which combine user

centred design, e-learning life cycle and layer evaluation methodological approaches).

3.3 Iteration ‘Delivery of Recommendations’

When evaluation results from previous iteration are not satisfactory, a formative empirical study

involving a large scale evaluation of the system as a whole can be carried out to obtain indicators

about the recommendations design and thus, understand their effect on the learner. This is meant to

get enough data for a meaningful statistical analysis. This study can also be seen as a rehearsal to

the summative evaluation that should be done when the whole system is finished. The methods

suggested here require more resources (time and participants) than in the previous iteration so that

statistical tests of significance can be applied.

The activity Context of Use (Ctx3) revises the current knowledge already obtained in the

previous iteration through interviews with educators complemented with the analysis of the data

collected from the interactions in the system during the previous iteration. In that case, the second

layer of the layered evaluation approach is applied as the data collected is interpreted.

To improve the activity Requirements specification (Req3), the scenarios and the

recommendations from the previous iteration can be revised with the updated context information in

a focus group or through individual interviews with educators.

In the activity Modelling (Mod3), the modelling of the recommendations revised in the previous

activity has to be checked in order to identify if there were suggested changes that are not covered

by the recommendation model. Thus, the evaluation of the modelling of the state of the world

regarding the recommendation needs identified (third layer) is carried out.

In the activity Publication (Pub3), recommendations modelled have to be instantiated in the

system so that they are ready to be used by the learners in a large scale setting. If appropriate,

especially if some of the applicability conditions depend on data mined or follow a rule-based

approach, the decision mechanism of the adaptation should be evaluated (fourth layer), for instance,

in a pilot study.

In the activity Usage (Us3), learners can interact in a functional prototype with the

recommendations obtained. Here, the above recommendations are offered when the conditions

defined in the recommendation model occur. User observational studies as well as experiments on

functional prototypes can be carried out. To evaluate the adaptation decisions applied, the fifth layer

has to be considered.

In the activity Feedback (Fdb3), the learners’ outcomes are to be collected and analysed from

data logs, questionnaires and interviews. If possible, the impact of recommendations should be

compared with an execution of the course without recommendations. The purpose here is to analyse

if the application of the recommendations has made a statistically significant impact or not. The

goal of the analysis is to find out those recommendations that did not perform well in the formative

evaluation, and thus, they need to be redesigned. Significant testing can be of help in this analysis.

As the learners’ interactions are to be collected, the first layer to evaluate the collection of input

data applies here.

An application in the EBIFE-Willow context of user centred design as defined by TORMES for

the iteration “Delivery of recommendations” (Iteration 3 in Table 1) is reported elsewhere (Santos

et al., 2014a). In this paper, we rather focus on summarising in the next Section the main outcomes

of the application of the practical guidelines proposed in this paper (which combine user centred

design, e-learning life cycle and layer evaluation methodological approaches).

14

3.4 Summary

The layout of the iterations and activities compiled in Table 1 and described in sections 3.1, 3.2 and

3.3 is shown in Figure 1.This figure explicitly introduces the phases of the e-learning life cycle and

the evaluation layers into the set of activities of the ISO-9241-210 user centred design cycle.

Previous interactions, which did not follow the user centred design approach, can be considered in

the activity Feedback of the first iteration. Furthermore, at the end of each iteration, the outcome is

to be checked to see if the system satisfies the specified design requirements. Once the iterations in

the user centred design are finished (e.g., after the third iteration in Figure 1), the recommendations

are ready to be evaluated empirically in a summative study. The red hexagons are used to point out

where the different layers of the layered evaluation approach apply.

Figure 1. Extended user centred design (UCD) cycle to support design and formative evaluation of

recommendations along the e-learning life cycle. Abbreviations used: Ctx: Context of use; Exp: Previous

experiences not following the UCD approach; Fdb: Feedback; Ly: Evaluation layer; Mod: Modelling; Pub:

Publication; Req: User requirements; Sys; System requirements satisfaction; Us: Usage.

4. Evaluating practical guidelines applicability

The practical guidelines proposed for the three iterations of the recommendation design and

evaluation cycle were applied in two very different contexts so as to evaluate their suitability to deal

with diverse situations in real-world online educational scenarios. These scenarios involved

contexts that differ both on the learning (different learning setting and contents) and the

technological side (different learning platform), as commented below.

The first context (DtP-dotLRN) corresponds to the course ‘Discovering the Platform’ (DtP).

This course has been developed following the ALPE methodology (Santos et al., 2007b), which

15

produces accessible Sharable Content Objet Reference Model (SCORM) 1.2 compliant courses and

is designed following the approach of learning by doing (Schank and Cleary, 1995), which means

that simple activities are defined to make use of the different platform services. It teaches how to

use the dotLRN platform to novice users. dotLRN is an open source collaboration oriented learning

management system, which was originally developed at the Massachusetts Institute of Technology

(MIT) and used in universities worldwide for its accessibility support, technological flexibility and

interoperability capabilities. For these reasons it has been the main learning platform considered

over the last decade in the research of the aDeNu group (Santos et al., 2007a).

The goal in the second context (EBIFE-Willow) is to offer a full e-learning course through a

learning system initially designed for blended learning (i.e. combining face to face teaching and

computer-based education). This system is Willow, a free-text computer assisted assessment system

that allows students to answer open-ended questions in natural language (Pérez-Marín et al., 2009).

In this context, two educators with experience in using Willow in blended learning settings had the

need for teaching the MOOC on ‘Search strategies in the Web with Educational Goals’ (EBIFE as

abbreviated in Spanish). The objective for integrating recommendations in Willow was to widen

Willow’s usage to a full e-learning context, where the physical presence of the educator was not

available (Pascual-Nieto et al., 2011). Here, recommendations are required to provide adaptive

navigation support in order to guide the learners in their interaction, covering those navigational

issues that were solved by the educators in the face-to-face session to introduce Willow. This is

meant to foster a proactive attitude of the learner which facilitates the usage of Willow without the

educator support. Thus, the design process has to embed the educators’ way of supporting their

learners during the course interaction as the idea is that the recommendations play the role of the

educator when she is not available. Details on how to provide adaptive navigation support in

Willow with recommendations involving an interdisciplinary team of software developers and

domain experts (i.e. educators, usability and accessibility experts, knowledge engineer, educational

support officers) are described elsewhere (Santos et al., 2014a).

The findings from the application of the design and evaluation practical guidelines proposed in

this paper in relation to these two different contexts (DtP-dotLRN and EBIFE-Willow) are

synthesised in Table 2. In particular, for each of the iterations and the user centred design activities

extended with the e-learning life cycle phases, it is compiled how the selected methods in the

practical guidelines were applied, the outcomes obtained and the layers evaluated. Note that there

were two rounds in the second iteration of EBIFE-Willow, as detailed in Section 4.2. Here A/B

pilot study refers to the comparison of the outcomes from a control group (without

recommendations delivered) vs. an experimental group (where recommendations are delivered if

appropriate).

16

DtP in dotLRN EBIFE in Willow

Iteration ‘Proof

of concept’

Iteration

‘Elicitation of

recommendations’

Iteration

‘Elicitation of

recommendations’

Iteration ‘Delivery of

recommendations’

Context of

use

Methods: Meeting

with 5 stakeholders

(ICT manager + 4

professors who

administer courses

in dotLRN).

Outcomes: Stakeholders’

agreement on the

context of use.

Evaluation layer: n/a

Methods: Questionnaire to 55

educators and 3

additional individual

interviews. To inform

the interviews, data

mining analysis on the

interactions from

previous iteration were

carried out to find

relevant patterns about

potential

recommendation needs.

Outcomes: 5 hours of

audio recording;

indicators for course

success, which were

identified in terms of

association rules and

decision trees.

Evaluation layer: The

interpretation of the

data collected in the

previous iteration

(Proof of Concept) was

evaluated (layer 2)

Methods: Joint

interview with 2

educators, who had

previously mined

interactions with

decision tree algorithm

in a past blended-

learning course with

133 learners.

Outcomes: Identification of

potential needs for the

course in an e-learning

setting and proposed

values for applicability

conditions to trigger the

recommendations.

Evaluation layer: The collection and

interpretation of the

data obtained from

2007-2008 interactions

of Willow was

validated (layer 2)

Round 1 & 2:

Methods: Joint interview

with the 2 previous

educators to revise the

context of use with the

outcomes from the

previous iteration/round.

Outcomes: No

modifications were done to

the context of use.

Evaluation layer:

Outcomes from Eval-Fdb

in the previous iteration

was interpreted before the

Ctx of round 1 of this

iteration, and data

processed in Eval-Fdb in

the first round was

interpreted before

considered in Ctx in the

second round (layer 2).

User requi-

rements

Methods: Brainstorming

session with 12

educators (experts in

psycho-education, e-

learning and

accessibility) to

identify factors that

affect learning

performance, and

observational study

with 10 learners

following a Wizard

of Oz to model

typical situations in

course activities.

Outcomes: Identification of six

factors to be taken

into account in the

recommendations

design.

Evaluation layer: n/a

Methods: Problems

identified by the 3

educators interviewed

were extrapolated to

produce 18 problem

scenarios that

generalised common

situations in an isolated

way. Then these

scenarios were

modified to include

potential solutions that

involve the delivery of

recommendations.

Outcomes: 18 solution

scenarios with 43

recommendations

identified and

modelled. These

recommendations

addressed educational

needs, promote the

active participation of

learners and take into

account accessibility

issues.

Evaluation layer: n/a

Methods: Definition by

the 2 previous

educators of a large

scenario including a

wide range of potential

problems for a fictitious

learner using Willow in

e-learning.

Outcomes: 1 scenario

with 12

recommendations

identified and

modelled. These

recommendations

tackled specific

educational issues very

often related to awaken

meta-cognitive features

during the learning

process.

Evaluation layer: n/a

Round 1 & 2:

Methods: Joint interview

with the 2 previous

educators to revise the

scenario and

recommendations elicited

in the previous

iteration/round.

Outcomes: Changes were

done to some of the

recommendations

description (applicability

conditions, text or both).

Evaluation layer: n/a

17

Modelling of

the design

solution

Methods: A focus

group with 3

educators proposed

a set of 13 sample

recommendations

addressing some of

the factors

identified. They

modelled the

recommendations in

terms of action,

object and text to be

shown.

Outcomes: 13

elicited

recommendations

described in terms

of the available

recommendation

model.

Evaluation layer: The 13

recommendations

elicited were

described in terms

of the categories of

the model (layer 3)

Methods: A focus

group with 6 educators

revised the

recommendations

(previously, they had to

categorise and rate each

recommendation

individually).

Outcomes: 32

validated and modelled

recommendations

rephrased in the form

‘recommend action –

on object – due to

reason’. Category

elements were refined

with a hierarchical

clustering.

Evaluation layer: The

32 recommendations

elicited were described

in terms of the

categories of the model

(layer 3)

Methods: A focus

group with 4 educators

revised the

recommendations

(previously, they had to

categorise and rate each

recommendation

individually).

Outcomes: 12

validated and modelled

recommendations

rephrased in the form

‘recommend action –

on object – due to

reason’. Proposal to

deliver 2 of the

recommendations by e-

mail instead of on the

user interface.

Evaluation layer: The

12 recommendations

elicited were described

in terms of the

categories of the model

(layer 3)

Round 1 & 2:

Methods: Recommendations

modelling was revised by

the 4 educators of the

previous focus group.

Outcomes: The 12 revised

recommendations were

described in terms of the

categories of the model.

Evaluation layer: The 12

recommendations to be

delivered were described in

terms of the categories of

the model (layer 3)

Publication

of the design

solution

Methods: Sample

recommendations

instantiated for an

observational study.

Outcomes: 13

recommendations

instantiated in

dotLRN.

Evaluation layer:

educators in the

focus group checked

the applicability

conditions (layer 4)

Methods: The

instantiation of the 32

recommendations in

dotLRN was checked.

Outcomes: The

feasibility of the

recommendations

instantiation in dotLRN

was analysed.

Evaluation layer:

educators in the focus

group checked the

applicability conditions

(layer 4)

Methods: The

instantiation of the 12

recommendations in

Willow was checked.

Outcomes: The

feasibility of the

recommendations

instantiation in Willow

was analysed.

Evaluation layer:

educators in the focus

group checked the

applicability conditions

(layer 4)

Round 1 & 2:

Methods: Recommendations were

instantiated in Willow. A

pilot execution was done to

check that

recommendations are

delivered as defined.

Outcomes: Recommendations

instantiated in Willow

Evaluation layer: it was

checked if the applicability

conditions of the rules were

properly established (layer

4)

18

Usage to

gather

evaluation

data

Methods: Observational study

with 40 learners

who were

recommended links

to actions in DtP.

Participants’

interactions were

observed. They also

had to fill in a

questionnaire.

Outcomes: Learners

interacted with the

recommendations in

dotLRN to

experience a typical

behaviour of the

SERS.

Evaluation layer:

System logs were

observed to check

that

recommendations

were properly

delivered (layer 5)

Methods: Wizard of

Oz to evaluate the

recommendations with

20 educators and 20

learners, including a

closed cardsorting.

Outcomes: Learners

were shown a running

prototype of a SERS,

but recommendations

were listed in paper to

be evaluated.

Evaluation layer:

Educators and learners

rated the value of the

delivery of the

recommendations

designed (layer 5)

Methods: Wizard of

Oz to evaluate the

recommendations with

15 educators and 15

learners, including a

closed cardsorting.

Outcomes: Learners

were shown the

recommendations

instantiated in Willow,

but recommendations

were listed in paper to

be evaluated.

Evaluation layer:

Educators and learners

rated the value of the

delivery of the

recommendations

designed (layer 5)

Round 1:

Methods: A/B pilot study

of a course run in a large

scale setting with 173

learners, randomly divided

in experimental group and

control group.

Round 2:

Methods: A/B pilot study

of a course run in a large

scale setting with another

204 learners, randomly

divided in experimental

group and control group.

Some of the

recommendations had been

modified.

Round 1 & 2:

Outcomes: Learners

interacted with the

recommendations in

Willow to perform the

tasks of the EBIFE course

Evaluation layer: system

logs were observed to

check that

recommendations were

properly delivered (layer 5)

Feedback

from evalua-

ting design

requirements

Methods: Analysis

of the information

gathered with

questionnaires and

data logs.

Outcomes: Feedback from

questionnaires

showed that

participants found

valuable the SERS

concept and

appreciated the

information

provided by the

recommendation

model.

Evaluation layer:

data collected from

participants

interaction with the

recommendations in

the DtP course in

dotLRN was

evaluated (layer 1)

Methods: Relevance

analysis with

descriptive statistics

done to the data

gathered from the

previous 20 educators

and 20 learners.

Outcomes: Feedback

from the

recommendations

categorisation and

rating showed high

scores by the

participants.

Evaluation layer:

educators and learners

feedback collected was

evaluated (layer 1)

Methods: Relevance

analysis with

descriptive statistics

done to the data

gathered from the

previous 15 educators

and 15 learners.

Outcomes: Feedback from the

recommendations

categorisation and

rating showed some

recommendations under

evaluated. These

recommendations

achieved low scores

from learners and

educators, so their

relevance needs to be

assessed in an empirical

study.

Evaluation layer:

educators and learners

feedback collected was

evaluated (layer 1)

Round 1 & 2:

Methods: Feedback

provided through

questionnaires and

interaction data was

analysed with significant

testing.

Outcomes: Learners

feedback and interactions

were empirical evaluated

regarding educational

effect, perceived utility and

system integration.

Evaluation layer:

information gathered from

participants interactions in

the EBIFE course in

Willow collected was

evaluated (layer 1)

Table 2. Application of the practical guidelines in DtP-dotLRN and EBIFE-Willow contexts

Next, there is more detail on the application of the practical guidelines in each of the two contexts.

19

4.1 Application of the practical guidelines in DtP-dotLRN context

Two iterations of the guidelines were carried out in the context of the course ‘Discovering the

platform’ run in dotLRN learning management system. A total of 104 educators and 70 learners

were involved. A summary of methods, outcomes and evaluation layers considered is compiled in

the first two columns in Table 2.

4.1.1 Iteration ‘Proof of concept’ in DtP-dotLRN context

The first iteration (reported in Table 2, column 1) was a proof of concept to understand the needs

for the recommendations in e-learning scenarios and get some feedback on the recommendation

model initially proposed in (Santos and Boticario, 2008). This first application of the user centred

design cycle was carried out with 20 educators and 50 learners. Details on this context are reported

elsewhere (Santos and Boticario, 2010). To start, a meeting with 5 educational stakeholders (an ICT

manager and 4 professors who administer courses in dotLRN) was carried out to define the context

of use of recommendations in e-learning scenarios (activity Context of use). After that (activity

User requirements), a brain storming session with another 12 educators (experts in psycho-

education, e-learning and accessibility) and an observational study with 10 learners following a

Wizard of Oz to model typical situations in course activities lead to the identification of the

following 6 factors for the recommendations design: i) Motivation for performing the task, ii)

Platform usage and technological support required, iii) Collaboration with the classmates, iv)

Accessibility considerations when contributing, v) Learning styles adaptations, and vi) Previous

knowledge considered. Following, 13 recommendations were proposed by 3 educators (activity

Modelling) in a focus group to provide runtime support considering some of the factors previously

identified. In particular, they provide platform usage and technological support (e.g., when the

learners enter the system, they are recommended to read the help section on the platform usage),

foster collaboration with classmates (e.g., read a thread of the forum with many relevant posts) and

learning styles adaptation (a recommendation is defined for each of the extreme values of the four

dimensions of the Felder and Silverman learning style questionnaire (Felder and Silverman, 1988),

provided that the corresponding questionnaires has been previously filled in). These

recommendations were described in terms of the action to be recommended on a specific object and

the text to be shown to the learner (e.g., “Fill in the learning style questionnaire” as shown in Figure

2). They were instantiated in the DtP course in dotLRN (activity Publication) and delivered to 40

learners who were taking part in an observational study (activity Usage). The goal of the

observational study was to get feedback on the participants’ experience with recommendations.

This feedback was obtained both from the analysis of the learners’ interactions as well as their

answers from a questionnaire that gather their opinion on the recommender and the relevance on the

elements used in the recommendation model to characterise the recommendations. Results showed

that participants had a positive perception of the recommendation approach and they also found

appropriate the recommendations modelling (activity Feedback).

Recommendations have been designed following the SERS approach (Santos and Boticario,

2011a), and thus, they are managed in terms of a message that suggests the learner to take an action

on an object in the course space, and provides the link to the place where the suggested action can

be performed. For instance, in the recommendation “Fill in the learning style questionnaire”, the

recommended object (i.e., the learning styles questionnaire) was linked to the platform service (i.e.,

the tool developed in dotLRN to ask the 44 questions of the Felder and Silverman learning style

questionnaire) where the recommended action (i.e., fill in) on that object can be done. This

approach follows current practice in the educational domain, where recommendations are usually

offered in the form of links (Romero et al, 2007). A specific area in the course space (see top right

side in Figure 2) was defined to list selected recommendations for each participant.

20

Figure 2. Recommendations area added to the DtP course in dotLRN (top right side of the figure)

Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1

evaluated collected data from participants’ interactions when the aforementioned 13

recommendations were delivered in the DtP course in dotLRN. The outcomes obtained from the

course run were related to the interaction context where recommendations were delivered. The

corresponding evaluation of the interpretation of these data is to be done in layer 2 of the following

iteration (see below). Moreover, there were no previous iterations whose collected data should be

interpreted. In turn, layer 3 evaluated the recommendations modelling when described in terms of

action recommended on a platform object. Recommendations designs were found consistent with

the existing models of the system. Educators in the focus group evaluated the applicability

conditions as defined by layer 4. In particular, they agreed on them. Finally, layer 5 evaluated the

delivery of the recommendations, showing that the designed adaptation was properly introduced in

the system interaction since recommendations were delivered according to their applicability

conditions.

4.1.2 Iteration ‘Elicitation of recommendations’ in DtP-dotLRN context

The second iteration (reported in Table 2, column 2) was conducted to discover the diversity of

recommendation opportunities and thus produce educational sound recommendations that can be

used to address issues of interest for personalised and inclusive learning. Details on this context are

reported elsewhere (Santos and Boticario, 2013). 84 educators and 20 learners were involved in this

iteration. The input was the previous iteration. The context of use was refined with the outcomes

from a questionnaire on personalisation support filled in by 55 educators and the information

gathered from individual interviews with 3 other educators (activity Context of use). For the later, a

data mining analysis using Weka’s implementations (Hall et al., 2009) of association rules and

decision tree algorithms on a previous similar course was done to identify indicators for course

success. In particular, resulting association rules showed that when learners had not carried out self-

assessment questionnaires on course contents, had not filled in the learning style questionnaire, had

not provided personal information (e.g., uploaded a photo), or had not used communication services

(e.g., posted a message in the forum or chatted), learners were likely not to be successful in the

course. However, when done, they did succeed. In the same way, the resulting decision tree clearly

differentiated the learners who had carried out the self-assessment and succeeded, from those who

did not. After that, the usage of the forums and filling the learning style questionnaire appeared as

relevant actions that impacted on the course success. With this information (gathered from the 3

educators interviewed), 18 scenarios (problem and solution versions) were produced to gather the

design requirements (activity User requirements). To clarify this, Table 3 shows a couple of these

21

scenarios, where the problem scenario (left) presents a common problematic situation, and the

solution scenario (right) shows some modifications of the scenario to solve the issues identified

with recommendations (i.e., RecX.Y, where X=scenario and Y=the order in it). These scenarios are:

1) Learners feel disoriented; do not know what they are expected to do in the course, 2) Learners

need personalised human support, especially when learners feel isolated, and 3) Default

configurations of the learning environment services may not be suitable for assistive technologies.

Respectively, the whole list of 18 scenarios (and the 43 recommendations proposed as solutions to

the problematic issues identified in them) address educational needs, promote active participation

and take into account accessibility issues.

Problem scenario Solution scenario

Scenario 1: Learners feel disoriented; do not know what they are expected to do in the course.

John is 25 years old and takes an on-line class related to a

new field he is interested in. Since he is not so familiar

with the course material, he is easily disoriented. He has a

hard time figuring out what was supposed to be the last

step, what is the next step and where he is currently as

compared to where he was supposed to be. He looks

around for clues. He feels unsure on how to proceed in

the course. He tries to do the first activity, but once done,

he would have expected some feedback and instructions

for the next step.

John is 25 years old and takes an on-line class related to a

new field he is interested in. Since he is not so familiar

with the course material, he is easily disoriented.

However, there is a message on the entry page of the

course that points him to the course plan [Rec1.1].

Moreover, another message tells him what the next step is

to be done in the course [Rec1.2]. He follows the

corresponding link and carries out the task. When this

task is completed another message congratulates him for

having finished the task and redirects him to the next one

[Rec1.3]. That makes him feel more confident on the

sequence of course activities.

Scenario 2: Learners need personalised human support, especially when learners feel isolated.

Anna is moderately familiar with distance learning but

not with e-learning. She is not comfortable with the

platform and she thinks it is complex. She tries to find her

way around by herself. The class has started and she feels

a bit lost. It seems that there are many learners in the class

and she is afraid that if she gets lost, nobody will help her

because they cannot see each other. There are only names,

and she thinks that they may not ‘be’ “human" people

ready to help her.

Anna is moderately familiar with distance learning but

not with e-learning. She is not comfortable with the

platform and she thinks it is complex. She tries to find her

way around by herself. The class has started and she feels

a bit lost. It looks that there are many learners in the class

and she is afraid that if she gets lost, nobody will help her

because they cannot see each other. Anna sees a message

reassuring her that there are people here “behind the

platform” to provide “human help”, and she is advised to

consult how to contact the educator of this course

[Rec2.1]. Anna sees a message suggesting she should

share her experience using the platform with her

classmates, and does not hesitate to write a post in the

forum for the other learners [Rec2.2]. She needed that

stimulus to post in the forum. For two weeks, nobody has

posted a new message in the forums. As Anna had

followed the previous suggestion, she is again proposed

to start some topic in the forum [Rec2.3].

Scenario 3: Default configurations of the learning environment services may not be suitable for assistive

technologies

22

Problem scenario Solution scenario Philip is new to the community of learners with disability

at his university. He has not used the platform before and

he feels a bit lost. He browses the community space for a

while and finds out a folder where his peers are sharing

useful information regarding technical aids. He has a

good document he found the other day on the Web, and

would like to share with his peers the link, but he has no

idea on how to do it. He gets bored and closes the session

on the platform.

Philip is new to the community of learners with disability

at his university. He has not used the platform before and

he feels a bit lost. However, he sees a message that tells

him that this is a new environment for all the users

[Rec3.1], and soon one gets used to it. After that, he

browses the community space for a while and finds out a

folder where his peers are sharing useful information

regarding technical aids. He has a good document he

found the other day on the Web, and would like to share

the link with his peers, but he has no idea on how to do it.

Then, he notices a message that suggests to read the

instructions on how to share a link in the platform

[Rec3.2]. He is using a screen reader, and the

instructions take that into account when describing the

steps to follow.

Table 3. Problem (left) and solution (right) scenarios from three situations (scenarios) collected from the

interviews

Following, a focus group with 6 educators revised the previously 43 recommendations proposed by

rating and categorising them and turned them into 32 validated and modelled recommendations

rephrased in the form “recommend action on object due to reason” to be delivered in the DtP

course (activity Modelling). Next, the feasibility of the recommendations instantiation in dotLRN

was analysed in the focus group (activity Publication). After that, a Wizard of Oz with 20

educators and 20 learners was set up to show recommendations delivery following the same

approach that was found appropriate in the iteration ‘Proof of concept’ (i.e., selected

recommendations are offered together in a specific area in the learning environment). Both

educators and learners were asked to categorise and evaluate these 32 recommendations listed in a

piece of paper (activity Usage). Finally, data was analysed with descriptive statistics to refine the

recommendations design (activity Feedback).

Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1

evaluated collected data from participants’ ratings on the recommendations designed and assembled

it with the rest of recommendation features. Layer 2 focused on the evaluation of the interpretation

of the data gathered in the previous iteration. Here, the positive outcomes from the ‘Proof of

concept’ iteration were assimilated for the recommendations design. Layer 3 evaluated the

recommendations modelling when described in terms of action recommended on a platform object.

New recommendations designs were also found consistent with the existing models of the system.

Educators in the focus group evaluated the applicability conditions as defined by layer 4. In

particular, they agreed on them. Finally, in layer 5, participants rated the value of the delivery of the

recommendations. Results showed that recommendations scores were high, both for learners and

educators, which leads to a positive perception of the designed adaptation. Thus, they were ready to

be delivered in a course, and thus, there was no need for a formative evaluation at a large scale with

the iteration “Delivery of recommendations”.

4.2 Application of the practical guidelines in EBIFE-Willow context

Regarding the EBIFE-Willow context (i.e., MOOC on ‘Search strategies in the Web with

Educational Goals’ in Willow a free-text computer assisted assessment system), two iterations were

carried out. A total of 21 educators and 525 learners were involved. A summary of methods,

outcomes and evaluation layers is compiled in the last two columns in Table 2.

4.2.1 Iteration ‘Elicitation of recommendations’ in EBIFE-Willow context

A similar goal than in the second iteration of the DtP-dotLRN context (i.e., eliciting educationally

oriented recommendations to be applied in a full e-learning scenario) was followed (and is reported

in Table 2, column 3) in the EBIFE-Willow context (Pascual-Nieto et al., 2011, Santos et al.,

2014a) with the participation of 21 educators and 148 learners.

23

Willow is a free-text computer assisted assessment system that allows students to answer open-

ended questions in natural language (Pérez-Marín et al., 2009). It follows the dialogue metaphor,

where an avatar that represents the system talks to the learner, who is in turn represented by another

avatar. When recommendations were integrated into Willow (Pascual-Nieto et al., 2011), the

outcomes from the ‘Proof of concept’ in DtP-dotLRN context were extrapolated to this one. In

particular, as shown in Figure 3, a new space was added in Willow to show the dialogue

corresponding to the recommendations delivery. On the top, the system suggests a list of

recommendations. On the bottom, the learner can provide feedback on the last recommendation

followed.

Figure 3. Recommendations dialogue in Willow (system at the top, learner at the bottom)

Initially (activity Context of use), 2 educators were involved in eliciting the recommendations. The

input that was considered here consisted in the educators’ experience in a previous blended learning

course, which was gathered in a joint interview. Previous to the interview, a data mining analysis

was carried out on the interactions of 133 learners in that blended-learning course, to help educators

identify the context of use in terms of the potential needs for the course in an e-learning setting. To

give some details on the usage of the data mining support, we can mention as an example that a

decision tree algorithm (implemented by Weka data mining tool (Hall et al., 2009)) was used to

classify learners who have used the reviewing functionality of Willow and those who have not used

it (despite both groups had spent time in the system). From this classification, a proposal for the

values for the applicability conditions that should trigger the recommendations in the new learning

setting was drawn. In this particular case, the analysis was useful to select a value for the number of

sessions that the recommendations should wait before being delivered. The idea behind is to deliver

the recommendation only when learners are expected not to start the review by themselves, to be

the less intrusive possible.

With that information, user requirements were specified (in the subsequent activity User

requirements) using the problem-solution scenario approach, similar to the one that was done in

the DtP-dotLRN context. Then, a large scenario was built which recap a wide range of potential

problems. These problems were avoided with the 12 recommendations proposed in the solution

scenario. They take into account education issues and thus are focused on suggesting the learner: 1)

to choose a lesson to study, 2) to start the study of the contents by asking the system for questions to

answer, 3) to study the concept estimated as less known by the learner, 4) to use the forum to share

a doubt, 5) to read a relevant thread of the forum, 6) to read the educators’ instructions about the

course, 7) and 8) respectively, to change the system’s and learner’s avatar (to make the learner

aware of the dialogue metaphor), 9) and 10) respectively, to look at the learner conceptual model

and the conceptual model of the class (to motivate the learner to keep answering questions), and 11)

and 12) to log in the system (either for the first time or when the learner has not entered for several

24

days). In this way, the aforementioned sample recommendation opportunity (i.e., when to

recommend the usage of Willow) turned into the recommendation 1 that is listed in first place in the

upper part of Figure 3. Here, recommendations focused on awaken meta-cognitive issues. For

instance, at certain situations, learners were recommended (recommendation 8) to change their

avatar (third recommendation in Figure 3) to make the learner aware of the philosophy of the

platform based on the dialogue metaphor between Willow and the learner, to motivate a more

personal interaction. In the same vein (i.e., meta-cognition), in other situations learners were

suggested to look at the conceptual model that describes learner’s individual progress

(recommendation 9), as well as the progress of the whole class (recommendation 10). In this way,

the corresponding recommendations can motivate the learner to keep working with the system so

that her conceptual model stands out the class one.

Following, a focus group with 4 educators revised (by categorising and rating) the

recommendations proposed in the previous scenario and modelled them in terms of the

recommendation model obtained in the ‘Proof of concept’ iteration in the DtP-dotLRN context.

Recommendations were also rephrased in the form “recommend action on object due to reason”

(activity Modelling). The next step that took place in the focus group was to check the feasibility of

the recommendations instantiation in Willow to confirm that they could be delivered in the EBIFE

course (activity Publication). After that, a Wizard of Oz with 15 educators and 15 learners was set

up to show recommendations delivery following the approach defined in Figure 3. Both educators

and learners were asked to categorise and evaluate these 12 recommendations listed in a piece of

paper (activity Usage). Finally, data was analysed with descriptive statistics to refine the

recommendations design (activity Feedback).

Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1

evaluated collected data from participants’ ratings on the recommendations designed and assembled

it with the rest of recommendation features. Layer 2 focused on the evaluation of the interpretation

of the data gathered in the selected blended learning course. The meaning of the prediction

attributes obtained in the data mining process was assimilated and translated into the

recommendations design. Layer 3 evaluated the recommendations modelling when categorised and

described in terms of ‘action recommended on a platform object’. Recommendations designs were

found consistent with the existing models of the system. Educators in the focus group evaluated the

applicability conditions as defined by layer 4. In particular, they agreed on them. Finally, in layer 5,

participants rated the value of the delivery of the recommendations. Results showed that although

some recommendations were highly scored (e.g., the above sample recommendation about the

usage of Willow shown in first place in the list of Figure 3, which was classified in the category

‘active participation’ and received one of the highest ratings both by the learners and educators),

some recommendations were under evaluated. These recommendations were those focused on

asking the learner to be proactive by using the forum to share her doubts and changing the avatars

that represent the user and the system (this latter is the third recommendation in the list of Figure 3).

This suggests that some refinements in their modelling might be needed to improve learners’

perception on them and thus, their usage (since recommendations are, by nature, to be freely chosen

to follow or not by learners). Provided that educators have designed the recommendations from

their educational experience and considering the learners’ needs in the centre, these

recommendations are supposed to have a benefit for the learner. If learners did not perceive benefits

from their modelling, something has failed in their design. For this reason, an empirical formative

study was carried out in the iteration ‘Delivery of recommendations’ aimed to discover problems in

their design and potential improvements to it.

4.2.2 Iteration ‘Delivery of recommendations’ in EBIFE-Willow context

Another iteration was conducted in the context EBIFE-Willow to deliver the recommendations

previously elicited (reported in Table 2, column 4). Details on this context are reported elsewhere

(Santos et al., 2014a). This iteration involved the same 6 educators who elicited and designed the

recommendations in the previous iteration, and 377 additional learners. The goal was to formatively

25

evaluate the recommendations design in a large scale study (with 2 rounds) to get feedback on the

usage of the recommendations so as to identify the changes to make in order to improve the

recommendations design. The input was the outcome from the previous iteration, that is, the 12

designed recommendations and the educators’ and learners’ feedback on their perception. With this

information in mind, the context of use previously defined was revised by the same 2 educators of

the previous iteration (activity Context of use). No modifications were done here. They also revised

the scenario and recommendations elicited (activity User requirements), without major changes.

Following, the modelling of the 12 recommendations was revised in a focus group by the 4

educators as in the previous iteration (activity Modelling). The categories and ratings obtained in

the feedback activity of the previous iteration were considered appropriate. Next, these

recommendations were instantiated in Willow (activity Publication). A pilot execution was done to

check that recommendations could be delivered in Willow as defined. After that, an A/B study (i.e.,

control group without recommendations vs. experimental group with recommendations) was carried

out at a large scale in two rounds (173 learners in the first round, 204 in the second), where learners

performed the EBIFE course tasks (activity Usage). Finally, participants’ answers to questionnaires

and interaction data were analysed in terms of educational effect, perceived utility and systems

integration (activity Feedback). From the findings of the first round, some of the recommendations

were redesigned for the second round. In particular, those which had not impacted on the learning

process (neither positively nor negatively) or had scored under average (although over 50%) in the

perceived utility. As a result, some changes were made to the recommendations’ applicability

conditions, text or both in order to better transmit the need of being proactive during the learning

activity, as learners do not have now a teacher that pushes them to take action, as it happened in the

blended-learning situation. Results from the second round confirmed the value of the changes done

as perceived utility improved for the recommendations that had lower scores in the first round

(while the high scores of the other recommendations remained high).

Adaptation features involved were evaluated in the corresponding layers. In particular, layer 1

evaluated collected data from participants’ interactions after the empirical formative study carried

out in each of the two rounds. Interaction data gathered was assembled with user information

gathered related to the interaction context. Layer 2 focused on the evaluation of the interpretation of

the data gathered. In the first round, this layer was applied after the feedback activity of the

previous iteration, when designed recommendations were rated by educators and learners previous

to their delivery. This feedback was analysed and used to improve user the requirements

specification. In the second round, this layer was applied after the previous feedback activity of this

iteration (i.e., in the first round), during the large scale delivery. In the same way, findings were

analysed and used to improve the user requirements. In turn, layer 3 evaluated the recommendations

modelling, and showed that they were in line with the rest of models in Willow. Layer 4 evaluated

the applicability conditions, confirming from the analysis of the recommendations interaction data

that appropriate recommendations were selected for delivery. Finally, layer 5 evaluated the delivery

of the recommendations by observing the system logs and checking that recommendations were

properly delivered. Here it can be said that benefits were found from recommendations utility,

showing an improvement on the learning performance measured in terms of learning effectiveness

(achievement of the learning goal), learning efficiency (resources to reach the learning goal and

activities successfully completed in time), course engagement (learners’ involvement terms of

connection behaviour) and knowledge acquisition (improvement of the learners’ knowledge).

Moreover, most participants perceived recommendations as useful, and not as an external

functionality of the system. This confirms that adapting the findings from the iteration ‘proof of

concept’ carried out in dotLRN was properly extrapolated to Willows’ dialogue metaphor.

5. Discussion on the practical guidelines applications

This paper extends previous work taking into account the existing needs that are to be addressed

when designing educationally oriented recommendations to be delivered in online courses. From

the state of the art analysis and available research experiences on technology enhanced inclusive

26

learning over the last decade (e.g., aLFanet, ADAPTAPlan, CISVI and EU4ALL projects) follows

that there is a need of having some practical guidelines that help educators in identifying and

designing those recommendations (see Section 2). In fact, there is the lack of reference systems in

the literature wherein potential recommendation needs have been previously identified. In this

context, proofs of concepts to establish the recommendations approach are needed (i.e., guide

educators in understanding the needs for recommendations in e-learning scenarios and demonstrate

the value of extending the adaptive navigation support in learning management systems with

recommendations). Additionally, there is also the need to establish the delivery approach to be used

to present the recommendations within the learning environment. Further, in order to provide the

recommendations in a personalised and user centred way (i.e., following a user centred design

approach), there are other methodologies that impinge on the design process, namely the e-learning

life cycle of personalised educational systems proposed in (Van Rosmalen et al., 2004) and the

layered evaluation of the corresponding adaptation features as compiled in (Paramythis et al., 2010).

To cope with these issues and fill in the gap found (and introduced before) between literature

demands (recommendations should focus on the learning needs and foster active learning) and

literature outcomes (educational recommender systems actually deliver mainly learning contents),

this paper provides the compilation of some practical guidelines for designing and evaluating

educationally oriented recommendations, which enrich the user centred design TORMES

methodology (based on ISO 9241-210) with those two complementary methodological approaches,

the e-learning life cycle and the layered evaluation approach. In order to keep its generality and

flexibility, as ISO 9241-210 does, TORMES does not oblige to follow specific user centred design

methods. However, through the guidelines described in this paper educators are guided throughout

the whole design and evaluation processes with some specific methods to apply. This does not limit

the generality of the user centred design approach but contributes to make an easier deployment by

educators. That said, proposed methods are to be considered plausible suggestions, and educators

may eventually follow other methods as appropriate.

The experiences carried out so far (i.e., in two very different educational settings, namely DtP-

dotLRN and EBIFE-Willow) show that the practical guidelines proposed can guide the

recommendations design and formative evaluation along the e-learning life cycle integrated with the

user centred design cycle. As a result, recommendations that go beyond simply recommending

learning objects and involve diverse actions within the learning environment services that address

specific educational needs, promote active participation of learners, take into account accessibility

issues and awaken meta-cognitive features during the learning process have been elicited. They also

serve as sample of the diversity of educationally oriented recommendations that can be offered in e-

learning scenarios to provide personalised and inclusive support. Recommendations were obtained

by combining the findings obtained with user centred design methods (such as interviews and

scenarios) with those obtained from a data mining analysis from interaction data on previous

courses. All these integrated to cope with the final goal, which is to design non-intrusive

recommendations that are meant to be delivered only when learners need them. Otherwise, learners

will not carry out the associated action, which is of relevance from the educators’ viewpoint when

derived from a learner centric perspective. Thus, both educators and learners need to be involved in

the elicitation process. Educators lead the elicitation process as they have wider experience and

knowledge, and have in mind the learners needs in an aggregated way. However, the design

proposed is complemented with the needs of individual learners who are also involved in the design

through the evaluation process and in terms of interaction data gathered form previous experiences.

Those experiences also show that the practical guidelines can support formative evaluations of

the recommendations that have been designed. For this, the aforementioned layered evaluation

approach for adaptive systems is embedded into the recommendations design process. To cope with

the latter and complement the piece-wise evaluation of the adaptation support, the work we have

done includes a large scale formative study, which compares the system with and without

adaptation (i.e. with and without recommendations). This is meant to understand how the learner

experience comes about in the recommendations process and get data for statistical analysis in order

27

to assess the effect of the recommendations designed. This empirical formative evaluation can be

seen as a rehearsal of the summative evaluation that should be done when the whole system is

finished. In particular, although this paper has not focused on describing in detail findings derived

from such a large scale study, these are reported elsewhere (Santos et al., 2014a).

Regarding the delivery approach, we have followed current practice in the educational domain

that use links to present the recommendations. In this line, the recommendations delivery has been

implemented in terms of a list of annotated links pointing at recommended actions on specific

objects within the learning environment, which are compiled in a centralised section in the e-

learning platform graphical user interface. This approach was followed in the DtP-dotLRN context

and reused (after extrapolated to consider the dialogue metaphor) also with success in the EBIFE-

Willow context.

The paper has focused on describing how the practical guidelines proposed support the

recommendations design and evaluation process along three iterations. These iterations have been

identified in aforementioned previous research experiences and consists of: 1) designing a proof of

concept for the recommendation approach and evaluating its perception by the users, 2) eliciting

educational recommendations from the practical experience of educators and designing them in

terms of the semantic recommendation model, and 3) delivering the recommendations previously

designed to formatively evaluate them in a large scale study. As they address different goals,

require different inputs and produce different outputs, as shown in Table 1, each of the iterations

has to be applied in a distinctive manner, using different methods for their design and testing

different adaptation issues in the evaluation layers.

To evaluate their coverage and applicability in real-world scenarios, we have applied them in

two disparate contexts that differ in both the learning (different learning scenario and contents) and

the technological (different learning environments) sides. Table 2 summarises the application of the

practical guidelines proposed in Table 1 for the required iterations in those two different contexts

(DtP-dotLRN and EBIFE-Willow). The delivery of the recommendations designed show benefits

for both learners and educators, as commented next.

On the one hand, the analysis of the results provided useful feedback to the design in terms of

the indicators measured dealing with the learning performance, the recommendations utility

perceived by the learners and the integration of the recommendations into the system. Details on

these learning results are described elsewhere (Santos et al., 2014a). Here we refer to some of them

to point out the expected benefits of well-designed recommendations when the practical guidelines

are taken into account for their design and formative evaluation. In particular, the outcomes of the

formative empirical study (comparing participants receiving recommendations, i.e. experimental

group, against participants not receiving recommendations, i.e. control group) in both rounds

showed that regarding the learning performance, participants were satisfied with the system, and

when receiving recommendations, their engagement, knowledge acquisition, learning efficiency and

learning effectiveness improved with statistical evidence. Regarding perceived utility, a third of the

recommendations were rated and, over 80% of them were considered useful by the learners. As for

system integration, results showed that the integration of the recommendations was done following

the usability principle of consistency and so, the addition of the new functionality (i.e.,

recommendations delivered) was not seen as additional plug-in the system. In fact, the good

usability levels of the system were kept, since the outcomes from the System Usability Scale

(Brooke, 1996) showed that for any of the empirical groups (control vs. experimental) there was an

average rating of over 71, which is higher than the abstracted average of 68 obtained from a set of

500 studies in (Sauro, 2011). The evaluation also showed that changes in the description of the

recommendations can be detected with the indicators evaluated and thus are helpful to understand

the behaviour of the recommendations for this particular context. This is of relevance with respect

to the work reported in this paper. From the analysis carried out in the first round of EBIFE

improvements on the recommendation design were done. These changes caused an improvement in

the recommendations effect in the second round, as discussed in section 4.2.2.

28

On the other hand, educators involved in both studies (DtP-dotLRN and EBIFE-Willow)

commented that thanks to the practical guidelines, they could reasonably manage the increase of

work during the course preparation (i.e., designing in advance the educational support to be

provided to the learners automatically by the system during the course execution). At the same time,

they highlighted the reduction in the tutoring support required during the learners’ interactions. In

particular, educators from EBIFE-Willow reported that they had turned the time required to prepare

and teach the face to face sessions in the blended learning approach (about 10 hours per edition,

adding up to 20 as there were two rounds) into the time spent to elicit and design the

recommendations (15 hours). In their view, it is plausible that these 15 hours would have been 10

times more if they would have not been supported by the practical guidelines. Moreover, they

acknowledged that the recommendations elicited can be reused in new course editions, where

another additional 10 hours should be spent if the blended approach is used. Thus, they found

benefits in terms of time reduction when the methodology is used for courses taught over several

editions. This shows that educators’ workload is shifted, reducing the tutoring support required

during each runtime execution of the course, while only increasing the time required when

preparing the course, which is to be done in advance to the course runs. The applications carried out

suggest that on the whole, a reduction of the educators’ workload can be achieved if

recommendations are used to support learners during their interactions in the course and educators

follow the practical guidelines proposed to design them. If the methodological support provided by

the guidelines does not exist, the time to be devoted for designing the recommendations can

increase dramatically (in words of the educators involved). We can envision that this payoff is

expected to be increased when the course is provided in large-scale settings like MOOCs, where

learning tasks are diverse, the number of learners is expected to be very high (sometimes several

thousands) and there are several repetitions of the same course. As reported here in the MOOC

EBIFE, in these situations, recommendations can provide timely personalised support that reduces

the educators’ workload during the learners’ participation in the course, in spite of the additional

work involved in applying the practical guidelines. As a result, it is expected that educators have

more opportunities to focus on attending those unforeseen issues that cannot be managed through

recommendations (e.g., adding clarifications to the wording of an exercise that is not well

understood by learners).

Related to this, it has to be said that the large involvement of educators reported in this paper

(125 educators counting all iterations and contexts) is required in these early stages as the field of

educational recommender systems is not yet mature, and thus, there is a need to identify appropriate

recommendation opportunities from current educational scenarios. When educationally oriented

recommendations are mainstream, the involvement of a low number of educators is expected to

produce the required personalised support for thousands of learners.

6. Conclusion

In this paper we have presented a set of practical guidelines for designing and evaluating

educationally oriented recommendations that are both based on educators’ experience and perceived

as adequate by learners (see Table 1). These guidelines integrate three different methodologies: i)

user centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised

educational systems, and iii) the layered evaluation of adaptation features. Those methodologies

reflect the state of the art in providing adaptive navigation support in educational scenarios. The

final purpose is to provide personalised recommendations in online courses supported by online

learning environments. These guidelines have been defined for three iterations of the

recommendation design and evaluation cycle: i) proof of concept, ii) elicitation of recommendations

and iii) delivery of recommendations. As a result, they cover a proposed set of methods to use,

expected outcomes and layers to apply in the evaluation of adaptive features for each of the user

centred design activities extended with the e-learning life cycle phases.

To evaluate the applicability of the guidelines proposed in practice, we have applied them in

two very different contexts, which involve different learning scenarios, contents and platform.

29

These contexts are the DtP course in dotLRN learning management system, and the EBIFE course

in Willow free-text adaptive computer assisted assessment system. A total of 125 educators and 595

learners took part in the user centred design and formative evaluation of 44 recommendations (104

educators and 70 learners for the 32 recommendations in the DtP-dotLRN context; 21 educators and

525 learners for 12 recommendations in the EBIFE-Willow context). Table 4 summarises (from

data included in Table 2) the learners and educators involved in each activity for each of the 4

contexts considered. When users are the same as in the previous activity, an equal sign (i.e., “=”) is

added to them so they are not counted twice in the total counting. In particular, this total counting

computes the total number of educators and learners per scenario, as well as in all of them.

Activity Context (C) Educators Learners

Context of Use

C1: Proof of concept in DtP 5 -

C2: Elicitation of recommendations in DtP 55+3 -

C3: Elicitation of recommendations in EBIFE 2 133

C4: Delivery of recommendations in EBIFE =2 -

User requirements

C1: Proof of concept in DtP 12 10

C2: Elicitation of recommendations in DtP =3 -

C3: Elicitation of recommendations in EBIFE =2 -

C4: Delivery of recommendations in EBIFE =2 -

Modelling of the

design solution

C1: Proof of concept in DtP 3 -

C2: Elicitation of recommendations in DtP 6 -

C3: Elicitation of recommendations in EBIFE 4 -

C4: Delivery of recommendations in EBIFE =4 -

Publication of the

design solution

C1: Proof of concept in DtP - -

C2: Elicitation of recommendations in DtP - -

C3: Elicitation of recommendations in EBIFE - -

C4: Delivery of recommendations in EBIFE - -

Usage to gather

evaluation data

C1: Proof of concept in DtP - 40

C2: Elicitation of recommendations in DtP 20 20

C3: Elicitation of recommendations in EBIFE 15 15

C4: Delivery of recommendations in EBIFE - 173 / 204

Feedback from

Evaluating design

requirements

C1: Proof of concept in DtP - -

C2: Elicitation of recommendations in DtP - -

C3: Elicitation of recommendations in EBIFE - -

30

Activity Context (C) Educators Learners

C4: Delivery of recommendations in EBIFE - -

TOTAL

C1: Proof of concept in DtP 20 50

C2: Elicitation of recommendations in DtP 84 20

C3: Elicitation of recommendations in EBIFE 21 148

C4: Delivery of recommendations in EBIFE =6 377

All scenarios 125 595

Table 4. Educators and learners involved in each activity (= means that these users are the same as in the

previous activity, and thus, do not have to be counted twice).

Results reported in this paper (see Table 2) illustrate that the practical guidelines were useful to

design and evaluate the recommendations elicited with the proposed user centred design

methodology along the e-learning life cycle. Their usage has shown benefits for both learners and

educators. Indirectly, this confirms the value of TORMES as regards to involving educators in the

identification of valuable recommendations for their educational scenarios. In this way, the

approach described in this paper addresses a critical issue in educational institutions worldwide,

which is to develop user centred scenarios mediated by the technology that respond to the specific

and evolving needs of learners. In particular, those needs derived from the information overload

existing in online learning environments, where learners are required to provide their own

contributions for educational purposes on the available learning services. This is of special

relevance in an increasing number of online courses where there ratio learner-educator is

unbalanced and very much in particular in the so-called massive open on-line courses (MOOCs),

which have to deal with large number of learners and course instantiations. Here, there is a need of

having educationally oriented recommendations that can be used to provide timely personalised

support without a significant amount of tutoring resources. Educators are thus asked to design in

advance the personalised support, which can be delivered during the course interaction through a

recommender system integrated with the corresponding learning environment following the service

oriented architectural approach proposed in SERS (Santos and Boticario, 2011a).

As part of future work, from the bases discussed in this paper, there are other development

avenues that are being explored as to designing educationally oriented recommendations. In this

sense, there is a wide range of peculiarities that can be considered in such educational foci. For

instance, the scenarios that have been considered in the work reported in this paper do not include

affective issues, although they are of relevance in e-learning scenarios (Fonseca et al., 2011). Some

use cases dealing with affective issues have been depicted elsewhere that deal with the

recommendation of users, learning activities and learning resources (Leony et al., 2013). Other

works show that emotional feedback in terms of recommendation rules can be used to improve e-

learning experiences (Shen et al., 2009). However, affect modelling in education still constitutes a

challenge (Porayska-Pomsta et al., 2013). When asked (Manjarrés-Riesco et al., 2013), educators in

this area have pointed out that applying recommendations into real practice that account for the

affective issues involved is beyond their capabilities. In particular, they have reported difficulties in

managing the need of affective support in face-to-face learning scenarios. For this reason, we are

currently extending our work to cope with designing affective recommendations. In particular, in

the context of the project MAMIPEC: TIN2011-29221-C03-01 (Santos et al., 2012) we are

investigating how emotional and affective issues can be taken into account when designing

educationally oriented recommendations. For instance, to cope with the situation “Difficulty to

understand the platform functionality leads to a state of panic and frustration, which can later lead

to dropout”. This situation came out when applying the scenario-based approach reported in this

paper and is compiled elsewhere (Santos and Boticario, 2013). To cope with this type of emotional

31

reactions (e.g., panic, frustration), we have extended TORMES methodology to elicit affective

educational recommendations and carried out an initial pilot study (Santos et al., 2014c). Results are

informing the affective recommendations design process and should be taken into account in order

to revisit the practical guidelines presented in this paper if design and evaluation of affective issues

are to be explicitly considered in them. The value of the guidelines to support the design of affective

educationally oriented recommendations is to be evaluated in the context of the project MAMIPEC

with an intelligent tutoring system that providing personalized guidance in arithmetic problem

solving tasks (Arevalillo-Herráez et al., 2014). In addition, we are also applying TORMES

methodology to elicit and design context-aware recommendations that do not only take as input

physiological and environmental information for the recommendation process, but also take

advantage of ambient intelligence in educational environments in order to deliver interactive

recommendations through two complementary sensorial actuators (Santos et al., 2015).

References Adomavicius, G., Tuzhilin, A. and Zheng, R. (2011) REQUEST: A Query Language for Customizing

Recommendations. Information Systems Research, 22(1), 99-117.

Al-Hamad, A., Yaacob, N. and Al-Zoubi, A.Y. (2008) Integrating ‘Learning Style’ Information into Personalised e-

Learning System. IEEE Multidisciplinary Engineering Education Magazine, vol. 3, no. 1, march 2008.

Arevalillo-Herráez, M., Arnau, D., Marco-Giménez, L., González-Calero, J.A., Moreno-Picot, S., Moreno-Clari, P.,

Ayesh, A., Santos, O.C., Boticario, J.G., Salmeron-Majadas, S., Cabestrero, R., Quirós, P. (2014). Providing

Personalized Guidance in Arithmetic Problem Solving. Proceedings of the 4th

International Workshop on

Personalization Approaches in Learning Environments, CEUR workshop proceedings, vol. 1181, p. 42-48.

Bates, B. and Leary, J. (2001) Supporting a range of learning styles using a taxonomy-based design framework

approach, Proceedings of the 18th annual conference of the Australasian Society for computers in learning in

Tertiary education, 9-12 December, p. 45-53.

Bevan, N. (2003) UsabilityNet Methods for user centered design. In: Jacko, J. and Stephanidis, C. eds. Human-

Computer Interaction: Theory and Practice (Part 1), Volume 1. Heraklion, Crete: Lawrence Erlbaum, p. 434-438.

Bieliková, M., Šimko, M., Barla, M., Tvarožek, J., Labaj, M., Móro, R., Srba, I., Ševcech, J. (2014) ALEF: from

Application to Platform for Adaptive Collaborative Learning. Recommender Systems for Technology Enhanced

Learning: Research Trends & Applications. Manouselis, N., Drachsler, H., Verbert, K., Santos, O.C. (eds.).

Springer, p. 195-225.

Blochl, B., Rumershofer, H. and Wob, W. (2003) Individualised e-learning systems enabled by a semantically

determined adaptation of learning fragments. In proceeding of the 14th International workshop on Database and

Expert Systems Applications, p. 640-645.

Boticario, J.G. and Santos, O.C. (2007) An open IMS-based user modeling approach for developing adaptive LMS.

Journal of Interactive Media in Education. Adaptation and IMS Learning Design. Special Issue, 2007/02, p. 1-19.

Boticario, J.G. and Santos, O.C. (2008) A standards-based modelling approach for dynamic generation of adaptive

learning scenarios. Journal of Universal Computer Science. 14 (17), p. 2859- 2876. doi: 10.3217/jucs-014-17-2859

Boticario, J.G., Rodríguez-Ascaso, A., Santos, O.C., Raffenne, E., Montandon, L., Roldán, D., Buendía, F. (2012)

Accessible Lifelong Learning at Higher Education: Outcomes and Lessons Learned at two Different Pilot Sites in

the EU4ALL Project. Journal of Universal Computer Science, 18(1), 62-85. doi: 10.3217/jucs-018-01-0062

Brooke, J. 1996. SUS: a ‘quick and dirty’ usability scale. In Jordan, P.W., Thomas, B., Weerdmeester, B.A. and

McClelland, A.L. Usability Evaluation in Industry. London: Taylor and Francis.

Brito, Patrick H. S., Ig Ibert Bittencourt, Aydano Pamponet Machado, Evandro Costa, Olavo Holanda, Rafael Ferreira

and Thiago Ribeiro. "A Systematic Approach for Designing Educational Recommender Systems." Educational

Recommender Systems and Technologies: Practices and Challenges. IGI Global, 2012. 232-256. Web. 15 Jan.

2014. doi:10.4018/978-1-61350-489-5.ch010

Brusilovsky, P., Peylo, C. (2003) Adaptive and Intelligent Web-based Educational Systems. International Journal of

Artificial Intelligence in Education, vol 13, p. 156-169.

Brusilovsky, P., Karagiannidis, C., & Sampson, D. (2004). Layered evaluation of adaptive learning systems.

International Journal of Continuing Engineering Education and Lifelong Learning, 14 (4/5), 402 - 421.

Burke, R. (2002) Hybrid Recommender Systems: survey and experiments. User-Modeling and User-Adapted

Interaction, 12, p. 331-370.

Buder, J. Schwind, C. (2012) Learning with personalized recommender systems. A psychological review. Computers in

Human Behaviour, 28, 207-216.

Dagger, D., O’Connor, A., Lawless, S., Walsh, E., Wade, V.P. Service-oriented e-learning platforms. From Monolithic

systems to flexible services. IEEE Internet Comput. 2007, 3, 28–35.

Dahlbäck, N., Jönsson, A., and Ahrenberg, L. (1993) Wizard of Oz studies: why and how. In Proceedings of IUI, 193–

200.

32

Dong, J., Martin, S., Waldo, P. (2001) A user input and analysis tool for information architecture. In CHI '01 Extended

Abstracts on Human Factors in Computing Systems (CHI EA '01).

Drachsler, D., Hummel, H.G.K. and Koper, R. (2009a) Identifying the Goal, User model and Conditions of

Recommender Systems for Formal and Informal Learning. In Journal of Digital Information 10(2).

Drachsler, H., Hummel, H. G. K., Van den Berg, B., Eshuis, J., Waterink, W., Nadolski, R. J., Berlanga, A. J., Boers, N.,

Koper, R. (2009b). Effects of the ISIS Recommender System for navigation support in self-organized Learning

Networks. Journal of Educational Technology and Society, 12(3), p. 122-135.

Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N. (2015) Panorama of Recommender Systems to Support

Learning. Handbook on Recommender Systems, 2nd edition (accepted).

Farzan, R. and Brusilovsky, P. (2006) Social Navigation Support in a Course Recommendation System. In proceedings

of 4th International Conference on Adaptive Hypermedia and Adaptive Web-based Systems.

Felder R. M. and Silverman L. K. (1988) Learning and Teaching Styles In Engineering Education, Engr. Education,

78(7), p. 674-681.

Fonseca A., Salete Leone Ferreira, B. , Fernandes Gonçalves, S., Toledo Coutinho, R.E.. Affectivity in the Process of

Distance Technical Education. Volta Redonda – RJ – Brazil - May 2011.

Gamboa Rodrı́guez, F., Pérez Silva, J.L., Lara Rosano, F., Caviedes Contreras, F., Miranda Vitela, A.I. (2011) A

student centered methodology for the development of a physics video based laboratory Interact. Comput., 13 (5):

527-548 doi:10.1016/S0953-5438(01)00031-5

Gassner, K., Jansen, M., Harrer, A, Herrmann K. and Hoppe, H.U. (2003) Analysis methods for collaborative models

and Activities, In B. Wasson, S. Ludvigsen, U. Hoppe (eds.), Designing for Change in Networked Learning

Environments, Proc. CSCL 2003, p. 411-420.

Gena, C. (2006) A User-Centered Approach for Adaptive Systems Evaluation, in Proc. of the Fifth Workshop on User-

Centred Design and Evaluation of Adaptive Systems held in conjunction with the 5th International Conference on

Adaptive Hypermedia and Adaptive Web-Based Systems (AH2006), 3-12.

Gena, C., Weibelzahl, S. (2007) Usability engineering for the adaptive web. In The adaptive web, Peter Brusilovsky,

Alfred Kobsa, and Wolfgang Nejdl (Eds.). Lecture Notes In Computer Science, vol. 4321, 720-762.

Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S., Persson, J., Cajander, A. (2003) Key principles for user-centred

systems design. In Behaviour & Information Technology, Vol. 22, Iss. 6.

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H. (2009) The WEKA Data Mining Software:

An Update; SIGKDD Explorations, Volume 11, Issue 1.

Hanover Research Council (2009) Best Practices in Online Teaching Strategies. Available at:

http://www.uwec.edu/AcadAff/resources/edtech/upload/Best-Practices-in-Online-Teaching-Strategies-

Membership.pdf.

Harrigan, M., Kravcik, M., Steiner, C. and Wade, V. (2009) What Do Academic Users Really Want from an Adaptive

Learning System?, The 17th International Conference on User Modeling, Adaptation and Personalisation (Trento,

Italy), 455-460.

Herlocker, J., Konstan, J., Terveen, L. and Riedl, J. (2004) Evaluating collaborative filtering recommender systems.

ACM Transactions on Information Systems, 22, 1 p. 5-53.

Hsu, C.K., Hwang, G.J., Chang, C.K. (2010) Development of a reading material recommendation system based on a

knowledge engineering approach. Computers & Education 55 (2010) 76–83

IDEO (2003) IDEO Method Cards: 51 Ways to Inspire Design. IDEO, Palo Alto.

ISO (2002) Ergonomics of human-system interaction - Usability methods supporting human-centered design. ISO/TR

16982:2002.

ISO (2010) Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems. ISO

9241-210:2010.

Karagiannidis, C., Sampson, D. and Cardinali, F. (2001) Integrating Adaptive Educational Content into Different

Courses and Curricula. In Educational Technology and Society, Vol. 4, No. 3.

Kelly, G. A. (1955). The psychology of personal construct. New York: Norton.

Kluver, D., Nguyen, T.T., Ekstrand, M., Sen, S., Riedl, J. (2012) How many bits per rating? In Proceedings of the sixth

ACM conference on Recommender systems (RecSys '12).

Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C. (2012) Explaining the user experience of

recommender systems. User Model User-Adapt Inter, 22, 441-504.

Konstan, J.A. and Riedl, J. (2012) Recommender systems: from algorithms to user experience. User Model User-Adapt

Inter, 22, 101-123.

Leony, D., Parada Gélvez, H.A., Muñoz Merino, P.J., Pardo, A., Delgado Kloos, C. (2013) A Generic Architecture for

Emotion-based Recommender Systems in Cloud Learning Environments. Journal of Universal Computer Science,

19(14): 2075-2092.

Lord, S.M., Prince, M.J., Stefanou, C.R., Stolk, J.D., Chen, J.C. (2012) The Effect of Different Active Learning

Environments on Student Outcomes Related to Lifelong Learning. International Journal of Engineering Education,

28(3), 606-620.

Manjarrés-Riesco, A., Santos, O.C., Boticario, J.G., Saneiro, M. (2013) Open Issues in Educational Affective

Recommendations for Distance Learning Scenarios. CEUR workshop proceedings, vol. 997, 26-33.

33

Manouselis, N., Karagiannidis, C., Sampson, D. (2014) Layered evaluation in recommender systems: a retrospective

assessment. Journal of e-Learning and Knowledge Society, vol. 10(1), p. 11-31

Mao, J.Y., Vredenburg, K., Smith, P.W. and Carey, T. (2005) The state of user-centered design practice.

Communications of the ACM, Volume 48 , Issue 3 (March 2005), p. 105-109.

Martin, L., Gutiérrez y Restrepo, E., Barrera, C., Rodríguez Ascaso, A., Santos, O.C., Boticario, J.G. (2007) Usability

and Accessibility Evaluations along the eLearning Cycle. In Mathias Weske Mohand-Saïd Hacid and Claude

Godart (Eds.), Web Information Systems Engineering (WISE 2007). Lecture Notes in Computer Science, 4832. p.

453- 458. doi: 10.1007/978-3-540-77010-7_44

Masters, K. (2011) A brief guide to understanding MOOCs. The Internet Journal of Medical Education, vol 1 (2).

Mulwa, C., Lawless, S., Sharp, M., Wade, V. (2013) The evaluation of adaptive and personalised information retrieval

systems: a review. Int. J. Knowledge and Web Intelligence, Vol. 2, Nos. 2/3, 138-156.

Muñoz-Merino, P.J., Delgado-Kloos, C., Fernández-Naranjo, J. Enabling interoperability for LMS educational services.

Comput. Stand. Interfaces 2009, 31, 484–498.

Paramythis, A., Totter, A., Stephanidis, C. (2001) A Modular Approach to the Evaluation of Adaptive User Interfaces.

In: S. Weibelzahl, D. Chin and G. Weber (Eds.) Empirical Evaluation of Adaptive Systems. Proceedings of

workshop held at the Eighth International Conference on User Modeling, p. 9-24.

Paramythis, A., Weibelzahl, S. and Masthoff, J. (2010) Layered evaluation of interactive adaptive systems: framework

and formative methods. User Modeling and User-Adapted Interaction, vol 20, (5), 383-453.

Pascual-Nieto, I., Santos, O.C., Perez-Marin, D., Boticario, J.G. (2011) Extending Computer Assisted Assessment

systems with Natural Language Processing, User Modeling, and Recommendations based on Human Computer

Interaction and Data Mining. International Joint Conference on Artificial Intelligence 2011, p. 2519-2524. doi:

10.5591/978-1-57735-516-8/IJCAI11-419

Pérez-Marín, D., Pascual-Nieto, I., Rodríguez, P. (2009) Computer-assisted assessment of free-text answers. The

Knowledge Engineering Review 24(4), p. 353-374.

Porayska-Pomsta, K., Mavrikis, M., D'Mello, S.K., Conati, C., Baker, R.S.J. (2013) Knowledge Elicitation Methods for

Affect Modelling in Education. I. J. Artificial Intelligence in Education, 22(3), 107-140.

Pu, P., Chen, L., Hu, R. (2012) Evaluating recommender systems from the user’s perspective: survey of the state of the

art. User Model User-Adap Inter, 22, 317–355

Romero, C., Ventura, S., Delgado, J.A. and de Bra. P. (2007) Personalised links recommendation based on data mining

in adaptive educational hypermedia systems, ECTEL 2007, p. 292-306.

Rosson, M. B. and Carroll, J. M. (2001) Usability engineering: scenario-based development of human computer

interaction. Morgan Kaufmann.

Ruiz-Iniesta, A., Jimenez-Díaz, G. and Gómez-Albarran, M. (2012) A Hybrid User-centered Recommendation Strategy

Applied to Repositories of Learning Objects. Special issue Recommender systems to support the dynamics of

virtual learning communities (RS4VLC). International Journal of Web Based Communities, 8 (3), 302-321.

Santos, O.C. (2009) Recommendations support in standard-based LMS. Proceedings of the 14th International

Conference on Artificial Intelligence in Education (AIED 2009), p. 775-776.

Santos, O.C. and Boticario, J.G. (2008) Users’ experience with a recommender system in an open source standard-

based learning management system. Lecture Note in Computer Science, vol. 5298, p. 185-204. doi: 10.1007/978-3-

540-89350-9_14

Santos, O.C. and Boticario, J.G. (2010) Modeling recommendations for the educational domain. Proceedings of the 1st

Workshop ‘Recommender Systems for Technology Enhanced Learning’ (RecSysTEL 2010), p. 2793-2800. doi:

10.1016/j.procs.2010.08.004

Santos, O.C., Boticario, J.G. (2011a) Requirements for Semantic Educational Recommender Systems in formal e-

learning scenarios. Algorithms 4(2), p. 131-154. doi:10.3390/a4030131

Santos, O.C., Boticario, J.G. (2011b) TORMES methodology to elicit educational oriented recommendations. Lecture

Notes in Computer Science 6738, 541–543. doi: 10.1007/978-3-642-21869-9_95

Santos, O.C., Boticario, J.G. (2012) (Eds). Educational Recommender Systems and Technologies: Practices and

Challenges. IGI Publisher. doi: 10.4018/978-1-61350-489-5

Santos, O.C., Boticario, J.G. (2013) User Centred Design and Educational Data Mining support during the

Recommendations Elicitation Process in Social Online Learning Environments. Expert systems. The Journal of

Knowledge Engineering (in press), doi: 10.1111/exsy.12041

Santos, O.C., Boticario, J.G., Raffenne, E. and Pastor, R. (2007a) Why using dotLRN? UNED use cases. In J.R.

Rodríguez and M. Palomo (Eds.) Proceedings of the FLOSS (Free/Libre/Open Source Systems) International

Conference 2007, p. 195-212, 2007, UCA Pub.

Santos, O.C., Boticario, J.G., Fernández del Viso, A., Pérez de la Cámara, S., Rebate Sánchez, C., Gutiérrez y Restrepo,

E. (2007b) Basic Skills Training to Disabled and Adult Learners Through an Accessible e-Learning Platform.

Universal Access in Human-Computer Interaction. Applications and Services. Lecture Notes in Computer Science

Volume 4556, 2007, pp 796-805. doi: 10.1007/978-3-540-73283-9_86

Santos, O.C, C. Barrera, E. Mazzone, J.G. Boticario (2010) Catering for Personal Autonomy with E-Mentoring

Supported by Recommendations. In Learning Design: Handbook of Research on Personal Autonomy Technologies

and Disability Informatics, edited by J Pereira. IGI Global, 191–203. doi:10.4018/978-1-60566-206-0.ch012.

34

Santos, O.C., J.G. Boticario, M. Arevalillo-Herraez, M. Saneiro, R. Cabestrero, E. del Campo, A. Manjarrés-Riesco, P.

Moreno-Clari, P. Quiros, S. Salmeron-Majadas (2012) MAMIPEC - Affective Modeling in Inclusive Personalized

Educational Scenarios. Bulletin of the IEEE Technical Committee on Learning Technology 14, no. 4 (2012): 35–38.

Santos, O.C., Boticario, J.G., Perez-Martin, D. (2014a) Extending Web-based Educational Systems with Personalised

Support through User Centred Designed Recommendations along the e-Learning Life Cycle. Science of Computer

Programming, vol. 88, 1 August 2014, pages 92–109, doi: 10.1016/j.scico.2013.12.004

Santos, O.C., Boticario, J.G., Manjarrés-Riesco, A. (2014b) An approach for an Affective Educational

Recommendation Model. Recommender Systems for Technology Enhanced Learning: Research Trends &

Applications. Manouselis, N., Drachsler, H., Verbert, K., Santos, O.C. (eds.). Springer, 123-143. doi: 10.1007/978-

1-4939-0530-0_6

Santos, O.C., M. Saneiro, S. Salmeron-Majadas, J.G. Boticario (2014c) A Methodological Approach to Eliciting

Affective Educational Recommendations. In Proceedings of the 14th IEEE International Conference on Advanced

Learning Technologies (ICALT 2014), 529-533 (2014) doi:10.1109/ICALT.2014.234.

Santos, O.C., Saneiro, M., Boticario, J.G., Rodriguez-Sanchez, C. (2015) Towards Interactive Context-Aware Affective

Educational Recommendations in Computer Assisted Language Learning. New Review of Hypermedia and

Multimedia (accepted)

Sauro, J. (2011) Measuring Usability with the System Usability Scale (SUS). Available from the following url:

http://www.measuringusability.com/sus.php

Schank, R. C. and Cleary, C. (1995) Engines for education. Hillsdale, New Jersey, Lawrence Erlbaum Associates.

Shaw, C., Larson, R., Sibdari, An Asynchronous, Personalized Learning Platform –Guided Learning Pathways. Sixth

International Conference of MIT’s Learning International Networks Consortium (LINC). June 16 – 19, 2013,

Cambridge, Massachusetts

Shen, L. and Shen, R. (2004) Learning Content Recommendation Service Based-On Simple Sequencing Specification.

ICWL 2004, p. 363-370.

Shen, L., Wang, M., Shen, R (2009) Affective e-Learning: Using “Emotional” Data to Improve Learning in Pervasive

Learning Environment. Educational Technology and Society, 12 (2), 176-189.

Sonwalkar, N. (2013) The First Adaptive MOOC: A Case Study on Pedagogy Framework and Scalable Cloud

Architecture—Part I. MOOCs FORUM, vol. 1 (P), 22-29.

Spencer, D. (2009) Card Sorting. Designing Usable Categories. Rosenfeld Media.

Tang, T. and McCalla, G. (2009) A Multidimensional Paper Recommender: Experiments and Evaluations. IEEE

Internet Computing 13(4): 34-41

UXPA (2012) The Usability Body of Knowledge. User Experience Professionals' Association. Retrieved from

http://www.usabilitybok.org/methods

Van Rosmalen, P., Boticario, J.G. and Santos, O.C. (2004) The Full Life Cycle of Adaptation in aLFanet eLearning

Environment. Learning Technology newsletter. Vol. 4, p. 59-61.

Van Velsen, L., Van der Geest, T., Klaassen, R., Steehouder, M. (2008) User-centered evaluation of adaptive and

adaptable systems: a literature review. The Knowledge Engineering Review, 23(3), 261-281.

Wang, P. (2007) The Analysis and Design of Educational Recommender Systems. In C. Crawford et al. (Eds.),

Proceedings of Society for Information Technology and Teacher Education International Conference 2007, p.

2134-2140.

Webb, E., Jones, A., Barker, P. and Van Schaik, P. (2004) Using E-Learning Dialogues in Higher Education.

Innovations in Education and Teaching Int'l, 41 (1), 93-103.

Weibelzahl, S., & Lauer, C. U. (2001). Framework for the evaluation of adaptive CBR-systems. In I. Vollrath, S.

Schmitt & U. Reimer (Eds.), Experience Management as Reuse of Knowledge. Proceedings of the 9th German

Workshop on Case Based Reasoning, GWCBR2001, 254-263.

Zaïane, O.R. (2002) Building a Recommender Agent for e-Learning Systems. Proceedings of International. Conference

on Computers in Education, Auckland, New Zealand, 3-6 December, p. 55-59.

Zins, A. H., Bauernfeind, U., Del Missier, F., Mitsche, N., Ricci, F., Rumetshofer, H. and Schaumlechner, E. (2004)

Prototype Testing for a Destination Recommender System: Steps, Procedures and Implications. In: Frew, A. J.

(ed.): Information and Communication Technologies in Tourism (ENTER 2004). Proceedings of the International

Conference in Cairo, Egypt, Springer, Vienna-New York, p. 249-258.