This article was downloaded by: [University of Liverpool]On: 09 October 2014, At: 02:49Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Open Learning: The Journal of Open,Distance and e-LearningPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/copl20

Facing the challenge in evaluatingtechnology use in mobile environmentsPatrick McAndrew a , Josie Taylor a & Doug Clow aa Institute of Educational Technology , The Open University ,Milton Keynes, UKPublished online: 11 Oct 2010.

To cite this article: Patrick McAndrew , Josie Taylor & Doug Clow (2010) Facing the challenge inevaluating technology use in mobile environments, Open Learning: The Journal of Open, Distanceand e-Learning, 25:3, 233-249

To link to this article: http://dx.doi.org/10.1080/02680513.2010.511959

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Open LearningVol. 25, No. 3, November 2010, 233–249

ISSN 0268-0513 print/ISSN 1469-9958 online© 2010 The Open UniversityDOI: 10.1080/02680513.2010.511959http://www.informaworld.com

Facing the challenge in evaluating technology use in mobile environments

Patrick McAndrew*, Josie Taylor and Doug Clow

Institute of Educational Technology, The Open University, Milton Keynes, UKTaylor and FrancisCOPL_A_511959.sgm10.1080/02680513.2010.511959Open Learning0268-0513 (print)/1469-9958 (online)Original Article2010Taylor & Francis253000000November 2010PatrickMcAndrewp.mcandrew@open.ac.uk

The process of developing innovative mobile approaches to informal and formallearning is challenging, not least in needing to satisfy stakeholders with diverseinterests in the technology, the pedagogy and the overall system. Some approachesto evaluation may focus on examining the nature and quality of learning thatoccurs, while other methods may take a user-centred approach to understandinteractions with the systems. In this paper we highlight a methodology thatattempts to address these two analytical issues in parallel, and to communicate theresults to stakeholders. The methodology is grounded in cultural historical activitytheory and is compatible with other views emerging that such evaluation can havemultiple levels. The method applies task analysis to examine the conflicts thatemerge when learners are interacting with technological systems in an informallearning setting. Results from a trial involving first-aiders are used to illustrate thetechniques as they were applied as part of a European project that developed acollaborative mobile learning environment. The method has been repeated in otherstudies and is suggested to provide a valuable tool to reflect on understanding andenable the sharing of perspectives on evaluation outcomes.

Keywords: mobile learning; activity theory; trials; task model

Introduction

Our experience of evaluating technology-enhanced collaborative learning over aperiod of years has highlighted a difficulty shared by many researchers. In complexmulti-disciplinary projects comprising a variety of professionals from different fields(e.g. teachers, researchers, software designers, programmers, system builders,pedagogy experts) it is hard to find an appropriate common language to enable theproject’s work to run smoothly and, in particular, to facilitate the evaluation. Carroll(2000) discusses this problem, and recommends the use of scenarios as boundaryobjects to help overcome the difficulties. Scenario-based design entails using concre-tisation – a concrete story of use. This story typically specifies a setting; objects;agents or actors; and their goals or objectives. They also have a plot. Scenarios maybe very short, focusing on a small piece of interaction with a system, or they may bequite elaborate and describe the complete system in action.

The main feature of scenarios is that they provide a coherent and concrete vision,and ‘not an abstract goal, not a set of requirements, not a list of features and functions’(Carroll, 2000, p. 50). Carroll also notes:

*Corresponding author. Email: p.mcandrew@open.ac.uk

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

234 P. McAndrew et al.

The problems of managing novelty and complexity in design are no doubt exacerbatedin cases [where] the technological issues [are] quite novel and tangled and the teamlarge, diverse and distributed. (2000, p. 8)

In working with learning technology, the case described by Carroll is not rare; thedevelopment of novel systems and the complexity of working across disciplines is anecessity if learning technology is to be adopted and refined. The focus of this paperis a European project where the team was multi-national, multi-lingual, multi-skilledand distributed across Europe, trying to build a system that orchestrated new andemerging mobile technology to construct a pedagogically sound mobile learningenvironment. The challenge of organising, managing, developing and testing such asystem is apparent and led to pressures to have in place robust and clear approachesfor defining and evaluating.

As leaders of the user requirements and evaluation work package, a starting pointwas to make use of scenarios, as prescribed by Carroll, to help communication and toenable pedagogy experts to communicate with system designers and builders. But asthe work developed, we found that a task model we had constructed as part of thedesign process (Taylor, Sharples, O’Malley, Vavoula, & Waycott, 2006) also provedits worth in terms of the implementation, or instantiation, of the scenarios, and thenled into an evaluation process that emphasised communication.

Some of the reflection on this and other work has been reported by Vavoula andSharples (2009), who set out six challenges restated here in brief as: capturingcontexts; identifying learning; satisfying ethical concerns; incorporating mobility;establishing an overview; and linking formal and informal learning. They then providea three-level consideration of the evaluation process as having micro, meso and macrolevels. The need to switch between these different perspectives provides an overallcommunication concern – how can those who engage at one level understand theimpact on those working at a different level? If tools can be developed that helpextract and share alternative interpretations of evaluation data, then this concern canbe addressed alongside other valid perspectives from different disciplines, and fromthe different roles and interests of stakeholders.

In the case of our evaluation work, the task model enabled us to develop a frame-work that attempted to satisfy various stakeholders when evaluating learning and tech-nology use in informal settings. It is a method for representing user activities(practices, strategies and conflicts) that emerge when interacting with technologicalsystems in an informal mobile learning setting. The task model represents both a semi-otic and a technological space, which are mutually dependent but which can bedescribed separately. The model is rooted in cultural historical activity theory, anddevelops Engeström’s (1987) extended model of human activity. A particular contri-bution is to build a visual representation that exposes the activity structure as viewedfrom more than one perspective.

We go on to provide some background, firstly to the project (MOBIlearn) and thento the first-aid user trial, before returning to the method in more detail.

The MOBIlearn project and first aid scenario

MOBIlearn was a large, multinational, European-funded research project involving 24partner organisations from Europe, Israel, Switzerland, the USA and Australia. Theconsortium comprised both industrial partners and universities, and brought together

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 235

areas of expertise in technical design and implementation, and pedagogy and evalua-tion. Taking a user-centred approach, the aim of the project was to define an architec-ture for a pedagogically sound mobile learning environment, and to evaluate aninstantiation of that architecture using available mobile technologies.

In Taylor and Evans (2005) we discuss some of the problems encountered in work-ing in such a large project, spread across the world, and how scenarios were used bothto help structure the project’s activities and to help focus design effort. Scenarios inthe project fulfilled a dual function:

● The first function was to assist in the process of ‘envisionment’ (Carroll, 1995)of the mobile learning environment.

● The second function was to begin considering basic requirements to enable usto progress towards the field studies that would provide us with user require-ments in the user-centred context.

The project had identified three target domains for study and development. Thesewere museums (including art galleries), first aid, and MBA students attending auniversity business course on a part-time basis. These domains are representative of arange of applications related to mobile learning, with particular reference to learningoutside the classroom.

The work with scenarios was carried out in the context of the socio-cognitive engi-neering design method (Sharples, Jefferey, du Boulay, Teather, & du Boulay, 2002),a user-centred approach that describes and analyses the complex interactions betweenpeople and computer-based technology, so as to inform the design of socio-technicalsystems (technology in its social and organisational context).

Figure 1 gives a picture of the flow and main products of the design process. It isin two main stages: a stage of activity analysis that sets constraints on the design andanalyses how people work and interact with their current tools and technologies; anda stage of design of new technology.Figure 1. Overview of the flow and main products of the socio-cognitive engineering design process.The focus for this paper is on the activity analysis aspect, shown in the four boxeson the left of Figure 1, and then how the analysis can be reapplied to evaluate theimplementation. General requirements were collected for each of the three scenariostrands, which were collated and stored in a database using a template based on the

Figure 1. Overview of the flow and main products of the socio-cognitive engineering designprocess.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

236 P. McAndrew et al.

Volere method (Haley, Nuseibeh, Sharp, & Taylor, 2004; Sharp et al., 2003). Therequirements contributed to a theory of use, along with the scenarios, and these twosources of information allowed us to specify the field studies that would feed into thetask model, and onward to the design of the mobile learning environment. In thispaper, we use the first aid scenario to describe the field studies, how they were run,how they were analysed, and what we were able to learn from this process. Themuseum scenario is considered in another paper (Sharples, Taylor, & Vavoula, 2007),which also presents the overall aims of the project and gives further information aboutthe framework adopted in the activity analysis.

The first aid scenario

The training of people prepared to give first aid to people suffering an accident at workwas chosen as one of the three main application areas for mobile learning by theMOBILearn project. Training is a requirement for anyone who wishes to take thenominated role as a first-aider and typically happens every three years during a one-day face-to-face workshop. In the United Kingdom, this training is provided byvolunteer organisations such as the St John’s Ambulance and British Red Cross, withsimilar systems in place in other countries.

User views were elicited through Future Technology Workshops (Vavoula &Sharples, 2007), and a series of detailed scenarios were built (Taylor & Evans, 2005).The scenarios fed into a requirements process jointly with scenarios from the otherapplication areas (museum visits, and supporting campus-based business students) tospecify a software solution able to support the training scenario.

The scenarios were developed into a plan for a trial using a group of first-aiders whohave the role of ‘designated first aider’ for their department within the Open UniversityUK (OU UK). This means that in addition to their normal job role they deal with firstaid incidents in their department or around the OU UK. Training is voluntary and, indiscussion with the first-aiders, it was found that whilst their training needs to beupdated, annually refreshed and tested, there was little opportunity to practise theirskills in the normal run of things at work. This was particularly the case for moredramatic incidents, such as heart attacks, when being able to take immediate and appro-priate action would be of critical importance. They met on a regular basis to refresh theirtraining, but felt they wanted to get a better feel for what it might be like to deal withan incident in reality. We felt that mobile technology could offer some possibilities forproviding stimulating training exercises, conducted in situ (rather than in an assemblyhall, or other meeting place), and the design of the trial aimed to address this issue.

The scenario that was developed was activity based: users would be faced withchallenges and would need to draw on knowledge, collaborate together andcommunicate to meet those challenges. The MOBIlearn system would support themin this, by offering content, access to a moderator and each other, and a range ofcommunication tools available through a single device.

Analysis methodology

Activity theory

Within the MOBIlearn project, a task model for mobile learning was developed as partof the socio-cognitive engineering design method (Figure 1). This method draws upon

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 237

socio-cultural theories of learning and, in particular, that of activity theory asexpressed by Engeström (for example, 1987).

Within activity theory, the introduction of a computer-based system to manage andsupport work practices ought to enable participants in that community to identify withobjects or information resources relevant to that work context. The significance of thisnotion emerges from the dialectic that is introduced – the process of interacting withobjects using computer-based tools enables transformations to occur in both theobjects and also in the individuals involved in carrying out work activities. Transfor-mations in objects tend to be physical in nature, whilst individuals involved in carry-ing out work activities using computer-based tools can experience changes in theirperceptions of the objects and also of the activity that they are performing. In theactivity theory literature, this notion of transformation has been discussed in relationto changes that occur between tools and tool-users, the activity being performed, andalso the objects being manipulated during activity (Engeström, 1993; Engeström &Escalante, 1996; Kaptelinin, 1996). Consequently, transformations in objects andactivities can result in changes in work practices. Such changes can become evidentin tool-usage behaviour. As a result, contradictions are bound to emerge in the waywork activities are carried out. This could have an impact on the kind of outcomesresulting from that work activity. Engeström (1993) positively highlights this line ofthinking by arguing that contradictions or discrepancies serve as a means by whichnew insights about developmental patterns of human practices emerge. Therefore,systems for supporting work practices should be designed with a technological infra-structure that is socially transparent and sensitive to both collaborative and individualworking styles, and preferences so as to accommodate discrepancies in workpractices.

Engeström’s activity system is illustrated in Figure 2. The extension of the trianglefrom the original Vygotskian ‘subject/object/tool’ triangle allows a more complexanalysis of the context of the activity by examination of, for example, the ‘subject/rules/community’ sub-triangle.Figure 2. Engeström’s (1987) extended activity system.

Figure 2. Representation of Engeström’s (1987) extended activity system.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

238 P. McAndrew et al.

Adapting the extended activity system

Some researchers have found the activity triangles inspiring, but have highlighteddifficulties in applying them in practice (Scanlon & Issroff, 2005) and the need toexplore ways of operationalising the approach. Papadimitriou, Tselios, and Komis(2007) consider the activity structure as a way to develop rich descriptions of separateactivity. Mwanza (2002a, 2002b) discusses ways in which the nodes of sub-trianglescan be formed into questions: for example, ‘in what way are the rules, and the commu-nity affecting the subject’s behaviour?’ The triangles can have different emphasis: forexample, in what way is the subject interacting with the community to interpret therules?’ We see our approach as moving further in this direction.

Several adaptations were necessary for us to use the representation effectively:

(1) The labelling of the triangle is interpreted as a snapshot of the system from aparticular point of view. This means that such labels can vary both in terms ofperspective and as further data allow refinement.

(2) An additional layer to the diagram is introduced to represent the correspondingtechnical activity underpinning the semiotic activity.

(3) The node descriptions on the triangle were not always helpful to engage peopleunaware of the underlying theory. The terms ‘rules’, ‘community’ and‘division of labour’ imply a political position that is relevant to the derivationof activity theory but unfamiliar and incongruous in the software designand implementation communities. To improve communication with that audi-ence, the bottom three nodes were described as ‘control’, ‘context’ and‘communication’.

In making this change, explained further in Sharples et al. (2007), the intention is toprovide a representation that is clearer across an interdisciplinary project, but we alsowanted to stay within the ethos of activity theory. The term ‘control’ relates to thenotion of ‘rules’ but also allows a cybernetic sense of control appropriate to thecomputing world. Control may be passing between the computer and the learner, orbetween programmes at the technological level, or between people at the semioticlevel. This is a richer concept than simply who can do what, and when, but thiselement is subsumed by it. Similarly, learning takes place within a context; and as ourlearning theory is socio-cultural, the context of community within which it takes placeremains of primary importance. On the other hand, context can have other things asso-ciated with it besides community; for example, many mobile computing applicationsare able to automatically sense aspects of the environment such as other people, otherdevices, other objects, location, proximity, and so forth.

The division of labour is often something to be negotiated. Again, this is a complexconcept that is further complicated by the presence of technology. The apportioningof activity between participants, and between participants and computers, will dependupon the communication links between them, so division of labour is not omitted butis instead subsumed by communication.

The task model: semiotic and technological spaces

The task model developed for the MOBIlearn project embodies a central concept ofmobile learning, derived from our empirical work, which is that:

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 239

● There is a clear separation between required functionalities and their embodi-ment in any specific technology.

In other words, in a work situation or a learning situation, people know what kinds offunctionalities (resources for learning) they would like to have available to them to beeffective, and will seek out particular tools as and when they need them (e.g. a webbrowser, a word processor, email, etc.). We also adopt the view of mobile learningexpressed by other authors, which is that:

● It is the people that are defined as ‘mobile’, not the devices around them (forexample, Sariola, Sampson, Vuorinen, & Kynäslahti, 2001).

Taylor et al. (2006) argue that mobile learning, then, is something people doon the move, using whatever they may have to hand to do it – handheld devices,personal digital assistants, laptop computers, telephones and notebooks – includ-ing paper. Mobile learning is not simply defined by the fact that someone has amobile device. These two principles lead us to a representation of a mobilelearning space that separates the technical view of the activity, and its enablingtechnology, from the view of the human learning activity, embedded as it is in asocial and cultural milieu (see Sharples et al., 2007; Taylor et al., 2006; for moredetail, see Sharples 2005). Learners move within these two spaces – the mentalspace that consists of required, or preferred, functionalities, and the space ofpossible embodiments of those functionalities in the form of devices. There is adialectic between these two spaces – if the learner sees that a device has a goodmatch to her requirements, she may choose to appropriate that technology and, inso doing, integrate it into her activities. As she integrates the device, her activitieswill be shaped by it (for an account of how this process of appropriation canoccur, see Waycott, 2005).

So there is a dialectical relationship between the technological activity-space, andthe more abstract semiotic learning-space. Learners enter the task of learning with anobjective – to augment knowledge and skills they may or may not already possess –and the output from this activity is a new set of knowledge and skills. But several otherimportant factors impinge on this simplified view of process. Arguably these otherfactors share the same dialectical relationship between a physical (or technological)domain, and a more abstract human and social (or semiotic) domain.

Figure 3 illustrates the top-level task model, instantiated to the first aid scenario.Because the task model is representing two spaces, however, this representation isactually one triangle laid upon another. Lifting them apart makes it easier to see thetwo levels, and to identify the points at which the dialectical relationship occursbetween the semiotic space and the technological space, as shown in Figure 4.Figure 3. The task model representing the first aid scenario.Figure 4. The task model showing two levels and dialectical relationships.This representation of the mobile learning task facilitates analysis of what ishappening at any given time, and is particularly helpful in evaluating learning eventsbecause it allows the evaluator to separate out those incidents that concern the techni-cal infrastructure from those that are associated with the learning event itself. We canalso see the impact of failures of technology on the learning activity – sometimes itmatters, and can be fatal for learning, and other times it does not.

The main purpose of the representation is to reveal contradictions and difficulties.So, for example, in an ideal world, the semiotic first-aider would never need to beexplicitly aware of the presence of the technological layer. It is just there,

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

240 P. McAndrew et al.

doing its job, facilitating her activity. If there is a breakdown, however, the semioticfirst-aider needs to become the technological mobile user in order to try to fix theproblem – if she can. In applying novel technology it is likely that some first-aiderslack competent mobile user personas that can step in at this point to fix things, so atechnological failure can be catastrophic.

Applying activity analysis in practice

We applied our two-layer analysis to a series of trials using software developed in theMOBIlearn project and known as the ‘MOBIlearn system’. For the first aid scenariothere were two linked trials with activity analysis used in each case to providefeedback and help us understand the issues that emerged.

Figure 3. The task model representing the first aid scenario.

Figure 4. The task model showing two levels and dialectical relationships.Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 241

Trial 1

In Trial 1, the first performance of the first aid scenario, the trial took place inpressured circumstances that are not untypical of experiences in developing researchsoftware within projects. In this case the project was nearing completion, the softwareintegration stage had been delayed, and there were performance issues and uncertaintyas to whether the software should be accepted for use. This led to compromises in thestructure of the trial in that the participants were not approved first-aiders, and thenumber of participants was kept low. The scenario was adjusted to apply to a groupof four technical support people from within a single department, all of them with aninterest in first aid but without formal certification.

In this trial there are two types of user: a task leader and a task participant. Thetask participants are expected to interact in pairs and as a complete group. The taskleader needs to configure the tasks and interact with all participants to guide them. Inbrief, an agenda rendered on mobile devices led participants through a series oftasks; for each task, they were directed towards appropriate content and tools withinthe MOBIlearn system. The task participants needed to carry out individual tasks(Quiz, testing their knowledge of first aid in relation to content in the system), pairedtasks (Challenge and Brainstorming, to work out a joint response to an emergencysituation) and work as a complete group (Initial chat and Get together and vote,when they would need to review each other’s suggestions). The task leader wasresponsible for constructing the agenda, overseeing activity during the chats andsetting tools to appropriate states; for example, by closing the vote at the end ofvoting.

A familiarisation session was run for all participants to give them general experi-ence with all facilities in the system. They were able to use a wide range of tools andwere encouraged to give usability feedback. The main trial session then used anarrower set of tools and encouraged feedback on functionality and task performance.The task participants were separate and physically distant, but worked together inpairs through the system. To aid evaluation, video recording captured the interactionof each pair, who were also watched by an observer who made notes. The task leaderwas part of the project team and made reflective notes. A debriefing session involvingall participants and observers was video-recorded. These sessions were then reviewedto highlight key incidents and identify the views of the participants.

Analysis of the trial

In broad terms, this trial – and other trials taking place on other scenarios – weresuccessful, in that users had accessed the system and carried out some tasks. However,there were some basic problems of system responsiveness, and users had difficultyfinding the correct menu and methods to select the tools that were expected by varioustasks. The first aid trial in particular had not been able to progress through all stagesin the assigned time. An initial reaction within the project was to generate somerecriminations and an urge to ‘fix’ problems with the user interface. There was a feel-ing that either the system could be blamed, along with its developers, or the specifica-tion could be blamed, along with those who built the scenarios. While this is in someways a natural reaction to faults late in a system’s development, the two-layer view ofactivity we have discussed offers a suitable means for determining and explainingmore subtle interactions between elements in the system that are causing contradic-tions for the user. The scenarios and planned actions can be represented in the semiotic

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

242 P. McAndrew et al.

layer of the activity system for the trial. Essentially, each of the six nodes of theexpanded Engeström triangle can be given a label or set of labels based on the viewof how the system should operate to meet the scenario. Such a labelling will not beunique but can be achieved in a rapid brainstorming fashion. The result should thenbe considered a tentative labelling that is open to revision during further analysis.Similarly, the technical view representing the system could become very complex ifall possible labels are applied and, in practice, only a partial labelling is feasible. Arapid approach to labelling again proved effective to generate node labels of thoseaspects of the technology likely to concern users in this particular trial. This meant thatthe technical and semiotic views were roughly balanced and related: a differentscenario would be expected to lead to both a different semiotic view and a differenttechnical view.

Having the related semiotic and technological views encourages the view thatcontradictions in the activity system can be addressed by adjusting both the technol-ogy and the scenario design. Presenting the evaluation results in this way also encour-aged shared understanding of the problem viewed across both layers and an approachof collaborative joint working towards a solution.Figure 5. Technological view of Trial 1.For this trial, we developed a technological view of the system (Figure 5) and asemiotic view (Figure 6). These use a representation of the activity triangle that hasmultiple labels at each node. In effect, each triangle shows many different interactionsthat can be separated out and used to identify and explain cases where differentaspects can act in support or contradiction. Having labelled the triangle, we looked atthe feedback and observations in relation to the technological view to enable us to spotwhere interactions worked well together, or were contradictory and so provided amismatch in how they were used.Figure 6. Semiotic view of Trial 1.For example, the section of the activity triangles shown in Figure 7 can beextracted from this overall triangle. This represents the need for the user to workthrough the agenda of tasks, link up with groups of other users, and choose the sametools to work with from the many available. Following the approach of Mwanza(2002a), these nodes can be considered to form a pair of questions:

Figure 5. Technological view of Trial 1.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 243

● How is it that the user at work operates with divided groups to work through theagenda?

● How is it that the user at work controls the many choices to work through theagenda?

Figure 7. Subsection of the technological activity triangle for Trial 1.The answers to these two questions are in conflict, as to work effectively in groups theusers must be using the same tool at the same time, while the many choices availableto the user in the system enabled individual users to choose different tools to achievethe same ends. Evidence for this conflict came from observations and feedback: as oneparticipant said, ‘Sometimes [I] don’t know who is there – too much information onthe screen’. And from another there was a ‘[need for] several communication channelsat the same time’. Viewed from the technological layer this means that the task infor-mation needed to ensure shared views; as feedback to the developers, this strength-ened their need to simplify some aspects of the interface. Viewed in the semiotic layer,however, this contradiction means that there is a need to construct the interactions insuch a way that people would act together more by adjusting the task design and theway co-workers interacted.

Figure 6. Semiotic view of Trial 1.

Figure 7. Subsection of the technological activity triangle for Trial 1.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

244 P. McAndrew et al.

This analysis approach was repeated for each case of concern or issue that camefrom the evaluation, with the aim to provide additional ways to explain the results ofour evaluation and then to explore where solutions could be found by adjusting thescenario design and the technical implementation. The result was to produce maps forboth supporting aspects and contradicting aspects. It is interesting to note that thesame feature, for example fixed location, can act both in support and conflict. In thiscase there is support between fixed location, working through an agenda, and the task.There is conflict between fixed location, the user at work and the user selections oftools.

By carrying out this analysis on first aid Trial 1 the outcomes could be fed into thedesign of first aid Trial 2, to improve its effectiveness and provide a clear basis to beaddressed in the second trial. Other observations from the trial led to the overallconclusions that for an improved match between the semiotic requirements and thesupporting technology there would need to be some changes to how the systemworked and to how the trial planned to use the technology. Changes were made toimprove usability whenever the user was faced with choices and to help provide statusinformation between users working remotely. At the same time, the way in which thetask leader acted to moderate different groups and alert them to events neededadjusting to the planned working environment. The lack of actual mobility in the trialsalso needed to be addressed. This generated a requirement added to the semiotic layerto share information about where each user was located. In turn, this generated newrequirements to capture that information at the technology level.

A joint effort, therefore, was made to revise the combination of the system andscenario for a retrial. A simpler interface was devised but also new technology wasadded to provide a means for synchronisation and sharing image information: mobilecamera phones with SMS text messaging were supplied to update participants outsidethe integrated MOBIlearn system. This split of technology initially may seem aretrograde step; however, it illustrates an important finding of the overall research thatmultiple device solutions were often appropriate in mobile situations.

Trial 2

The scenario of Trial 1 was adjusted to encourage greater mobility and encouraginggreater continuity in communication to support a uniform choice of tools. While theoriginal trial had sought to prove the technology, this analysis gave the project theconfidence to introduce its key research questions into the second trial. This meant wecould address whether the system could genuinely operate in a mobile environment,and whether the first aiders could benefit from carrying out designated tasks inrelation to their first aid experience.Figure 8. Pictures used in support of the second first aid trial.A second trial was carried out involving six participants and one team leader, allfrom the first aid community. Participants worked in pairs carrying out assessments ofimmediate actions based on cases presented on a mobile device relevant to theirimmediate location, Figure 8 shows images forming part of each case matched to threeof the locations (multimedia laboratory, refectory, and café). They then discussed andshared plans for actions with the other groups.

As in the first trial, a familiarisation session took place to allow participants to getinitial experience with the technology. The main trial then took place two days laterover a period of about 1.5 hours. Each pair in the trial was monitored by an observerwho could also provide technical support, if it was requested. The observers used a

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 245

simplified observation card to record their observations, in particular looking forevidence of synchronisation of activities (contra-indications being confusion andworking on tasks different from other pairs) and working on task (contra-indicationsbeing discussion of system aspects and technical problems). The moderator completeda reflective report. All participants, the moderator and observers then took part in astructured group feedback session that was digitally videoed for analysis. Participantsalso completed a questionnaire and there was a final debrief of the observing team.

This trial produced more positive results, with feedback generally being morefocused on the possibilities for use in practice and less on the technology difficulties.All participants agreed that the session had ‘gone really well’ and that it had been‘Good fun – able to relax and learn more’. The moderator reported:

The trial seemed a great success in teaching terms. The participants all engaged with theactivities set, and clearly showed a grasp of first aid principles in the work they uploadedto the system.

Both the provision of SMS contact and using pictures within tasks were positivelycommented on. It was pointed out by one user that with ‘SMS you have a warningsignal, you can really hear that something is going on’. Using the camera phoneincreased interaction with the environment – those using the phone stood up andlooked around before deciding on the picture to take. No one felt inhibited by carryingout this sort of activity in a relatively public location: ‘As soon as you are involved inthe task [in a public place] you forget about other people there’.

The messaging improved the synchronisation between groups to greatly reduce thecontradiction that was evident in the first trial where groups would be trying tocommunicate using different tools and at different times. The pictures themselvesadded to the involvement in the activity: ‘Uploading pictures is particularly helpful.… Sharing things is most helpful, with the person you were working with and theother groups’. Having made this semiotic change in the scenarios and provided thetechnology to support it, we could see it in action and have a much greater feel forseeing the evidence for mobile learning that was the overall goal of the project.

The model of challenge-based training received strong support. It was felt to behighly applicable to first aid training, especially as the current approach only reviewstraining at three-year intervals for re-certification. It was commented that: ‘For quieterpeople this is … much better than normal first aid training’. All saw the potential intaking part in such events run at about six-month intervals; the existing approach

Figure 8. Pictures used in support of the second first aid trial. Reproduced with permissionfrom the photographer Philip Downes, Open University.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

246 P. McAndrew et al.

meant that training was often never used, while being faced with a challenge meantthey would gain confidence from being able to recall key points. Sharing views andresults was highly valued and confidence was gained by praise from each group forthe others’ assessment within the chat. Learning outcomes were not assessed as partof the trial. In general, the trial had acted to confirm the value of previous training andto raise confidence that everyone was ‘thinking exactly the same thing’. However,during the feedback session it emerged that before the trial only one participant (withspecialist defibrillator training) knew the location and availability of a defibrillator ifit was needed for heart attack cases. All participants had picked up this additionalinformation during the trial by sharing the group views and discussion.

Steps in implementing the methodology

We have now used this approach to both plan a technical implementation (McAndrew& Taylor, 2000) and in reporting and running evaluation studies including thosedescribed above. We feel that the activity triangles augmented by developing separateviews offer us considerable power to explain situations and to gather complex collab-orative situations into relatively compact representations. We have achieved this byusing the approach flexibly and accepting that it can only be a partial and potentiallybiased view of the real situation. Our experience has encouraged us to develop a tenta-tive position on how to implement our approach.

(1) Agree a set of labels that have meaning to those involved (our own labellingreplaced rules, community and division of labour, with control, context andcommunication).

(2) Produce a semiotic triangle linked to scenarios of use.(3) Determine the aims for the trial based on a semiotic view.(4) Produce a technological triangle linked to the features in a system.(5) Determine the main technological issues.(6) Gather information from a trial.(7) Relate that information first to the technological view.(8) Match through to the semiotic view.(9) Use both together to explain and adjust the situation.(10) Iterate through steps two to nine.

Throughout this approach, the evaluator needs to remain involved and flexible.

Reflections and conclusion

Large-scale projects such as MOBIlearn have inherent problems that, if ignored, caneffectively sabotage collaborative effort as partners fragment and focus on dischargingtheir individual responsibilities. Evaluators are only too well aware that, as projectsdisintegrate, fingers of blame are often pointed in various directions, including at theevaluators for delivering the message.

This paper has discussed a method that not only helps stakeholders unify theirdisparate views, but also allows a representation to capture the innate complexity oftechnology-supported collaborative learning in a mobile environment. In other words,communication is not achieved at the cost of over-simplification. This proves to beimportant in the evaluation phase, in which the representation can simultaneously

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 247

display where systems and their activities are succeeding in their goal of supportinglearners, as well as displaying points of failure and possible consequences. A reviewby Wali, Winters, and Oliver (2008) of the framework as presented by Sharples et al.(2007) considers it a weakness that only one application is described, and also that thetwo-level analysis is more complex than using a single activity system. The furtherexample given here shows that the method can apply in other circumstances (we havealso applied it to subsequent work examining the production of open educationalresources). In our view, the two-level analysis – rather than introducing complexity –provides a representation that separates out the technological complexities from thesocial ones and provides a way to support communication of results.

Further augmentation of this extended activity system may well be a fruitfulavenue to explore. Possible extensions include deepening the analysis by developingtools to help with drawing diagrams and with logging the sequence of diagrams as wemove to ever more fine-grained representations. Whilst there is a general sense ofhaving gone ‘deep enough’ for a given purpose, we have not yet systematically minedthe scenarios to see what happens as you go deeper. It will be interesting to seewhether this simply becomes too complex to manage or whether, with computersupport, it yields more interesting perspectives. A similar process of two-stage analy-sis of evaluation data was also applied in working with a system to support reuse ofopen educational resources. In that case (McAndrew, Santos, & Godwin, 2007) theanalysis both revealed the tension in the rules and community concerning the tools andcircumstances for editing of shared content, and also helped communicate these to thedevelopment and academic teams.

The dimensions of communication we have considered in this paper are betweentechnological and the semiotic (design) points of view. Vavoula and Sharples (2009)draw attention to three levels – micro, mesa and macro – each of which needs to beunderstood in relation to the others. A suggestion that we have yet to explore is toconstruct three distinct activity representations using the terminology appropriate toeach level. We believe that attempting to do this would be a worthwhile exercise andlikely to reveal new perspectives; however, comparing two activity systems thatrepresent the same situation from different perspectives is already a fairly complexexercise, and extension to three views may cause implementation problems.

From the point of view of the project, we feel that the task model and evaluationapproach served their purpose well, and that we managed to analyse a very compli-cated situation to improve not only the user experience, but also the first aiderexperience. Analytic approaches that aid evaluation and improve communication arenot commonplace and so the development of shared activity representations is apotentially valuable contribution to the evaluators’ toolset.

AcknowledgementsThe authors would like to acknowledge the European Union for financial support through theMOBIlearn project IST-2001-37187. They would also like to thank all members of theMOBIlearn project team, and the participants in the studies.

ReferencesCarroll, J. (1995). Scenario-based design. New York: John Wiley & Sons.Carroll, J.M. (2000). Making use: Scenario-based design of human-computer interactions.

Cambridge, MA: MIT Press.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

248 P. McAndrew et al.

Engeström, Y. (1987). Learning by expanding: An activity theoretical approach to develop-mental research. Helsinki: Orienta-Konsultit.

Engeström, Y. (1993). Developmental studies of work as a test bench of activity theory: Thecase of a primary care medical practice. In S. Chaiklin & J. Lave (Eds), Understandingpractice: Perspectives on activity and context (pp. 64–103). New York: CambridgeUniversity Press.

Engeström, Y. & Escalante, V. (1996). Mundane tool or object of affection? The rise and fallof the postal buddy. In B. Nardi (Ed.), Context and consciousness: Activity theory andhuman-computer interaction (pp. 325–333). Cambridge, MA: MIT Press.

Haley, D., Nuseibeh, B., Sharp, H.C., & Taylor, J. (2004, September 6–10). Theconundrum of categorising requirements: Managing requirements for learning on themove. In Proceedings of the 12th International Requirements EngineeringConference (RE’04), Kyoto, Japan (pp. 309–314). Washington, DC: IEEE ComputerSociety Press.

Kaptelinin, V. (1996). Computer-mediated activity: Functional organs in social and develop-mental contexts. In B. Nardi (Ed.), Context and consciousness: Activity theory andhuman-computer Interaction (pp. 45–68). Cambridge, MA: MIT Press.

McAndrew, P. & Taylor, J. (2000, September). Organising for a knowledge network. InProceedings of Human Computer Interaction 2000, Sunderland (Vol. 2, pp. 99–100).London: British Computer Society.

McAndrew, P., Santos, A., & Godwin, S. (2007, March 26–28). Opening up for OpenLearn:Issues in providing open educational resources. In Proceedings CAL ‘07 Development,Disruption and Debate Conference, Dublin. Retrieved September 8, 2010, from http://kn.open.ac.uk/public/document.cfm?docid=11936.

Mwanza, D. (2002a). Conceptualising work activity for CAL systems design. Journal ofComputer Assisted Learning, 18(1), 84–92.

Mwanza, D. (2002b). Towards an activity-oriented design method for HCI research andpractice. PhD thesis, The Open University, Milton Keynes.

Papadimitriou, I., Tselios, N., & Komis, V. (2007). Analysis of an informal mobile learningactivity based on activity theory, In G.N. Vavoula, A. Kukulska-Hulme, & N. Pachler(Eds), Proceedings of Workshop Research Methods in Informal and Mobile Learning(pp. 25–28). London: Institute of Education.

Sariola, J., Sampson, J.P., Vuorinen, R., & Kynäslahti, H. (2001, May 2–5). PromotingmLearning by the UniWap Project within higher education. In Proceedings ofInternational Conference on Technology and Education (ICTE 2001), Florida StateUniversity, Tallahassee, FL. Retrieved September 8, 2010, from http://www.icte.org/T01_Library/T01_254.pdf.

Scanlon, E. & Issroff, K. (2005). Activity theory and higher education: Evaluating learningtechnologies. Journal of Computer Assisted Learning, 21(6), 430–439.

Sharp, H., Taylor, J., Lober, A., Frohberg, D., Mwanza, D., & Murelli, E. (2003, 29 Septem-ber–1 October). Establishing user requirements for a mobile learning environment. Paperpresented at the Eurescom Summit, 2003, Evolution of Broadband Services, Heidelberg,Germany.

Sharples, M. (2005, April). Learning as conversation: Transforming education in the mobileage. Paper presented at the Conference on Seeing, Understanding, Learning in the MobileAge, Budapest, Hungary.

Sharples, M., Jefferey, N., du Boulay, J.B.H., Teather, B., & du Boulay, H.H. (2002). Socio-cognitive engineering: A methodology for the design of human-centred technology.European Journal of Operational Research, 136, 310–323.

Sharples, M., Taylor, J., & Vavoula, G. (2007). A theory of learning for the mobile age. InR. Andrews & C. Haythornthwaite (Eds), The Sage handbook of e-learning research(pp. 221–247). London: Sage.

Taylor, J. & Evans, D. (2005). Pulling together: Keeping track of pedagogy, design andevaluation through the development of scenarios – a case study. Learning, Media andTechnology, 30(2), 131–145.

Taylor, J., Sharples, M., O’Malley, C., Vavoula, G., & Waycott, J. (2006). Towards a taskmodel for mobile learning: A dialectical approach. International Journal of LearningTechnology, 2(2/3), 138–158.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014

Open Learning 249

Vavoula, G.N., & Sharples, M. (2007). Future technology workshop: A collaborative methodfor the design of new learning technologies and activities. International Journal ofComputer Supported Collaborative Learning, 2(4), 393–419.

Vavoula, G. & Sharples, M. (2009). Meeting the challenges in evaluating mobile learning: A3-level evaluation framework. International Journal of Mobile and Blended Learning,1(2), 54–75.

Wali, E., Winters, N., & Oliver, M. (2008). Maintaining, changing and crossing contexts: Anactivity theoretic reinterpretation of mobile learning. ALT-J, o16(1), 41–57

Waycott, J. (2005). Appropriating tools and shaping activities: The use of PDAs in the work-place. In L. Hamill, A. Lasen, & D. Diaper (Eds), Mobile world: Past, present and future(pp. 119–139). Berlin: Springer-Verlag.

Dow

nloa

ded

by [

Uni

vers

ity o

f L

iver

pool

] at

02:

49 0

9 O

ctob

er 2

014