RECALL Project 2009-504970-LLP-1-2009-1-UK-KA3-KA3MP

Evaluation of route learning software on Android for people with disabilities – Jacqui Lewis, Camelia Popescu, Penny Standen and

Maria Saridaki

The project is partially funded under Key Activity 3: Informational and Communication Technologies (ICT) Lifelong learning programme of the European Commission The content of this project does not

necessarily reflect the position of the Commission, nor does it involve any responsibility on its part

Project Partners: Nottingham Trent University – UK

Greenhat Interactive Ltd – UKBID Services - UK

Marie Curie Association – BulgariaUniversity of Athens – Greece

CPPC – Romania

RationaleEvidence presented by the Disability Rights Commission (DRC) indicates that people with disabilities experience considerable disadvantage in terms of transport and travel. For example:• People with disabilities are unable to use 70 per cent of buses and 40 per cent of

the rail network.• 60 per cent of people with disabilities have no car in the household, compared

with 27 per cent of the general population.• Almost half (49%) of people with disabilities are totally reliant on public transport.• Over half (56%) of people with disabilities said that they would like to go out

more.• People with disabilities are twice as likely to turn down a job due to travel

difficulties.• More than half (52%) of people with disabilities expressed some difficulties in

getting to all essential services such as GPs and hospitals.• The research phase of our project showed that travel skills and confidence to

learn new routes are core skills for leading independent lives; showing users their exact location relative to where they need to get to can help people who struggle with spatial skills.

Project Aims

• To counteract exclusion that occurs when people with learning disabilities leave statutory services, exacerbated by the anxiety of carers about their vulnerability in the community.

• To overcome physical and psychological barriers to independent travel and community inclusion.

• To reengage people into active citizenship• To increase access to community activities by developing

independent travel skills

Objectives

• The project utilizes location based services in assistive technologies to allow people with learning disabilities and sensory impairments to develop lifelong learning skills of: route planning, memory, concentration, stress management, time-management & understanding targets and deadlines.

• Three modes of RECALL allow our target audience to plan and use routes to reconnect with community opportunities; and to play to reinforce the connection between maps and the real world.

Description of Application• Provides a tool for developing and

supporting routes that involves and enables the route learner

• Allows carers and trainers to have additional confidence

• Uses mainstream and contemporary location aware technology

• Provides other travel training elements such as reminding the user when to leave

• Provides geographically based prompts

• Allows specific safety based prompt e.g. at road junctions

• Provides a means to contact a carer in the event of an emergency

• Not a replacement for travel training, but another tool trainers can use to engage and keep users on task

Route Mate vs Satnav approach

Route Mate attempts to use technology to enhance route learning– Route creation is

part of the process– User defines their

own waypoints– Supports cognitive

map formation

Traditional route guidance (location based) turn by turn instruction– Route generation is

automatic– Instructions

presented to the user– Suppresses cognitive

map formation

Route Mate - Console

Using a basic desktop/laptop and internetconnection, the user, parent or carer can:• Load the routes from a device• Preview and edit routes• Duplicate routes• Save/load routes to/from PC• Save new and modified routes to the phone• Delete routes from the phone.

Console

Telephone Start Up

Plan Mode•Plan mode gives the user the option to create a new route, or load and modify an existing route with the help of a parent, carer or trainer. They can enter the start point using selection on a map and can also enter the start time of their journey, set daily alarms and end address of their new route. •An emergency contact can be specified and points of interest can also be set between the start and end points using the phone’s camera to break the journey up into a number of smaller routes connected by key landmarks and more effectively scaffold its learning.

Route Mate ‘Use’ Mode•The user can rehearse the route a number of times accompanied by a trainer or teacher before independent travel. The Use Mode allows the user to travel more independently and rely less on the application and more on their own skills.•To facilitate independent movement, the screen turns off while travelling between points of interest in the Use Mode.•This is important because over reliance might be dangerous for example looking down at the screen whilst crossing a road unaccompanied.

Route Mate ‘Use’ Mode

Challenge Mode - GamificationTo teach/reinforce the concept of maps and route learning, a games approach has been adopted that:• Allows students to insert

different icons to waypoints in planning mode (e.g., treasure)

• The student is then challenged to find this treasure in the real world, using game play around treasure hunts, or pirates, for example.

• Promotes the connection between the map and the real world

Project Evaluation

Completed• Issues Tracking for

functionality• Iterative Testing for

product usability and accessibility

• Piloting• Soft outcomes

measurement

Ongoing/Future:• Scientific Review

• Internal and external impact measurement against key performance indicators

Scientific Review – RoleThe external evaluator will:1. ensure that we are able to demonstrate the scientific

quality of our work so that others can trust our outputs by being able to provide enough evaluation detail to reassure them that our conclusions are sound and to enable them to judge whether our products would be suitable for their client group

2. monitor our use of accepted inclusive design standards and also the use of the issues tracker for bug fixing

3. produce a state of the art review of other developments to test for product innovation

Qualitative Evaluation –internal and external

Further qualitative evaluation will take two approaches to outcome measurement, both of which will measure the delivery of the project against a number of key performance indicators for measurement of sustainability and social impact of the project and its results.•the first of these will be conducted internally with project partners •the second externally with project stakeholders.

Usability Testing methodologyBetween month 9 and month 30 of the project, iterative testing was conducted by all partners using different tools for experts and final users. Tools:• Instructions for deploying testing • List of tasks that users should attempt with Route Mate • Observations sheet to record any usability issues• Template to collect data Throughout this period of testing an issues tracker was used by all partners and experts to record any problems with functionality of the device. This allowed the programmer to bug fix as we went along to keep the process flowing, so that the device would be ready for piloting

Testing Tasks

• Testing with final users was conducted in teams including one assistant (to assist users and to ensure their safety) and one evaluator (to observe and record the data). Spontaneous feedback from the participants and their facilitators was recorded. The participants were given training in the use of the phone and were shown how to navigate the application.

• Relatively quiet and short roads were chosen. A shorter route allowed more time to be spent gathering feedback. The length of routes chosen for testing varied between 200-1600 m and the time spent on the road varied between 10 – 30 minutes.

• At least two testing sessions were carried out with every user and between 5 to 10 users tested the application in each country, following the same set routine of tasks.

Usability Testing ResultsAnalysis of Tasks The main characteristics of Route Mate tested were:• Open/ close the application • Use the virtual keyboard• Use the map (using arrows, using zoom buttons)• Set points• Take pictures• Use panic button• Take a route A 4 level scale for registering answers was used:Level 1= could not complete even with helpLevel 2 = requiring considerable physical or verbal help to completeLevel 3 = requiring a little physical or verbal help (eg prompts) but could otherwise complete the task

unaidedLevel 4 = completed with no help The amount of time spent looking at the device (percent of total time of walking) was:• 22 cases between 70% -90% • 6 cases between 20%-50%.

Testing Results

• Analyzing the main characteristics of Route Mate we found out that for the users the application was in most of the cases easy and engaging to test and only in a few cases were no answers registered because of a functionality error.

• Open/close the application was completed mostly with no help - 20 responses • Use the virtual keyboard was completed mostly requiring a little physical or

verbal help -18 responses• Using the map (using arrows, using zoom buttons) was completed mostly

requiring a little physical or verbal help - 17 responses• Set points was completed mostly requiring a little physical or verbal help - 20

responses• Take pictures was completed with requiring a little physical or verbal help - 16

responses• Use panic button was completed mostly requiring a little physical or verbal

help - 21 responses. (4 non answers were recorded because of errors)• Take a route was completed requiring a little physical or verbal help - 13

responses. (8 non answers were recorded because of errors)

Main areas of physical difficultyTaking pictures was one of the characteristic where the answers varied the most depending on which version of Route Mate was tested. This required considerable physical or verbal help to complete in some cases – 7 responses. Using the virtual keyboard and using the map (using arrows, using zoom buttons) required sometimes considerable physical or verbal help to complete depending of the disability of the users:

• Using the virtual keyboard –required considerable physical or verbal help to complete - 6 responses

• Using the map (using arrows, using zoom buttons) required considerable physical or verbal help to complete - 8 responses

Users’ Typical ReactionsSome examples of reactions and behaviours of users:

• Users weren’t always sure of menu layout and where to find the Route Mate icon.

• Users were unsure how to locate the ‘close keyboard’ icon to be able to progress to the next screen.

• When moving around the map screen in Plan mode, accidental clicking meant the option to add a point kept coming up, which caused confusion with users

• Some users found it difficult to know where they were on the map while taking a route and, therefore, how to proceed.

• Difficulties with using the keyboard due to fine motor limitations. This cannot be rectified within the project – it is a hardware issue.

Functionality

• An online issues tracker (Red Mine) was used to allow the software engineer to be constantly updated on functionality problems encountered during iterative testing.

• These issues were then prioritised and bugs fixed accordingly.

• Testers were then informed as each issue was dealt with and signed off.

Issues Tracker

Issues – Bug report

Functionality – Main Issues reported

Main issues have been: - Crashes / freezes - usually a coding mistake. Crash reporting tool (ACRA) integrated into the app to allow analysis of the events leading up to the crash - Photos not being recorded - only occurred on some devices, not others. Image capture code was rewritten. - Off route warning not consistently working - code corrected.

Quite a few issues were determined to be device specific, so it was decided to pilot with a consistent configuration. Galaxy Nexus devices were used as they have an up to date version of android, and don't have any manufacturer (e.g. Sense UI (HTC) or TouchWiz (Samsung)) or carrier (e.g. hard coded home page, Telco apps) additions to the operating system.

Piloting TasksSCENARIOOutline Scenario• 3-5 Sessions• 5-10 end users

Piloting Scenario1st Session: Training and first use of Route Mate• Training by the researchers - max 15 mins• Soft Outcomes Star™ used by End Users (physical disabilities) or caretakers

(learning disabilities) - max 15 mins• Piloting Process - Participant observation • Familiarize with the software• Create first route on console - by facilitator/end user• Use first route - by facilitator/end user/researcher• End of route, return to the premises - by facilitator/end user/researcher• Debriefing Process - max 15 mins

Piloting Tasks contd2nd Session: Semi-Autonomous use• Piloting Process - Participant observation • Use existing route - by facilitator/end user/researcher• End of route, return to the premises - by facilitator/end

user/researcher• Debriefing Process - max 15 mins

End Session - Semi-Autonomous (if possible)• Piloting Process - Participant observation • Use existing route - by end facilitator/end user/researcher• End of route, return to the premises - by facilitator/end

user/researcher• Soft Outcomes Star™ used by End Users (physical disabilities) or

carers (learning disabilities) - max 20 mins• Observational checklist - Likert Scale - max 20 mins

Observational checklist - Likert Scale

• Documents feedback - open questions• Closed type questionnaire with interval scales• Invites open comments from both end users and

facilitators

SOFT OUTCOMES STAR ™ tool

The soft outcomes star™ is a tool to measure and record the difference a service/training has made to clients by showing progress made in qualitative areas of life across a period of time such as from the beginning to end of a course delivered.

The tool is used with clients in a non-threatening way to help diagnose where they might need more support as well as what they have achieved. It can also be used as evidence of value of a project or course and to demonstrate a return on their investment to project funding agencies in areas that are hard to quantify.

Piloting Results – Likert Scale•Carers and users were very positive about Route Mate. •General attitude towards Route Mate was positive even in cases of technical issues•Users are willing to use Route Mate when the app has been finalised•Most of them credited Route Mate with an increase in confidence and in helping relatives and families to be reassured that users are safe when travelling independently using the app. The panic button was one of the most important features for the family

Piloting Results – Likert Scale•There were insignificant differences in the way caretakers (carers, teachers, trainers, support workers) answered the questionnaire. •The only demographic characteristic that was found to be important is experience in technology. •Experienced caretakers found Route Mate to be usable and easily accessible and had more positive attitude towards it. •Users that had previous experience with technology scored better •All categories of users showed significant improvement in the second measurement•Older users performed worse in the first measurement but improved more than the other categories in the final measurement•Impact was the same across disability categories

Piloting Results - Star Tool and Observation

• Users with intellectual disabilities show no qualitative differentiation in contrast with individuals with other kinds of impairments/disabilities, providing us with an important indication that Route Mate can be an equally useful learning and assistive tool for users with different disabilities

Piloting Results – Star Tool

Star Tool: All categories increase significantly in the second measurement

Piloting Results•During the piloting sessions and as the technical problems started to slowly resolve:–Stress levels decreased–Users’ confidence increased–Users felt encouraged and motivated to use Route Mate.

•As expected, previous experience with mobile technology had an effect on users’ self confidence and therefore their ease with Route Mate.

•Games and game based learning elements reduce the steepness of the learning curve and increase satisfaction and motivation

Piloting ResultsReasons that increased the levels of stress:•Technological problems •Disbelief about technology’s reliability in general•Feelings of frustration about not being able to access the right technology (smartphone) because of prohibitive costs and accessibility •Risk (fear of being victimized when holding an expensive mobile)According to Soft Outcomes Star™ findings, these stress levels can be reduced through repeated use of Route Mate.

Piloting Methodology Results - ImpactEnd users could clearly see the immediate impact that Route

Mate could have on their lives:• “Route Mate will be really helpful next year when I go to

college” • “Will let my mother know that I can get there safely”, End users could see the potential that Route Mate could have

on expanding and improving their daily lives:• “I can see my friends more often” • “I can learn new and unfamiliar routes”

Piloting Methodology Results - Motivation

• Motivation to use Route Mate was also increased) by the “Where to go Next?” screen and the photo gallery, which according to most of the users made the route more interesting and enjoyable

• Motivation from using Route Mate: two cases where the use of Route Mate prompted them to improve themselves or at least triggered the desire to improve themselves. One student with learning disabilities who was unable to read or write. After the first session with RM she expressed to her parents the desire to learn how to read. In a second case, after a RM session, the end user started to search on the internet for similar programs and assistive technologies.

Piloting Methodology ResultsBugs and optimization:•The intermediate stops have to be actual places, things that the end users can recall instantly and not commands or actions. •The pop up info about the next stop has to appear before the users reach a street, otherwise they tend to freeze in the middle of a street raising important safety issues. •For some students with sensory impairment or fine motor control issues, the keys were too small •Individuals in wheelchairs could not use Route Mate while moving. They had to stop, use the application and then continue their journey

Piloting Methodology ResultsProposals for further improvement:• To add a voice command or to be able to

activate audio commands• To add haptic feedback as an alternative for

people with visual impairments• To enhance the more game-like aspects of

the app.

Thank you for listening!

RECALL team

http://recall-project.eu