Taking Care of Our Elders Through Augmented Spaces

Edgardo Aviles-Lopez, Ismael Villanueva-Miranda,J. Antonio Garcıa-Macıas

Computer Science DepartmentCICESE Research Center

Km. 107 Carretera Tijuana-EnsenadaEnsenada, Baja California, Mexico{avilesl, ivillanu, jagm}@cicese.mx

Luis E. Palafox-MaestreFacultad de Ciencias Quımicas e IngenierıaUniversidad Autonoma de Baja California

Calzada Universidad #14418, Mesa de OtayTijuana, Baja California, Mexico

lepalafox@uabc.mx

Abstract—As the population ages worldwide, more and moreefforts are being made toward providing the best conditionspossible for our elders. In these efforts, technology plays a veryprominent role. We present our proposal for ambient-assistedliving spaces where an underlying sensorial infrastructure,coupled with sentient visors, can aid individuals in many dailytasks that promote a safer and healthier lifestyle. We alsodiscuss some scenarios based on our contributions that servesas a showcase of what can be accomplished.

Keywords-ambient assisted living; sentient visors; ubiquitouscomputing

I. INTRODUCTION

In the course of the next years, worldwide populationwill constantly increase its average life expectancy. We canexpect elderly forming a large part of the population as wecan witness in some countries. Elders have special needsgiven the fact that the aging process will come with aconsiderable decline in capabilities making them prone tofalls, forget things, and others. As technology advances sodo the opportunities to come up with new tools that candrastically improve the lifestyle of our elders. Along withthe particular requirements in assisting the elders, we mustthink in how they will interact with the tools and howto take advantage of the solutions available today so ourcontributions can be applied in the short term.

We start this article by discussing ambient assisted living;its scope, benefits and applications. Then, in Section 3 weintroduce sentient visors, a technology designed to interactwith services and applications in simple and easy means;in particular, its approach to elder users. In Section 4 wetalk about our sensorial infrastructure for helping the elders.Later in Section 5, we exemplify those concepts throughsome scenarios in which we are currently working. Finallywe present some conclusions and future work.

II. AMBIENT ASSISTED LIVING

Ambient Assisted Living (AAL) is a research area thathas been gaining importance recently. Its main goal is toapply ambient technology to assist individuals in performingeveryday activities. The social impact of AAL grows even

Figure 1. Homecare system domain.

more if it is applied on scenarios that involve physicallyimpaired individuals and new alternatives on helping themto maintain certain degree of independence [1].

This technology opens the possibility to implement appli-cations that offer specialized services according to specificapplication scenarios such as: assistance for daily activities,health monitoring, emergency systems and many others.According to Oppermann [2], AAL represent a new gen-eration of systems that must comply with the followingrequirements:

• Invisible (embedded in clothing, watches, eyeglasses)• Mobile (ability to be transported by the user)• Spontaneous (ability to communicate dynamically with

many nodes)• Heterogeneous (the system must integrate several dif-

ferent types of nodes)• Context-aware (ability to interpret user actions)• Proactive (ability to infer behavior according to the

activities of the user)• Natural communication with the user, such as voice and

gestures.

• Adaptive (ability to react to uncommon situations thatmay occur)

In Figure 1, we show the three types of assistance inthe homecare system domain introduced by Kleinberger [1],each of which presented as a subset: emergency assistance,autonomy enhancement, and comfort.

AAL systems development has focused on improving thequality of life on individuals with health problems and/orelderly people. In medical environments, AAL systemscan provide assistance to patients with different kinds ofcognitive impairments such as Alzheimer’s or Parkinson’sdisease.

In elderly individuals, improving life quality through AALenvironment is a goal that has been pursued recently [3].This goal has gained greater importance because of theworldwide increase in average life expectancy [4]. For thisreason, the assistance of elderly individuals for activitiesof daily living (ADL) has become an important socialproblem that can be solved (presumably) through the useof technology-augmented spaces such as AAL systems.

AlarmNET is an AAL system designed to provide as-sistance through a smart healthcare environment. It specifi-cally targets assisted-living residents and others whom maybenefit from continuous remote healthcare monitoring [5].AlarmNET’s network manages an audit trail and continu-ous medical history while preserving resident’s autonomy,comfort and privacy. Thus, improving security and qualityof life. The AlarmNET system [6] has been implemented asa network of MicaZ sensor nodes, Stargate gateways, iPAQPDAs and PCs. Customized infrared motion, temperature,light, pulse and blood oxygenation sensors were used. Soft-ware components include: a TinyOS query processor andsecurity modules for the motes; AlarmGate, an embeddedJava application for power management, privacy, security,queries and client connections; Java resident monitoring andsensor data applications for PDAs and PCs; and a circadianactivity rhythm analysis program.

III. SENTIENT VISORS

To further accomplish ambient assisted living we need tointegrate many different devices into the environments of theusers. But with this seamless integration of devices a newchallenge of human-computer interaction arises. This is, howcan users interact with devices that are “invisible” to them?And they are not just physically invisible but cognitivelyinvisible as well. We also need to consider the scenario inwhich users of an augmented environment need to knowwhat augmented physical objects are available for them touse, where they are, and once they encounter one of those,how to communicate with them. The interaction problemis further complicated if we talk about users of advancedage, as they commonly present difficulties for interactingwith traditional software applications; so the migration from

Figure 2. Scenario of use and anatomy of a sentient visor.

an already known paradigm of explicit interfaces to inter-action with augmented objects may not be trivial. Eldersneed a clear and simple interaction mechanism that, takingin account their personal profile, presents them with theinformation they need when, where, and straightforwardlyas they need it.

For instance, an elder person with diabetes may need toknow the sugar contents of a particular food. For this user,the vitamins, minerals, fat, and other food elements are notas important as the sugar is. Even more, it would be usefulto have a mechanism that allows the environment to knowthe personal dietary history and profile of the user, so itcan present her with the digested food characteristics, aningestion recommendation according to the diet indicationsgiven by his physician and stored in her profile.

Ambient displays are based on the idea of augmentedenvironments and continuously present information in a verynatural and simple way [7]. Those devices show informationby physical means such as lights, sounds, images, and others.Users just need to have a glance or listen to them to benotified of an important incoming message or other infor-mation, so the interaction is very simple and straightforward.There is no need to push a button, select a menu, enter acommand, nor any of the mechanisms present in traditionaluser interfaces, so they are very appropriate for elders.

Following the idea introduced by ambient devices, inour research group we have been working in what wecall sentient visors. Sentient visors are user-carried devicesthat have a screen and a camera (so for instance, a tabletcomputer or a smartphone could be currently used as asentient visor [8]). Those devices enable the interactionwith augmented objects; to start an interaction the user justhas to point the sentient visor to the object and the actualinteraction takes place through an augmented reality scenariorendered on top of the video feed captured by the camera.

In Fig. 2 we can see the anatomy and scenario of use

of the sentient visor concept for our previous example. Toaugment a physical object (in this case a package of food),we just need to attach an automatic identification tag (suchas bar codes, RFID, or others) and an augmented realitytag, so there is no need for complex modifications on thesubjects to be digitally enabled. Once the visor identifies theobject been pointed at, it asks the environment for the sceneto draw which will provide the needed interaction. Finally,the user is informed of the food characteristics through agraph and other indicators in 3D drawn over the video inthe sentient visor. As the visor is not tied to the applicationand asks for what to draw, it can be extended to fit manydifferent situations and application needs.

IV. AUGMENTED SPACES FOR HELPING THE ELDER

There are two main technical components in our proposal:a sensorial infrastructure and sentient visors. In the followingsubsections we will detail how they work, and then in sectionV we will show through scenarios how they can be integratedto help in activities of daily life.

A. Sensorial infrastructure

Even when living at a carefully designed elderly-safehome, our elders are always at risk and accident-pronedue to their ever decreasing abilities. We think that severaltechnological elements can help reduce these risks and avoidaccidents, or if accidents happen, caretakers can have a fasterresponse. We have designed and implemented a platformbased on the architecture shown in Figure 3, includingmodules to support the following functionalities: indoorlocation, video monitoring, behavior analysis, fall detection,voice commands, and decision-taking. We will now give adescription of each of these modules.

The indoor location module estimates the user’s positionbased on the signal’s intensity of surrounding Wi-Fi accesspoints. Based on the received signal, a neural networkperforms location estimation. The neural network has to betrained before the location service can be used, training isdone by measuring Wi-Fi signal strengths at different pointsin the house and associating the measured signal strengthlevels with a specific location. The service has a functionthat enables the neural network to be trained by receivingthree inputs: data (MAC address, signal strength), latitudeand longitude. In our implementation, we took over 4,300readings across 50 different locations using a laptop HPDV4000 as a client with an Intel IPW2200 wireless networkcard. On the client-side, we installed a Java application tosend the information to the server. This information includesa dataset consisting of the MAC address and Wi-Fi signalstrength (i.e., 00:13:46:f2:fc:b9,-67), latitude andlongitude. Then using a k-means method, the readings wereclustered to have a more localized reading. Once we hadthe clusters, these were used as input to a neural network.The goal we pursued when we did this was to increase

Figure 3. Our sensorial infrastructure.

the precision and reduce the processing time of locationestimation.

Voice commands provide a very natural way of interac-tion where user effort is minimized. In our platform, wehave implemented a subsystem where a sensor networkcaptures sound, determines if it is within the human voicerange, and if it is then forwards it to a base station (afull-fledged computer) to analyze it and determine if it isa command. For the implementation of this subsystem, wehave installed a wireless network of small and cheap nodeswith sound sensors. Reliability is an important factor, as it isimportant not to miss a command issued by the user, so wehave deployed redundant nodes where sensing areas overlap;we made this decision based on the facts that nodes arefailure-prone and have a low price. By doing this, if a noderandomly fails we have other ones that could perform itscapture and transmit commands without disrupting overallsystem functionality, thus assuring reliable voice capture. Ina previous article we gave more in-depth details about theimplementation and experimental results of using this voicecommand subsystem [9], so we will not repeat them herefor the sake of brevity.

The activity detection module offers the possibility toinfer certain low-level activities through the use of dataprovided by an accelerometer sensor. The activities arecategorized as follows:

• standing up (from a seated position)• seating down (from a standing up position)

• walking• climbing steps• laying down (from a seated position)• seating down (from a laying down position)• falling downActivity detection is a very crucial task for overall system

functionality, it actually starts most of the higher levelservices provided. For instance, if certain activity is detectedwithin a predetermined danger zone, this module sends asignal to the decision-taking module (which we will describelater) in order to take an appropriate action.

Given the fact that during the aging process, there isa considerable decline in motor capabilities, older adultsbecome more prone to falls. Thus, it is important to note thatfall detection is a very important function that is performedin the activity detection module that we mentioned earlier.Falls can occur in a variety of ways, whichever the case maybe, this module must send a warning signal to the videomonitoring and to the decision-taking module.

The video monitoring system serves as an auxiliarymethod for other systems. For instance, if the fall detectionsystem indicates that there is a high probability that the userhas fallen, an alert is issued to the caretaker and a video feedis activated, so it can be corroborated if the fall effectivelytook place or if it was a false alarm.

Dependent on the other modules, a behavior analysismodule makes inferences, takes proactive actions, and issuesalarms regarding the behavior of the users. Behavioral anal-ysis includes determining patterns of conducts to facilitateproactivity. For instance, suppose that a person wakes updaily at about 7 AM, issues a voice command to open thecurtains, proceeds to the bathroom and issues a command tohave running water for taking a shower, and stays for roughly30 minutes in the shower; then all these series of dailyrecurrent events can serve to indicate the system to open thecurtains and have the water running when it is appropriate(also, taking into account location and movement). Also,if something deviates from the expected behavior, e.g., itis already 8:30 PM and the person is still in bed, then analarm is issued and the caretaker is informed to take theappropriate actions.

The decision-taking module is arguably the most impor-tant one; this is because this module is in charge of takingspecific actions that support target users. The decision istaken considering input signals provided by other modules.Once the decision has been taken, this module sends theappropriate signal to a specific actuator in order to performa specific task, i.e. opening a faucet, making a phone call,sending a SMS message, activating an alarm.

V. SCENARIOS

We will now present some scenarios that illustrate howour platform can aid in many situations that can promotesafer and healthier lifestyles for our elders.

Figure 4. An aid for taking medications.

A. Taking medications

People who need to take many medications often strugglewith remembering schedules and amounts of medication totake. Sometimes a medication is not taken simply becausethe person did not remember to do it, other times a medica-tion is taken more than once, or not in the right amount.All these situations can lead to adverse, sometimes veryserious consequences. In fact, polypharmacy is known tocause many problems worldwide [10].

Mr. Favela is a typical senior of advanced age who needsto take many medications daily. However, he often forgetsto take these medications, sometimes he forgets which oneshe already took and takes the same one twice or thrice,etc. Being concerned with this situation, his son livingabroad gave him a cell phone that not only allows him tohave conversations more often with his father, but also hasinstalled sentient visor software for helping with medicationintake. Now, when in doubt of which medication to take,Mr. Favela just needs to point at the bottle to see if it isthe right one: the right bottle will show a green glow andthe amount to take, while a wrong bottle will show a redglow and a clock counting backwards to the time for thenext dose, as depicted in Figure 4.

The software also features audible alarms, so when Mr.Favela hears an event, he can point his cell phone at thebottles. Mr. Favela’s physician can remotely use the samesoftware to change the dosis by altering the number of pillsto be taken or by setting a different schedule.

The medicine bottles have two important elements: anRFID tag to help retrieving the schedule and dosage infor-mation, and a label on top of the cap for orientation andhelping in drawing the glow and other visual informationpresented in the display.

Figure 5. Dietary care showing the nutritional values and recommendation.

B. Dietary care

Due to her diabetic condition, Mrs. Tentori needs to bevery careful with her diet. With this in mind, she downloadeda system for her iPod Touch that allows her to see thenutritional contents of products she is about to buy or eat.

When she is at the supermarket, she points at cans tosee graphic charts indicating the nutritional values (e.g.,carbohydrates, fat, sugar, etc.) of the products, which canallow her to make smarter buying decisions. Since she hasslight visual impairments, it is always difficult for her toread directly at the small letters of the labels in the cans; sobig and colorful charts provide her with better at-a-glanceinformation (see Figure 5). Later during the day, she meetssome friends for dinner at a restaurant. When she’s presentedthe menu, she points at it with her iPod to get an augmentedview where, as she picks some items, a table adds the amountof calories and sugar contents of her selection; if somevalue goes above her allowed daily intake, a visual alarmis presented. That way she knows what items she can takefrom the menu without incurring dietary problems. All thisdata is always remotely available so if her physician needsto know further about the history of the food she ingestedhe just needs to open his Web browser.

C. Indoors guide

During her frequent visits to health institutions, Mrs.Martinez gets disoriented and needs assistance in findingthe doctor’s office, the in-hospital pharmacy, etc. Luckilyfor her, many of the public institutions she goes to haveaugmented systems to help her.

Her doctor sent her to a clinic for specialized studiesregarding a kidney condition she has. Once at the clinicshe sees a “welcome” sign at the entrance (which has beenaugmented with a special label) and points at it with her cellphone. Once her location has been determined, a window is

Figure 6. Interactive indoors guide.

displayed asking if she allows the ambient system to retrieveselected information from her medical record, she consentsand she is instructed to point at the aisle to her left, whereshe will be directed to Dr. Moran’s office. She then seesan arrow (similar to that in Figure 6) drawn on top of thereal-time video feed she is getting from the phone’s camera,and as she follows the arrow she gets to the right office.

If there is a problem, and Mrs. Martinez suffers a fall, thesoftware is immediately aware of this and sends an alert tothe hospital crew with the location and estimated urgencyof the accident.

D. Physician’s aid

During her morning rounds visiting patients at the hospi-tal, Dr. Lugo always carries with her a tablet-sized computer.With this device, once she stands in front of the patient’s bedshe is able to view, overlayed on top of the live video feed,different types of charts and graphs indicating the patient’sbody temperature, heart rate, blood pressure, as well as inforelated to the treatment he is receiving.

VI. CONCLUSIONS AND PERSPECTIVES

Parting from the ambient assisted living concept, wehave strived towards an augmented space that combines asensorial infrastructure and sentient visors. We believe ourproposals will contribute in improving the lifestyle of theelderly. At the moment, we are developing proof of conceptsbased on the scenarios discussed in this article using a work-ing prototype of our platform. We are also exploring newscenarios and applications. We plan to conduct adoption andusability tests with a group of elder users to obtain feedbackon how effective and appropriate are our contributions.

Although there are many useful technologies available tosupport an augmented space (e.g., WSN, RFID, AR, andothers), there is a need for mechanisms to better detect and

communicate the activity of the users. By instance, in ourmedications scenario, we would highly increase the value ofour proposal by knowing if the user took or not the pills theapplication said him. In this way, we could alert his familyand physician of his behaviour, or we could auto-request thepharmacy for medication resupply.

ACKNOWLEDGMENT

The work presented in this paper was funded with grantsfrom CONACyT, the Mexican Council for Science andTechnology.

REFERENCES

[1] T. Kleinberger, M. Becker, E. Ras, A. Holzinger, andP. Muller, “Ambient intelligence in assisted living: Enableelderly people to handle future interfaces,” in UniversalAccess in Human-Computer Interaction. Ambient Interaction,2007, pp. 103–112.

[2] R. Oppermann, R. Rashev, and K. Kinshuk, “Adaptability andadaptivity in learning systems,” 1997.

[3] D. Hanak, G. Szijarto, and B. Takacs, “A mobile approach toambient assisted living,” in Proc. IADIS Wireless Applicationsand Computing, Lisbon, Portugal, 2007.

[4] United Nations. (2009) World population prospects the2008 revisions. [Online]. Available: http://www.un.org/esa/-population/publications/wpp2008/wpp2008 highlights.pdf

[5] G. Virone, A. Wood, L. Selavo, Q. Cao, L. Fang, T. Doan,Z. He, R. Stoleru, S. Lin, and J. A. Stankovic, “An assistedliving oriented information system based on a residentialwireless sensor network,” in Proc. 1st TransdisciplinaryConference on Distributed Diagnosis and Home Healthcare(D2H2 2006), Arlington, VA, USA, 2006, pp. 95–100.

[6] A. Wood, G. Virone, T. Doan, Q. Cao, L. Selavo, Y. Wu,L. Fang, Z. He, S. Lin, and J. Stankovic, “ALARM-NET:Wireless sensor networks for assisted-living and residentialmonitoring,” Department of Computer Science, University ofVirginia, Tech. Rep. CS-2006-11, 2006.

[7] J. Favela, M. Tentori, D. Segura, and G. Berzunza, “Adaptiveawareness of hospital patient information through multiplesentient displays,” International Journal of Ambient Comput-ing and Intelligence, vol. 1, no. 1, pp. 27–38, 2009.

[8] A. Wright, “Get smart,” Communications of the ACM, vol. 52,no. 1, pp. 15–16, 2009.

[9] L. E. Palafox-Maestre and J. A. Garcıa-Macıas, “Wirelesssensor networks for voice recollection in ubiquitous homeenvironments,” in International Symposium on Wireless andPervasive Computing (ISWPC 2009), Melbourne, Australia,Feb. 2009.

[10] L. Baro-Manuela, “Deterioro de la movilidad fisica y con-tinuidad de cuidados,” in Comision para el desarrollo de laenfermerıa en el servicio andaluz de salud (CODAE), Apr.2002, vol. 3, no. 2.