Designing A Smart Shopping-Aid System Based on Human-Centered Approachimpact.asu.edu/cse494sp09/Smart Shopping.pdf · · 2009-04-05Designing A Smart Shopping-Aid System ... his/her

Designing A Smart Shopping-Aid System

Based on Human-Centered Approach

Hiroshi Tamura1, Tamami Sugasaka2, Kazuhiro Ueda3

1 R&D Division, Hakuhodo Inc.

4-1 Shibaura 3-Chome, Minato-Ku

108-8088, Tokyo, Japan

tamdai AT userstudy.org

2 Fujitsu Laboratories Ltd.

1-1 Kami-Odanaka 4-Chome, Nakahara-Ku

211-8588, Kawasaki-Shi, Japan

sugasaka.tamami AT jp.fujitsu.com

3 Graduate School of Arts and Sciences

The University of Tokyo

8-1 Komaba 3-Chome, Meguro-Ku

153-8902, Tokyo, Japan

ueda AT gregorio.c.u-tokyo.ac.jp

Abstract We introduce our human-centered approach for de-

signing a ubiquitous computing system which aims at providing

a better experience for shoppers at a supermarket. We investi-

gated grocery shopping processes by using ethnographic research

techniques and understood the processes with details. Further,

we constructed Three-Phase Model, or TPM, which describes a

shopper’s behaviors and various states of mind which can be dis-

tributed into three phases. We also described a user scenario in

each phase which successfully contributed to the system design

by giving a clear picture of user experiences.

1 Introduction

Retail outlets are a promising area of application for ubiquitous computingsystems. There have already been a variety of systems developed notonly for research purposes but also for business purposes [1]. One of themost famous cases is the Extra Future Store by Metro AG (http://www.future-store.org/). The store has employed a variety of embeddedand mobile computing systems to improve a shopper’s experiences duringhis/her shopping trip within the store as well as to track and managegrocery store inventory both from the distribution center and within thestore [2]. From the view of a shopper’s experiences, he/she has come toexploit the services as everyday activities, for instance, some 24 percentof shoppers utilize PSA (Personal Shopping Assistant), which is a mobile

Hiroshi Tamura, Tamami Sugasaka, Kazuhiro Ueda “Designing A SmartShopping-Aid System Based on Human-Centered Approach” Vol. I No. 3 (Dec.

2007). ISSN: 1697-9613 (print) - 1887-3022 (online). www.eminds.uniovi.es

Hiroshi Tamura, Tamami Sugasaka, Kazuhiro Ueda

kiosk terminal embedded with a shopping cart, and some 48 percent usethe interactive kiosks furnished at a constellation of places throughout thestore including the produce, meat, and wine sections [2].

On the other hand, many shoppers have a feeling of distaste towardthe service automation at a storefront due to the anticipation that it willfurther decrease human-based interaction services [1]. One of the reasonsyielding such disaffection is that the services provided by the systemsentirely fulfill the shoppers’ expectations. For instance, several systemsproposed in this area adopt shopping list management as a principal ser-vice to enhance a shopper’s experience [3] [4] even though every shopperdoes not necessarily buy items strictly according to his/her list. Rather,as [5] revealed, 93 percent of shoppers do not shop exactly by their spec-ified lists. It was also noted that, ”on average, purchases by list shoppersexceeded the categories on their lists by about 2.5 times.” Then whatare the promising alternatives? We consequently determined that we hadto investigate shoppers’ behaviors and state of mind changes during theirshopping trips in order to uncover evocative design information that show-case worthy experiences for shoppers.

Since 2003, we have promoted an empirical design project of ubiquitouscomputing systems at a supermarket, named SSA, or Smart Shopping-Aid. We have secured an operating store, Bonrepas Momochi-branch(http://www.bonrepas.co.jp/) in Fukuoka City, located in the south-west region of Japan, as the project site. We had anticipated SSA to be atruly human-centered design project. Therefore, we conducted a generalsurvey regarding grocery shopping both in a quantitative and qualitativemanner, thorough fieldwork, moment-to-moment analyses, and systemsdevelopment and deployment according to the precedent research results.

In this paper, we introduce our design process and implications whichwill contribute to intrinsically useful ubiquitous computing systems atsupermarkets.

2 Investigation in Shopping Process

Shopping process is a long-lasting research topic in retail marketing. Ac-cording to Takahashi [6], about 70 percent of items at a supermarket andabout 80 percent at a supercenter were bought without preexisting plans.Meanwhile, according to our survey conducted in the Tokyo metropolitanarea in 2005, respondents who were homemakers1 ranging in age fromtheir 30s to 40s answered that about 62 percent of them mostly planned

1The leaders of grocery shopping in Japan are married women, notably homemakers.Furthermore, about 80 percent of them answered to our survey that each of themshopped at a certain grocery store at least once a couple of days. This implies thatthe major objective of her grocery shopping was not to do bulk buying but rather buythings for her mael arrangement for that particular day. Under these cultural contexts,we should note that our research focused on the shopping processes of homemakers withwhich they arranged their dinner menus on a day-to-day basis.

56

Designing A Smart Shopping-Aid System Based on

Human-Centered Approach

Figure 1: Influential factors of dinner menu planning; Bars with verticalstripes are relevant to ”planning at home” and with horizontal stripesare relevant to ”planning at storefront”. The survey was conducted inthe Tokyo metropolitan area in January 2005. (Sample size 486, multipleanswers allowed of this question)

their dinner menus at home and about 27 percent at storefronts [7]. Thesetwo facts implied a question: ”Why are grocery shoppers buying so manyunplanned items even though a majority of them already had preset itemsthey were determined to purchase?” We speculated that grocery shopperstended to gradually articulate their plans along with their shopping tripswhich were neither necessarily limited to at-home nor in-store but ex-tended to a consolidation of both. One of the reasons came from anotherresult of our survey which showed that there were almost equal affectsof major factors from both sides to their dinner menu planning (Figure1). Past works had already pointed out that there existed a same kindof phenomenon, although the mechanisms were still at all unapparent[6]. We, therefore, decided to investigate in shopping processes which hada wide distribution from at-home to in-store, and conducted thoroughethnographic research on them.

2.1 Research Design

In-depth research was conducted individually with nine informants (twoof which were for pilot studies) from August to September 2005. Theinformants were all female homemakers, ranging in age from their 30sto 50s, and chosen from customers of Bonrepas Momochi-branch. Weinformed them that we would observe their holistic shopping processesfor dinner arrangements, and requested that they shop as they normallywould. The research procedure consisted of four steps described as follows:

1. Pre-interview at home (30-60 min.): interview with an informantregarding her rituals and attitudes of everyday grocery shopping;checking whether she had a shopping plan or not; and observing herpreparation for the expedition, i.e. looking in the refrigerator,

57


Figure 2: Encolpia: our original wearable video recording system equippedwith 150-degree wide vision CCD. The system was designed as minimally-invasive to informants. The bottom right is a snapshot of recorded images.

2. Participant observation at the storefront (20-50 min.): accompany-ing an informant on her shopping trip on an entrance-to-exit basisby using contextual inquiry techniques [8],

3. Participant observation at home (30-120 min.): describing the in-formant in the home-situation regarding storing, using, processing,and cooking purchased items with contextual inquiry techniques,and

4. Post-interview at home (30-60 min.): debriefing of what was notclarified during the previous steps.

We also asked each informant to wear a video recording system duringthe participant observation at the storefront for posterior analysis. Thecamera, named Encolpia (Figure 2), had originally been developed for thispurpose, with the feature of 150-degree wide vision and real time MPEG-4encoding functions.

2.2 Findings from the Observations

A series of observations uncovered that a shopping process at the store-front was never uniform but the process dynamically changed correspond-ing to each shopper’s context; her behavior and state of mind abruptlytransformed along with her shopping trip. Even if she had her shoppinglist, she never did just a rundown of it. Rather, it was used as one of theartifacts including articles, price tags, and in-store signs, with which sheiteratively interacted to articulate her plan until the end of the shopping.We discovered that there were generally three phases across two majorcontext shifts in the process. Observed cases which implied existence ofthe first phase are shown below.

58



– Right after starting her shopping, an informant directly went to thedeli floor, closely looking at some items, saying ”they’re very helpfulto plan my dinner menu as well as to know how to cook them!” (aninformant in her 30s)

– As an informant already had planned her dinner menu at home bydrawing upon a magazine article, she just looked around the meatfloor right after initiating her shopping, then went to the producefloor and started choosing items referring to the assortment of themeat floor in her memory. (an informant in her 40s)

– (An utterance of an informant) ”I basically start shopping from whatI don’t have to forget to buy”. (an informant in her 40s)

There seemed to exist a warm-up phase right after initiating her shop-ping regardless of whether she had her plan or not at the moment. In thisphase, she mainly replenished what she had already recognized as well asdeveloped her plan for a main dish of the day. She was so enthusiastic tobuy what she had to buy without omission that, we could speculate, shefelt pretty tense during this phase.

Observed cases which implied existence of the second phase are shownbelow.

– (An utterance of an informant) ”Because I had decided to cookhashed rice as today’s main dish, I wanted to choose an appropriatepackaged beef for it. I compared several packages with each other,and decided to choose this one because this seemed the most fresh.”(an informant in her 30s)

– (An utterance of an informant; after picking up a package of aroids)”As I have decided today’s main dish packed a hard punch, I thinkboiled aroids with soy and sugar as a side dish will go nicely withthe main dish.” (an informant in her 40s)

This phase was the closest to what we had conceived as the typicalgrocery shopping: Each informant implemented her plan by choosing andpicking up specified items. In this phase, she fulfilled her plan which hadbeen made in the previous phase as well as developed and carried outher plan for side dishes which went nicely with the main dish. We couldsee that she concentrated on how she could suffuse her plan during thisphase, e.g. to buy higher quality items with cheaper prices. We couldalso confirm that her state of mind in this phase got less tense than in theprevious phase.

Observed cases which implied existence of the third phase are shownbelow.

– After declaring that they finished their shopping for the day, someinformants still continued to look around aisles and found what theywould like to buy.

59


– (An utterance of an informant) ”I’m always feeling that I might for-get to buy what I have to buy today, so I usually scour for somethingto tell me so in the closing stage of my shopping” (an informant inher 30s)

There seemed to exist a wrapping-up phase before terminating hershopping. In this phase, she tended to buy items which were not neces-sarily for that particular day. She was also willing to take new articlesand reduced items to try them out. In other words, this phase acutelytriggered her impulse buying. We could see she was relaxed and feelingfun during this phase since, we speculated, she was freed from the missionof the shopping for the day.

2.3 Findings from the Video Ethnography

We did moment-to-moment analyses of the video data of the seven infor-mants (two were omitted due to lack of their video data). We normalizedtheir shopping durations because the lengths differed from each other,split them into five divisions, and plotted the items chosen according toactual uses in each division (Figure 3). As the result, ”replenishment” gotthe highest in the first and the second division, ”foodstuff for main dish”was the highest in the third division, ”foodstuff for side dish” was thehighest in the fourth division, and ”replenishment” again and ”others”including pure impulse buy (without any assumptions of use) were thehighest in the final division. To have compared this result with the resultin the previous section, we could understand that the first and the seconddivision roughly corresponded to the first phase, the third and the fourthdivision to the second phase, and the final division to the third phase.

Consequently, we defined Three-Phase Model, or TPM (Figure 4):From the first to the second phase, it was divided in the wake of startingto buy a foodstuff for a shopper’s main dish, and, from the second to thethird phase, it was divided in the wake of finishing to buy what a shopperhad to buy for that particular day.

3 Informing Design

As a premise for TPM, what kind of information does a shopper need ineach phase? We speculated implicit information needs in each phase basedon the tasks as well as the state of mind, both of which are clarified in theprevious section at the outset. We then described an experience scenarioreferring to the information needs. After that, we identified principalservice types included in the scenario. We also examined when and howeach service type should become activated.

60



Figure 3: Changes in the rates across five time-divisions according to theinformants’ actual uses of items chosen in each division.

Figure 4: Three-Phase Model; describing consecutive changes of a shop-per’s behaviors and his/her states of mind.

61


3.1 Information Needs

3.1.1 Information Needs in the First Phase

Since a shopper experiences a heavier mental load during this phase, in-formation delivered to him/her should be focused on the tasks as well asminimal. We assumed that there are two major information needs:

– information as resources for him/her to plan a main dish, and

– information to remember to buy planned items.

3.1.2 Information Needs in the Second Phase

Since a shopper tends to concentrate his/her mind on examining indi-vidual articles in this phase, information delivered to him/her should bedetailed as well as thorough. We assumed that there are two major infor-mation needs:

– information as resources for him/her to plan side dishes, and

– information to have him/her ground to choose each article, espe-cially for dinner arrangement.

3.1.3 Information Needs in the Third Phase

Since a shopper is freed from obligations and consistency in his/her shop-ping on the day in this phase, information delivered to him/her shouldmake him/her curious. We assumed that there are two major informationneeds:

– information as resources for him/her to plan yet another dish, e.g.snack, appetizer and dessert, which is beautiful addition to his/herdinner arrangement, and

– information to have him/her know articles unseen yet intriguing forhim/her.

3.2 Experience Scenarios

Given that we would develop a mobile kiosk terminal embedded with ashopping cart which would be feasible to operate in a supermarket at thattime, we described an experience scenario featuring a variety of servicescorresponding to the information needs in each phase.

3.2.1 Experiences in the First Phase

I come to the supermarket with the thought that a meat dish is better fortoday’s dinner menu because I cooked fish yesterday. I rent out a mobilekiosk terminal embedded with a shopping cart, turn on the system, and

62



check today’s fresh recommendations on the screen. I discover that theprice of every pack of pork is reduced 20 percent, therefore I move torecommendations on the screen to retrieve recipes of pork dishes. Thereare plenty of appealing pork dishes varying by parts and seasonings. SinceI would like to have a hefty dish today, I choose ”pork spareribs grilled withbarbecue sauce.” I bookmark the recipe and head down to the meat floor tosee pork spareribs. I pick up a carton of milk and a pack of sausages, bothof which I have already planned to buy, on the way. While I’m passingover the egg section, I happen to remember that there are just three eggsleft in the refrigerator at home, and I, therefore, take a pack of eggs whichis also recommended on the screen.

3.2.2 Experiences in the Second Phase

I arrive at the meat floor and notice that there are two types of porkspareribs. I scan a bar-code attached to one of the pricier type by usinga bar-code reader which is embedded with the cart. Then a descriptionregarding the article appears on the screen. I discover that the type ofpork has been raised on organic foods, so that it seems healthier than theother one. I decide to buy the organically-raised one and take a pack withan adequate amount.

Because I remember barbecue sauce is now out of stock at home whichis necessary to cook the meal, I go to the dry grocery floor. I notice thatrecommendations on the screen have changed from fresh foods to packagedfoods. I see through them all and notice a brand of barbecue sauce isincluded. It is not the one I usually buy, therefore I retrieve the descriptionof the article and realize that it is made entirely with organic ingredients.Although it is at a reduced price for this week, but still a little pricier thanthe usual one, I decide to buy just to try it out. While taking a bottle,I notice that three new recommendations are flashed in rotation at thecorner of the screen. One of them is a jar of mustard. I think mixingmustard with barbecue sauce sounds good. I remember that there remainsenough amount of mustard at home though, so I don’t have to buy it today.

Then I head for the produce. I feel something plain will be better fora side dish because the pork spareribs are somewhat greasy. I look intothe screen and find that there are many recipes using in-season vegetablesready, which suit me. I prefer ”tomato salad with boiled spinach,” there-fore I display the ingredients by touching the particular icon. I confirmthat all the ingredients, except spinach, are stocked at home. I directlyaccess the spinach shelf, and select a lively-leafed bunch. I also bookmarkthis recipe for remembrance’ sake.

3.2.3 Experiences in the Third Phase

I have gathered most of today’s dinner menu, so I settle on the idea that Iwill stroll down the aisles to see what I had better buy. I again notice that

63


instances on the screen transformed. There are new articles of the weeknow. I check them one by one, find a brand of low-calorie ice cream, andget happy to know my favorite mango taste is lined up. I go straight downto the ice cream freezer and pick up two cups of mango and chocolate tasterespectively for everybody in my family for after-dinner dessert.

On the way to the checkout, I stop by the tofu section and notice thatthere are sales rankings of the section for the last week displayed on thescreen. I realize that I have not tried out the most popular one. I hesitatebut bring it forward, although it seems attractive really. Then I go straightto the checkout counter.

At the checkout counter, the two recipes bookmarked are automaticallyprinted out and I can obtain them for later use. Looking back on my entirejourney, I should say what an attractive supermarket it is!

3.3 Allocation and Configuration of Services

We identified five service types included in the scenario. We also specu-lated the best allocation and configuration of the service types by exam-ining when and how each service type would become activated.

3.3.1 Recipe Recommendations

As recipes seem effective information to a shopper’s comprehensive plan-ning, they will tend to be used in an earlier part of the process, i.e. recipesfor a main dish should be activated in the first phase, and those for sidedishes in the second phase. Those for yet another dish, e.g. snack, appe-tizer, and dessert, may be effective in the third phase though.

3.3.2 Product Recommendations

Although product information seems valuable throughout the process,each phase will favor different kinds of information. In the first phase,information regarding a sale on fresh food may help a shopper to develophis/her plan for a main dish. In the second phase, information which willenable a shopper to compare prices, qualities, and features for an arrayof choices may help a shopper to fulfill his/her plan for a main dish aswell as for side dishes. In the third phase, information regarding new andluxury articles may help satisfy his/her curiosity about grocery items.

3.3.3 Location and Preference Based Recommendations

While the previous two service types aim at communicating from theretailer, information taking account of a shopper’s individual context, in-cluding where he/she is and what he/she prefers, may contribute to pro-vide more appropriate information to a shopper him/herself. We assumethat this service type will take an active role especially in the third phase

64



Figure 5: SmartCart: a working prototype system developed accordingto the scenarios described in the previous section.

by itself as well as will exhibit the multiplier effect by linking up with theprevious two service types througout the process.

3.3.4 Bookmark Functions

This service type will complement the previous three service types. Sinceit aims at helping a shopper to return to information which he/she sawearlier in the process, it will be accessible at any time.

3.3.5 Scan-to-deliver Information

This service type will make an article itself as a trigger to retrieve its de-tailed information by using a handy scanner attached to the cart togetherwith the items barcode. It may be especially useful from the second phaseonward because it will provide a shopper with information which enableshim/her to choose and evaluate articles.

4 Subsequent Study

We developed a mobile kiosk terminal embedded with a shopping cart,named SmartCart (Figure 5), as a working prototype system. We alsoconducted an operation test at the project site for about two weeks inSeptember 2006. About fifty customers used the system during theirshopping trips, and about twenty among them cooperated with our par-ticipant observations. We will report the details of the system as well asthe evaluation of the test in the near future.

65


5 Conclusion

In this paper, we reported our human-centered approach for designing aubiquitous computing system which aimed at providing better experiencesfor shoppers at a supermarket.

We investigated in the shopping process by using ethnographic re-search techniques, understood the processes with details, and constructedTPM which describes a shopper’s behaviors and states of mind changedistributed into three phases. We also described a user scenario in eachphase which successfully contributed to a system design by giving a clearpicture of user experiences.

Although we didn’t give details, we had already developed a prototypesystem based on the design information described in the previous sectionsand conducted an operation test at the project site. We are now investi-gating in the data which was acquired from the test. In the near future,we will introduce the details of the system as well as the evaluation ofthe system from the view of how the system had an effect on shoppers’behaviors and their state of mind.

We believe that understanding people’s cognitive processes, like TPM,will hugely contribute to practical applications of ubiquitous computingtechnologies. Through this series of empirical research, we could learnhow to manage a design process which is not limited to designing shop-ping support systems but also to a variety of system designs includingelectronic tourist guide, intelligent office, and smart educational medium.We suggest critical points of view in the process as follows:

– Observing user’s behaviors and emotions throughout his/her pro-cess: It is not at all apparent for an observer, and even for a userhim/herself, how a user’s behaviors and emotions shift respondingto changing situations. It is critical to understand his/her behav-iors and emotions as a cognitive process through fieldwork coveringhis/her entire process in a projected field.

– Discovering metrics relevant to user’s behaviors and emotions: Ifwe can succeed in discovering robust metrics which are relevant touser’s behavioral and/or emotional states, we will be able to man-age a system much more easily. Although we did not explain in thispaper, we simultaneously measured shoppers’ actions and biologicsignals with a variety of digital sensors, and successfully identifiedpurposeful metrics (see details in [9]). Meanwhile, a key suggestionhere is that we should understand user’s behavioral and/or emo-tional shifts in qualitative aspects at first, and after that, move intoan investigation of the metrics. We often fail to explain user’s be-haviors and emotions with a model consisting of predefined metrics,so that we had better keep in mind that qualitative explorationshelp discovering adequate metrics.

66



– Developing services leveraging user’s cognitive process: If we canclearly model user’s cognitive processes according to such the met-rics, we can easily imagine new service ideas including when and howthey should be activated. In such the case, user scenario is a usefultool specifying a system design because it enables us to describe co-herent cases with rich user context, i.e. we can anticipate a systemworking closely with practical use by referring to the scenario.

– Evaluating actual use of a system and bringing it into improvement :An introduction of a new artifact sometimes causes transformationsof user’s behavioral and/or emotional processes through emerginguser-artifact interactions. In fact, in our operation test, some un-expected usages occurred in a particular phase. We must not thinka project terminated when a system once has developed, rather, weshould remind ourselves that it comes the time when we are fac-ing a chance to get back to the starting point to investigate user’sbehaviors and emotions under a new circumstance.

As Abowd and Mynatt stated, one of the most major motivationsfor ubiquitous computing is to support the informal and unstructuredactivities typical of much of our everyday lives [10]. Through research inapplications at real retail outlets, we hope we will be able to contributetoward the solution of such difficult problems.

67

Bibliography

[1] George Roussos. Building Consumer Trust in Pervasive Retail. 2004.In Proceedings of International Workshop Series on RFID. Availableon line from http://www.slrc.kyushu-u.ac.jp/rfid-workshop/

roussos-paper.pdf-

[2] Kirthi Kalyanam, Rajiv Lal and Gerd Wolfram. Future Store Tech-nologies and Their Impact on Grocery Retailing. 2006. In ManfredKrafft and Murali K. Mantrala (eds.) Retailing in the 21st Century.Springer-Verlag, Berlin Heidelberg, ISBN 3-540-28399-4.

[3] Erica Newcomb, Toni Pashley and John Stasko. Mobile Computing inthe Retail Arena. 2003. In Proceedings of Conference on Human Fac-tors in Computing Systems, ACM Press, New York, ISBN 1-58113-630-7.

[4] Panos Kourouthanassis, George Roussos. Developing Consumer-Friendly Pervasive Retail Systems. 2003. IEEE Pervasive Computing,Vol 2, No 2, ISSN 1536-1268.

[5] Art Thomas, Ron Garland. Susceptibility to Goods on Promotionin Supermarkets. 1996. Journal of Retailing and Consumer Services,Vol 3, No 4, Elsevier, Amsterdam, ISSN 0969-6989.

[6] Ikuo Takahashi. Zouho Shouhi-sha Koubai Koudou: Kouri-Marketing he no Shazou (in Japanese). 1999. Chikura Shobou, Tokyo,ISBN 978-4-805107-61-4.

[7] Hiroshi Tamura, Tamami Sugasaka, Hirohisa Naito, MinoruSekiguchi, Satoko Horikawa and Kazuhiro Ueda. Exploring Every-day Activities for Pervasive Decision-Aid. 2006. In Proceedings ofPICMET’06, Portland International Center for Management of En-gineering and Technology, Portland.

[8] Hugh Beyer, Karen Holtzblatt. Contextual Design: DefiningCustomer-Centered Systems. 1998. Academic Press, San Diego,ISBN:1-55680-411-1.

[9] Hiroshi Tamura, Tamami Sugasaka. Harmonizing Human Eyes withDigital Sensors. 2007. In Proceedings of The Third InternationalEthnographic Praxis in Industry Conference, University of CaliforniaPress, Berkeley.

[10] Gregory D. Abowd, Elizabeth D. Mynatt. Charting Past, Present,and Future Research in Ubiquitous Computing. 2000. ACM Trans-actions on Computer-Human Interaction, Vol 7, No 1, ACM Press,New York, ISSN:1073-0516.

Documents

Designing A Smart Shopping-Aid System Based on Human-Centered Approachimpact.asu.edu/cse494sp09/Smart Shopping.pdf · · 2009-04-05Designing A Smart Shopping-Aid System ... his/her