Measuring and evaluating real service operationswith human-behavior sensing: a case study in a

Japanese cuisine restaurant

Tomohiro Fukuhara∗, Ryuhei Tenmoku∗, Takashi Okuma∗, Masanori Takehara†, and Takeshi Kurata∗∗Center for Service Research,

National Institute of Advanced Industrial Science and Technology (AIST)1-1-1 Umezono, Tsukuba, Ibaraki, 305-8568 Japan†Graduate School of Engineering, Gifu University,

1-1 Yanagito, Gifu-shi, Gifu, 501-1193 Japan

Abstract—A case study of human-behavior sensing in aJapanese cuisine restaurant is described. The aim of this study isto create a suite of human-behavior sensing that facilitate peoplewho are in real service fields to understand current operations,and make plans for improving operations. We created a prototypeof the suite, which consists of several component technologiessuch as pedestrian dead-reckoning (PDR), sensor data fusion(SDF), service operation estimation (SOE), service field simulator(SFS), and data visualization. We had anexperiment in a Japanesecuisine restaurant, and found that the suite assisted managers andwaiting staff to understand current state of operations, and tomake a plan for improving operations. An overview of the suite,and results of experiment are described.

I. INTRODUCTION

Today service industries play an important role in economy.As growth of service industries, a new field called “servicescience” or “service engineering” are emerging[1][2]. Forimproving efficiency of service industries, efficient measuringmethods of service operations are needed. In industrial engi-neering (IE), various measuring methods have been developpedsuch as work sampling and time and motion study. Althoughthese techniques allow us to measure both of macro andmicroscopic state of work in the field, it is hard to measureoperations for long time, which is needed to observe changesof operations.

We have been creating technologies that enbales peopleto measure and evaluate work operations instantly by usingvarious sensors[3]. By using sensors, we can collect variousdata from service fields continuously. In case of long termobservation, cost for measuring operations can be decreasedusing sensors compared to traditional IE techniques. We cre-ated a suite of technologies that can measure and analyze bothof human-behavior data and business data such as POS (point-of-sales) data in service fields.

We had an experiment using the suite in a Japanese cuisinerestaurant, and found that the suite assisted waiting staff to (1)measure their operations, and (2) facilitate them to create aplan for improving operations. Through this experiment, weobserved changes of behavior data, and an effect of changeswhich appeared in accounting data of the restaurant.

This paper is organized as following sections. Section IIreviews the related work. Section III describes an overview

of the humamn-behavior sensing suite. Section IV shows anoverview of experiments, and their results. In Section V,we discuss the experiment results. Section VI summarizesarguments of the paper, and describes future work.

II. RELATED WORK

For measuring efficiency of work operations, various tech-niques have been developed in industrial engineering. Motionand time study1[4], and work sampling[5] are major techniquesto observe work operations in the field. Although these tech-niques are good way to obtain real data from the field, itis hard to apply these techniques to long term observations.Because changes of operations can be seen through a longterm observation in service fields, continuous and automatedtechniques to measure operations are needed.

Burger et al. proposed a testbed called ServLab2 for de-signing and evaluating service operations and fields by usingvirtual reality (VR)[6]. Simo et al. also proposed a testbedcalled SINCO for designing service operations and fields usingVR[7]. Although our suite includes a technology called servicefield simulator (SFS) which uses an omni-direction immersivedisplay to visualize a virtualized service field, our aim is tomeasure actual operations in real service fields.

Ueoka et al. proposed a scheme called CSQCC (Computer-supported Quality Control Circle) which aims to improveoperations by using human-behavior sensing technologies.They reported a case study of human-behavior measuring inthe restaurant[8]. In this paper, we also aim to assist workersto measure their operations.

The aim of our research is to create a suite of technologiesthat can facilitate managers and employees in service fields tomeasure and evaluate their operations effectively. For achievingthis aim, we create a prototype of the suite, and apply the suitein real service fields.

III. HUMAN-BEHAVIOR SENSING SUITE

The suite consists of following component technologies.

1) Interactive 3D indoor modeler1It is also called as time and motion study.2http://www.servlab.eu/

gotoタイプライターテキスト

gotoタイプライターテキスト

gotoタイプライターテキスト

gotoタイプライターテキスト19th Korea-Japan Workshop on Frontiers of Computer Vision, pp113-116 (2013)

2) Pedestrian dead-reckoning (PDR)3) Sensor data fusion (SDF)4) Service operation estimation (SOE)5) Service field simulator (SFS)6) Data visualization

Figure 1 shows an overview of the suite.

Interactive 3D indoor modeler allows users to create a 3Dmodel of the service field. Users can create a 3D model ofthe field based on still images of the field taken by digitalcameras[9]. Because the modeler estimates parameters of acamera automatically, users can easily create a 3D model ofthe field from images. We have created various 3D models ofservice fields such as restaurants, Japanese style hotels, andelderly-care facilities3.

Pedestrian dead-reckoning (PDR) allows users to track thelocation and orientation of a person in the service field[10].We have created a sensor which contains multiple sensorssuch as barometer, magnetometer, accelerometer, gyroscope,thermometer, and an RFID reader in a box. By using thesesensors, PDR estimates position, orientation, and velocity of aperson.

Sensor data fusion (SDF) also estimates the location andorientation of a person. By compared to PDR, SDF estimateslocation and orientation more precisely by combining severalsensor data[11]. SDF uses PDR data, RFID data, cameraimages, 3D model of the field, and so on. By applyingmap matching technique[11] to these data, SDF estimates theposition and orientation of a person in the field precisely.

Service operation estimation (SOE) estimates types ofoperation of a person. In service fields, operations can becategorized into several types, for example in a Japanesecuisine restaurant, taking orders, serving food and drink,carrying dishes, and accounting[12]. SOE estimates these typesof operations from PDR and speech data of workers which isrecorded by a voice recorder. From speech data, utterance of aworker is detected by using the voice activity detection (VAD)method[13], and the utterance data is used for estimatingtypes of operations. SOE uses a machine learning techniquecalled AdaBoost[14] to estimate operation types. We achieved81% of precision for operation types in a Japanese cuisinerestaurant[12].

Service field simulator (SFS) allows users to explore aservice field interactively[15]. SFS provides users a virtualizedservice field using an omni-direction immersive display and aPDR sensor. Users can walk through the virtualized servicefield.

Data visualization allows users to browse both of humanbehavior data and various field data such as POS (point-of-sales) data. Figure 2 shows screen images of the visualizationtool. Our visualization tool provides several visualization ofservice fields such as the number of customers in dining areas,the number of orders, and staying time of workers in each area.

3Some of models can be viewed at our Website:http://unit.aist.go.jp/cfsr/cie/hubsense/mris/Demo-e/Videoclips-e.html

3D model Still images

Waiting staffPDR sensor

Voice recorder

Sensor data fusion (SDF)

Video

cameras

Service operation

estimation (SOE)

3D visualization of service

operations and fieldPOS data

Active

RFID tags

Interactive 3D indoor modeler

Service field

simulator (SFS)

Fig. 1. Overview of the human-behavior sensing suite. The suite consistsof human-behavior sensors, 3D model of the service field, service operationestimetor (SOE), and data visualization.

Staying time in each areas

A staff is represented as a blue

arrow. Trajectory of the staff is

displayed.

Customers are

represented

as green icons. POS data

Number of orders accepted by a staff

Orders of customers can be

checked in the right window.

Fig. 2. Screen images of the visualization tool. The upper window showsthe visualization of an employee and customers in the restaurant. The lowerwindows show the summaries of data. The right down window shows the ratioof staying areas of an employee, and the left down window shows the numberof orders that an employee accepted.

IV. EXPERIMENT IN A JAPANESE CUISINE RESTAURANT

A. Overview

We had an experiment of the suite in a Japanese cuisinerestaurant. The aim of the experiment was to observe changesof behavior of waiting staff, and its effect which can beappeared as account data.

Table I shows the terms of the experiment. We had twoterms. In the first term, we aimed to observe ordinary oper-ations of the waiting staff. After the first term experiment,we provided behavior data of the staff and account data ofthe restaurant to QC circle members with the visualizationtool. Figure 3 shows a scene of QC circle where memberschecked their operations and account data in the first term,and discussed plans for improving operations in the second

TABLE I. TERMS OF THE EXPERIMENT HELD IN A JAPANESE CUISINERESTAURANT.

Stage Term1st term 12 (Wed) to 18 (Tue) January, 20112nd term 3 (Thu) to 9 (Wed) February, 2011

Fig. 3. Discussing scene of QC circle members. Waiting staff discussed plansfor improvement by watching data visualization tool.

term. After making a plan, QC circle members implementedthe plan in operations of the restaurant. We observed effectsof the plan in the second term.

B. Results

As results, we describe following data obtained from twoterms: (1) the stay ratio of waiting staff in dining areas, (2)the number of additional orders per customer, and (3) the walkdistance of the staff in a day.

Figure 4 shows the stay ratio of waiting staff in diningareas. The data showed the average stay ratio during the firstand the second terms respectively. The figure shows the stayratio during evening hours (18 to 22 o’clock) in the secondterm was increased compared to the first term. At 19 o’clock,the ratio was 51.3% in the second term, and 43.9% in the firstterm. We observed differences of stay ratio between two terms.

Figure 5 shows the number of additional orders per cus-tomer. The data showed the average during the first and thesecond terms respectively. The figure also shows the numberof additional orders in evening hours was increased afterimplementing the plan. For example, the number of additionalorders at 19 o’clock was 1.2 in the second term, meanwhile0.7 in the first term. Throughout evening hours, 1.7 additionalorders are requested in the second term compared to the firstterm.

Figure 6 shows the walk distance of the staff. This figureshows the average during the first and the second termsrespectively. At 19 o’clock, walk distancec was 1, 400 metersin the second term, and 1, 360 meters in the first term. Totaldistance through evening hours were 6, 620 meters in thesecond term, and 6, 420 in the first term. The differece was200 meters between two terms. We found that there were fewdifferences with respect to walk distance between two terms.

30%

35%

40%

45%

50%

55%

11 12 13 14 15 16 17 18 19 20 21 22

Sta

y r

ati

o i

n d

inin

g a

rea

s (%

)

Hour

1st term 2nd term

Fig. 4. The stay ratio of waiting staff in dining areas. The data is averageof the 1st term and 2nd term espectively. The ratio of evening hours (18-22o’clock) increased in the second term.

0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

11 12 13 14 15 16 17 18 19 20 21 22

Nu

mb

er

of

ad

dit

ion

al

ord

ers

pe

r cu

sto

me

r

Hours

1st term 2nd term

Fig. 5. Number of additional orders per customer. The data is average ofthe first and the second terms respectively. The figure shows the number ofadditional orders in evening hours (18 to 22 o’clock) increased in the secondterm.

V. DISCUSSION

In this experiment, we observed changes of both of be-havior of the staff and account data of the restaurant. Weobserved increases of the stay ratio in dining areas, and thenumber of additional orders per customer in evening hours.Our suite observed changes of behavior of the staff, andaccount data of the restaurant. These results can be relateddirectly. Furthermore, our suite revealed that there were fewdifferences of walk distance between two terms.

The results obtained from this experiment imply thatefficiency of service can be improved by introducing anappropriate plan which is planned based on observed data.In many service fields, managers make plans for improvingoperations based on their insights and experiences. Althoughthese insights and experiences are important for achievingefficiency in some case, it is difficult to apply this approacheto other fields because there is no objective data. Contrary,we aim to measure actual data in the restaurant, i.e., behaviordata of the staff and account data of the restaurantin. Based on

0

500

1,000

1,500

2,000

11 12 13 14 15 16 17 18 19 20 21 22

Wa

lk d

ista

nce

(in

me

ters

)

Hour

1st term 2nd term

Fig. 6. Walk distance of waiting staff a day. The data is the average of thefirst and the second terms.

actual data, members of the QC circle were able to decide theplan to increase the staying time in dining areas. This approach,which is based on data, can be applied to other fields becauseit is obvious for everyone. Our results indicate that managerscan improve their service operations based on observed data,but without relying on their insights and experiences.

Our future work is to improve the suite so that anyonewho are in service industries can use the suite. There are twoissues: (1) usability of the suite, and (2) feedback function ofthe system.

The first is the usability of the suite. In service industries,various people are working whose age and education are quitedifferent. There exist people who are not familiar with sensorsand computers. In this experiment, waiting staff were notfamiliar with sensors at the first time, so we had to assistthem to use sensors. For allowing anyone in service fields touse our suite, we have to improve the usability of the suite.

The second is a feedback function of the data. In thisexperiment, behavior and account data were not presentedduring the experiment terms, we only provied the data at theQC circle. If people can easily check their behavior data beforeor after their work, we assume that the data will motivate themfor improving their work operations. For allowing people tocheck their behavior data instantly is our future work.

VI. CONCLUSION

In this paper, we presented a human-behavior sensingsuite that measures service operations of workers in realservice fields. Our suite, which consists of several componenttechnologies such as interactive 3D modeler, PDR, SDF, andSOE, enabled workers to measure their service operations in areal service field. We had an experiment in a Japanese cuisinerestaurant, and observed changes of operations, and effect ofthe changes which appeared as account data. We considerthat measuring and providing real data in service fields canfacilitate managers and employees to improve their efficiency.Our future work is to improve the usability of the suite, andto provide a feedback function for users.

ACKNOWLEDGMENT

This work was supported by the Ministry of Economy,Trade and Industry (METI) of Japan. The authors thank GankoFood Service Co., Ltd., and staff in the Ganko Ginza 4-chomerestaurant for their cooperation with our experiment.

REFERENCES[1] Innovate America: National Innovation Initiative Summit and Re-

port. The Council on Competitiveness, U.S., May 2005, (available athttp://www.compete.org/publications/detail/202/innovate-america/, ac-cessed 2013-01-09).

[2] J. Spohrer and P. P. Maglio, “The emergence of service science: Towardsystematic service innovations to accelerate co-creation of value,”Production and Operations Management, vol. 17, no. 3, pp. 238–246,2008. [Online]. Available: http://dx.doi.org/10.3401/poms.1080.0027

[3] T. Kurata, M. Kourogi, T. Okuma, T. Ishikawa, R. Ueoka,R. Tenmoku, and K. Makita, “Human-behavior sensing and vi-sualization for service quality control,” in Proceedings of CSCW2012 Workshop: Exploring collaboration inchallenging environments:From the car to the factory and beyond, 2012, (available athttp://workshops.icts.sbg.ac.at/cscw2012/).

[4] L. C. Pigage and J. L. Tucker, Motion and time study, ser. TheUniversity of Illinois Bulletin, R. W. Fleming and B. D. Dennis, Eds.Institute of Labor and Industrial Relations, University of Illinois, 1954,no. 24.

[5] P. S. Ray, Handbook of Industrial and Systems Engineering. CRCPress, 2005, ch. 7, pp. 7–1–7–6.

[6] T. Burger, K. J. Kim, and T. Meiren, “Visualizing and testing serviceconcepts,” in Proceedings of the first international symposium onservices science, 2009, pp. 149–159.

[7] R. Simo, M. Satu, K. Essi, and L. Antti, “A laboratory concept forservice prototyping: Service innovation corner (sinco),” in Proceedingsof ServDes 2012: Service Design and Innovation Conference, 2012,(available at http://www.servdes.org/conference/servdes-2012-research-papers/, accessed 2013-01-09).

[8] R. Ueoka, T. Shinmura, R. Tenmoku, T. Okuma, and T. Kurata,“Introduction of computer supported quality control circle in a japanesecuisine restaurant,” in Proceedings of 4th International Conference onApplied Human Factors and Ergonomics (AHFE). USA Publishing(DVD), 2012, pp. 6632–6641.

[9] T. Ishikawa, K. Thangamani, M. Kourogi, A. P. Gee, W. Mayol-Cuevas,K. Jung, and T. Kurata, “In-situ 3d indoor modeler with a cameraand self-contained sensors,” in Proceedings of the 3rd InternationalConference on Virtual and Mixed Reality, ser. VMR ’09. Berlin,Heidelberg: Springer-Verlag, 2009, pp. 454–464.

[10] M. Kourogi, T. Kurata, and T. Ishikawa, “A method of pedestrian deadreckoning using action recognition,” in IEEE/ION PLANS 2010, 2010,pp. 85–89.

[11] T. Ishikawa, M. Kourogi, T. Okuma, and T. Kurata, “Economic andsynergistic pedestrian tracking system for indoor environments,” inInternational Conference of Soft Computing and Pattern Recognition(SOCPAR ’09), 12 2009, pp. 522–527.

[12] R. Tenmoku, R. Ueoka, K. Makita, T. Shimmura, M. Takehara,S. Tamura, S. Hayamizu, and T. Kurata, “Service-operation estimationin a japanese restaurant using multi-sensor and pos data,” in Proceedingof APMS 2011 conference, 2011, parallel 3-4: 1.

[13] M. Takehara, S. Tamura, R. Tenmoku, T. Kurata, and S. Hayamizu,“The role of speech technology in service-operation estimation,” inProceedings of the International Conference on Speech Database andAssessments (Oriental COCOSDA 2011), 2011, pp. 116–119.

[14] Y. Freund and R. E. Schapire, “A decision-theoretic generalization ofon-line learning and an application to boosting,” Journal of Computerand System Sciences, vol. 55, no. 1, pp. 119 – 139, 1997.

[15] J. Hyun, Y. Habuchi, A. Park, T. Ishikawa, M. Kourogi, and T. Kurata,“Service-field simulator using mr techniques: Behavior comparison inreal and virtual environments,” in Proceedings on ICAT2010, 2010, pp.14–21.