DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances Recognition by Deep Learning

DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appliances

Recognition by Deep Learning

〇Yuta Takahashi1, Naoki Shirakura1, Kenta Toyoshima1, TakuroAmako1, Ryota Isobe1, Jun Takamatsu1 and Keiichi Yasumoto1

1. Nara Institute of Science and Technology

The Tenth International Conference on Mobile

Computing and Ubiquitous Networking (ICMU2017)

S7: Application, October 5, 2017

Background

❖Increasing home appliances

2

❖Many IR remote controllers▪ Need to learn how to use

▪ High management costsIncrease user’s burden

Unification of remote controllers

3

Network Multiple IR controller

Increase the number of appliances

The interface becomes complicated

Method for selecting home appliance is important!

✔ Management cost

Home appliance selection

4

❖With special attachments

❖Without special attachments

✔ Accurate selection

Cost of devices/markers

Selectable distance

Voice [Pan 2010] Vision [Kong 2016]

Intuitiveness ✔ Intuitiveness

IR control

Burden for wearing

• IR LED [Neßelrath 2011]

• IR transmitter [Tsukada 2004]

• QR [Ullah 2012]

Dedicated device is needed

DeepRemote (Proposed)

5

❖Home appliance selection

Object recognition by deep learning

→ Intuitive and robust

❖Two units

▪Control unit

▪Deep learning unit

❖Network

Home network (ROS)

Control unit

6

✓ Hand-held-type remote controller

❖Main processor

▪ Raspberry Pi3 (with Wi-Fi module)

❖Capturing a home appliance

▪ Front camera

❖Control interfaces

▪ Four buttons

▪ Gestures (right & left rotation)

Deep learning unit

7

✓ Image recognition

❖Main processor

▪ Laptop PC with Core i5 (Ubuntu)

▪ Distribute the calculation load of control unit

❖Recognition Model

▪ VGG16 [Simonyan 2014]

TV

Recognition

result

Deep

learning unit

Control unit

Technique of training

❖Requirements of deep learning with zero-base

8

Huge dataset A lot of time

❖Fine tuning

Trained model Arranging model

Re-training

Experiment

9

Experimental environment

10

✓Living room in smart-home facility

❖Five home appliances▪ Fan (IR)

▪ Air conditioner (IR)

▪ TV (Network)

▪ Audio player (IR)

▪ Air purifier (Network)

❖Training data▪ Took 20 images each appliances at P1, P2 and P3

▪ 20 × 5 (appliances) × 3 (positions) = 300 images

Example of training data

11

P1

P2

P3

Air

purifierAir

conditioner

Audio

player Fan TV

Training

❖Model▪ VGG16 trained ImageNet

❖Dataset▪ Three hundred images

▪ Five classes

12

❖Optimizer▪ SDG

❖Loss function▪ Categorical cross entropy

Evaluations

1. Classification accuracy

▪ Captured 50 images of each appliance in each position

▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images

13

2. Response time

▪ Measured time since pushing button until return the result

▪ Evaluated at the same time as 1.

3. User test

▪ Verified control time of home appliances

Appliance Precision [%] Recall [%] F-measure [%]

P1

Air purifier 100.00 68.00 80.95

Audio player 83.33 90.00 86.54

TV 96.15 100.00 98.04

Air conditioner 57.47 100.00 72.99

Fan 86.96 40.00 54.80

Average 84.78 79.60 78.66

P2

Air purifier 81.63 80.00 80.81

Audio player 100.00 98.00 98.99

TV 92.59 100.00 96.15


Fan 100.00 90.00 94.74

Average 90.69 90.40 90.45

P3

Air purifier 76.36 84.00 80.00

Audio player 100.00 58.00 73.42

TV 71.43 80.00 75.47


Fan 100.00 62.00 76.54

Average 80.19 73.60 74.11

1. Classification accuracy of home appliances

14

P1:Effect of

black door?

P3: Too near?

Evaluations




15

2. Response time



3. User test


2. Response time

16

(n=750)

Maximum time: 3.07 [sec]

Minimum time: 1.72 [sec]

Stable recognition about

two seconds

Evaluations




17

2. Response time



3. User test


3. User test to assess control time (1/3)

Experimental conditions

Five participants

Position: P1

Targets: Fan, Air conditioner, TV and Audio player

18

❖Control time ❖Comparison

Holding Power on DeepRemote Original

vs


19

F:86% F:98% F:77% F:55%

High → ✔ about 5 seconds

Lower than the 85% → over 10 secondsAccuracy

Accuracy requires

85% over


20

DeepRemote: Sum of each test > All

Original: Sum of each test < All

Changing time

DeepRemote < Original

Sum Sum

31.533.7

11.7

7.7

(- 2.1)

(+ 3.0)

Conclusions

❖DeepRemote▪ Smart device for intuitively control the home appliances

▪ Deep learning method for home appliance selection

❖Results of evaluation▪ 81.07% classification accuracy on average

▪ Average of response time is 1.97 seconds

▪ Time of power on an appliance takes 5 seconds

▪ 85% accuracy is required

▪ Time of changing target is lower than original remote controller

21

Appendix

22

Future works

❖Improving recognition accuracy▪ Cropping object

❖Investigating energy consumption

❖Accuracy of distinguishing similar appliances

❖Experiment in long-term usage▪ User teaches correct label when the system recognizes

wrong

23

Detail conditions of user test

❖Compare DeepRemote and original remote controller▪ measure control time

▪ Remove “air conditioner” (no original controller)

▪ Five participants (males in the 20s)

▪ Perform three times in each measurement (one participant performs30 times)

▪ Participant’s position: P1

24

1) Measure one appliance control time

(start) → (power on an appliance)

2) Measure all appliances control time

(start) → (power on the fan)

→・・・ → (power on the audio player)

Confusion matrix of P1

25

Classified

Air purifierAir

conditioner

Audio

playerFan TV

True

Air purifier 34 6 9 0 1

Air

conditioner 0 50 0 0 0

Audio player 0 1 45 3 1

Fan 0 30 0 20 0

TV 0 0 0 0 50


26

Classified

Air purifierAir

conditioner

Audio

playerFan TV

True


Air



Fan 0 5 0 45 0

TV 0 0 0 0 50


27

Classified

Air purifierAir

conditioner

Audio

playerFan TV

True


Air



Fan 0 19 0 31 0

TV 3 7 0 0 40