저 시-비 리- 경 지 2.0 한민

는 아래 조건 르는 경 에 한하여 게

l 저 물 복제, 포, 전송, 전시, 공연 송할 수 습니다.

다 과 같 조건 라야 합니다:

l 하는, 저 물 나 포 경 , 저 물에 적 된 허락조건 명확하게 나타내어야 합니다.

l 저 터 허가를 면 러한 조건들 적 되지 않습니다.

저 에 른 리는 내 에 하여 향 지 않습니다.

것 허락규약(Legal Code) 해하 쉽게 약한 것 니다.

Disclaimer

저 시. 하는 원저 를 시하여야 합니다.

비 리. 하는 저 물 리 목적 할 수 없습니다.

경 지. 하는 저 물 개 , 형 또는 가공할 수 없습니다.

M.S. THESIS

Virtual Keyboard with

Gyroscope Sensor of Smartwatch

스마트 워치의 자이로스코프 센서를 이용한

가상 키보드 기법

2017년 2월

서울대학교 대학원

컴퓨터공학부

최 준 혁

M.S. THESIS

Virtual Keyboard with

Gyroscope Sensor of Smartwatch

스마트 워치의 자이로스코프 센서를 이용한

가상 키보드 기법

2017년 2월

서울대학교 대학원

컴퓨터공학부

최 준 혁

Abstract

Virtual Keyboard with

Gyroscope Sensor of Smartwatch

Junhyeok Choi

Dept. of Computer Science & Engineering

The Graduate School

Seoul National University

Both keyboard and watch are widely used equipments that makes human

life convenient. First, a keyboard is an input device that helps interaction

between human and computer. We usually make use of keyboard, one of the

most proper tools to represent human language, to express our intent to tar-

get device. Next, a watch has been played an important role for human life

so that the people can wear it on their wrist and check exact time anywhere,

anytime. Besides providing exact time to user, a watch has been improved

with additional features like stop watch, alarm. A smartwatch, which is the

cutting edge technology in watch field, has quite good processing power,

capabilities of communication and supports several in-built sensor so that

expected to have great potentiality. This paper proposes a system called Vir-

tual Keyboard that uses the in-built gyroscope in smartwatches to recognize

i

human key typing motion. When an user wearing smartwatch on her wrist

tries to make typing motion on any surface, for instance, desk, the watch col-

lects gyroscope data. Our system leverages the data to predict the behavior

of wrist, classify the intended key, and generate proper virtual keystrokes to

target devices. We implemented our virtual keyboard module on LG watch

Urbane platform, and evaluated the performance of system with a few rea-

sonable constraints. Results show that English characters can be identified

by reducing candidates into maximum 2, with an average accuracy of 45.1%.

In this research, we obviously found the possibility of the whole new way to

provide alphabet input with smartwatch. Future work is focused on refining

the prototype of our system. We are planning to get rid of several constraints

iteratively with the goal of offering a new user-experience that complements

keyboards and touch-screens.

Keywords: Virtual Keyboard, Input device, Smartwatch, Sensor, Gyroscope

Student Number: 2015-21272

ii

Contents

Abstract i

Chapter 1 Introduction 1

Chapter 2 Background 5

2.1 Smartwatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 In-built Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Accelerometer . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.2 Gyroscope . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 3 Related Works 9

3.1 Acoustic Emanation . . . . . . . . . . . . . . . . . . . . . . . 9

3.2 Keystroke Inference with Smartwatch . . . . . . . . . . . . . 10

Chapter 4 Virtual Keyboard System 12

4.1 Gyroscope and Wrist Rotation . . . . . . . . . . . . . . . . . 14

4.2 Keypress Timing Detection . . . . . . . . . . . . . . . . . . . 15

4.3 Keystroke Classifier . . . . . . . . . . . . . . . . . . . . . . . 16

4.4 Virtual Keystroke Generator . . . . . . . . . . . . . . . . . . 18

iii

Chapter 5 Evaluation 19

5.1 Experiment Environment . . . . . . . . . . . . . . . . . . . . 19

5.2 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5.4.1 Difference in Typing Style . . . . . . . . . . . . . . . . 23

5.4.2 Dictionary Matching . . . . . . . . . . . . . . . . . . . 23

Chapter 6 Conclusion 25

Bibliography 27

초록 30

Acknowledgements 32

iv

List of Figures

Figure 2.1 Structure of gyroscope. . . . . . . . . . . . . . . . . . 7

Figure 2.2 x, y, z-axis of gyroscope in LG watch urbane smartwatch. 7

Figure 4.1 Virtual keyboard system architecture. . . . . . . . . . 13

Figure 4.2 Keys located in left side of keyboard. . . . . . . . . . 14

Figure 4.3 Wrist rotation for x, y and z-axis. . . . . . . . . . . . 15

Figure 4.4 Pressing q and a sequentially. . . . . . . . . . . . . . 16

Figure 4.5 Pressing q and r sequentially. . . . . . . . . . . . . . 16

Figure 5.1 Gyroscope sensor data of x, z-axis for word ‘assert’. . 20

Figure 5.2 Gyroscope sensor data of y-axis and magnitude for

word ‘assert’. . . . . . . . . . . . . . . . . . . . . . . 21

Figure 5.3 Each keypress will be classified into one of 9 parts. . 22

v

List of Tables

Table 4.1 Sign of inductive angular velocity from rotation. . . . 14

Table 5.1 Keypress detection rate. . . . . . . . . . . . . . . . . . 20

Table 5.2 Keystroke classification accuracy. . . . . . . . . . . . . 22

vi

Chapter 1

Introduction

Imagine the following scenario. Alice is sitting in her desk, writing out her

report. Meanwhile, she finds that her keyboard is broken and she cannot type

words with it anymore. Fortunately, she are almost done with her report, but

she should correct several expressions. And there is no much time for meeting

with her customer. She suddenly fingers her smartwatch, and starts typing

on her desk surface, not her keyboard. Surprisingly, words that she typed on

her desk appear on her monitor, Alice can complete her report on time. How

could this kind of thing ever happen?

Keyboard is an input device that human uses to represent her intent in

order to interact with computer. Users who know how to type with keyboard

can express the words they want to tell in easy and quick way. By virtue

of this convenience, everyone using computer is also using keyboard. That

means everyone who is working with computer already know how to input

text with keyboard.

1

Despite these advantage of keyboard, there exist environments not proper

to type input with keyboard. For example, mobile devices like smartphones

and smartwatches are not suitable to adopt keyboard as input device, though

they are able to do so, because portability is one of the most important feature

of them. So input devices other than keyboard are adopted in these environ-

ments. We are interested in smartwatches, which leverage touch-screen[1, 2, 3]

or microphone as a way to make communication between human and device.

These kinds of input methods are good choice, of course. They utilize some

features of smartwatches properly as input device. But these methods have

some weak points, either.

[1], [2] are input methods that brings touchscreen QWERTY keyboard of

smartphones to small touchscreen of smartwatches. To cope with small screen

[1] aggregates keys on same vertical lines, for example, q, a, z, [2] takes natural

QWERTY keyboards and introduced additional feature that if a user touches

small keyboard, keys near that place will be zoomed in for the user to touch it

in delicate manner. But this kinds of works that bring traditional keyboards

to small screen introduce considerable inconvenience for user experience. To

overcome this inconvenience, [3] proposed new method that introduces touch

gesture. But it has considerable disadvantage that users who are used to

typing with QWERTY keyboards have to be skilled with the new way to

type. The case of using microphone takes audio records of human voice and

has some processing for the acoustic data. In result the voice is transformed

to text message. This is good way to input texts like words or sentences, but

has weak points in case of a single key level input that cannot be inferred

from dictionary or the context.

2

We propose a whole new scheme that can input text with smartwatches.

We regard in-built sensors of smartwatch as a side channel that can be used

to infer the movement of the hand taking behavior of typing. A user considers

some surface other than keyboard as a virtual keyboard, then she can make

typing behaviors in same way as typing a common QWERTY keyboard. All

that the user need is just smartwatch on her wrist as usual.

Users who are used to typing with traditional keyboards don’t have to

make any effort to be skilled with the new input method. Besides, the vir-

tual keyboard can be connected with any other target devices with wireless

communication interfaces which is provided by smartwatches, like WiFi or

Bluetooth. That is, virtual keyboard with smartwatch can be the input device

of any equipment that is capable of wireless communication.

We implemented the system we designed and evaluated it with LG watch

urbane smartwatch[4]. Results show that English characters can be identified

by reducing candidates into maximum 2, with an average accuracy of 45.1%.

[5] proposed PhonePoint Pen which is a new input method that a user

grabs mobile phone like a pen and write in air to take some memos. Our

ultimate goal is consistent with the PhonePoint Pen with the prospect of

developing a new input technology for personal devices that can identify

typing motions not only on random surfaces but also in air.

This paper is composed with following chapters. Chapter 2 describes

smartwatch and in-build sensors to help understand. Chapter 3 introduces

some related works which have similar vision or methodology with this work.

Chapter 4 describes structure, design principles and implementation of the

Virtual Keyboard System in detail. Chapter 5 evaluates the result of this

3

work. Chapter 6, finally, concludes the result of this work and proposes some

visions for future works.

4

Chapter 2

Background

2.1 Smartwatch

To measure exact time is traditionally one of the most important task for

human life, results in advancement of clock.

Beginning with closet clock and table clock, needs for portability brought

about wrist watch. It has been a long history for watch manufacturers to ex-

tent the capability of watch beyond basic function of time keeping, resulting

in introducing several additional features like stop watch, alarm. The cutting

edge of the technology is smartwatches. Smartwatch is another trend that mi-

grates functionalities from smartphones. By nature, it has the form of watch,

which means you are able to wear it on your wrist everyday life. Besides, it has

considerable computing power and capability on wireless communication. It

provides synchronization with smartphone, enabling additional features like

push notice, text message, and some functions working with its in-built sensor

5

support.

One of the most notable feature of smartwatch is its nature of wearable

device. That is, smartwatch is the most near-placed computer with our body,

especially the wrist wearing it. While a user is playing with other behaviors,

the smartwatch is worn on her wrist and keep processing. So in-built sensor

is consistently recording the movements of the user and her behaviors are

conserved as time sequence data. As we can recover the human body move-

ments by processing these kinds of data, the smartwatch can work as a side

channel to get some information about human behavior.

We built an Android Wear application implementing the system we de-

signed, and used LG watch urbane smartwatch to evaluate our system. The

smartwatch is powered by Android Wear OS[6], a variation of standard An-

droid OS that supports essential portion of whole Android functionalities

but suiting better to limited processing power and battery life. Also, it pro-

vides various in-built sensor support like gyroscope, accelerometer, compass,

PPG[7], and Barometer. Especially, we extract data from gyroscope for the

purpose of our work. The watch provides some communication interfaces like

WiFi and Bluetooth Low Energy, so we can connect it with other devices

capable of wireless communication.

2.2 In-built Sensor

Andrdoid Wear framework provides interface to collect the data from smart-

watch in-built sensors. We describe two sensors, accelerometer and gyroscope

that have essential portion of this work.

6

2.2.1 Accelerometer

An accelerometer is an electromechanical device that measures accelereration

forces. The sensor returns x-, y- and z-axis accelerations given to smartwatch

for that time. Some previous works[13, 14], we introduced them in Chapter 3,

mainly exploited accelerator to inference keystroke. The array of accelerations

in x- and y-axis from streaming sensor data corresponds to each movement.

With a double integral, It will be integrated into a displacement array. The

displacement can be used to inferring the watch movement. But there are

some weak points of working with accelerometer. The acceleration data is

coarse and noisy in natural. It contains the accelerations due to unexpected

movements, and even the acceleration of gravity. So proper way of filtering the

unneedful components is necessary, or the noise will make error cumulation.

To sum up, exploiting accelerometer enables estimating real displacement of

the watch, but needs refining the raw data to pick up necessary components.

2.2.2 Gyroscope

Figure 2.1 Structure of gyroscope.

Figure 2.2 x, y, z-axis of gyroscope

in LG watch urbane smartwatch.

A gyroscope is a device that uses earth’s gravity to help determining

7

orientation. The in-built gyroscope sensor of smartphones measures the rate

of rotation in rad/s around a device’s x-, y-, and z-axis. As a user will wear

smartwatch on her wrist, we can infer the rotation of wrist from obtained

gyroscope data. We differentiate our work with the previous works, that did

not exploit gyroscope or adopted as some portion of learning features, by

exploiting the gyroscope as main component. We found some insights while

considering about the natural characteristics of gyroscope for this work.

The works mainly exploited accelerometer use the absolute value like dis-

placement, calculated by integration. But our work exploited gyroscope fo-

cuses the relative rotation of wrist, in result less error prone than accelerom-

eter.

Based on 3 axes described in Figure 2.2, clock-wise rotation would intro-

duce negative angular velocity, and counter clock-wise rotation would intro-

duce positive angular velocity.

8

Chapter 3

Related Works

A smartwatch is relatively recent technology and there are not many works

that researched it in depth. There are no previous researches dealing with

new kind of input method like our virtual keyboard system, but the field of

keystroke inference has been researched consistently. Many works are about

privacy leak of keystrokes that victim typed, through the side channel attack

exploiting various sensors.

3.1 Acoustic Emanation

The sound produced by electrical devices have been used to leak information

of users’s activities. There are various works dealing with emanation attacks

that aim to infer keystrokes and proved to be effective[8, 9, 10, 11]. These

works recorded the acoustic sound emanating from the keyboard, and make

some processing to classify which key was pressed serially. Asonov.[8] was the

9

first work dealing with keystroke inference attack with acoustic emanation.

They extracted the acoustic signal components with FFT processing, and

adopted them as features for supervised learning to classify each keystroke.

Zhuang.[9] classified each key with unsupervised manner, Berger.[10] pro-

posed dictionary based attack which compares data, collected from online

phase, from the result of learning with dictionary. Zhu.[11] tried to recover

the keystrokes by combining the data collected with several smartphones

which is placed near by the keyboard of victim. All above works leverage

difference of acoustic sound to classify the different keystrokes. So they are

susceptible to noise and have limits that the adversary should preliminarily

equip the additional recording devices near the keyboard.

J. Liu.[12] put a single smartphone near the keyboard, and traced the

keystroke by exploiting TDoA(Time Difference of Arrival) of keystroke sound

between the two different microphones of the phone. This is less prone to

noise but has same limitation that the smartphone should be located near

the keyboard, so that the attack scenario is unrealistic for adversary to carry

out.

3.2 Keystroke Inference with Smartwatch

MoLe[13] stands for Motion Leaks through Smartwatch Sensors. It installs

malicious applications that can access the sensor data of the victim’s smart-

watch, and collects the accelerometer data when the victim types on her

keyboard. It fits the point cloud, built with the sensor data, to the templates

preliminarily made on offline phase. Then it can leak the victim’s motion.

This work utilizes accelerometer for two different goals. First, it uses z-axis

10

value to detect the key press timing. This is possible because when a user

takes typing motion, the wrist would move up and down. Next, it uses x-,

y-axis value to calculate the displacement of smartwatch. The displacement

helps inferring the location of wrist and hand so that the adversary can clas-

sify the typed keystroke. Finally, there are blank spaces because the victim

would not wear the smartwatch on her right hand. Authors fill in the blank

with dictionary based bayesian inference to infer the complete words.

X. Liu.[14] infers keystroke with the microphone and accelerometer data

of smartwatch. Main difference of this work compared to MoLe is that they

used acoustic key press sound to detect key press timing, other than ac-

celerometer. This perspective is in line with the works we introduced in 3.1.

As we can find above, almost related works are based on microphone and

accelerometer, and they are research of security field aim to leak privacy of

victim. We differentiate our work by mainly using gyroscope. And as we are

not assuming the attack scenario, we have some advantages with achieving

the cooperation with the user and be able to fit some practical parameters

like classifying thresholds.

11

Chapter 4

Virtual Keyboard System

Figure 4.1 represents the architecture of the virtual keyboard system we

designed. This system consists of three submodules, keypress timing detector,

keystroke classifier and virtual keystroke generator.

The first phase, keypress timing detector, catches the intent of user to

press the key. It infers the timings of keypress events and hands over the

result to the keystroke classifier. The second phase, keystroke classifier, plays

role of guessing the intended key by integrating all contextual information

during the keystroke timing. The last phase, virtual keystroke generator,

takes output of the classifier and generate the proper keystroke events to the

target devices.

We leverage in-built gyroscope sensor of the smartwatch. When a user

tries to type some keys, her hand movements cause variation of gyroscope

sensor values. Our insight is to find connection between wrist motion when

typing a key and following gyroscopic transition.

12

Figure 4.1 Virtual keyboard system architecture.

We made some assumptions that helps our research. First, we assume

that user wears the smartwatch on her left wrist. Second, the user types

keyboard with standard manner, with her elbow hold on to desk, or other

surface, and press each key exactly. Third, when user is typing, the hand

is periodically return to the basic position, which means four fingers except

thumb is above the virtual ‘A’, ‘S’, ‘D’, ‘F’ position of QWERTY keyboard.

Cause user is wearing smartwatch just on her left wrist, we have limitation

13

Figure 4.2 Keys located in left side of keyboard.

that we cannot obtain information of right hand typing. Our ultimate goal

is proposing virtual keyboard capable of typing all alphabets on QWERTY

keyboard. But this time we decided our work scope to classify the alphabets

that can be typed with left hand only, shown in Figure 4.2.

4.1 Gyroscope and Wrist Rotation

counterclockwise clockwise

x-axis - +

y-axis - +

z-axis + -

Table 4.1 Sign of inductive angular velocity from rotation.

Change on gyroscope sensor value of smartwatch is strongly related to

rotation of the user’s wrist. Figure 4.3 shows the rotation of wrist over each

axis, x-, y- and z. Note that Figure 2.2 and Figure 4.3 have opposite position

over x- and y-axis. In result, gyroscope sensor will have negative angular

velocity when wrist rotates over x- and y-axis to counterclockwise direction.

14

Figure 4.3 Wrist rotation for x, y and z-axis.

In contrast, when wrist rotates over z-axis to clockwise direction, it will have

positive angular velocity. To sum up, we can infer direction of wrist rotation

by comparing the angular velocity measured from gyroscope sensor.

4.2 Keypress Timing Detection

Keypress timing detection module has a goal of guessing which portion of

gyroscopic data stream has the user’s intent to pressing keys. We set up

following hypotheses to characterize key press behavior.

At the key press timing, First, smartwatch will rotate rapidly compared

to idle time, so that gyroscopic value of each axis will make some changes.

Second, wrist of user moves up and down slightly because of the clicking

motion, which results in y-axis rotation.

15

4.3 Keystroke Classifier

Keystroke classifier module has a goal of taking key press timing from pre-

vious module and gyroscopic data during that time window as input, and

inferring the indened key at that moment. We can guess it from relative

position between previous position of hand and the next key user is typing.

Figure 4.4 Pressing q and a sequentially.

Figure 4.5 Pressing q and r sequentially.

Figure 4.4 and Figure 4.5 show shape of hand and wrist when movements

over vertical and horizontal line happen, respectively.

16

In the case of movement over vertical line, z-axis rotation happens. Fig-

ure 4.4 shows that wrist moves in order to press next key on other vertical

line. As we assumed that user’s elbow is held on to desk, wrist revolves around

the elbow. This rotation induces z-axis angular velocity, as we can find on Ta-

ble reftable:sign. Considering the direction of rotation, Table 4.1 shows that

negative z-axis angular is induced by clockwise rotation like line 1 → 2, 1 →

3, 2 → 3. In constrast, positive angular velocity is due to counterclockwise

rotation, for example, 3 → 2, 3 → 1, 2 → 1. If there is no meaningful angular

velocity on z-axis, we regard that no vertical movement happened and next

key would be located on the same vertical line with previous key. Both line

1 → 2 and line 1 → 3 case bring out positive angular velocity, so we would

distinguish them with the degree of variation.

In the case of movement over horizontal line, x-axis rotation happens.

Figure 4.5 shows that the arm rotates in order to press next key on other

horizontal position. Considering the direction of rotation, Table 4.1 shows

that positive x-axis angular is induced by clockwise rotation like q → w → e

→ r → t. In constrast, negative angular velocity is due to counterclockwise

rotation, for example, t → r → e → w → q. If there is no meaningful angular

velocity on x-axis, we regard that no horizontal movement happened and

next key would be located on the same horizontal line with previous key. We

can distinguish the keys with different distance from previous key with the

degree of variation.

To sum up, we take a look at x- and z-axis variance of gyroscopic sensor

data and estimate relative position for vertical and horizontal between pre-

vious key and next key. Integrating those results, we can infer the next key

17

that user intended to press.

4.4 Virtual Keystroke Generator

This phase is to generate the genuine keystroke to the target devices with

the virtual stroke taken by last phase. We implemented a simple key event

generator that is working on windows OS, our target device.

18

Chapter 5

Evaluation

5.1 Experiment Environment

We measure the accuracy of our approach through two experiments. In the

first experiment, we test the keypress timing detection module. In our second

experiment, we test the keystroke classifier module. Both experiments were

conducted by two demo users wearing smartwatch by typing some words and

analyzing the sensor data.

Our demo application was developed and installed on the LG watch ur-

bane, which records gyroscopic sensor data for given time. The participants

type some words composed with keys can be typed with left hand. The partic-

ipants were sufficiently noticed for the assumptions we made on 4. in advance.

Though we are testing virtual keyboard application, the participants typed

on a real keyboard in order to collect the ground truth. Finally we obtained

more than 1000 samples and did some analysis to evaluate our work.

19

5.2 Detection

Figure 5.1 Gyroscope sensor data of x, z-axis for word ‘assert’.

The participants typed some words whose length varies between 5 let-

ters to 15 letters, including both strings which are in dictionary and not in

dictionary. With the all 1245 samples we obtained, 1250 keystroke was de-

tected. Among the keystrokes, 1021 are found to true positive, which means

the keystrokes matched to ground truth. That is, we can detect the keypress

motion with average accuracy of 81.7%.

Asonov [8] VK MoLe[13]

78.85% 81.7% 55.76% (93.9% for left hand)

Table 5.1 Keypress detection rate.

Other works, leveraging acoustic emanation and acceleration sensor, also

researched keypress detection with indirect sensor data. Asonov. reported

78.85% of accracy using acoustic signals emanated during typing. But ac-

cording the experiment of [14], which reproduced the experiment of Asonov.,

it showed the overall accuracy of only 35.75%, which is significantly lower

20

than the report. MoLe, using acceleration, reported keystroke detection rate

for each character. The average detection rate for all characters is 55.76%.

But that is because poor detection rate for characters being typed by right

hand, which is not detectable with smartwatch on user’s left hand. If we limit

scope to the characters reachable with left hand, the detection rate increases

to 93.9%. It shows acceleration has good features to represent keypress mo-

tion.

5.3 Classification

Figure 5.2 Gyroscope sensor data of y-axis and magnitude for word ‘assert’.

Our classifier decides target keypress considering x- and z-axis angular

velocity. As we assume that the watch will return to basic position periodi-

cally, each keypress can be denoted by relative position with ’d’ key, which is

on the center of keyboard. In result, our classifier match each keypress with

one of the 9 parts described on Figure 5.3. Each classification result will have

maximum candidates of 2 keypresses. With the same samples, 562 among

1245 keystroke was classified as true positive. So the average accuracy was

21

about 45.1%.

Figure 5.3 Each keypress will be classified into one of 9 parts.

There are some works that leverage sensors of smartwatch to infer words

typed with keyboard. The main difference of these works and ours is that

they focused on inferring English words other than English character. Our

classifier is key-oriented so that we evaluate the result with unit of each key.

In contrast, MoLe and X. Liu. evaluate the result with unit of each word.

We discuss about this on 5.4.2.

Key-oriented Word-oriented

VK MoLe[13] X. Liu.[14]

45.1% / 2 candidates30% / 1 candidates

50% / 6 candidates

54.8% / 5 candidates

63% / 10 candidates

Table 5.2 Keystroke classification accuracy.

22

5.4 Discussion

We discuss about some topics that we noticed while we are working on our

evaluation.

5.4.1 Difference in Typing Style

We adopt some threshold values in implementing our application, like key-

press duration, sensor signedness boundary.. etc. But over the different partic-

ipants, there are different fit values to maximize the accuracy of our models.

And the thresholds are found to have consistency within the samples col-

lected by each participant. We consider that is because difference in typing

style among the people. Fortunately, our system is user-friend application

and do not have malicious intent. So that we can have adjusting phase which

fits the threshold values for the user wearing smartwatch. That will improve

the accuracy of detecting and classifying phase, and be able to provide better

user experience.

5.4.2 Dictionary Matching

Cause our work is key-oriented, there is some troubles to assess our work with

others directly. In order to compare our approach with the previous works

introduced before, we constructed a dictionary matching algorithm that gives

score to words according to matching characters. We chose 10 words that can

be typed with only left hands and made an additional dictionary matching

phase after the keypress detecting and keystroke classifying phase. The result

showed that our application have 40% chance to narrow down the typed word

to only 5 possibilities. That is, even with the word-oriented view, our work can

23

be considered to have analogous level of inferring typed words compared to

(50% / 6 candidates), or (54.8% / 5 candidates) of previous works described

in Table 5.2.

24

Chapter 6

Conclusion

In this paper, we proposed new way to input alphabet text by taking typing

motion, even if the user do not have real keyboard. Users who have prelimi-

nary experience of standard QWERTY keyboard do not have to study for this

new way to input text, cause our system aims to have essentially same func-

tionalities with existing keyboard. We utilized smartwatch sensor as a side

channel to predict behavior of user’s hand. Previous works that researched

about keystroke inference mainly exploited acoustic sound or accelerometer,

but we differentiate our work by leveraging gyroscope to estimate the relative

position between the intended keystrokes.

Though we found that gyroscope has considerable potentiality for this

work, the virtual keyboard we implemented is just a beginning of our ultimate

plan. There are some constraints that we made while designing the system,

and the precision of virtual keyboard may not be proper to common use.

But we shed a light on the field of keystroke inference using gyroscope sensor

25

of smartwatch and proposed the new type of input methodology. We are

planning to conduct more works to get rid of the constraints we set and

improve the accuracy of our system.

26

Bibliography

[1] Minuum, Typing on Smartwatch, http://minuum.com/minuum-on-

smartwatch

[2] S. Oney, C. Harrison, A. Ogan and J. Wiese, “ZoomBoard: A Diminu-

tive QWERTY Soft Keyboard Using Iterative Zooming for Ultra-Small

Devices,” Proceedings of the SIGCHI Conference on Human Factors in

Computing Systems, Paris, France, April - May 2013. pp. 2799-2802.

[3] H. Cho, M. Kim and K. Seo, “A text entry technique for wrist-worn

watches with tiny touchscreens,” in Proceedings of the adjunct publica-

tion of the 27th annual ACM symposium on User interface software and

technology, Honolulu, Hawaii, USA October 2014. pp. 79-80.

[4] LG Watch Urbane, http://www.lg.com/us/smart-watches/lg-W150-lg-

watch-urbane

[5] S. Agrawal, I. Constandache, S. Gaonkar, R. R. Choudhury, K. Caves

and F. DeRuyter, “Using mobile phones to write in air,” in Proceedings

of the 9th international conference on Mobile systems, applications, and

services, Bethesda, Maryland, USA, June - July 2011. pp. 15-28.

27

[6] Android wear, https://developer.android.com/wear/index.html

[7] Photoplethysmogram, https://en.wikipedia.org/wiki/Photoplethysmogram

[8] D. Asonov and R. Agrawal, “Keyboard acoustic emanations,” in Pro-

ceedings of IEEE Syposium on Security and Privacy, Oakland, California

USA, May 2004. pp. 3-11.

[9] L. Zhuang, F. Zhou and J. D. Tygar, “Keyboard acoustic emanations

revisited,” ACM Transactions on Information and System Security., vol.

13, no.3, October 2009.

[10] Y. Berger, A. Wool and A. Yeredor, “Dictionary attacks using keyboard

acoustic emanations,” in Proceedings of the 13th ACM conference on

Computer and communications security, Alexandria, Virginia, USA, Oc-

tober - November 2006. pp. 245-254.

[11] T. Zhu, Q. Ma, S. Zhang and U. Liu, “Context-free Attacks Using Key-

board Acoustic Emanations,” in Proceedings of the 2014 ACM SIGSAC

Conference on Computer and Communications Security, Scottsdale, Ari-

zona, USA, October - November 2014. pp. 453-464.

[12] J. Liu, Y. Wang, G. Kar, Y. Chen, J. Yang and M. Gruteser, “Snooping

Keystrokes with mm-level Audio Ranging on a Single Phone,” in Pro-

ceedings of the 21st Annual International Conference on Mobile Com-

puting and Networking, Paris, France, September 2015. pp. 142-154.

[13] H. Wang, T. T. Lai, R. R. Choudhury, “MoLe: Motion Leaks through

Smartwatch Sensors,” in Proceedings of the 21st Annual Interna-

28

tional Conference on Mobile Computing and Networking, Paris, France,

September 2015. pp. 155-166.

[14] X. Liu, Z. Zhou. W. Diao, Z. Li and K. Zhang, “When Good Becomes

Evil: Keystroke Inference with Smartwatch,” in Proceedings of the 22nd

ACM SIGSAC Conference on Computer and Communications Security,

Denver, Colorado, USA, October 2015. pp. 1273-1285.

29

초록

키보드와 손목시계는 모두 인간의 삶에 편리함을 주는 도구들이다. 먼저 키보

드는 인간이 컴퓨터와의 상호 작용을 목적으로 자신의 의도를 표현하기 위해

사용하는 입력도구이며, 인간의 언어를 표현하기에 가장 적합한 도구 중 하나로

널리 사용되고 있다. 다음으로 손목시계는 인간이 손목에 착용하여 언제든지 시

간을알수있도록하는도구로,인간의삶에있어서항상중요한역할을해왔다.

손목시계의 기본적인 기능은 정확한 시간을 파악하는 것이지만, 이에 그치지 않

고스탑워치,알람등의다양한기능이추가되며발전되어왔다.현재손목시계의

최첨단 기술으로 볼 수 있는 스마트워치는 상당한 수준의 프로세싱 능력, 통신

능력 및 내재되어 있는 다양한 센서를 가지고 있기 때문에 더욱 많은 잠재력을

가지고 있을 것으로 전망된다. 이 논문은 스마트워치에 내재된 자이로스코프 센

서를 이용하여 인간의 키 타이핑 모션을 인식할 수 있는 새로운 입력 방식으로

가상 키보드 시스템을 제안한다. 사용자가 스마트워치를 착용한 채로 키보드가

아닌 곳, 예를 들면 책상과 같은 표면에 타이핑을 하면 자이로스코프로 측정한

타이핑할때의손목의움직임을통해어떤키를입력하려했는지구분하여대상

디바이스에 가상 키 입력를 전송한다. 우리는 설계한 가상 키보드 모듈을 LG

watch Urbane플랫폼에서구현하였으며,몇몇제약조건을적용한환경에서그

성능을 평가하였다. 그 결과 45.1%의 정확도로 글자들을 최대 2개의 후보까지

줄이는 방식으로 구분해낼 수 있었다. 우리는 이 연구를 통해 스마트워치를 이

용한 새로운 입력 방식에 대한 가능성을 확인하였다. 우리의 미래 연구는 현재

설계한 기법의 프로토타입을 보완하고, 연구 과정에서 설정하였던 몇몇의 제약

조건을 순차적으로 제거해나가는 것이며, 궁극적으로 키보드나 터치스크린을

30

대체할 수 있는 새로운 입력 기술을 제안하는 것이다.

주요어: 가상 키보드, 입력도구, 스마트워치, 센서, 자이로스코프

학번: 2015-21272

31