MoLe: Motion Leaks through Smartwatch Sensors

Authors: He Wang, Ted Tsung-Te Lai, Romit Roy Choudhury

Presentation: Joe Sirrianni

Central Question

• This work tries to answer the following question:

Can an accelerometer and gyroscope data from smart watches be mined to infer the words that a user is typing?

Assumptions and Setup

Assumptions:• Smart watch is on the left hand.• Only individual words are being typed.

Setup:• Samsung Gear Live smart watch

Approach: Use the motion of the hand to determine which letter are being typed.

Initial analysis: using camera to measure distance

MoLe System Overview

Subproblem: Keystroke Detector

• Used bagged decision trees to determine if a Z-axis acceleration was a key press or not.

Subproblem: Key-Press Location Estimation

• Created a process to determine key press location

• The process had 5 steps:1. Find Gravity and define

coordinates2. Estimate and Remove Gravity 3. Estimate hand location and

calculate project acceleration4. Calibrate by mean removal5. Kalman smoothing – Remove

remaining gravity error.

Subproblem: Point Cloud Fitting

• Essentially matching incoming data to training data. • Used Convex hulls to overlay the training data on the incoming data

Subproblem: Inferring the typed word

• This involved estimating:• The total number of keystrokes (The size of the word)• Consecutive Characters that won’t be detected by motion (“er”, “re”, “ea”,

”fa”)• Accounting for watch displacement (accounting for minute orientation

differences)• Accounting for transitioning between characters (v -> a, r -> a)• Keystroke intervals – how long it takes to type• Matching to dictionary words – had to estimate letters typed by right hand.

Experiment

• Training set – 2 people typed 500 common words• Testing set – 8 different people typed 300 common words

Results: How good is MoLe at guessing each word?

• User who typed more toward guidelines had better accuracy.

Results: Keyboard Variant

• Does using a different keyboard affect the results?• They trained the model on a laptop keyboard and tested with a user

on a desktop keyboard.

• They found that using a different keyboard did not affect the accuracy significantly.

Take-aways

• Wearable technology can leak information that can be exploited!• Smart watches can be used to guess the words a user is typing just

from the accelerometer and gyroscope data.

Criticisms

• I think this technology is more suited for breaking passwords than reconstructing blocks of text.

• Both the @ symbol and the TAB button are typed by the left hand.

Bonus: Can you guess the correct sentence?

Answer

The most profound technologies are those that disappear - Mark Weiser 1991