Upload
fxpal
View
500
Download
0
Embed Size (px)
DESCRIPTION
PACER assists fine grain interactions on touch screen camera phones
Citation preview
PACER: Fine-grained Interactive Paper via Camera-touch Hybrid Gestures on
a Cell Phone
Chunyuan Liao, Qiong Liu, Bee Liew, Lynn Wilcox
FX Palo Alto Laboratory
ACM CHI ConferenceAtlanta, GA, U.S.A.
4/15/2010
Scenario One
I should email Jenny about this interesting article, especial the pulses in this curve
2
Scenario Two
I want to search for the definition of θ in this book
3
Typical Solutions Step 1: Switch to digital media
On-the-fly conversion Capture pictures and apply OCR (Optical Character Recognition)
Retrial of the digital version Find the digital version by file system browsing or web search
Step 2: Locate the specific content Navigate to the specific page and find the figure
Step 3: Interact Mark the interesting region and email Type θ for full text search
4
Problems Media switching
On-the-fly conversion Inefficient, no contextual information, low quality
Finding the digital version File system browsing and typing on cell phones are inconvenient The keyword-based search may be inaccurate
Location & interaction on cell phones Not integrated with paper
Lose the working context already established on paper Redundant document navigation No direct interaction with the paper
Small screen, lower display quality and inconvenient input
5
Video Demo PACER: Paper And Cell phone for Editing and Reading
Better integration of paper and cell phones
6
http://www.youtube.com/watch?v=HEwwx1spujk
Paper vs. Cell phones
7
Seamless integration of their complementary affordances
High display qualityFlexible spatial arrangementInstant accessibilityHigh robustness
Dynamic renderingRich digital interactionDigital communication
Lower display qualitySmall display sizeInconvenient inputLower robustness
Static displayNo computation capabilityNo digital communication
+Computer-like UX on paper
to put paper and digital media on more equal footing for smooth integration
The State of the Art
8
Interaction target granularity Text patches [Erol, 08] Pre-defined map regions [Rohs,04]
Interaction styles Point & Click [Hare,08]
Role of paper Paper as transient input source [Arai,97]
Paper as a proxy of digital documents [Liao,08][Weibel,08][Tsandilas,09]
Recognition mechanisms Barcodes [Rohs,08], RFID [Reilly,06] Anoto [Liao,08][Weibel,08][Tsandilas,09]
Text visual features [Hull,07][Liu, 08]
Image visual features [Lowe,04] [Liu,09]
•Fine granularity
•Rich interaction stylesUser-specified arbitrary content and actions
•Generic documents
Computer-like UX:
PACER
Overview of PACER
9
Highlights Generic paper documents
No barcodes, not RFIDs, no special devices
Text (language-independent), pictures, graphics
Rich gesture-based interaction styles Hybrid camera and touch input
Fine-grained interaction Individual Latin words, Chinese/Japanese
characters, math symbols, user-specified map places and image regions
+
Camera phone + normal printouts
Gestures for fine-grained interaction
Architecture of the PACER system Client-server architecture Data flow
Print Register Capture Recognize Retrieve Interact
10
Overview of the PACER Interface
11
+
Physical-digital InteractionMapping
Fine-grainedCommand
System
1. Semi-real time processing 1. Loose Registration2. Hand Jitter Handling3. Hybrid Camera-touch Gestures
Application
Content-based Physical-Digital Interaction Mapping
Similar to Augmented Reality (AR)
Content-based approach Local image visual features
SIFT [Lowe,04], FIT[Liu,09]
Robust to partial documents, occlusion, scaling and rotation
Generic document content types
Advantages of physical-digital linkage Rich contextual information Persistent digital info. associated with
paper
Feature Extraction
Feature Matching
image registration
12
Highlights (1): Semi-Real-Time Processing Challenge: Recognition and transmission are too
expensive for continuous document tracking ~300ms for 320x240 pictures with a 2.8GHz 4-core CPU ~1000 ms for a complete SOAP call transaction
Solutions: Fast algorithms optimized for cell phones
[Wagner,08]
Remote recognition + local motion detection Recognition is slower but more accurate Camera-based motion detection is faster but accumulates errors Independent of background content (good to loose registration!)
13
PACER Gestures Simulate pen-based computer interfaces [Hinckley,05]
Pointer: an individual word and character Underline, Bracket, Vertical Bar: a text line, sentence and chunk Lasso, Marquee: an arbitrary document region Path: a route Free-from
14
Pointer Underline Bracket Vertical Bar
Lasso Marquee Path Free-form
Highlights (2): Loose Registration Mobile AR UIs not optimized for fine-grained interaction
Inaccurate image registration Low image quality of cell phone video frames
Low resolution, out-of-focus, bad lighting conditions, distortion
Hand jitter and fatigue
15
displaced overlay illegible content
Highlights (2): Loose Registration Our solution
Replace raw frames with high quality images Perform recognition only when required
background-independent motion detection
Advantages over the strict registration of normal AR UIs Robust to inaccurate image registration Better legibility Less demands on phone-paper coordination More flexibility for user interface designs
Stable zoom levels User-changeable control-to-display ratio
16
perfect overlay
Highlights (3): Hand Jitter Handling Hand jitter affects fine-grained interaction too
Inherent with direct freehand pointing Hand-held projector [Forlines,05], Laser pointer [Olsen,01]
Solutions
17
Physical pointer Logical pointer
Filter (Zoom-and-pick [Forlines,05])
Beautification (REXplorer [Kratz,09])
Snap-to-object
Highlights (4): Hybrid Gestures Direct touch manipulation is faster and more intuitive for
within-thumb-reach content
Embodied vs. Touch interaction Touch can enhance fine-grained interaction
18
Highlights (4): Hybrid Gestures Our proposal:
Embodied gestures for faster and coarser navigation Touch gestures for slower and finer navigation Automatically switch with touch down/up
19
Video Demo Hybrid camera-touch gestures
20
http://www.youtube.com/watch?v=E9hR5D_mQvs
Video Demo of PACER Applications Highlights
Fine-grained content manipulation Contextual information Rich and intuitive interaction with paper Generic document content types
21
http://www.youtube.com/watch?v=PNqUcC0YZ78
Preliminary User study (1) Task
Select designated words and pictures in a marked printout
Participants Six colleagues not affiliated with
PACER
Settings 4 testing pages 400+ database pages 2 sessions x 16 trials
22
Tracking tag
Preliminary User study (2) Overall feedback was positive
Novel ideas, useful for mobile settings
Document recognition 81.6% accuracy for the 1st shots Failure sources
23
Motion blur Out of range Shadow
Preliminary User study (3) Embodied vs. Touch
Embodied gestures are faster but more difficult to learn Caused by inadvertent phone movement and unfamiliar mental model
Touch gestures are more familiar but may be hard for one-handed operations
Loose vs. Strict registration Feedback on loose registration was positive
The retrieved high quality documents were appreciated Lower mental and physical demands Depends on the completeness of the digital models
24
Preliminary User study (4) One vs. two-handed input
One-handed input is more flexible for simultaneous manipulation of paper and the phone
Two-handed input is more stable and easy for touch gestures
25
Participant-initiated two-handed interaction
Conclusion & future work PACER is a cell phone-based interactive paper system
Generic paper documents linked to rich digital information Flexible interaction with the hybrid camera-touch gestures Fine-grained content manipulation
Future work Understand embodied/touch and one/two-handed interaction Investigate loose vs. strict registrations in more application
scenarios Integrate PACER with other devices like mobile projectors and
digital pens Explore more application areas
26
Thank You! Acknowledgement
Our colleague participants Don Kimber and Tony Dunnigan CHI reviewers
More resources
http://www.fxpal.com/paperui/
27