View
222
Download
2
Category
Preview:
Citation preview
1
Pen-CentricShorthand Interfaces
Charles C. TappertSeidenberg School of CSIS, Pace
University
2
Themes of Presentation
Online Handwriting Recognition and Pen Computing Tutorial
Historical Research – undertaken for the Palm-Xerox Patent Infringement Lawsuit
Recent Research - Enhanced Pen-Centric Shorthand Interfaces can have benefits DPS dissertation could extend M.S. thesis
3
Enhanced Pen-Centric Shorthand Interfaces
Can use word/phrase shorthand to speed text input
Can provide critical infrastructure for many pen-centric applications
Can enhance natural pen-centric interactions for many applications
Will have greatest impact on the utility of applications running on small mobile devices
4
Part 1: Online (Pen-Centric)Handwriting Recognition
Written Languages and Handwriting Properties The Fundamental Property of Writing Handwriting Recognition Difficulties Online (Pen-Centric) Handwriting Recognition Online more accurate than Offline Recognition Online Info Can Complicate Recognition
Process Design Tradeoffs / Design Decisions Computer Problems in English
5
Written Language and Handwriting Properties
Alphabet Letters, digits, punctuation, special
symbols Writing is a time sequence of strokes
Stroke – writing from pen down to pen up Usually complete one character before
beginning the next Spatial order – e.g., in English left to right
6
Fundamental Property of Writing
Differences between different characters are more significant than differences between different drawings of the same character
This makes handwritten communication possible
Can there be exceptions – say, different characters written identically?
7
Fundamental Property of Writing
in English
Property holds within subalphabets of uppercase, lowercase, and digits, but not across them
“I”, “l”, and “1” written with single vertical stroke
“O” and “0” written similarly with an oval
8
Handwriting Recognition Difficulties
Shape, size, and slant variation Similarly shaped characters – U and
V Careless writing
in the extreme, almost illegible writing Resolving difficult ambiguities
requires sophisticated recognition algorithms, syntax/semantics
9
Electronic tablets invented in late 1950s Digitizer and display in separate surfaces
Pen Computers arrived in 1980s Combined digitizer and display Brought input and output into one surface Immediate feedback via electronic ink Created paper-like interface
Online (Pen-Centric) Handwriting Recognition
10
Tablet Digitizers – Dynamic Information
Pen down – indication of inking X-Y coordinates as function of time
Sampling rate: 100 points per second Resolution: 200 points per inch
11
Early Pen-Centric Interface
Different surfaces for input and output
Rand system, about 1959
12
Pen Computers
IBM vision Paper-like interface,
1992
Microsoft Tablet PC Launched, 2001
13
Pen-Centric PDAs
Early Palm Pilot
Palm Tungsten T3 and Sony Clié TH55
14
Online (Pen-Centric) Handwriting Recognition
Machine recognizes the writing as the user writes
Digitizer equipment captures the dynamic information of the writing
Stroke number, order, direction, speed A stroke is the writing from pen down to pen up
15
Online (Pen-Centric) more accurate than Offline (Static)
Recognition
Can use both dynamic and static information
Can often distinguish between similarly shaped characters E.g., 5 versus S where the 5 is usually
written with two strokes and the S with one stroke
16
Online Information Can Complicate Recognition Process
Large number of possible variations E can be written with one, two, three, or four
strokes, and with various stroke orders and directions
A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)
17
Online Information Can Complicate Recognition Process
Other variations
18
Online Information Can Complicate Recognition Process
Segmentation ambiguities character-within-character problem lowercase d might be recognized as a cl if
drawn with two strokes that are somewhat separated from one another
19
Design Tradeoffs/Decisions
No constraints on the user Machine recognizes user's normal
writing User severely constrained
Must write in particular style such as handprint
Must write strokes in particular order, direction, and graphical specification
20
English Writing Styles Handprint
Uppercase – about 2 strokes per letter Lowercase- about 1 stroke per letter
Cursive Script Usually less than 1 stroke per letter Delayed crossing and dotting strokes
21
Computer Problems in English
Constrained Handprint Printing one symbol per box – form filling Printing on lines – symbols can touch or
overlap Unconstrained Handprint
No lines and symbols can touch or overlap
Cursive Script Mixed Printing and Cursive
22
Computer Problems in English
23
Part 2Shorthand in Pen-Centric
PDAs
Famous Uses of Shorthand Historical Shorthand Alphabets Pen-Centric Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Shorthand System
M.S. thesis that could be extended into a DPS dissertation
24
Background
Famous writings throughout history were effectively written in a style of shorthand Cicero’s orations Martin Luther’s sermons Shakespeare’s and George Bernard Shaw’s
plays Samuel Pepys’ diary Sir Isaac Newton’s notebooks
25
Historical Shorthand Alphabets
We first review the history of shorthand systems prior to pen computing
Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases”
Shorthand can be traced back to the Greeks in 400 B.C.
26
Historical Shorthand Alphabets
We focus on shorthand alphabets that might be appropriate for PDAs
We review two types of shorthand Geometric shorthand
Small number of basic shapes Shapes reused in multiple orientations
Non-geometric shorthand shorthand
27
Historical Shorthand Alphabets
Ancient Greeks – 400 BC Tironian Alphabet – 63 BC John Willis’s Stenography – 1602 Gabelsberger Alphabet – 1834 Moon Alphabet – 1845
28
Tironian Alphabet, 63 B.C.Non-Geometric
29
Stenography Alphabet, 1602
30
Stenography Alphabet, 1602
Basic Shapes and Orientations
31
Gabelsberger Cursive-Style, 1834
Non-Geometric Alphabet
32
Moon Geometric Alphabet, 1845
33
Other Historical Shorthand Systems
Phonetic alphabets Pitman (1837), was popular in UK Gregg (1888), was popular in USA
Systems for the blind Braille (1821)
34
Pen-Centric Shorthand Alphabets
Some of the earliest were for CAD/CAM symbols represent graphical items and
commands Others developed for text input on
small consumer devices like PDAs that have limited computing power
We review geometric and non-geometric shorthands appropriate for small devices
35
Pen-Centric Shorthand Alphabets
Historical alphabets presented above could be used for machine recognition symbols drawn with a single stroke (except “K” in Tironian and “+” in
Stenography) In addition to shape and orientation,
online systems can use stroke direction to differentiate among symbols
36
Pen-Centric Shorthand Alphabets
Geometric Pen-Centric Shorthands Organek – 1991 Allen – filed 1991, patent 1993 Goldberg (Xerox) – filed 1993, patent 1997
Non-Geometric Pen-Centric Shorthands Graffiti (Palm Computing) – 1995 Allegro (Papyrus) – 1995
37
Organek Alphabet, 1991
38
Organic Alphabet, 1991 Basic Shapes and
Orientations
One shape in 4 orientations.
This gives 8 directions that together with 3 lengths provide 24 symbols.
A second wheel provides additional symbols.
39
Allen patent, filed 1991
40
Allen patent, filed 1991 Basic Shapes and
Orientations
41
Goldberg patent, filed 1993(“unistroke symbols”)
42
Goldberg patent, filed 1993 Basic Shapes and
Orientations
43
Goldberg patent, filed 1993
5 Basic shapes4 Orientations
2 Stroked Directions40 Possible Symbols
Designed for Speed of Input and Maximum Symbol Separation
44
Shorthand Alphabet Design
How would you design a shorthand alphabet?
What would be the design criteria?
45
Design of Graffiti Alphabetfor the Palm Pilot
Small alphabet Uppercase, digits, special symbols
One stroke per symbol to avoid segmentation difficulty
Separate writing areas for letters and digits to avoid same-shape confusions
46
Graffiti Alphabet, 1995
47
Graffiti Mimics Keyboard Input
Character by character input Mode shifts for
Uppercase Special characters
Eyes can focus on application’s insertion point rather than on input area
48
Graffiti Alphabet Design
What was the additional design criterion?
49
Graffiti Alphabet Design
Designed for ease of learning 20 letters exactly match the Roman
alphabet 6 remaining ones match partially
50
Graffiti Alphabet: 11 of 26 characters
have alternate variations
51
Frequently Confused Characters
52
Other Low Performance Characters
53
Symbol Overlap Comparison
54
Graffiti Recognition Accuracy Study
55
Allegro Alphabet (Papyrus), 1995 (now Microsoft)
56
Simplified Design Tradeoffs/Decisions
for Graffiti and Allegro PDA Alphabets
Small alphabet one case rather than both upper and lowercase
One stroke per character (character = stroke) allows machine to recognize each character upon pen lift
Small number of writing variations per letter preferably only one
Separate writing areas for letters and digits avoids confusion of similarly shaped letters and digits
High correspondence to Roman alphabet for ease of learning
non-geometric, might not actually qualify as shorthand
57
Commercially Successful Shorthands
Similar to the Roman alphabet Easy to learn Graffiti used in Palm OS devices
notably the Palm Pilot & Handspring models Allegro used in Microsoft Windows devices
Geometric alphabets not successful
58
Current Commercial Systems
Company/System Writing Style
Palm Computing/Graffiti*
Special Shorthand Alphabet
Microsoft/Papyrus Allegro
Special Shorthand Alphabet
CIC/Jot Relatively Unconstrained Handprint
Microsoft Relatively Unconstrained Handprint and Cursive
*A few years ago Palm switched from Graffiti to Graffiti2, Graffiti2 is basically Jot licensed from CIC.
59
Jeff Hawkins, innovator 1979 BSEE Cornell, 1979-1986 worked at Intel and GRiD 1986-1987 ABD BioPhysics doctoral program, U.C. Berkeley 1987- back at GRiD he created GRiDPAD, first pen computer 1992 formed Palm Computing, 1993 created first handwriting reco
software product for a mobile handheld - Casio’s Zoomer 1995 Palm Computing bought by U.S. Robotics 1996 created PalmPilot, first PDA with Graffiti shorthand alphabet
(over a million shipped in 18 months, a 66% market share, and the fastest growth of any computing product in history, faster than the TV and the VCR)
1997 U.S. Robotics bought by 3Com (sued by Xerox for patent infringement)
1998 left Palm to form Handspring, 1999 launched the Visor handheld 2000 Palm Computing spun off by 3Com 2002 created what is now the
Redwood Center for Theoretical Neuroscience 2003 Handspring (with Hawkins, et al.) acquired by Palm Computing 2005 Founded Numenta to build the ultimate brain-like machine
60
Palm-XeroxPatent Infringement Lawsuit
The nine-year old battle between Palm and Xerox over handwriting recognition ends in 2006, see article.
Palm pays Xerox $22.5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology
Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements.
61
ConclusionsPalm-Xerox Patent Infringement
Lawsuit
Invalidity Historical research showed that Goldberg
alphabet not so unique Even though the patent was accepted as valid,
these arguments narrowed the scope of the patent
Infringement Analyses and comparisons of the Goldberg and
Graffiti alphabets showed major differences Result was favorable settlement for Palm
62
Pen-Centric Word/Phrase Shorthand
such as Chatroom Shorthand
Further increase speed of text entry
Potential applications Where input speed important Where word/phrase abbreviations
occur frequently – e.g., email
63
Chatroom Shorthand Examples
CU See you, or Cracking up
CM Call me
@TEOTD At the end of the day
^5 High five
2nite Tonight
LOL Laughing out loud
ASAP As soon as possible
B/C or BC Because
64
Allegro/Chatroom Shorthand System
Developed for M.S. dissertation Student was hearing impaired Developed as output component of
communication system Handwriting to text to speech
Two input writing areas One for Allegro (all-purpose) One for chatroom-like or user-defined
words/phrases
65
Allegro/Chatroom Shorthand System
Stroke acquisition GUI
allegro strokerecognition
alphabet
sentence accumulator
Sentence display and spoken output
allegro strokelibrary
user-defined stroke library
a single stroke
other strokerecognition
word/phrasecharacter
done?no
yes
meaning
is it
66
Allegro/Chatroom Shorthand System
67
Allegro/Chatroom Shorthand System
M.S. Thesis Experimental Results
Allegro/Chatroom pen-centric shorthand input is faster than typing text and is comparable to typing text and chatroom shorthand characters
68
ConclusionsPen-Centric Shorthands
Pen-centric interfaces should use shorthand, and especially word/phrase shorthand where appropriate, for fast text input
Benefit of shorthand interfaces Provides critical infrastructure for many pen-
centric applications Enhances natural pen-centric interactions for
many applications Has greatest impact on the utility of
applications running on small mobile devices
69
ConclusionsHandwriting Recognition
Graffiti and Allegro greatly simplified the recognition problem
Handprint problem not completely solved Even with IBM’s ThinkWrite, CIC’s Jot, and
Microsoft products Cursive script problem clearly not
solved
70
References W.B. Huber, S.-H. Cha, C.C. Tappert, and V.L. Hanson, "Use of
Chatroom Abbreviations and Shorthand Symbols in Pen Computing," Proc. 9th Int. Workshop on Frontiers in Handwriting Recognition, IWFHR 2004, Tokyo, Japan, October 2004.
W. Huber, V. Hanson, S. Cha, and C.C. Tappert, "Common Chatroom Abbreviations Speed Pen Computing," Proc. 11th Int. Conf. Human-Computer Interaction, Las Vegas, NV, July 2005.
C.C. Tappert and S. Cha, "Handwriting Recognition Interfaces," Chapter 6, pp. 123-137, in Text Entry Systems, Scott MacKenzie and Kumiko Tanaka-Ishii (Eds.), Morgan Kaufmann, 2007.
C.C. Tappert, C.Y. Suen, and T. Wakahara, "The state-of-the-art in on-line handwriting recognition," IEEE Trans. Pattern Analysis Machine Intelligence, Vol. PAMI-12, pp. 787-808, August 1990.
C.C. Tappert and J.R. Ward, "Pen-Centric Shorthand Handwriting Recognition Interfaces," Proc. 1st Int. Workshop on Pen-Based Learning Technologies, Catania, Italy, May 2007.
Recommended