Evaluation of Multimodal Input for Entering Mathematical

Equations on the Computer

Lisa Anthony,Jie Yang, Kenneth R. Koedinger

Human-Computer Interaction Institute

Carnegie Mellon University, Pittsburgh, PA

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Programming constructs or special syntax

Linearization of mental representation

Difficult to revise structure in template-based editors

Computer-Based Math Tools

MATHEMATICA

MICROSOFT EQUATION EDITOR

Maple V, Mathematica, Matlab; Microsoft Equation Editor, …

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Problems:►Large learning curve►Tied to keyboard and

mouse input

Computer-Based Math Tools

MATHEMATICA

MICROSOFT EQUATION EDITOR

Maple V, Mathematica, Matlab; Microsoft Equation Editor, …

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Notations already exist for paper-based math

Affordance for spatial representation

Especially good for students learning math

Problem: recognition accuracy

Pen-Based Computing

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Solution: Multimodal Input

Increased robustness►Better recognition accuracy with multiple input

streams (Oviatt, 1999; pen gestures + speech)

Consider both repair-only and simultaneous input in both streams

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Motivation for User Study

Literature says typing is faster (Brown, 1988) ►Compared inputting paragraphs of English text►Math is different domain

Little evaluation done of pen-based equation input►Systems constrained by recognition accuracy

(Smithies et al, 2001)

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: DesignInput method

Equation complexity► Number of characters in equation► Number of “complex” symbols (e.g., √ and ∑)

Keyboard-and-mouse Used keyboard and mouse with Microsoft Equation Editor (MSEE).

Handwriting-only Wrote in Windows Journal program; without recognition.

Speech-only Dictated into microphone; without recognition or visual feedback.

Handwriting-plus-speech

Both spoke and wrote in series or in parallel (user choice); without recognition.

“Recognition” means system tries to interpret user input.

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Participants

48 paid participants (27 male, 21 female)

Undergraduate/graduate, full-time/part-time students at Carnegie Mellon

Native English speakers only

Most (33 of 48) had no experience with MSEE before study

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Procedure

Entered math equations on TabletPC►36 equations (7 + 2 practice per condition)►Conditions counterbalanced across participants

Instructions for each condition►No prompting for specific ways of expressing

equations►5 min “explore time” for MSEE

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Measures

Time per equation

Number of errors per equation (corrected and uncorrected)

User preferences before and after session

Equation entry screen inhandwriting condition.

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Sample Stimuli

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Results

Average time in seconds per equation by condition. Error bars

show 95% confidence interval (CI).

05

101520253035404550

Keyboard-and-mouse

Handwritingonly

Speech only Handwriting-plus-speech

Condition

Tim

e (s

eco

nd

s)

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Results

0

0.5

1

1.5

2

2.5

Keyboard-and-mouse

Handwritingonly

Speech only Handwriting-plus-speech

Condition

Nu

mb

er o

f E

rro

rs

Mean number of user errors made per equation by condition. Error bars

show 95% CI.

Human-Computer Interaction Institute Anthony, Yang, Koedinger

User Study: Results

00.5

11.5

22.5

33.5

44.5

5

Keyboard-and-mouse

Handwriting only Speech only

Condition

Sco

re (

1 to

5)

Preference questionnaire rankings of each condition on a 5-point Likert scale. Error bars show 95% CI.

Pre-test Post-test

00.5

11.5

22.5

33.5

44.5

5

Keyboard-and-mouse

Handwritingonly

Speech only Handwriting-plus-speech

Condition

Sco

re (

1 to

5)

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Conclusions

Handwriting faster, more efficient, and more enjoyable to novice users than standard keyboard-and-mouse

Handwriting-plus-speech faster and better liked than keyboard-and-mouse

Handwriting-plus-speech not much worse than handwriting alone, so multimodal may be a winner for technology reasons

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Further Analyses

Transcription of spoken input as corpus for generation of language model

Consistency across and within users in handwriting and speech► Ambiguity resolution► Self correction► Pausing and synchronization in multimodal input

Greater variability within speech condition than within handwriting condition

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Further Analyses

Transcription of spoken input as corpus for generation of language model

Consistency across and within users in handwriting and speech► Ambiguity resolution► Self correction► Pausing and synchronization in multimodal input

Greater variability within speech condition than within handwriting condition

(Supported by Oviatt et al, 2005)

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Questions?

Project Webpage:

http://www.cs.cmu.edu/~lanthony/research/multimodal/

Pittsburgh Science of Learning Center:

http://www.learnlab.org/index.php

Human-Computer Interaction Institute Anthony, Yang, Koedinger

ReferencesBrown, C.M.L.: Comparison of Typing and Handwriting in “Two-Finger

Typists.” Proceedings of the Human Factors Society (1988) 381–385.

Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576–583.

Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53–67.

Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4–14.

Human-Computer Interaction Institute Anthony, Yang, Koedinger

Other References in PaperAnderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.:

Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167–207.

Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557–582.

Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63–68.

Microsoft.: Microsoft Word User’s Guide Version 6.0 (1993), Microsoft Press.

Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257–285.