Master’s Thesis

A Framework for Mobile Paper-based Computingby

Tomas Sylverberg

LITH-IDA-EX--07/004--SE

2007-02-14

Master’s thesis

A Framework for Mobile Paper-based Computing

by

Tomas Sylverberg

LITH-IDA-EX--07/004--SE

Supervisor : Dr. Erik BerglundDept. of Computer and Information Scienceat Linkoping University

Examiner : Dr. Erik BerglundDept. of Computer and Information Scienceat Linkoping University

Abstract

Military work-practice is a difficult area of research where paper-based ap-proaches are still extended. This thesis proposes a solution which permitsthe digitalization of information at the same time as work-practice remainsunaltered for soldiers working with maps in the field. For this purpose, amobile interactive paper-based platform has been developed which permitsthe users to maintain their current work-flow. The premise of the solutionparts from a system consisting of a prepared paper-map, a cellular phone,a desktop computer, and a digital pen with bluetooth connection. Theunderlying idea is to permit soldiers to take advantage of the informationa computerized system can offer, at the same time as the overhead it in-curs is minimized. On one hand this implies that the solution must belight-weight, on the other it must retain current working procedures as faras possible. The desktop computer is used to develop new paper-drivenapplications through the application provided in the development frame-work, thus allowing the tailoring of applications to the changing needs ofmilitary operations. One major component in the application suite is asymbol recognizer which is capable of recognizing symbols parting from atemplate which can be created in one of the applications. This componentpermits the digitalization of information in the battlefield by drawing onthe paper-map. The proposed solution has been found to be viable, butstill there is a need for further development. Furthermore, there is a needto adapt the existing hardware to the requirements of the military to makeit usable in a real-world situation.

Keywords : pen-gesture recognition, paper-based user interfaces, ubiq-uitous computing, software engineering, digital pen

iii

iv

Acknowledgements

I would like to thank my supervisor, Erik Berglund, for proposing such aninteresting topic for this master’s thesis. Further, I would like to thank OlofLeifler for all the support and help with LATEX. It has been a pleasure to beable to discuss my problems with Anders Larsson and Per-Ola Kristenssonwho had many interesting ideas to share. Further, a big thank you to allthe people who have tested the applications and have thereby given me theambition to work even harder to improve the system.

I would also like to thank my parents, my brother and my girlfriendwho all have supported me during this thesis. Without your support thisthesis would not have been what it is today.

v

vi

Contents

1 Introduction 11.1 Research Question . . . . . . . . . . . . . . . . . . . . . . . 31.2 Ubiquitous Computing . . . . . . . . . . . . . . . . . . . . . 31.3 Research Approach . . . . . . . . . . . . . . . . . . . . . . . 4

1.3.1 Supporting Development of Interactive Maps . . . . 51.3.2 Supporting Extensible Maps . . . . . . . . . . . . . 51.3.3 Supporting Different Coordinate Systems . . . . . . 61.3.4 Supporting Different Interaction Devices . . . . . . . 61.3.5 Supporting Symbol Recognition . . . . . . . . . . . . 61.3.6 Supporting Plug and Play of Components . . . . . . 7

1.4 Paper Summary . . . . . . . . . . . . . . . . . . . . . . . . 71.5 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 System Design 92.1 Component Based Development . . . . . . . . . . . . . . . . 92.2 The Development Framework . . . . . . . . . . . . . . . . . 11

2.2.1 Desktop Applications . . . . . . . . . . . . . . . . . 112.2.2 Cellular Phone Application . . . . . . . . . . . . . . 132.2.3 Digital Pen and Paper . . . . . . . . . . . . . . . . . 14

3 Implementation 173.1 Development Platform . . . . . . . . . . . . . . . . . . . . . 17

3.1.1 Java 2 Micro Edition . . . . . . . . . . . . . . . . . . 173.1.2 CLDC and MIDP . . . . . . . . . . . . . . . . . . . 19

vii

viii CONTENTS

3.2 Symbol Recognition . . . . . . . . . . . . . . . . . . . . . . 203.3 Symbol Format . . . . . . . . . . . . . . . . . . . . . . . . . 243.4 Use of XML in the Project . . . . . . . . . . . . . . . . . . 24

4 Conclusion 27

5 Article 295.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.2.1 Symbol Recognition in the Field . . . . . . . . . . . 315.3 The Kartago Project . . . . . . . . . . . . . . . . . . . . . . 315.4 Interactive Maps in the Field . . . . . . . . . . . . . . . . . 335.5 System Overview . . . . . . . . . . . . . . . . . . . . . . . . 345.6 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5.6.1 Support Complex Pen-gestures . . . . . . . . . . . . 355.6.2 Robust . . . . . . . . . . . . . . . . . . . . . . . . . 355.6.3 Support Low-powered Devices . . . . . . . . . . . . 355.6.4 Easy to learn . . . . . . . . . . . . . . . . . . . . . . 355.6.5 Easy to maintain . . . . . . . . . . . . . . . . . . . . 35

5.7 Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 365.7.1 Symbol Language . . . . . . . . . . . . . . . . . . . . 365.7.2 Recognition Process . . . . . . . . . . . . . . . . . . 36

Related Work . . . . . . . . . . . . . . . . . . . . . . 36Algorithms . . . . . . . . . . . . . . . . . . . . . . . 38

5.8 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.8.1 Participants . . . . . . . . . . . . . . . . . . . . . . . 415.8.2 Material . . . . . . . . . . . . . . . . . . . . . . . . . 415.8.3 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . 415.8.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . 415.8.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . 42

Error . . . . . . . . . . . . . . . . . . . . . . . . . . 42Latency . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.10 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 43

Bibliography 44

Chapter 1

Introduction

Ever since the advent of computers, engineers have continuously managedto reduce the size of the computer systems. The first computers occupiedentire rooms, and it wasn’t until the desktop computers made their en-trance, that computers became available to the general public. The thirdgeneration of computing is called ubiquitous computing, a term coined byMark Weiser in the 1990’s [1]. The ubiquitous computer is supposed tobecome invisible at the same time as it provides the benefits of using acomputerized system. An important implication of ubiquitous computingis that the computer system should allow people to perform their normaltasks in their normal environment, but with computer support.

In this context a natural mean of interaction is paper and pen, whichhas formed part of people’s everyday life for a long time. Today, althoughcomputers are making their entrance in people’s life, work-practice stillinvolves the use of paper and pen. For instance, military work-practice,relies heavily on working with maps. Soldiers draw on maps, and discussstrategies looking at maps. Current work-practice, however, implies thatif the operators at headquarters want to have a complete overview of thesituation, it is necessary for all units to report their findings manually. Thearguments for having the information in a digital form is growing strongeras critiquing systems are being developed.

Paper as a tool for interaction possesses various features which makes

1

2

it attractive. A system which is based on a paper application is easy totransport, and it has low weight. Further, paper has a high resolutionwhich is demanded in, for example, military applications. In the future,a combination of paper and pen interaction can become reality in elec-tronic paper. The technology at this stage, however, proposes the use ofdigital pens together with specialized paper in order to get a version ofelectronic paper. Whereas computer systems which use computer screenshave problems with low resolution and low portability, this is not the casewith paper which combines a higher resolution, and the portability of pa-per [2]. When electronic paper, in its true sense, will become available theability to provide visual feedback directly from the computer system willbe added.

A further benefit from using paper is that it provides a natural mecha-nism for graceful fallback, which is a feature that can be important in, forexample, military applications. If the computer system fails to function,the information is still available on the paper. For military purposes theweight of the soldiers equipment is crucial. Laptops and tablet PCs arehighly portable, but their weight is considerably higher than a system con-sisting of, for example, a cellular phone, paper and pen. Currently thereis no other solution than paper maps which allows the use of a large 1,high-resolution image.

A probable reason why computerized systems haven’t been acceptedin the military is their weight, low resolution, and unnatural interactionmodels. This calls for the development of applications which better suitsthe specific needs of this user group. The goal with the project was todevelop a tool for military use in which a digital pen could be used asan interaction device to work on maps. The main benefit from such asystem was expected to be an augmented map representation together withdigitalization of the information.

1The size of a paper map can typically be far bigger than A3-format. Even very bigpaper maps are wearable and can easily be folded.

1.2. Ubiquitous Computing 3

1.1 Research Question

The main research question was to investigate in which way it is possibleto support the use of an online paper-driven computer system in militarywork-practice, where multiple users should be able to interact and shareinformation. The requirements on the solution are implicitly decided bycurrent work-practice, and limitations of the technology. To be usable, thesolution must fulfill the following requirements:

• The solution must be light-weight

• The solution should be based on paper maps

• The solution should facilitate the development of new interactive pa-per maps, thus enabling rapid prototyping

• The application should support typical pen-based interaction, for in-stance symbol writing

In this thesis a solution to the military’ needs can be found in the formof a framework for paper-driven applications. This thesis describes howa solution to the problem was developed. The thesis is multidisciplinaryin the sense that it treats several disciplines within Computer Science.Ubiquitous Computing, Paper-Based Interfaces and Software Engineeringare all fields that are related to this work.

1.2 Ubiquitous Computing

One of the ideas behind ubiquitous computing is the integration of comput-ing into normal life in a way which requires less attention on the computerinteraction while the benefits from using the system are substantial [3].When developing ubiquitous computing solutions the need for explicit in-teraction should be kept to a minimum, which in turn means that theinteraction with the computer virtually melts together with the environ-ment [4]. The idea to hide computer interaction can be traced back toMark Weiser who in 1991 wrote “The most profound technologies are thosethat disappear” and“They weave themselves into the fabric of everyday life

4

until they are indistinguishable from it” [1]. He also writes that the mainproblem of getting the computer out of the way is not the GUI, but ratherthe whole context of using the computer [5]. One way of taking the com-puter out of its current context is to make it mobile. Therefore, mobilityis an important issue in ubiquitous computing. It permits the computer’susers to use the computer anywhere. Technology available nowadays al-lows users to use their cellular phones in much the same way as if theywould use a desktop computer. However, there is a need for improvementwhen it comes to how interaction is carried out. Most systems rely heav-ily on a touch screen, which uses traditional desktop system based menustructures. As pen-based applications become more extended the need todevelop pen-based user interfaces becomes more important [6]. Given thatthe use of pens to interface a computer system is relatively new, there isa need to investigate how a user interface which permits easy navigabilitythrough the system can be constructed, given that there is no possibilityto hide inactive menu items. A further design consideration when workingwith ubiquitous computing is that the designer can not expect to keep theusers attention during longer periods of time [4]. The implication of thisis that the system must be very easy to understand, and require a smallamount of time to learn. Existing experimental systems usually requireheavy equipment, and alter work-practice, thus forcing the users to learnto work with the system. Such a system is the BARS system [7] which usesa head-up display and a mouse to interact with a laptop computer.

1.3 Research Approach

Within the context of this thesis the focus is set on the traditional papermap as the central work tool in an interactive system for the soldiers’ work-flow.

The hardware platform that the project work with is paper maps usingan Anoto pen [8], paper, and a cell phone for communication and feedback.There are several reasons why this hardware platform has been chosen.First and foremost, it is highly wearable and uses to great extent technol-ogy which people is familiar with. Further, the development of new cellularphones with computer-like qualities is everyday more present which pro-

1.3. Research Approach 5

vides a platform where computing power will be improved at the sametime as the platform is maturing. The choice to use an Anoto pen over anyother device is natural since this technology is the only one which currentlyallows to build a system which, except for a computer2, does not requireany additional tools than the special paper3 and the pen to work. Alto-gether this provides us with a very light-weight system that also fits in aleg pocket or similar. However, there are also a series of software designissues to solve.

1.3.1 Supporting Development of Interactive Maps

Paper-maps typically cover limited regions, and are very detailed. Thisimplies that in a typical setting, several paper-maps are needed to coverthe battlefield. In a ubiquitous computing setting, with augmented maps itis necessary to individually prepare the map for each region. This calls fora method which permits rapid preparation of paper-maps. Furthermore,each paper-map can be seen as its own application which means that dif-ferentiated behavior could potentially be built-in to the maps. To be ableto support both of the discussed ideas a component based approach towarddevelopment of new applications was adopted. The underlying idea was topermit an inexperienced user to build different paper-map applications byselecting building blocks from a menu.

1.3.2 Supporting Extensible Maps

One important aspect which was thought of was to allow new functionalityto be added to an already working map. To make this possible a method torepresent a map had to be developed. For this purpose a XML-format wasdeveloped which should support short development time, at the same timeas it should be expandable. Further reasons to use a XML-format is that itprovides a common platform for data-exchange between different platforms.The map representation was heavily influenced by the component basedapproach. The combination of a XML-driven application structure together

2Computer here is used in its broadest sense, including any kind of device which hascomputational power such as PDA,cellular phone, embedded system

3The paper has a special pattern printed on it.

6

with a component based approach allows the creation of applications on thefly. Focusing on how the development time for new paper-maps could beaccelerated an application which automatically generates files in the XML-format was developed. This also allows to comply with the wish to allowinexperienced users to develop applications. Further, the application limitsthe need to have knowledge about programming on the J2ME platform toa minimum.

1.3.3 Supporting Different Coordinate Systems

One important aspect when working with maps are different coordinatesystems, and how to handle them. For this thesis, the simplifying assump-tion that the coordinate system can be considered flat for small geographicregions was adopted. For more specialized uses it might be necessary toexchange the current representation with a more general one.

1.3.4 Supporting Different Interaction Devices

To be able to support different interaction devices the concept of an inputdevice hook was introduced into the system. This design decision permitsthe user to allow any kind of interaction device to be connected to thesystem as long as appropriate drivers are available.

1.3.5 Supporting Symbol Recognition

One crucial component to make pen-based interactive maps a good solutionis symbol recognition. It became obvious that to be able to support thesoldiers’ current work-practice it was necessary to allow them to use theirregular symbols when drawing on maps. As a result a literature study wasstarted, and based on the information found about existing systems a partlynew approach to symbol recognition was chosen. The choice to develop adifferent solution than the ones provided by various researches was based onthe underlying platform, which has limited computational power. Further,symbol recognition is often based on glyphs rather than complex figureswhich automatically disqualified many of the methods, since they couldnot work with this kind of figures. Symbol recognition is a complex field in

1.4. Paper Summary 7

which entire studies focus on, for example, recognizing numbers, or othervery specific fields [9]. For the purpose of this thesis a symbol recognizerwhich would allow the recognition of virtually any kind of symbol hadto be developed. The choice to use a template-based approach towardsymbol recognition, meant that it was necessary to represent symbols in adata format. There are very different ways how this can be carried out,sometimes bitmaps are used, other times database entries are used. In thiscase XML was found to be an appropriate format.

To be able to quickly add new symbols to use with the symbol recog-nition a new desktop application was developed with this in mind. Thesymbol recognizer was first developed as a desktop application, since itwould be more efficient to test the algorithm in a desktop setting. The firsttests of the symbol recognition algorithm resulted in a desktop applicationwhich can be used to evaluate how well symbols are recognized, since thesame algorithm was ported to the Java 2 Micro Edition platform.

1.3.6 Supporting Plug and Play of Components

The last step in the development process was to develop a set of plug-inswhich would support the military in their work in the field. The resultis an application which can recognize a set of military symbols, attachinformation to the map coordinates related to the symbol, and give auditoryfeedback of what the system has detected. Other plug-ins makes it possibleto send SMS/MMS messages, download new maps through web-servicesand draw on the map. An important application is to show informationwhen clicking on areas which have been triggered with information.

The component based approach has permitted the development of newplug-ins knowing that a basic system is working and that any additionalfeatures can be implemented if time permits it.

1.4 Paper Summary

This thesis also contains a paper manuscript, focused on the symbol recog-nition part of the work performed within the context of the work. Thepaper is written with the intent of submitting it to a renowned interna-

8

tional conference. The following text is the abstract of the article, thearticle can be found in Chapter 5.

We present a robust on-line handwritten recognition system that recog-nizes NATO symbols drawn on paper maps using a digital pen and a mo-bile phone. Our user evaluation shows that very high accuracy is achieved.Furthermore, our proposed system has low latency running on off-the-shelfmobile phones. The recognition system is part of a research effort in aug-menting existing work-practices of soldiers in the field with very lightweightcomputer support.

1.5 Glossary

CDC The Connected Device Configuration is part of the Java runtimeenvironment used by, for example, TV top-set boxes.

CLDC The Connected Limited Device Configuration is one part of theJava runtime environment used by, for example, cellular phones.

J2ME The Java 2 Micro Edition is a Java platform intended for deviceswith limited computational power

J2SE The Java 2 Standard Edition is the Java platform which is mostlyused for desktop computers

JVM The Java Virtual Machine

KVM The Kilobyte Virtual Machine is a virtual Java machine with lim-ited memory footprint

MIDP The Mobile Information Device Profile is one part of the Javaruntime environment used by, for example, cellular phones

MIDlet A Java program which requires MIDP to be able to execute. Theconcept is similar to an Applet in J2SE

Pen-gesture In this context equivalent to a number of strokes which makeup a symbol

XML Extensible Markup Language

Chapter 2

System Design

The specific constraints dictated by the way military work, together withthe hardware constraints dictated how the system was developed. Thefollowing presentation of the system describes the key components to un-derstand the system. A general overview is given in figure 2.1.

2.1 Component Based Development

The use of paper-based maps in military applications imply that eachpaper-map must individually be crafted as an application. For this pur-pose, three support programs were developed for a desktop computer. Theapplication soldiers would use is executed on a cellular phone with supportfor MIDP/CLDC 1.

Military use of maps has traditionally been very extended, but there hashardly been any development of its uses. Ubiquitous computing permitsnew applications which can improve the use of maps in military practice.To support differentiated applications within the same support frameworka component based approach was chosen. One benefit from a componentbased approach is that it supports incremental development. Further, com-ponents can be developed separately and then connected together [10]. This

1Mobile Information Device Profile/Connected Limited Device Configuration

9

10

XML

Kartago System Symbol Creator

Cellular phone application

Symbol recognizer application

Figure 2.1: System overview

flexibility allows the adaption of the system’s functionality which givesshorter development time. Another possibility is to develop small compo-nents which are combined into more complex systems. This functionalityhas been used in various components to create complete sub-applications(plug-ins).

Currently the cellular application is controlled using a stylus pen, butthe system also supports the use of a digital pen. When a digital pen isconnected, the system provides dual input possibilities, which provides afallback to stylus if the digital pen would stop working. In a more generalsetting an application developer could provide different ways to interactwith the system based on the hardware that is available. The frameworksupports any kind of input device to be used as long as it complies to aninterface. How the device handles its input to comply with the interface is aproblem the device driver programmer has to consider. Since the softwarecan run on very different platforms it might be necessary to support some

2.2. The Development Framework 11

kind of fallback if certain functions are missing, for example, a plug-inwhich permits the user to compose a MMS message allows cellular phoneswith a compatible camera to attach a picture taken with the camera to theMMS message. The photo could potentially also be connected to an areaon the screen. A user with a cellular phone without a camera will only beable to attach photos that are already stored in the phone.

2.2 The Development Framework

The implementation can be divided into two branches, one for desktopusers, and the other for cellular phone users. In the desktop environmentthere are three applications 2, which can be seen as a development frame-work.

2.2.1 Desktop Applications

The Kartago System Editor is the core of the desktop application develop-ment framework (see figure 2.2). The program permits a user to load animage and on top of the image draw boxes and polygons. The drawn figurescan then be connected to functions (plug-ins). The actual connection offunctions is achieved by choosing appropriate plug-ins from a drop-downmenu whose items are loaded based on what plug-ins are available3 in thedevelopment area. It is also possible to submit arguments to the plug-inswhich can modify the plug-in’s behavior. The result of the editing can besaved to a XML file which can then manually be installed on the cellu-lar phone. Another option is to put the file in a database and retrieve iton-demand using web services.

2Two of the applications are developed in two different versions- supporting differentinteraction methods

3The plugins are in a separate namespace to avoid name-clashes. When a new pluginhas been developed it will appear in the drop-down box automatically.

12

Figure 2.2: Left: Main window. Right: Connecting a plug-in to an area

Figure 2.3: Left: Main window. Right: Creating a new symbol

2.2. The Development Framework 13

The Symbol Creator is an application which is used to create symbolsand save them in a XML-format (see figure 2.3). For each symbol it ispossible to specify if it should be able to rotate the symbol, how far itshould be between different rotations etc. The most important function is,however, the creation of new symbols which can be used for the symbolrecognition engine 4. A symbol is created by drawing its representationon a canvas. One important issue is that the user must draw the symbolin the same order as the end-user should do it. As soon as the drawingis done the result is sampled to a specified number of equidistant samplepoints. The symbol is also translated and scaled. By creating differentsets of symbol files it is possible to use different templates depending ondifferent situations which makes recognition more reliable.

The Symbol Recognizer is an application to test how well the recognitionof the symbols work in a desktop setting (see figure 2.4). It uses the samesymbol recognizer as the cellular phone application which assures its useas an indicator of how well symbols created with the framework will becorrectly identified on the cellular phone platform. One important featureis that the application supports dual input, it can be controlled by anydevice that the system recognize as a pointer device, as well as with anAnoto digital pen 5.

2.2.2 Cellular Phone Application

The Kartago System for Cellular Phones is an implementation which usesthe framework and shows that the framework is a viable solution to use.Further, it demonstrates various of the plug-ins that have been developed.The application presents the user with a map which has hot-spots 6. Thefunctionality of the system depends entirely on what kind of functionalitythe current map has been equipped with. The functionality of a map isdefined in a XML-file which is created using the development platform. Inthe most primitive application, a map has no additional functionality which

4The symbol recognition is template-based, and uses an algorithm which permits thedetection of complex symbols.

5For the Anoto digital pen a driver had to be written which in real-time processesthe streaming data from the pen.

6Hot-spots are areas on the map which are augmented with functionality.

14

Figure 2.4: Recognizing a symbol. The figure in the lower-right corneris the one drawn by the user. The template of the identified symbol isshown in the upper-left corner. In the lower-left corner the identifier of theidentified symbol is shown.

means that the application will only show the map with no possibility tointeraction. When hot-spots are introduced the scenario changes consider-ably. The plug-in library provides various functions which can be connectedto the map. The plug-ins usually extend the use of the application, andprovide a method to tailor applications to different scenarios. There are,for example, plug-ins to recognize military symbols and attach informationto them, draw on the map, send SMS/MMS- messages in various ways,play sounds etc.

2.2.3 Digital Pen and Paper

For this project a digital pen which uses Anoto technology [8] has beenemployed. There are several approaches to how the use of pen technology

2.2. The Development Framework 15

can be used together with computer systems. Tablet PCs have a touchscreen, when doing gestures on the screen with a stylus the pointer on thescreen follows the stylus’ movements. Taking a step toward the solutionused in this thesis it is worth mentioning systems that use paper togetherwith a digital pen, but where the paper has to be placed on top of a specialdevice which detects what the user is doing with the pen. Anoto usesan approach which only requires a specialized digital pen together with aspecially prepared paper [8]. This approach provides an extremely easyway to interface computer applications [11]. The digital pen communicatesthrough bluetooth. This meant that a driver for a cellular phone 7, as wellas, a driver for desktop computers were developed. The optical sensors inthe pen recognize where on the paper the user has made a gesture, withwhat pressure, and when the gesture was made. For this project a digitalpen which uses streaming was used. Earlier versions used a tick-box tosignal to the system to send the saved strokes to a device. Further, oldersystems had a very slow response time, which implied that the systemcould not be used for a setting where responsiveness is important. Thedigital pen used in this project does not have any of the two problemsits predecessors had. Instead it demonstrates good responsiveness togetherwith high-accuracy localization on the paper. It stands, however, clear thatthe digital pen, in its current form, is not suitable for military missions sinceit is a sensible piece of equipment.

The digital pen used in this project needs a specially prepared paperto be able to function. The preparation of the paper consists in printing apattern on the paper. The pattern consists of small dots with a nominalspacing of 0.3mm [8]. These dots form a unique pattern which is interpretedby the digital pen. There are various types of patterns, some are used forstreaming data, whereas others are used for batch-processing of data. Thepattern covers a region greater than Europe and Asia combined [8] whichmeans that there are good possibilities to create applications with very bigpaper formats.

7A bluetooth service had to be developed

16

Chapter 3

Implementation

This chapter introduces the development platform and discusses some of thetechnologies used. Symbol recognition is a complicated area. Therefore, inthis chapter a description of how the symbol recognition was implementedin this project is presented. XML has been used extensively in the project,and therefore a short explication to how it has been used is included.

3.1 Development Platform

The project is implemented using Java in two different flavours. The mainsystem, which runs on a cellular phone, is programmed in Java 2 MicroEdition 1 whereas the rest of the programs are programmed using Java 2Standard Edition 2.

3.1.1 Java 2 Micro Edition

To better understand how development of Java applications for small de-vices is done a brief introduction is given in the following text. The J2MEcontext can be seen in figure 3.1.

1From know on J2ME2From know on J2SE

17

18

Figure 3.1: J2ME versions

J2ME is a Java version especially designed for small devices. It currentlysupports a broad range of devices such as PDAs, TV set-top boxes, cellularphones, printers and embedded systems. The differences between J2MEand J2SE are important since it is not possible to directly compile codewritten for J2SE on J2ME. Within J2ME there are differences betweenwhat can programmatically be done depending on the deployment platform.Different devices may use different parts of the framework (as can be seenin figure 3.1. These are called configurations. The basic functionality of thedevelopment platform is, as has been mentioned earlier, similar to J2SE.However, there are some differences given that J2ME runs on less powerfulhardware and therefore has more limitations.

One important aspect of programming in J2ME is that although manyof the same classes exist in J2SE, they usually do not have the same meth-ods. Important differences can be found in, for example, the String andVector classes. Further, depending on deployment platform there may evenbe differences within J2ME when it comes to what functionality the classesprovide.

3.1. Development Platform 19

3.1.2 CLDC and MIDP

Since applications that use J2ME potentially have limited I/O possibilitiesthe platform uses a different approach to GUI-programming. The program-ming model is depicted in figure 3.2.

Figure 3.2: Programming model for cellular phones

In a normal desktop environment users are used to work with systemsthat support multiple windows open at the same time. While J2SE providesthe Swing and awt packages for GUI-programming, this is not true for theJ2ME development environment. For the platform used in this project, aJava-enabled Sony Ericsson cellular phone, the lcdui package is central toGUI- programming.

The Connected Limited Device Configuration or CLDC is a J2ME con-figuration which is designed for devices with limited memory, low process-ing power and primitive graphical capabilities [12]. More advanced devices(such as advanced PDAs and TV top-set boxes) use a different profile, theConnected Device Profile (CDC).

Currently the CLDC and CDC are the only two J2ME configurations,and together they lay out the foundation of the J2ME framework [13]. Al-though applications developed for the two different platforms are both bun-dled as J2ME applications there are fundamental differences. The CLDCuses KVM 3 while the CDC uses JVM 4, which means that the CLDC hasa smaller memory footprint, at the expense of being more limited.

The CLDC used for this project is version 1.1 which incorporated var-ious enhancements compared to its predecessor CLDC 1.0. The most im-

3Kilobyte Virtual Machine4Java Virtual Machine

20

portant feature in the new version for the work with this project was thesupport for floating point, in earlier versions only integers were supported.

Mobile Information Device Profile 2.0 5 is as of writing the last imple-mented version of the profile. MIDP is a supplement to the CLDC whichprovides the parts of the system libraries that are not provided by theCLDC profile. Additionally, it provides the possibility to create graphicalapplications, use persistent storage etc. The MIDP is targeted at cellularphones, and the corresponding profile for PDAs is called PDAP. [13]

There are important differences between MIDP 1.0 and MIDP 2.0, inthe later it is possible to make applications run in fullscreen mode [14].This feature has extensively been used in the project since cellular phoneshave quite small screens. Other features that have been used in the projectand are new for MIDP 2.0 are, for example, sound support and supportfor other media. The development of the profile facilitates development ofadvanced applications. When programming for a cellular phone the basicentity is a MIDlet which is a MIDP application. MIDlets provides a limitedway of laying out GUI components and the navigation is based on switchingbetween different displayable items [15].

In addition to the API provided by MIDP/CLDC there are optionalpackages which give further support for hardware, communication, cryp-tology etc.

3.2 Symbol Recognition

One important component for this project is the symbol recognition plug-in. It uses a symbol recognition package, which was developed, and addedto the framework in order to allow recognition of military symbols that aredrawn with any kind of input device6. The symbol recognition packageis template-based which means that for each symbol a template is usedto identify each symbol. This is an important feature since it is hardto find gestures that are easy to memorize [16]. Further, it is possibleto claim that the use of symbol recognition should be done to maintain

5From know on MIDP.6Input device can here be interpreted as any device who’s input can be transformed

into an ordered set of points.

3.2. Symbol Recognition 21

Figure 3.3: Elastic matching (left) versus proportional matching (right)

current work-flow, in this sense, having to learn how to draw totally newsymbols to represent already known ones would be undesirable. Further,the representation on a paper map would no longer be maintained whichwould be an important drawback. When a user draws a symbol the strokesare saved as a sequence of positions which are sampled to be equidistant. Asimilar approach has been used in the SHARK2 system [17]. This approachstands in contrast to the one used by Tappert, who uses elastic matching[18]. Whereas proportional matching has a one-to-one relation betweenthe sample points, the elastic matcher, depending on lookahead, is allowedto take the closest point from the template (see figure 3.3). The problemassociated with Tappert’s approach for this thesis is that the patterns arerelatively similar, which would cause the elastic matcher to do erroneousassumptions about how different the pattern actually is from the template.

To be able to compare the reference pattern with the pattern drawn bythe user it is necessary to scale the pattern so that the two patterns havethe same size. The scaling is done by observing that the longest side of a

22

bounding box around the drawn pattern should always remain constant. Tobe able to draw patterns anywhere in a coordinate system it is necessaryto translate the drawn pattern to the origin. When the drawn patternhas passed the before mentioned steps, the distance between the drawnpattern and the reference pattern in each point is compared and summedup. Relying on the belief that the start- and endpoints when drawingare drawn with greater care than in between is taken into account as aweight for the total distance. The symbol language supports rotation, thuspermitting a symbol to be drawn rotated 7. The plug-in for the cellularphone generates rotated templates which are added to the symbols libraryif a symbol should be allowed to be rotated.

There are various approaches to symbol recognition, some approachesare easy to implement but can give poor results, other are very hard toimplement, and the symbol recognition does not necessarily improve sub-stantially. When developing a symbol recognizer for a cellular phone variousaspects had to be accounted for. A cellular phone has limited computa-tional power, and the support for floating point operations is generallybased on emulation8. Further, the J2ME maths library is limited com-pared to the J2SE library, for example, it does not provide support for thearcus functions. Therefore, for the symbol recognition in this project, itwas necessary to implement a couple of arcus functions.

Rubine [19] uses features of the collection of points to recognize patterns.The symbol recognizer has been used in many applications, for example theSATIN toolkit uses the gdt implementation of Rubine’s recognizer [20]. Ru-bine’s approach has several disadvantages, it is possible to analytically findcases when the algorithm will not be able to distinguish one symbol fromanother. Further, the algorithm requires several trigonometric operationswhich are slow on a small device. This is an issue on devices with limitedcomputational power. Reducing calculation time by using look-up tablesimplies incremented memory requirements and reduced precision. A morepragmatic reason not to use Rubine involves that it is not template-based,which means that the system has to be trained. A template-based approach

7There are several reasons to not activate rotation. It decreases the recognition rate,and it may introduce ambiguity between different military symbols.

8It is probable to see cellular phones with Floating Point Units (FPU) as these devicesare getting more advanced.

3.2. Symbol Recognition 23

eliminates the need to train the system, instead each user must learn howa symbol is represented. For the recognition of military symbols some ofthe features would become meaningless since the military symbols can beconsidered quite similar. It is even possible to analytically construct sym-bols that are completely ambiguous to the Rubine recognizer’s feature set.The dependency of a feature such as initial stroke angle means that if theuser uses an inappropriate angle of attack when starting the pen-gesture itcould directly lead to an unintended response from the system. However,adding some of the features proposed by Rubine would probably improvethe recognition rate even further, especially for rotated symbols.

How input is represented in different systems can vary, for example, theSATIN system [20] saves the information as (x,y,t) tuples. A system whichtakes into account time might not be usable in certain settings. A soldieron the field wearing thick gloves will probably not draw as fast as he woulddo sitting at a desk. Instead of taking into account when a certain strokewas generated, the order of the strokes is considered. This approach hasthe additional benefit that order can be implicitly saved, which means thatno extra data storage is necessary for the information. The downside of asystem which takes into account the time-aspect or order is that how thesymbol was drawn becomes important, as opposed to a system which onlystudies the final result of the drawing.

Most military symbols are compound symbols, which means that theycan be seen as various separate symbols. One approach to recognize com-pound symbols could be to incrementally recognize each part of the symbol.This could be achieved by on pen-up events triggering a timer which wouldforce the user to continue drawing within a specified amount of time inorder to recognize the following strokes as part of the initial symbol. Theapproach in the implemented symbol recognizer instead looks at the entiresymbol when it is drawn and ignores pen up and pen down which resultsin a continuous symbol (see figure 3.4). The downside with this approachis that the representation of large, and detailed symbols require a highersample rate than what would be necessary if the recognizer would use basicprimitives to create the symbols. The benefit of the approach is that eachsymbol gets its unique representation, and there is no need to take intoconsideration where a detected symbol part is located compared to the restof the symbol’s parts. Further, using a set of primitives to create complex

24

Figure 3.4: Continuous recognition (lower path) versus incremental recog-nition (upper path)

symbols would require a greater effort from the user in order to be able tocreate new symbols.

3.3 Symbol Format

As has earlier been discussed a template based approach toward symbolrecognition has been chosen. This implies that a prototype of the symbolmust be represented in some form. The system provides support to loadsymbols from a XML-format. In an effort to use similar technologies whereapplicable the XML-format was chosen over alternative representations. Asymbol is currently represented by various attributes, such as an image file,a sound file, and a ordered set of points defining the symbol. For other usesit might be necessary to adapt the format to support different attributes.

3.4 Use of XML in the Project

One important component in the system design is the use of a XML-formatto represent functionality, as well as, basic entities.

Most of the functionality of the cellular application is controlled byXML-files. The entire behavior of the system is determined by a XML-file

3.4. Use of XML in the Project 25

which defines which functions should be launched when specific areas havebeen clicked on. Further the XML-file also contains information on whereto locate the background image that should be used.

The structure of the basic XML-file is a general description of the appli-cation followed by a set of active elements which in turn specify functions.Active elements are the basis for the click-and-run approach. An activeelement can, for example, be a path which is located somewhere on thepaper. Connected to the enclosing area of the path a set of functions, withtheir corresponding arguments, can be found. The functions are separatecomponents that should be dispatched when clicking on the correspondingarea. It is up to each plug-in developer to require the arguments neededto be able to implement the desired functionality. One example of whenthe use of an argument could be handy is when a predefined message (forexample using SMS/MMS) should be sent to a predefined number.

One important benefit of using XML is that it permits the separation ofthe functionality of the paper from the graphical representation of it. It is,for example, possible to restrict what operations different users are allowedto execute by providing different XML-files. This way it is possible to createapplications where one user is able to draw and modify the functionality ofthe paper, while another only can see the changes. This is an example ofhow the use of separated user interfaces come into play, since two differentviews of the same information can be created [21]. It is also a powerfulway to reduce the information the system provides to the amount neededto perform the required tasks.

The Open Geospatial Consortium provides a standard for geospatialand location based services called OpenGIS R© [22]. The standard dictates aspecific format to represent geometrical data. Whereas the OpenGIS R© hassupport for most of the situations where some representation of geospatialdata should be used it was not optimal for this project. The main reasonis that the standard is very extensive, and only a small percentage of thespecification would be used. Further, this application provides additionalfunctionality which requires its own representation. This specializationcalled for the development of the XML-format which is currently used.

26

Chapter 4

Conclusion

This thesis was set to investigate in which way it is possible to support theuse of an online paper-driven computer system in military work-practice.Several important aspects were detected and a system which to a high-degree complies with the needs’ of the military was implemented. Further,the implementation suggested that the same kind of technology could eas-ily be applied to other areas within the field of paper-driven computersystems. Problems similar to the ones the military experience when tryingto use computers in their daily work can also be applied to areas such as,firefighting, surveying, waiting on customers etc. The framework presentedin this thesis was first intended for paper-driven map applications. Duringthe development of the system it has, however, become clear that the sametechnology employed for this solution is also viable for other solutions. Theefficiency when it comes to developing new applications with this frameworkmakes it an interesting solution for all applications that may benefit frompaper and pen as its interaction device. The support for gesture recogni-tion through the symbol recognition package is another utility which can beused in many different settings and can make the work with the computersystem more transparent to the user. The symbol recognizer has a veryhigh accuracy rate, which makes it suitable for recognition of pen gestureseven in difficult settings. The use of paper and pen as interaction devicechanges various concepts of how computers may be used. The possibility

27

28

to have a cellular phone in the pocket, and use its services without the needto consult its display is a powerful concept which integrates computers fur-ther into everyday life. Soldiers with no computer experience could usethe cellular application to its full extent without having to know anythingabout the underlying technology. Whereas, it can be argued that this im-plies lower training costs it also implies a significant improvement in work-practice. It is known that in situations of great pressure it is common thata regression toward well-known response patterns occurs. This suggeststhat the methods soldiers use in the battlefield should be as well-known tothem as possible. In this sense pen and paper are natural and powerfultools. Whereas the cellular application in this project uses some of thetechnologies currently available, there are many other that are interestingwithin the military field. It is, for example, possible to connect a GPS-receiver to the cellular phone and thereby be able to monitor the soldier’sprogress, sending customized warning messages depending on the locationof the soldier etc. The need for context-awareness in pervasive computingis disccused by, among others, Satayanarayanan [23]. Another possibilityis to improve communication between the soldiers. Even more interestingis the possibility to connect cellular phones to a critiquing system whichgets its input from a centralized server which merges the observations ofall the connected cellular phones, and sends out information to supportthe decision-making in the battlefield. The development of digital pens hasmade them a viable choice for applications in which it is crucial to havea high-accuracy on the readings. For military applications it is howevernecessary to make the digital pens work in hard conditions where dust,and water are part of everyday use. The need for further studies of howthe system would work in a real-life setting is essential to be able to furtherimprove the applications, such a study could also give some clues on whatto focus on to further develop the concept of paper-based computing.

Chapter 5

Drawing on Paper maps:Robust On-line SymbolRecognition ofHandwritten NATOSymbols using a DigitalPen and a Mobile Phone

5.1 Abstract

We present a robust on-line handwritten recognition system that recognizesNATO symbols drawn on paper maps using a digital pen and a mobilephone. Our user evaluation shows that very high accuracy is achieved.Furthermore, our proposed system has low latency running on off-the-shelfmobile phones. The recognition system is part of a research effort in aug-menting existing work-practices of soldiers in the field with very lightweight

29

30

computer support.Keywords: pen-gesture recognition, mobile phones, digital pen, paper-

based user interfaces, ubiquitous computing.

5.2 Introduction

There is still extensive use of ordinary paper products within the militaryand advanced computer systems are sometimes unused [24]. The use ofcomputer support is, naturally desirable, given satisfactory cost and mo-bility characteristics. For the soldier in the field, weight and cost can beimperative, design parameters that a computer system such as a commu-nications, planning, or critiquing system needs to overcome.

The current and ongoing computerization of physical paper provides uswith a new technical platform for such systems. In this project, as part ofthe Kartago research framework, we explore the development of a paper-map-based computer support tool providing very lightweight and low costcharacteristics. The Kartago research framework focuses on the traditionalpaper map as the central hardware device and explore the augmentationof paper as a means of providing interactive computer support. Soldiers inthe field use paper maps extensively. Current work-practice involves usingpaper maps for orientation, observation and collaboration. Observationsare marked with a military symbol language on the paper map and arealso reported to headquarters.

The ability to recognize drawn symbols such as NATO symbols is essen-tial in a paper-map-based computer system for soldiers in the field. Symbolrecognition is a non-trivial area of research, further challenged by the mil-itary domain and by use in the field.

In this paper we present our work in enabling symbol recognition for theKartago System in its current form, a printed paper map used with a digitalpen (Anoto) and a mobile phone as the system components. This systemprovides pen-based interactivity on a paper map to a computer systemrunning on a mobile phone via wireless communication. The system fits ina soldier’s leg-pocket and it retains all positive aspects of traditional papermaps.

Our user evaluation results show that very high accuracy (98% before

5.3. The Kartago Project 31

training, 100% after minimal training) is achieved in the Kartago symbolrecognition running on off-the-shelf mobile phones. Consequentially, weare confident that the symbol recognition part of the Kartago system willprovide sufficiently ease-off-use in the field.

5.2.1 Symbol Recognition in the Field

Armies invest enormous sums in technology to keep up with the technolog-ical development. Yet, in the field, it may still not be to bold a statementto say that paper still rules. Whereas paper maps have several advan-tages over ordinary computerized systems, they do not provide automatedconnectivity to other military systems.

One important leap toward the next generation of soldier equipment isthe digitalization of the soldiers’ work. For instance, when a soldier wantsto report an observation it could be done by drawing with a pen on a papermap. One of the issues with such a system is how to support the recognitionof symbols.

Symbol recognition for military applications requires a different ap-proach. One important aspect is that the input data can vary considerablyin quality due to the special conditions in which military operations takeplace. For example, how well a symbol is drawn depends to great extent onwhat input device is used, if the symbol is drawn by a person who stands orsits, if the canvas is supported by a hard surface underneath or not. Thissuggests that a solution must be robust and permissive to bad data.

Although symbol recognition is not a new phenomena, it unveils thefull potential of a mobile military computerized system. Annotations on amap can be captured and used locally, or by remote systems, for instance,by command and control systems.

5.3 The Kartago Project

Kartago is a research project aimed at a mobile paper-based or paper-likeinteractive map platform. The aim is to provide a platform for military op-erations which is usable in the field for tasks such as planning, simulation,communication, and documentation. It is an alternative approach to digi-

32

Figure 5.1: Drawing NATO symbols on a paper map with a digital Anotopen.

talization of map-related tasks which focus on augmenting the traditionalpaper map as means of digitizing map-related work practice.

The research project envisions that paper or a paper-like material (suchas a flexible screen) can be the central artifact in an interactive-map systemwhere networking, communication and user interactivity can be added ona level as advanced as that of an everyday computer.

The goal is to achieve a printable leg-pocket sized system with a map-sized interface (folded and carried in hand or in a leg-pocket) with symbol-based interaction, and information/communication task management (seeFigure 5.2).

A highly relevant part of the Kartago platform is symbol managementand symbol recognition for the use of hand-written symbols in interactivesystems such as planning and critiquing systems. Being able to use thesymbol language already in place is an important part of a mobility plat-form for the military.

5.4. Interactive Maps in the Field 33

Figure 5.2: The system in its natural workplace.

5.4 Interactive Maps in the Field

In a regular computer-enabled staff environment, a multitude of systemscan provide decision support for staff members. Mixed-initiative planningsystems can help staff officers analyze a course of action with respect tophysical limitations, rules of engagement or other known constraints (e.g.,[25] [26][27][28] [29]). Collaboration systems can analyze the interactionsbetween multiple commanders [30] and thereby help them synchronize theiractions.

The usefulness of these systems could be extended greatly if they al-lowed for field officers to access the same information that is available ina regular staff environment through the use of pen and paper . Further,the availability of observations from the field can increase the usefulness ofthese systems even further.

34

5.5 System Overview

The outlined solution uses paper maps together with a cellular phone, andan Anoto pen. The pen deduces where the user is drawing by a special pat-tern, which is printed on the paper map. This technology allows the soldierto have a full-size paper map which can be folded and put in a pocket. Thecellular phone used is a normal bluetooth cellular phone which supportsJ2ME. Altogether this hardware configuration assures that the hardwareis easily wearable. This paper focuses on the pen-gesture recognition com-ponent of a larger system intended for military use.

Figure 5.3 illustrates the high-level relationship between the paper maps,the digital pen, the cell phone, and an optional server.

Figure 5.3: Overview of the pen-gesture recognition component. The userdraws pen-gestures with digital pen on a paper map. The pen is connectedwirelessly to a cell phone, which in turn is connected to a server over themobile network (e.g. GSM, EDGE, 3G, etc.).

Within the framework for Kartago System for Cellular Phones, thesymbol recognition is one important component. In a case implementationthe symbol recognition allows the soldier to draw a symbol, and get itrecognized. When the symbol is recognized the soldier is allowed to positionthe symbol and can, optionally, add information which can be used by acritiquing system. The information collected during this work is attached

5.6. Requirements 35

to a XML-representation of the map which can be sent through SMS/MMS,or web services to other soldiers, or to headquarters.

5.6 Requirements

5.6.1 Support Complex Pen-gestures

Military symbols are typically complex compound symbols consisting of acombination of various pen-strokes, see Figure 5.4 for examples. Thereforethe pen-gesture recognizer must be able to handle multiple pen-strokes.

5.6.2 Robust

Since the pen-gesture recognition system is intended to be used in the field,it must be robust to distortions and maintain high-accuracy.

5.6.3 Support Low-powered Devices

Yet another difficulty is the limited processing power of cell phones whichlimits the amount of calculations that can be performed. Also, mobilephones typically lack a floating point unit (FPU).

5.6.4 Easy to learn

The recognition system should work with a minimal amount of trainingrequired by the users.

5.6.5 Easy to maintain

Many recognition systems are based on training-data where several writingsamples for each symbol are obtained into a database. A side-effect of thedata-driven approach is that it is harder to introduce new symbols since itis not enough to submit one ideal template symbol (e.g. from a handbookof symbols). Instead several symbols need to be written using a specializedsoftware. As such, a requirement is also that it is easy to maintain the

36

symbol set and add new symbols as needed with minimum effort requiredfrom the users.

5.7 Recognition

5.7.1 Symbol Language

Cooperation between different countries’ armies creates new challenges tothe world of symbol representation. Whereas NATO has one symbol lan-guage, individual countries may use a different system. In order to supportdifferent military symbol languages a special symbol format was developed,which permits the symbol recognition engine to be loaded with differentsymbol languages. Figure 5.4 shows a selection of the symbols the systemrecognizes.

The symbol format is based on XML, which is a natural choice giventhat the system, in which the symbol recognizer is one part, is XML-driven.The system provides support to load symbols expressed in the XML-format.A symbol is represented by various attributes, such as an image file, a soundfile, and a ordered set of points defining the symbol. The symbol recognizeronly require the ordered set of points, together with information regardingwhether the symbol could be rotated, and how far it should be betweeneach rotation of the symbol. The rest of the specifiers are optional andare used by other parts of the application to enhance the user’s experience.The implication of requiring an ordered set of points is that drawing orderbecomes important. It is therefore necessary for the user to learn in whichorder a symbol should be constructed.

5.7.2 Recognition Process

Related Work

A common pen-gesture recognizer is the Rubine recognizer [19]. The Ru-bine recognizer is the “classic” statistical linear machine that extracts afeature vector from the input signal and attempts to find the most sim-ilar class by statistical inference. The features in the Rubine recognizer

5.7. Recognition 37

Figure 5.4: Selection of primitives and NATO symbols.

38

are, for example, initial angle, and the length of the stroke [19]. The Ru-bine recognizer has been integrated into many toolkits, for instance SATIN[20]. The Rubine recognizer approach is practical for a few pen-gesturesbut is increasingly less robust as the number of symbols increase. For in-stance, the dependency of a feature such as initial stroke angle means thatif the users initiated an inappropriate angle of attack when starting thepen-gesture it could directly lead to an unintended response. Analyticallyanother downside with the Rubine recognizer is the possibility to createtwo geometrically distinct symbols that are completely ambiguous to theRubine recognizer’s feature set.

Algorithms

The symbol recognition is template-based and the symbol templates areused by the recognizer to identify the user’s intended symbol. User inputis saved as ordered strokes and is equidistantly sampled and normalized inscale and translation to permit comparison with the reference pattern.

The actual symbol recognition is performed when the unknown symbolhas passed the preprocessing steps for recognition. For each template sym-bol, the distance between the pattern drawn with the digital pen and thereference symbol pattern is calculated by the scoring function q (Equation5.1). To improve recognition accuracy we also give extra weight to thestart and end point comparisons between the input signal (drawing trace)and the template pattern. In Equation 5.1 α is the weight of the start andend points. In the summation the distance between each ordered pair issummed up:

q(x, y) = (1− α)(m∑

i=0

‖xi − yi‖2) + αγ(x, y) (5.1)

where x and y are two patterns, the subscripted xi and yi componentsrefer to individual sample points, ‖2 ·2‖2 is the L2 (Euclidean) norm andγ is a function defined as:

γ(x, y) = ‖x0 − y0‖2 + ‖xm − ym‖2 (5.2)

5.7. Recognition 39

The recognition engine is based on the concept of proportional matchingpresented by Kristensson and Zhai [17]. Points are matched on a one-to-one basis as opposed to the elastic matching approach which was used insome early handwriting recognition systems [18]. There are three primaryadvantages to proportional matching in relation to elastic matching [17]: 1.Time complexity of the proportional matching algorithm is linear in relationto the number of sample points. By contrast, elastic matching has quadratictime complexity. 2. Proportional matching is easier to implement. 3.Proportional matching is easier to work with analytically. For instance, tocompare elastic matching scores against each other, the scores need to benormalized. However the normalization procedure for elastic matching isnon-trivial [31]. Other approaches to recognizing on-line writing (mainlydata-driven handwriting recognition) have been proposed (e.g. [32, 33],also see Tappert et al. [34] for an extensive review). Since our systemis template-based rather than data driven new symbols can be added bysimply inserting an ideal template of a symbol (e.g. from a handbook) intothe recognizer’s set of reference symbols. To make proportional matchingwork well in a military setting we found it necessary to introduce a couple ofmodifications of the concept. For military applications in the field, specialcare has to be taken to how a stroke should be represented. Several systemssave strokes as (x,y,t) triplets. In a military setting a symbol recognitionthat depends on movement dynamics is probably not reliable. A soldier inthe field wearing thick gloves will probably not draw as fast as he woulddo sitting at a desk. Furthermore, expert users have dramatically differentmovement patterns than novice users or users under considerable stresswho may suffer considerable hand tremor. As a result, instead of takinginto account when a certain stroke was generated, the order of the strokesis considered. This increases the tolerance toward disruptions in the inputprocedure. Furthermore, given the risk of unwanted pen-up events therepresentation of the symbols discards any pen-up and pen-down data.Figure 5.5 shows the result of how a symbol is represented internally, whenpen-up and pen-down is discarded.

As has earlier been mentioned, most military symbols are compoundsymbols, which means that they can be seen as various separate symbols.One approach to recognize compound symbols could be to incrementallyrecognize each part of the symbol. This could be achieved by on pen-up

40

Figure 5.5: Graphical representation (left) and internal representation(right) of two symbols.

events triggering a timer which would force the user to continue drawingwithin a specified amount of time in order to recognize the following strokesas part of the initial symbol. The approach in the implemented symbolrecognizer instead looks at the entire symbol when it is drawn and ignorespen-up and pen-down which results in a continuous symbol.

The downside with this approach is that the representation of large, anddetailed symbols require a higher sample rate than what would be necessaryif the recognizer would use basic primitives to create the symbols. Thebenefit of the approach is that each symbol has a unique representation,and there is no need to take into consideration where a detected symbolpart is located compared to the rest of the symbol’s parts. Further, usinga set of primitives to create complex symbols would require a greater effortfrom the user in order to be able to create new symbols.

As is always the case when developing recognition-based systems thereare multiple conflicting dimensions of goodness: high accuracy, ease-of-use,fast recognition response time, etc. In this application we prioritized highaccuracy (robust recognition) at the cost of initial ease-of-use.

5.8. Evaluation 41

5.8 Evaluation

There are many research questions when developing a new recognition sys-tem for soldiers in the field. In this paper we have focused on the mostfundamental questions: 1. Is recognition robust enough to be practical?2. Can the system recognize symbols fast enough to run on off-the-shelfmobile phones?

5.8.1 Participants

Fifteen participants were recruited. They were told to draw the twentycompound pen-gestures depicted in figure 5.4. The ages ranged from 27 to50 (average 40). 11 were female and 4 were male.

5.8.2 Material

The participants were asked to reproduce 20 pen-gestures (Figure 5.4).Each pen-gesture was re-sampled into 100 equidistant sample points.

5.8.3 Apparatus

An Anoto pen model Maxell DP-201 was used as the digital pen. TheAnoto pen was used on paper sheets with Anoto’s background patternimprinted. Using the background paper the Anoto pen transmitted abso-lute and relative positional coordinates to a desktop computer executingthe recognizer software. The performance tests were conducted on a SonyEricsson P990i mobile phone with a 250 MHz 32-bit ARM9 CPU.

5.8.4 Procedure

The experiment was run in two sessions. In the first session participantswere shown how to draw the symbols in Figure 5.4 and afterwards in-structed to reproduce them. The second session was performed after usershad been allowed to practice drawing the symbols that were difficult to, orhad failed to draw, in the first attempt.

42

5.8.5 Results

Error

As can be seen in Table 5.1 the error rate at the first test was well overthe acceptable level (2%). After minor training the error rate reduces tozero. The results show that after minimal training very high accuracy isachievable with our proposed NATO symbol recognition system, and it isclearly practical for prototype testing in the field.

Table 5.1: Error rates before and after trainingFirst Test Second Test

Error Rate 2% 0%

Latency

Latency was calculated by recognizing five randomly chosen symbols fromthe symbol set in Figure 5.4. The average latency was 589 ms. Latencyvaried from 492 ms up to 703 ms. The variation can be attributed to howcomplex the input data was to process (re-sample and normalize). We wantto emphasize that the latency data points given above are of a preliminarynature since they were obtained without using any pruning or indexingmechanisms. We are in the process of developing a filter that prunes outunlikely patterns early in the recognition process to reduce latency.

5.9 Conclusion

A symbol recognizer capable of recognizing for instance hand-drawn NATOsymbols is an essential component in a paper-map-based computer systemsupporting information and communication tasks for soldiers in the field.This paper show that the presented Kartago symbol recognizer provides avery robust solution. High accuracy is achievable using the symbol recog-nizer after minimal training.

The Kartago system illustrates that interactive and mobile computersystems with lightweight and low-cost characteristics can be provided by

5.10. Acknowledgements 43

augmenting paper maps. Combined with technologies included in off-the-shelf mobile phones such as GPS, Bluetooth and GPRS/3G we can supportadvanced information tasks in the field.

The fact that the system is wearable while still providing the use of full-size maps suggests that Kartago is a usable tool in military work practice.The work presented here is a step towards real-time critiquing and computersupport in the field.

With the positive results of our symbol recognizer we are planing toimplement and evaluate the use of the Kartago with a critiquing system,which would allow us to study the use of interactive paper maps in the fieldusing traditional paper maps.

5.10 Acknowledgements

This work was in part sponsored by the Swedish National Defence College,Anoto AB and Santa Anna IT Research Institute AB (SICS Linkoping).

Bibliography

[1] Mark Weiser. The computer for the 21st century. SIGMOBILE Mob.Comput. Commun. Rev., 3(3):3–11, 1999.

[2] Simon Downs. Is it a book, is it a screen, no it’s... graphics and theinterface in electronic paper. Digital Creativity, 16(1):31–42, 2005.

[3] Xerox. Avilable at http://sandbox.xerox.com/parctab/csl9501/paper.html.accessed 5 december.

[4] Anind K. Dey, Peter Ljungstrand, and Albrecht Schmidt. Distributedand disappearing user interfaces in ubiquitous computing. In CHI ’01:CHI ’01 extended abstracts on Human factors in computing systems,pages 487–488, New York, NY, USA, 2001. ACM Press.

[5] Mark Weiser. Some computer science issues in ubiquitous computing.Commun. ACM, 36(7):75–84, 1993.

[6] Jr. Allan Christian Long, James A. Landay, and Lawrence A. Rowe.Implications for a gesture design tool. In CHI ’99: Proceedings of theSIGCHI conference on Human factors in computing systems, pages40–47, New York, NY, USA, 1999. ACM Press.

[7] Mark A. Livingston and et al. An augmented reality system for mili-tary operations in urban terrain.

[8] Anoto AB. Available at: www.anoto.com. accessed december 5, 2006.

44

BIBLIOGRAPHY 45

[9] Joseph LaViola, Randall Davis, and Takeo Igarashi. An introductionto sketch-based interfaces. In SIGGRAPH ’06: Material presented atthe ACM SIGGRAPH 2006 conference, page 18, New York, NY, USA,2006. ACM Press.

[10] Yu David Liu and Scott F. Smith. A formal framework for componentdeployment. SIGPLAN Not., 41(10):325–344, 2006.

[11] Mario Kolberg. Programming a pvr with pen and paper. Con-sumer Communications and Networking Conference, 2006, 2:1332–1333, 2006.

[12] Sun. Available at: java.sun.com. accessed november 28, 2006.

[13] James White. An introduction to java 2 micro edition (j2me); javain small things. In ICSE ’01: Proceedings of the 23rd InternationalConference on Software Engineering, pages 724–725, Washington, DC,USA, 2001. IEEE Computer Society.

[14] Christopher Williams and Mark Burge. Midp 2.0 changing the faceof j2me gaming. In ACM-SE 42: Proceedings of the 42nd annualSoutheast regional conference, pages 37–41, New York, NY, USA, 2004.ACM Press.

[15] Ian Utting. Problems in the initial teaching of programming using java:the case for replacing j2se with j2me. SIGCSE Bull., 38(3):193–196,2006.

[16] Jr. A. Chris Long, James A. Landay, Lawrence A. Rowe, and JosephMichiels. Visual similarity of pen gestures. In CHI ’00: Proceedingsof the SIGCHI conference on Human factors in computing systems,pages 360–367, New York, NY, USA, 2000. ACM Press.

[17] Per-Ola Kristensson and Shumin Zhai. Shark2: a large vocabularyshorthand writing system for pen-based computers. In UIST ’04: Pro-ceedings of the 17th annual ACM symposium on User interface soft-ware and technology, pages 43–52, New York, NY, USA, 2004. ACMPress.

46 BIBLIOGRAPHY

[18] Charles C. Tappert. Cursive script recognition by elastic matching.IBM Journal of Research and Development, 26(6):765–771, 1982.

[19] Dean Rubine. Specifying gestures by example. In SIGGRAPH ’91:Proceedings of the 18th annual conference on Computer graphics andinteractive techniques, pages 329–337, New York, NY, USA, 1991.ACM Press.

[20] Jason I. Hong and James A. Landay. Satin: a toolkit for informal ink-based applications. In UIST ’00: Proceedings of the 13th annual ACMsymposium on User interface software and technology, pages 63–72,New York, NY, USA, 2000. ACM Press.

[21] Chris Vandervelpen and Karin Coninx. Towards model-based designsupport for distributed user interfaces. In NordiCHI ’04: Proceedingsof the third Nordic conference on Human-computer interaction, pages61–70, New York, NY, USA, 2004. ACM Press.

[22] Open Geospatial Consortium Inc. Available at:www.opengeospatial.com. accessed november 30, 2006.

[23] M. Satyanarayanan. Pervasive computing: Vision and challenges.IEEE Personal Communications, pages 10–17, aug 2001.

[24] David R. McGee, Philip R. Cohen, and Lizhong Wu. Something fromnothing: augmenting a paper-based work practice via multimodal in-teraction. In DARE ’00: Proceedings of DARE 2000 on Designingaugmented reality environments, pages 71–80, New York, NY, USA,2000. ACM Press.

[25] Juan Fdez-Olivares, Luis Castillo, Oscar Garcıa-Perez, and FranciscoPalao. Bringing users and planning technology together. experiencesin SIADEX. In 16th International Conference on Automated Planningand Scheduling (ICAPS 2006), 2006.

[26] Marcel A. Becker and Stephen F. Smith. Mixed-initiative resourcemanagement: The AMC Barrel Allocator. In Proceedings 5th Interna-tional Conference on Artificial Intelligence Planning and Scheduling(AIPS-2000), 2000.

[27] Karen L. Myers, Peter A. Jarvis, W. Mabry Tyson, and Michael J.Wolverton. A mixed-initiative framework for robust plan sketching.In Thirteenth International Conference on Automated Planning andScheduling (ICAPS-03), 2003.

[28] John L. Bresina, Ari K. Jonsson, Paul H. Morris, and Kanna Rajan.Mixed-initiative planning in MAPGEN: Capabilities and shortcom-ings. In Proceedings of the IJCAI-2005 Workshop in Mixed-InitiativePlanning and Scheduling, 2005.

[29] Stephen F. Smith, David W. Hildum, and David R. Crimm. Comirem:An intelligent form for resource management. IEEE Intelligent Sys-tems, 20(2):16–24, 2005.

[30] Karen Myers, Peter A. Jarvis, and Thomas Lee. CODA: Coordinatinghuman planners. In Proceedings of the 6th European Conference onPlanning (ECP-01), 2001.

[31] A. Marzal and E. Vidal. Computation of normalized edit distance andapplications. IEEE Trans. Pattern Anal. Mach. Intell., 15(9):926–932,1993.

[32] E.J. Bellegarda, J.R. Bellegarda, D. Nahamoo, and K.S. Nathan. Afast statistical mixture algorithm for on-line handwriting recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,16(12):1227–1233, 1994.

[33] Daiki Okumur, Seiichi Uchida, and Hiroaki Sakoe. An hmm implemen-tation for on-line handwriting recognition - based on pen-coordinatefeature and pen-direction feature. icdar, 0:26–30, 2005.

[34] C. C. Tappert, C. Y. Suen, and T. Wakahara. The state of the artin online handwriting recognition. IEEE Trans. Pattern Anal. Mach.Intell., 12(8):787–808, 1990.

Avdelning, InstitutionDivision, Department

DatumDate

SprakLanguage

2 Svenska/Swedish

4 Engelska/English

2

RapporttypReport category

2 Licentiatavhandling

4 Examensarbete

2 C-uppsats

2 D-uppsats

2 Ovrig rapport

2

URL for elektronisk version

ISBN

ISRN

Serietitel och serienummerTitle of series, numbering

ISSN

Titel

Title

ForfattareAuthor

SammanfattningAbstract

NyckelordKeywords

Military work-practice is a difficult area of research where paper-basedapproaches are still extended. This thesis proposes a solution whichpermits the digitalization of information at the same time as work-practice remains unaltered for soldiers working with maps in the field.For this purpose, a mobile interactive paper-based platform has beendeveloped which permits the users to maintain their current work-flow.The premise of the solution parts from a system consisting of a preparedpaper-map, a cellular phone, a desktop computer, and a digital penwith bluetooth connection. The underlying idea is to permit soldiersto take advantage of the information a computerized system can offer,at the same time as the overhead it incurs is minimized. On one handthis implies that the solution must be light-weight, on the other itmust retain current working procedures as far as possible. The desktopcomputer is used to develop new paper-driven applications through theapplication provided in the development framework, thus allowing thetailoring of applications to the changing needs of military operations.One major component in the application suite is a symbol recognizerwhich is capable of recognizing symbols parting from a template whichcan be created in one of the applications. This component permitsthe digitalization of information in the battlefield by drawing on thepaper-map. The proposed solution has been found to be viable, butstill there is a need for further development. Furthermore, there is aneed to adapt the existing hardware to the requirements of the militaryto make it usable in a real-world situation.

HCS,Dept. of Computer and Information Science581 83 LINKOPING

2007-02-14

—

LITH-IDA-EX--07/004--SE

—

http://www.ep.liu.se/exjobb/ida/2007/dd-d/004/

2007-02-14

A Framework for Mobile Paper-based Computing

Ett ramverk for mobilt pappersbaserat datoranvandande

Tomas Sylverberg

pen-gesture recognition, paper-based user interfaces, ubiquitous com-puting, software engineering, digital pen

50

Copyright

SvenskaDetta dokument halls tillgangligt pa Internet - eller dess framtida ersattare - under 25 ar franpubliceringsdatum under forutsattning att inga extraordinara omstandigheter uppstar.

Tillgang till dokumentet innebar tillstand for var och en att lasa, ladda ner, skriva utenstaka kopior for enskilt bruk och att anvanda det oforandrat for ickekommersiell forskningoch for undervisning. Overforing av upphovsratten vid en senare tidpunkt kan inte upphavadetta tillstand. All annan anvandning av dokumentet kraver upphovsmannens medgivande.For att garantera aktheten, sakerheten och tillgangligheten finns det losningar av teknisk ochadministrativ art.

Upphovsmannens ideella ratt innefattar ratt att bli namnd som upphovsman i den omfat-tning som god sed kraver vid anvandning av dokumentet pa ovan beskrivna satt samt skyddmot att dokumentet andras eller presenteras i sadan form eller i sadant sammanhang som arkrankande for upphovsmannens litterara eller konstnarliga anseende eller egenart.

For ytterligare information om Linkoping University Electronic Press se forlagets hem-sida http://www.ep.liu.se/

EnglishThe publishers will keep this document online on the Internet - or its possible replacement -for a period of 25 years from the date of publication barring exceptional circumstances.

The online availability of the document implies a permanent permission for anyone toread, to download, to print out single copies for your own use and to use it unchanged for anynon-commercial research and educational purpose. Subsequent transfers of copyright cannotrevoke this permission. All other uses of the document are conditional on the consent of thecopyright owner. The publisher has taken technical and administrative measures to assureauthenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned whenhis/her work is accessed as described above and to be protected against infringement.

For additional information about the Linkoping University Electronic Press and its pro-cedures for publication and for assurance of document integrity, please refer to its WWWhome page: http://www.ep.liu.se/

c© Tomas SylverbergLinkoping, February 22, 2007