Eulexia A Wearable Aid For Spell-checking

Eulexia: A Wearable Aid For Spell-checkingSEBASTIAN CHEAH, BUSHENG LOU, MICHELLE WU, PHOEBE GAO, and ANDREW DU,University of California, San Diego

With Eulexia, we wanted to create an application that would help close the learning gap between dyslexic and non dyslexic

individuals. Word processors can to spell checking, but often we find ourselves in situations (homework assignments, whiteboarding during company meetings, etc) that require handwriting. We envision Eulexia as a solution to this issue by providing

a ubiquitous form of spell checking for hand written documents. Our prototype is currently limited to accurate processing of

typed text and neat handwritten text, but access to a better OCR engine would allow us to easily achieve our vision.

Additional Key Words and Phrases: Dyslexia, Ubiquitous Computing, Optical Character Recognition (OCR), Dictation, Spell-

check and Suggestions, Levenshtein Distance

ACM Reference Format:Sebastian Cheah, Busheng Lou, Michelle Wu, Phoeba Gao, and Andrew Du. 2015. Eulexia: A Wearable Aid For Spell-checking.ACM Trans. Appl. Percept. 0, 0, Article 0 ( 2015), 15 pages.DOI: 0000001.0000001

1. INTRODUCTION

With 3-7% of the population diagnosedand 20% exhibiting degrees of symptomsdyslexia continues tobe one of the most common learning disabilities in America. Characterized by inaccurate word recog-nition during reading and writing despite the student possessing normal intelligence, dyslexia oftencorrelates to dysgraphia as well. Dysgraphia mainly entails trouble with written expression, and oftenpeople misspell words when writing by hand [Peterson and Pennington 2012]. Due to this cognitivedisorder, misspellings in the classroom have become commonplace. Existing non-ubiquitous applica-tions to aid dyslexic students with language and visual processing include spellchecking features inWord processors and voice assistants such as Amazon Echo. Our ubiquitous approach is to implementa wearable solution for correcting spelling via a Google Glass application.

The general idea is to offer the wearer a simple and easy-to-use spellchecker that can be utilized indaily classroom tasks such as when taking notes or completing classwork. By integrating our appli-cation into a wearable, we provide a method for the user to correct his or her own spelling throughsimple steps: tapping to take a picture and swiping between spelling suggestions. People who sufferfrom dyslexia do not have issues with audio processing, so we also provide a text-to-speech feature thatenables the user to listen to the spelling of each suggestion.

In section 2, we will describe the motivations and background behind our work.In section 3, we will outline how we approached designing features for dyslexic users.In section 4, we will discuss the system development of our application, specifically the architecturedesign, the technologies we used, and the features we implemented.In section 5, we will detail how we tested and evaluated our application.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by othersthan ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from [email protected]© 2015 ACM. 1544-3558/2015/-ART0 $15.00DOI: 0000001.0000001

ACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.

0:2 • S. Cheah, B. Lou, M. Wu, P. Gao, and A. Du

In section 6, we will cover our collaboration, namely our team structure and how we solved issues.In section 7, we will conclude our work and discuss future directions for our research.

2. MOTIVATION AND BACKGROUND

2.1 The Problem

Dyslexia is a condition that affects how the brain processes written and spoken language, which meansit specifically has an impact on writing, spelling, and speaking; however listening is not affected. Dueto this, students with dyslexia require methods of learning alternative to what conventionally existsfor the general population (i.e. listening to audio versions of a text since reading comprehension isdifficult).

A major side effect of this invisible illness is its tendency to affect an individual’s chances of educa-tional success due to early experiences with the condition. Take, for example, two students, Jack andJill. Jack suffers from a mild form of dyslexia and was never diagnosed. Jill is not affected by dyslexia.They both go through early elementary school, Jack struggling to keep up with other kids due to hisimpaired reading and writing abilities, not really knowing why he is unable to learn as well as theother students. Jill excels in all subjects, with a natural affinity for reading and obtaining new knowl-edge. An important note is that Jack possesses intelligence akin to Jill, and simply becomes deficientbecause he does not have the resources that he needs accessible in order to learn in an alternative way.As the years go by, Jack begins to believe he simply is not as smart as the other students and givesup trying on his assignments-slacking off and eventually dropping out of high school. Jill continuesto succeed in school-earning straight A’s, acceptance into a top university, and landing a competitiveposition at a top company. This opportunistic gap is the motivation behind our application, Eulexia.

The technical problem-or the reason why this problem does not have a trivial solution-is the fact thatdyslexia cannot be cured. Individuals suffering from this condition simply learn to deal with dyslexiain their own ways over time; however far too often, this happens too late in the game. We need totarget weaknesses caused by dyslexia in earlier stages of development so that effects do not growexponentially. Each individual with dyslexia is also unique, which is another reason why this problemrequires a ubiquitous solution, one that modifies itself continuously to fit each user’s needs.

2.2 Previous Solutions

2.2.1 Spell checking on computers. Modern word processors have built-in spell-checking featuressuch as Microsoft Word and Google Docs, and the latter also comes with a custom dictionary featurewhich the user can use to add their commonly misspelled words. Though this feature is useful forolder dyslexics (and are currently used by them), none of these word processors process handwrit-ing. Younger students may not regularly complete schoolwork on computers and are often assignedworksheets in the classroom, and they are limited in resources to verify their spelling.

2.2.2 Text-to-Speech. There are multiple existing text-to-speech applications for documents on thecomputers, which is useful to dyslexics because they have no issues with processing audio. DCODIAis a mobile iOS text-to-speech application built for dyslexic students [Hall 2014]. A student takes apicture of printed text and the application then dictates the words in the text aloud from the image.While this approach is viable, the user must take out their mobile phone in order to position the cameraand take a picture, and this only works for printed text, not handwriting.

2.2.3 Voice assistants. Amazon Echo, a wireless speaker and voice command device from Ama-zon.com, and Apple’s Siri provide another alternative solution for dyslexic students to spell-check.Users can simply query these devices for the correct spelling of a word (i.e. ”Siri, how do you spellACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.

Eulexia: A Wearable Aid For Proofreading • 0:3

dyslexia?”) [Amazon 2015] [Sheehan 2012]. However, this method becomes reliant on user pronunci-ation and there is no secondary option if multiple words have the same pronunciation but differentspellings. As solely voice assistants, these devices do not come with a camera feature and cannot pro-cess written text. Constantly asking for correct spelling also becomes tedious and inefficient for thosewith dyslexia because the problem is not that they do not know how to spell a specific word, it is thatthey do not realize when they have misspelled one.

3. DESIGN

3.1 Design Idea

Before we could begin to design our application, we needed to familiarize ourselves with the targetuser’s needs. To do so, we interviewed Gini Keating, a UX designer with dyslexia who works at Qual-comm Inc. Through this interview, we gathered the following information on dyslexia, as it pertains toher case:

—Spelling is still a challenge for her and does not improve over time.—She spells phonetically, so she often misspells the same word in numerous different ways. However,

she spells Spanish words correctly.—Text-to-speech feature to spell out loud is very helpful (she sometimes uses Amazon Echo). She also

uses the speech-to-text feature on her smartphone.—She uses a custom dictionary on her word processor to give her spelling suggestions.—During company meetings, her coworker writes on the whiteboard for her because one of her autistic

coworkers will be too distracted by misspelled words.—She would like to either see a rectangle around misspelled words or blur/grey out the other text.

We then had Keating try on Google Glass to locate any problems she has with its current userinterface design. She could read words easily and had no issues with the font or default white color.

With this gathered information, we decided to keep the existing Google Glass font (Roboto) as wellas its white color with the slight shadow behind it. Since a high contrast leads to easier readability, weset the background of each slide to black. In order to achieve a real-time effect, we intended to leavethe camera in live preview mode while the user writes. When a misspelled word is detected, a red boxwould appear around the word and the user can then tap to view spelling suggestions in list format,with the incorrect spelling in red color (Fig.1.). We also decided to include a text-to-speech feature forthe user to listen to the different spelling suggestions as well as a custom dictionary that prioritizeswhich suggestions to display first based on previous selections.

3.2 Design Prototypes

Our prototype consists of two specific features. First, we would like to provide an image overlaid withbounding boxes over the misspelled words. Second, using a misspelled word as reference, provide sug-gestions for the correct spelling. Figure 1 shows an example of how we visualized the Google Glass towork.

3.3 Design Challenges

3.3.1 User Interface Design. One of the main challenges we faced when designing the user inter-face was adopting existing design standards with Android and Google Glass applications. Because ourteam had limited experience developing for Android, we had to initially spend some time to learn ba-sic Android application flow. In addition, Google Glass has it’s own design and user flows that builtoff of existing Android libraries that we had to learn. For example, the card view was one of Google



Fig. 1. Initial Prototype Designs

Glasss main drivers to getting user gestures, navigating between different parts of the application,and passing information between activities.

Another challenge for the design, involved the limitations of Google Glass’s screen size. Because thescreen was only large enough to fit part of the user’s view, we had to make sure that we didn’t overloadthe user with information that made the application difficult to understand.

We overcame this by prototyping some card views that were simple to understand, provided shortinstructions to guide the user in the application flow, and offered simple signifiers – such as a scrollingbar to indicate that they can navigate through a list – to assist them they in easily discovering newparts of the application.

3.3.2 Word Recognition. Word Recognition, also known as optical character recognition (OCR), willbe the key component in the system design. There are many different kinds of OCR methods, likeonline OCR or local OCR (i.e., using Tesseract library). We initially started by trying out local OCRthrough (OpenCV and Tesseract), but because Glass has limited support and uses an outdated AndroidSDK, we spent a good amount of time trying to fix build issues and overall OCR library integrationin our application. After we finally had local OCR working, we found that the results were extremelypoor. We tried doing basic image manipulation to improve the quality of the OCR, but still faced thesame issues that we initially ran into.

Our next option was using A9T9’s Online OCR API which did extremely well in recognizing printedtext, but did not do so well with handwritten text. We detail this challenge more in depth in latersections (along with its trade-offs), but overall given the time constraints of the course, we decided toproceed with this library.

3.3.3 Spell checking and suggestions. For spell checking, there are also many methods. For ex-ample, there is already a library called spelling checker in Android, which will need internet to runsuccessfully. We need pay more attention to the decision for using a internet method or a local method,as well as how and when to display the suggestion.

3.4 Evolution of Our Idea

Our idea evolved over time both from setbacks and also from valuable feedback. Initially, we envisionedEulexia as an augmented reality application, with incorrect spellings being detected in real time andproviding a list of word suggestions. After interviewing Gini Keating, we decided to add in dictation forthe suggested spellings. We also initially aimed for our application to conduct OCR locally so that it isindependent of an Internet connection. However, the Tesseract library did not produce usable results,so we opted for an online API, which made our word recognition much more accurate.ACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.


Fig. 2. Timeline

Our original UI design also evolved to work cohesively with the UX. Prior to development of Eu-lexia, we were unfamiliar with the Google Glass UI and believed that a list of spellings on one cardview would be sufficient. However, Glasss native card views show that lists on one card view are onlyaccessible via voice command, so for easy usability we switched to have each card view contain a singlespelling suggestion, and the user simply presses on the side of the Glass to hear the spelling out loud.

We also moved away from our original plan to have an augmented reality application because GoogleGlass does not posses the amount of processing power needed to achieve this. Ideally, we would preferthe live camera preview to be on as the user writes and then box misspelled words, but the Glassoverheats after a period of time, which currently limits this feature.

3.5 Timeline

Our timeline mainly featured work regarding two important functional modules: OCR and spell-check.Without either one, our application would be unable to implement the features we seek to provide. Ourfirst priority as a group was to implement each module independently, and then combine the two, tak-ing extracted text from OCR and passing it to the spell checking module. Our second priority thenconsisted of features useful to the user, such as user navigation, image capture aids, and audio dicta-tion. The timeline, that we set out to complete also involved user feedback from dyslexic individuals.

Overall, we have followed the timeline, illustrated in Figure 2, fairly closely. A mishap with ourinitial OCR solution lead us to scrap most of our work in progress, forcing us to transition to otherOCR libraries. On the other hand, spell checking and suggestions was implemented and integratedseamlessly. Unfortunately, we were not able to demonstrate the application to dyslexic individuals forfeedback due to time constraints.

3.6 Final Design

For our final design, we changed many aspects from our initial prototype. Instead of a card view con-taining both the misspelled word and the spelling suggestions below it, we changed the views to con-tain one word per card. This design choice to switch to large lettering was due to Google Glass’s limitedscreen real estate. The incorrectly spelled word as well as each spelling suggestion is displayed in thecenter of the screen and a simple single line instruction indicating the possible gesture as a captionon the bottom (Figure 3). The user can also swipe down to return to any previous card view, whichis one of Google Glass’s native gestures. The scroll bar at the bottom of the spelling suggestions cardindicates that there are more suggestions and that the user may swipe right and left to swap betweensuggestions.



Fig. 3. Reading and Correcting a misspelled word

Fig. 4. Module view of architecture

4. SYSTEM DEVELOPMENT

4.1 Architecture

Eulexia is designed as an Android application that uses the Glass Development Kit (GDK) previewrelease add-on. With the GDK, Eulexia can be implemented as Glassware for use with the GoogleGlass Explorer Edition. Also, we use additional libraries and APIs to implement certain features to bediscussed later.

The architecture design must support the following functions:

(1) Take an image(2) Process image for text(3) Spell-check input text(4) Provide suggestions for misspelled words(5) Audio feature for spelling of suggestions(6) Save commonly misspelled words

The architecture is split into functional modules or domains in relation to the above list. To illus-trate, Figure 4 shows each module with their respective transitions. Typically, each module receives aninput and provides an output to a user or module. This way, changes are isolated within each module,ACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.


Fig. 5. Comparison between Online OCR Service with different images

allowing for easier debugging efforts. The transitions between each module is aided with the use of theGDK, allowing us to create transitions and read gestures from the user.

4.2 Technology used

4.2.1 Google Glass Explorer Edition. The application is created for use with the Google Glass Ex-plorer Edition. We note that the Google glass allows us to work with its audio speaker, gesture com-mands, and camera. These all will play a role in incorporating features for our Glassware.

The gesture commands are they key feature for the user to interact with our application. The userwould be prompted for certain gestures to navigate in the application. As for the audio speaker, itallows us to provide a dictation aid when providing suggestions.

The camera on the Google Glass has some limiting features we have to consider when implementingEulexia. First, the camera takes pictures up to 5 megapixels uncompressed. This will affect the time itwill take to process the image during the OCR (Optical Character Recognition) process. Compressioncan help speed up the processing times, but leads to degradation of quality and accuracy of results.With this in mind, we consider accuracy as a top priority for the application when communicatingresults to the user. No amount of compression is utilized, leading to the best possible image qualityat the cost of processing times. On a different note, users may have difficulty positioning the camerawhen taking a picture of text. To counteract the problem with the camera capturing a small range ofthe users vision, we provide a live camera preview feature. We make this feature readily available,but optional in order to alleviate possible overheating problems. Additionally, the camera hardware islimited to a single focus, leading to some low quality text in images. Nevertheless, before processingthe image with OCR, we provide an image preview of the image taken for the user to verify that thepicture taken contains the text they want to spell check.

4.2.2 OCR Service. An accurate OCR service is necessary to provide precise results for spell check-ing. If this feature mistakenly extracts text from images, a spell checker would lose its ability to helpand inform the user. We decided upon the OCR online API from a9t9 (Autonomous Technology). Thisonline API is basically an extension of the Microsoft OCR library, which is easier to use and gives bet-ter results than Tesseract, previously our first choice for an OCR library [Technology 2015]. We alsoconsider other services, but research into other OCR online services yielded lesser results most of thetime. In our research, we note results comparing many services, shown in Figure 5. Comparison be-tween Tesseract Online (a variant of Tesseract) and Abby Cloud SDK (which is similar to a9t9’s OnlineOCR) show a clear difference in accuracy. Results were based on the Levenshtein distance between



Table I. Edit distance calculationparameters

Parameter CostRemove a character 95Insert a character 95

Swap adjoining characters 90Substituting a character 100

Change case of a character 10Suggestion Cost Threshold 140

Fig. 6. Application Flow Simplified

Fig. 7. Capturing an image with the viewfinder

output and truth input strings (this will be discussed later). Our test cases involve a low quality imageof text (ideally close to a low quality scan of a text document), and we prefer an OCR solution capable ofproviding accurate results. Despite this, we note that there is a limitation with current OCR solutionsused with images taken from a camera, as they do not consistently provide accurate results. Also toconsider, since this is an online service, it may take some time to upload the image using the Internet.To alleviate these processing times, loading bars are used to provide feedback to the user.

4.2.3 Spell-check and Suggestions. For spell checking and suggestions, the Jazzy library is used.Jazzy is an open source spell checker, providing a set of APIs relevant to our project. Basically, Jazzytakes as input a dictionary or word list with one word per line, to populate a dictionary hash map.We check against this hash map when determining whether or not a word is misspelled. As for sug-gestions, the Jazzy library utilizes the suggestion strategy from Aspell (the standard spell checker forthe GNU operating system). Basically, suggestions are based on thresholding an edit distance valuebased on a variant of the Levenshtein distance between two strings [Atkinson 2004]. The edit distancebetween two strings is defined as the number of operations, involving interchanging two adjacent let-ters, changing one letter, deleting a letter, or adding a letter, to transform one string into the other.To further clarify the default edit distance algorithm, please refer to Table I. Each operation has acost associated with it. Suggestions are generated and threshold-ed with costs less than the indicatedthreshold value. The default configuration follows closely with Aspells suggestion strategy of returningsounds-like suggestions up to two edit distances away.ACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.


4.3 Features

The first feature we show the user in the application flow is the camera view to assist them in takingpictures shown in Figure 7. We provide a short instruction on the homescreen to signify that theuser is able to switch back and forth between camera view and the home screen. Once the user ison the camera view they are able to see a live preview through the Google Glass’s camera. We chosethis feature because as we started testing the application, we realized that taking pictures without apreview was much more difficult than we initially thought. The images were sometimes out of viewand we would have to then take multiple pictures because words were cut off. With the live preview, itshows the user the actual image they would be capturing, so they are able to adjust to include/excludewords that they want to check for spelling.

The next core feature of our application is the A9T9 Online OCR API based off of Microsoft’s OCRengine. We use this API to process images that are taken with the Google Glass. We first start bysaving the image and uploading it to A9T9. After, we then parse the JSON response back and continuewith our application flow. One of the trade-offs of using an online library was that our application wasnow dependent on having an internet connection. This caused testing and future development of theapplication to be much more difficult as we had to make sure that Glass had proper internet connectionwith each run of the application. Google Glass did not have capability of using on-campus WiFi as itrequired secondary authentication which Glass did not support at all, so we relied heavily on mobilehotspots. This was a problem in itself as we then had to schedule times to meet to develop with Glass(see more in the Challenges section). And though this new dependency slowed development process,this same feature can also be seen as a benefit for the application overall. One of the major stress pointsof Glass is that it overheats when it does heavy computation, and it does overheat very frequently. Sohaving an intensive computational process such as OCR on the server-side (as opposed to local OCR)means that we can avoid having Glass overheat at that point in the application flow.

Another point that should be taken note of is that this is an OCR library and not an ICR (IntelligentCharacter Recognition) library, so A9T9 and our application mostly works on just printed characters.Our results have been fairly good with using this library, but for handwritten text it fails because itsimply wasn’t designed to support it. Though this is definitely an issue for our application, we see itmore as an trade-off because we looked into many libraries apart from A9T9 and most seemed to benot as reliable as the current one we were using. In addition, due to the time constraints of the course,integrating and testing the pros/cons of all of them did not fit with the timeline of our project, so wehad to make a decision on the library we were going to use and then move from there. We stand by thisfeature decision because it there were other features that needed just as much attention as OCR, andin the future, we could definitely come back and swap out the library for an ICR library easily becauseour application was designed to have loosely coupled modules.

After character detection is complete and processed back into our application flow, we then checkfor any misspelled words. The user is prompted with card views of words that they misspelled andthey are then able to swipe forward and back to view all the words that were misspelled in the imagecaptured (as shown in Figure 8. We chose card views because it matches the existing design standardsof Google Glass application and it provides the proper discoverability, signifiers, and gestures to allowthe user to easily figure out how to navigate the card views given the minimal screen space of theGoogle Glass. From this view, we offer a small instruction at the bottom of the view to tell users thatthey are then able to tap on a misspelled word card view which will then prompt them with suggestedcorrect word spellings (as shown in Figure 9). We decided to use the open-source library Jazzy to detectincorrectly spelled words and offer their respective correct suggestions (see more in the Jazzy section of



Fig. 8. Misspelled Words Found

Fig. 9. Suggestions for the misspelled word ”wayst”

the article). One major feature that Jazzy provides is that it allows us to use a custom dictionary so thatit allows us to account for words that aren’t in normal dictionaries such as slang and abbreviations.

The next major feature that Eulexia provides is that it offers text-to-speech spellings of words. Thisfeature was inspired through our interview with a person with dyslexia. She mentioned that she hadtrouble with homonyms such as (”Waste” vs. ”Waist”), and that to overcome this barrier she would con-stantly use Amazon Echo to ask Alexa how to spell the given word in the context of a sentence. BecauseAmazon Echo needs to be stationed somewhere, it can’t be used on the go and so our application buildsoff this feature by porting it to Google Glass.

In the suggested words card view the user is able to swipe back and forward to see scroll throughword suggestions for their misspelled word. We then provide a small instruction to the user that theyare able to long press on a word to hear the suggested word being spelled out character by character, asseen in Figure 9. Through our interviews and research, we found that this feature was key to includein our application because it allows one with dyslexia to draw their focus away from their eyes andinstead allow them to use hearing to be able to interpret and then spell the word correctly.

Finally the last major feature of our application is that it memorizes words that have been misspelledfrequently. We store misspelled words in a map and then use a priority queue to have that same wordoffered higher in the suggested words list the next time a user misspells that word. We draw thisfeature from our research because it builds off the fact that people with dyslexia still have a hard timespelling words that they commonly misspell. In fact, in one of our interviews with a dyslexic person, wenoticed that the user actually misspelled her own name (even though she has probably wrote it downACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.


Fig. 10. Testing process flow

countless times throughout her whole life). Her name wasn’t a common one, so it wouldn’t appearin a normal dictionary, but in combination with Jazzy’s custom dictionary capabilities and our wordmemorization feature, our application could detect whenever she incorrectly spells her name and thenoffer her the corrected name spelling as the first suggestion provided – which would be therefore beextremely customized to the user’s experience.

5. TESTING AND EVALUATION

5.1 Testing Process

The testing process follows closely with the flow of the application, which is illustrated in Figure 10.First, the application begins with the title screen, which has some instructions and an applicationlogo. The user then simply taps to enter a live camera preview with the viewfinder. After capturingand accepting the picture, the application will run the online OCR module, with a loading screenindicating progress with saving the image and receiving the OCR response. With the OCR response,we would get a string a words extracted from the image with or without misspelled words. The nextmodule for testing would be the spell check. The application will send the response from OCR to thespell check module. The spell check should return a array of misspelled words, which can be used tocheck whether the results are expected. Then for each misspelled words, the user can tap and go to thesuggestion module, with a list of suggested words related to each misspelled words.

5.2 White-box Testing

White-box testing, also known as clear box testing, is used to test internal structures (i.e., each module).For the main module, we create possible inputs and send them to the special module and then checkwhether we get the desired outputs.

For the OCR module, we write some neat handwriting on the whiteboard, consisting of some correctwords and misspelled words. Then we use the Google Glass to take a picture, and send the pictureto the OCR module. In the debug mode, we can easily print out the response from OCR and checkwhether it detect the words in the whiteboard correctly. One interesting we find in the testing is that,the a9t9 OCR will detect the new lines as escape characters such as ”/r/n”, which is not expected. Tosolve this, we remove all the escape characters ”/r/n”.

For the spell check module, a string of words are used as input to the spell check module. Testsinclude several misspelled words in the string. The expected results should be a hashmap with mis-spelled words as the key and maps with a list of suggestion words. To avoid too many suggestion wordsslowing down the speed of the application, we only capture the suggestions up to two edit distancesaway, as previously mentioned in our system design.



5.3 Black-box Testing

Black-box testing is used to examine the functionality of the application without peering into its inter-nal structures or workings. In the testing process, we do some role playing to implement the Black-boxtesting. Behaving as a user, we tested our application with writing from a whiteboard and typed texton a laptop screen. In this part of testing, we mainly focus on the correctness and the processing timesof our application. Also we evaluated the user experience of our application (i.e., how long it will takeusers to get used to our application and what difficulties they may meet when running) and the poten-tial issues, such as Google Glass overheating and limited battery life with high power consumption.

5.4 Evaluation

5.4.1 Performance. The recognition works near-perfect on print type words, the correction rate canbe more than 95% when testing with 10 words. For neat handwriting, the OCR also works satisfactorily.The spell check module works well and we can update our dictionary for future use. The text-to-speechworks great; we can modify the speech speed to make it more user-friendly. Also in the processing ofrunning the application, we provide operation instructions to help the user navigate forward withinour application. Since we built our application in a clear and concise way, users would soon learn howto operate the application properly. Also, we provide loading screen when processing OCR and spellingchecking to inform the users of the current running status and the current application progress (suchas saving pictures or receiving OCR responses).

5.4.2 Limitations. The limitation of the application mainly consists of the OCR results. The speedof OCR is still an issue, currently it may need 10-20 seconds to finishing OCR and spell check. We needfast speed and less time running OCR if we want the application be used in real world applications.Also, only print type words and neat handwriting words can be recognized. Detecting messy hand-writing will easily result in mistakes, citing a need for more powerful and accurate OCR methods. Forhardware and device limitations, the Google Glass itself has some weaknesses. It will generate a lot ofheat, causing the device to overheat in some cases. Additionally, wearing the Google Glass for too longwill cause discomfort with the right ear, because the battery and other hardware situated on the rightside makes it heavier than typical eyewear. The battery can also be a problem: with full battery, thegoogle glass can last for around 2 hours. Finally, another limitation is the fact that the application hasto be connected to the Internet to function fully. Without it, the OCR module will be unable to processimages.

6. COLLABORATION

6.1 Contributions

6.1.1 Application Contributions

—Phoebe Gao: Native OCR, result caching, user interface design, gestures, camera—Michelle Wu: User interface design, testing/debugging, interview—Andrew Du: Native OCR, live preview, interview—Sebastian Cheah: Online OCR, spellcheck, suggestions—Busheng Lou: Online Spelling check,testing/debugging, evaluation

6.1.2 Report Contributions

—Phoebe Gao: Executive summary, motivation and background, evolution of ideas, screenshots—Michelle Wu: Introduction, previous solutions, design, collaboration, conclusionACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.


—Andrew Du: Features, conclusion—Sebastian Cheah: System development, Document formatting—Busheng Lou: Design challenges, testing and evaluation, future work

6.2 Collaboration Tools and Problems

Our team split into smaller subgroups to work on different aspects of our application. For OCR imple-mentation, Phoebe, Andrew, and Sebastian worked together and for the user interface design, Phoebeand Michelle collaborated on user experience and gestures. Michelle and Andrew also made up thesub-team that scheduled an interview with the dyslexic UI researcher. Busheng tested and debuggedthe written code. During our weekly meetings, we worked together to brainstorm application ideas, so-lutions to current implementation problems, as well as troubleshoot. The largest problem we face wasschedule coordination, which we overcame by constant online communication via Facebook Messengerand pairing up with team members with similar schedules. Github was our repository host of choice,and each team member working on a different part of the application created a different branch fromour main repository. Once a feature was fully functional, the responsible team member created a pullrequest where it was reviewed before merging his or her code with the master branch. For final testing,we went through the whole application flow, and each time we got bugs, we isolate the conditions toreproduce it, and then review the problem through general debugging techniques. Some bugs were toocomplicated to debug, so we refer to the sources who worked on similar functionalities to help solve thebugs.

7. CONCLUSION AND FUTURE WORK

7.1 Conclusion

We first started this project by expanding out problem spaces that we noticed throughout our lives.Given the technologies we were able to choose from and our course goal to help others through ubiq-uitous computing, we tackled the problem space of assisting users with dyslexia. This space in itselfis extremely challenging to address because the dyslexia is a disability that is rarely discussed evenif one was diagnosed with it or even if one has shown symptoms with it. In one of our user interviewswith a person with dyslexia, she mentioned that even though she and her colleagues knew that shehad this disability, it was not a comfortable subject to bring up. In fact, she argued that dyslexia is atype of disability that one would try to hide, and if a person had dyslexia and was somewhat consciousof it, they would not be willing to get themselves tested or diagnosed with it mostly due to the neg-ative connotations that revolve around the disability. So with that said, one of our major challengeswas designing for a group of users that we had no prior experience of working with. It was extremelydifficult to design in a space that could not be simulated easily. We couldn’t address the basic designquestions such as: What was it like to read/write with dyslexia? What were their pain points? Howdid they feel about existing solutions? What problems haven’t been addressed yet? And are there anyunderlying problems that stem from reading/writing with dyslexia? And even trying to find people thatwere dyslexic to interview/test our application was one pitfall that we encountered.

One pitfall that we initially tried to avoid in the beginning of the project was being too solutiondriven. We tried to stray away from this common engineering misstep by collaborative brainstorminguses cases, user flows, challenged points in the process of writing, and conducting user interviews.However, as the project dragged on throughout the course, we ran into issues with technical imple-mentations of libraries with Google Glass and even the limitations on the hardware with Glass itself.We spent a good amount of time trying to overcome these issues to get back on track, but in doing so,we derailed ourselves from constantly involving the customer to having the general thought of them in



mind. Though this may seem as a poor design decision, we believe it was the right one to make becauseit allowed us to push out the essentials of our application and build from there. We made this choicedue to the time constraints of the course, and a tradeoff of this decision is that we weren’t able to testthe user experience of the application thoroughly. But this problem isn’t too big of an issue because wecould then test users in the future, and having that foundation of good design process in the beginningof the project allows us to be confident that the decisions that we make now are geared towards theuser.

One of the core features of Eulexia was character recognition. We addressed this problem in thecurrent application version by utilizing an online OCR library. Though it works great for printed text,it doesn’t work nearly as well for handwritten text so a major future direction of the project wouldbe to instead use an ICR library to detect handwritten text. In addition, the interface of displayingmisspelled words and providing more context to the user about where the misspelled words located,and how we should then go about informing the user of correct spellings could also be improved.

Eulexia does well in providing initial steps in ubiquitous computing towards helping users withdyslexia. Though there were hardware and software limitations with Google Glass, we tackled eachproblem by keeping the user in mind and made design decisions and tradeoffs based off of our initialresearch with our customers. And overall, we are pleased with the progress and insights that we ableto provide to others that choose to address this problem in the future with our product.

7.2 Future work

Ideally, our vision is for Eulexia to become a fully ubiquitous application. Future work can be donein terms of offering the user different modes of spell checking to choose from. Once Google Glassimproves its processing power, one feasible mode can be real-time spell checking, in which the livepreview feature of Glass is left on for the duration of the user’s note taking. Once the user misspells aword, the Glass will beep, indicating the error and the user can then stop writing and perform a spell-check to make a correction, all achieved in real-time. The benefit of this feature is that it is similar toa non-dyslexic person changing his or her mind and returning to correct their mistake. The mode wecurrently have implemented is to be used when a user has a long trail of thought and does not want tobe disturbed until he or she is finished writing. Once done writing, he or she performs the spell-checkto receive all the misspelled words displayed at one time, and can then jump from word to word.

Improving our currently existing mode entails more customization towards the user’s needs. Dyslexiavaries in different degrees among people, so a multi-sensory approach is key to developing the optimalspell checking application. This means catering to the user in both visual and auditory cases, and ap-proaches to achieving this include having customization options with different font choices and colors,as well as allowing users to choose between highlighting words, displaying them in different colors, orboxing/circling words. Offering a definition or sentence usage feature for each suggestion is anotherarea where customization can be key. An audio spelling may not be enough for the user to decidewhich spelling he or she intended, so pulling up an example usage can give more context. We can alsolook into examining the context of the sentence that the user wrote the misspelled word in. A machinelearning algorithm that can adjust to the user’s writing style is a more efficient way of predicting whichspelling suggestion the user will choose.

Another core feature, spell-checking and suggestion generation, can be improved upon. First, somewords can be detected as misspelled despite the user disapproving of the application’s conclusion. It ispossible to modify the dictionary word list in memory and on disk to help with this use case. Jazzy sup-ports this partially, with word additions to the dictionary, but missing the opposite: deletion support.With no way of reverting an addition without manually modifying the dictionary word list directly, wedid not include this in the initial application. In the future, we can modify the Jazzy library, and recom-ACM Transactions on Applied Perception, Vol. 0, No. 0, Article 0, Publication date: 2015.

Eulexia: A Wearable Aid For Proofreading • 15

pile a new version with this functionality in place. If we do this, we must consider how to support suchfeatures with the existing gestures available and remaining to us in our implementation of a GoogleGlass app. Second, the loading of a dictionary word list into memory versus disk is an ongoing issue.We can look into newer algorithms with a suitable balance for use with the system specifications ofthe Google Glass. Such algorithms may have an effect on dictionary modification support, as we haveseen implementations utilize specifically formatted dictionaries generated using a fixed word list. Itcan lead to the loss of addition and deletion operations on word lists. Nevertheless, we can extend uponthe base classes already in Jazzy, providing our own implementations in the future.

REFERENCES

Amazon. 2015. Ask Alexa. (2015). https://www.amazon.com/gp/help/customer/display.html?nodeId=201549800Kevin Atkinson. 2004. Aspell Suggestion Strategy. (2004). http://aspell.net/man-html/Aspell-Suggestion-Strategy.htmlJoshua Hall. 2014. DCODIA app helps student with dyslexia read, saves families thousands. (2014). http://techpoint.org/2014/

08/dyslexia-technology-software/Robert L. Peterson and Bruce F. Pennington. 2012. Developmental dyslexia. (2012). http://pediatrics.uchicago.edu/chiefs/DBP/

documents/reading\%20pdf/Dyslexia.Peterson.pdfMichael Sheehan. 2012. IOS 5 TIP: USING SIRI AS A SPELL CHECKER AND SPELLING

ASSISTANT SPELLING TEST RESULTS! (2012). http://www.hightechdad.com/2012/01/23/ios-5-tip-using-siri-as-a-spell-checker-and-spelling-assistant-spelling-test-results/

Autonomous Technology. 2015. ”Hosted Microsoft OCR library”: Free OCR API. (2015). http://blog.a9t9.com/2015/09/ocr-api.html


Online Appendix to:Eulexia: A Wearable Aid For Spell-checkingSEBASTIAN CHEAH, BUSHENG LOU, MICHELLE WU, PHOEBE GAO, and ANDREW DU,University of California, San Diego

A. KEY PROBLEMS ADDRESSED

A.1 Difficulties with writing for individuals with dyslexia

This obstacle can hinder performance and confidence in school, or the workplace, putting dyslexic in-dividuals at a great disadvantage. Eulexia would be able to help balance out the playing field and evenprevent young dyslexic students from falling behind in school and losing faith in their own abilities.

A.2 Differentiation between correctly spelled and misspelled words

For a dyslexic, one issue with trying to learn the correct spellings of words is that even though theymight be looking at the correct spelling of a word, they would still not be aware of how to spell itbecause they might not be able to process the characters like everyone else. Eulexia addresses thisissue by using the microphone on the Google Glass to spell out the word. In one interview, we foundthat the interviewee found it useful to look at words as shapes rather than individual characters.This could be a potential future direction to investigate different ways in which we can help dyslexicindividuals get a better handle on both reading and writing.

A.3 Tendency for repeat mistakes

This problem exists not only within the dyslexic population, but it exists within the entire human pop-ulation as well. Eulexia addresses this issue by keeping track of the user’s most frequently misspelledwords so that it makes the learning process smoother and more organized.

B. DESIGN IDEAS

B.1 Ubiquity

We wanted our application to integrate flawlessly into the workflow of dyslexic individuals, so weoriginally came up with the idea to use augmented reality to highlight the user’s misspelled words inreal time. However, we were not able to achieve this due to the current technological limitations of theGoogle glass. Instead, we came up with the idea of allowing the user to manually take a picture of thetext in order to check for misspelled words. We reasoned that this might actually be a better option inthe end, since it’s a lot less obtrusive and it allows the user to be in control of when he or she wants toverify his/her work.

B.2 Personalization

We really liked the idea of allowing personalization within Eulexia, so we came up with the idea ofkeeping track of the users’ most commonly misspelled words and providing the user with differentmediums of presentation (such as text or audio) for spelling suggestions.

c© 2015 ACM. 1544-3558/2015/-ART0 $15.00DOI: 0000001.0000001


App–2 • S. Cheah, B. Lou, M. Wu, P. Gao, and A. Du

C. IMPLEMENTATION

C.1 OCR

For OCR, we used a9t9’s (Autonomous Technology) online OCR API to process images that are capturedwith the Google Glass’ built in camera.

C.2 Spell-check

For spell-check, we used the Jazzy library and downloaded a dictionary from Jazzys website. Withgreater computational power on the Glass, we would be able to use larger, more advanced dictionaries.

C.3 Word Caching

We used local storage (on the Google Glass) of a hash map to keep track of most commonly misspelledwords for the user.

D. FEATURES

(1) Detect misspelled words(2) Spelling suggestions(3) Audio playback of spellings(4) Word caching

E. EVALUATION

Our prototype looks very promising, but because of current hardware limitations and time constraints,we were unable to implement all of our design ideas. We are extremely glad to say that the OCRfor our prototype is working on neatly handwritten text, and we believe Eulexia has the potential tomake a big difference in the lives of those who are affected by dyslexia. In particular, its features ofaudio playback and word caching will allow Eulexia to cater to the dyslexic user and personalize theirlearning experience.

F. FUTURE DIRECTIONS

Future directions include finding a better OCR engine that is more adept at recognizing handwriting.Perhaps even a machine learning model would be appropriate in order to allow personalization ofthe character recognition. We also hope to do more research on different types of dyslexia and offerdifferent modes to better suit users with different needs. If Google glass ever improves its processingpower, we can also look into providing a real time mode.


Documents

Eulexia A Wearable Aid For Spell-checking