Literature Review: To explore text input techniques forsmart watches to overcome the occlusion and fat finger

problem

Amruth AkojuSOFTENG 702

Final Report2014

ABSTRACTThe ’fat finger’ and occlusion problems still plague the smartwatch technology because we cannot ignore the fact that ourfingers are simply too large for text input on a small screensize. There are however ways to either minimise the prob-lems by optimising the text input keyboard [1, 6, 4] or movingthe input to other parts of the smart phone like the wristbandas proposed by Funk et al [2]. Other ways to minimise theproblems would be to make better use of the finger by dis-tinguishing between certain parts of the finger as proposedby Huang et al [3]. This technique allows the input fieldsto be much smaller on the smart watch and therefore at leastgreatly minimising the ’fat finger’ problem. All the reviewedliteratures propose solutions which have their weaknesses andfurther research needs to take place in order to combat theseweaknesses or by providing a hybrid solution.

INTRODUCTIONThe aim of this review is to identify means to solve the fatfinger and occlusion problems that plague the smart watchdevices. The fat finger problem defines that a normal sizedfinger covers up multiple buttons on the screen and there-fore making the click ambiguous. Occlusion problem hap-pens when or finger hides important parts of the screen as weinteract with the device. This review focuses on text inputand not general input like the gesture based system proposedby Xiao et al [8].

The first section will go into currently researched techniqueswhich work right of a native touch-sensitive smart watch suchas the Sony SmartWatch 2 [1] without the need for additionalcomponents.

The second section will go into techniques that require theuse of additional components other than the touch-sensitivesmart watches currently available on the market.

The two sections are separated because the native techniquesare likely to be easily implemented in the market whereas

the techniques that require additional equipment will requiremuch more research and testing to be marketable.

Figure 1. Prototype smartwatch text entry on Sony SmartWatch 2.

NATIVE TEXT INPUT TECHNIQUESThis section will analyse novel text input techniques for smartwatches where the solution is software based and thereforethe technique could potentially be implemented in to mosttouch-sensitive smart watches. The papers are identified asrelevant since they offer novel text input techniques whichtry to solve the fat finger and occlusion problems without theneed for extra components other than a touch-sensitive smartwatch.

Keyboard OptimisersDue to the availability of various touch-sensitive smart watchmodels released in the market right now, it would increase thedevelopment speed of novel text input techniques if the ap-proach does not require additional custom hardware.Dunlopet al. promote the use of an optimised layout of text char-acters, modelled by them, which attempts to make good useof the limited screen real estate on a smart watch without theneed of additional components (see Figure 1) [1].

MacKenzie et al have a similar technique of optimising a key-board layout by grouping multiple characters to the same but-ton [6]. Their design uses four keys to represent the 26 alpha-bets whereas Dunlop et al have six in their design. It is quitedifficult to state which one has the higher words per minute(WPM) rate because Dunlop et al lack the data to be able toclearly state their potential WPM. However, MacKenzie et alhave claimed their user testing participants have clocked in20 WPM [6].

The main points to gather from the two papers is that reducingthe number of buttons and optimising the layout could poten-tially make typing more efficient because the buttons can belarger without sacrificing input field screen space. Havinglarge buttons also avoids the issue of the fat finger problemcompared to having an entire miniature QWERTY keyboardon a small screen where the buttons are hard to tap individu-ally.

A point of note regarding the technique proposed by Dunlopet al is that their argument focuses on making text input on thesmart watch more efficient in terms of speed. However, theirlayout appears to occlude the text input area when the userattempts to access the top three buttons. The paper concludesthat the grouping of the characters is optimal under their rank-ing system and due to this more independent research needsto take place in order to find a better layout in order to solvethe occlusion problem.

At this point in the review, MacKenzie et al have a tech-nique which appears to be more efficient than the text inputtechnique proposed by Dunlop et al. However, in order totruly compare the two techniques we will need to considerhow the user testing was conducted, mainly because the tech-nique proposed by Dunlop et al lacks data on the WPM rate.Dunlop et al. claim their technique has potential since theyconducted a detailed analysis of variations of text characterarrangements and they have ranked them based on a disam-biguation score. Under their specifications, a text arrange-ment with a lower disambiguation score allowed a new userto easily understand the layout and make fewer errors [1].

MacKenzie et al claim their text input technique avoids thefat finger and occlusion problems, and therefore increases theefficiency of text input, based on the conclusions made fromusability testing conducted on six participants. The six par-ticipants managed to clock in approximately 20 WPM whichis double that of another technique that will be looked at laterin the review. While Dunlop et al, have optimised their key-board layout based on iterative testing, MacKenzie et al haveperformed statistical testing and made use of Huffman algo-rithm in order to make their four key layouts functional [1,6].

A major noteworthy point is that MacKenzie et al use a handcoded frequency table, and characters are placed into a treestructure and upon the pressing one of the four keys, the treeis truncated to identify the most appropriate word [6]. Thefrequency table is hand coded which means it could have biasand inaccuracies that could make the technique inefficientwhich is unwanted even if the fat finger and occlusion prob-lems are solved. Future research into this technique couldinvolve creating vigorous statistical testing to find the mostoptimised frequency table thus solving the noted problems.

Figure 2. Shown is a ”1Line Keyboard” optimisation layout proposed byYat Li et al

Keyboard Optimiser for other devicesPapers by Dunlop et al and MacKenzie et al look into op-timising a keyboard layout for smart watches however wecould expand our way of thinking by looking at text inputtechniques for other devices. Devices such as a tablet whichis also plagued by limited screen space when the normal QW-ERTY keyboard is displayed. A paper published by Yat Li etal. proposes an input technique which layouts a QWERTYkeyboard into a single line consisting of eight keys (see Fig-ure 2) [4].

Reducing the number of keys that are available to the usermeans we could have larger buttons whilst still leavingenough screen area for the input field. Their eight buttonsystem would definitely solve the fat finger problem if thebuttons are sufficiently large. The occlusion problem is alsosolved if the eight keys are positioned below the input fieldso the fingers will not cover the text field unlike the techniqueproposed by Dunlop et al.

The major issue with reducing the number of buttons meansthat words with the common sequence of key presses wouldcause ambiguity. Yat Li et al also state that their techniquedisambiguates the input the user enters by narrowing downthe most acceptable word as the keys are press in sequence[4]. They claim that their technique could allow normal usersto type 30 words per minute which is surprising fast for a con-densed form of a QWERTY keyboard layout (see Figure 2).

A difference between the technique proposed by Yat Li et aland the other two techniques that were looked at previously isthat the other two techniques use a condensed keyboard withthe keys arranged in alphabetical order instead (see Figure 1).Considering that a condensed QWERTY keyboard clocked30 WPM while an alphabetically ordered keyboard clockedin about 20 WMP suggests that future research will need tofocus more on QWERTY layouts.

The technique proposed by Yat Li et al has an average WPMrate of 30 and this claim is supported by a test conducted on50 people in lab conditions. The tests revealed that partic-ipants were able to input text fairly quickly and made rela-tively few errors. The most interesting contribution that thispaper has made in this review is that the technique proposedby this paper successfully reduces the screen space occupiedby the technique [4].

Therefore this technique has the potential of solving the fatfinger and occlusion problems. However, more research intoporting this technique to smart watches will need to take placein order to truly know whether an optimised QWERTY key-board is the step in the right direction.

The idea of using reducing the number of input buttons doesappear to increase efficiency however more research into thisarea would be required to achieve a truly efficient optimisedkeyboard arrangement technique. Another technique that willbe addressed in the following sub-section is the ZoomBoardwhich comes under the group of techniques with no addi-tional component requirement. However this technique dif-fers from the previous three techniques by providing an un-touched QWERTY keyboard to the user on a smart watch.

Figure 3. ZoomBoard on a watch-sized device. The keyboard is fullyzoomed out by default (A). When users press a key, the keyboard iter-atively zooms in (B and C), until the keys are a size that is comfortableand accurate (C). After the desired character is entered, the keyboardresets (D). Users may also swipe to the left to delete, to the right for aspace, and up to switch to a symbols keyboard. [7]

Zoom BoardThe previous sub-sections reviewed techniques to solve thefat finger and occlusion problem through keyboard optimisa-tion and key arrangement. There is another technique that ex-ists in the research area which is called the ZoomBoard. Thepaper presented by Oney et al proposes a technique whichusers are already familiar with, requires no additional compo-nents, and solves the fat finger and occlusion problem. Zoom-Board (see Figure 3) introduced by Oney et al. is a techniquein which users are able to zoom into a QWERTY keyboardin one tap and in the consecutive tap could select one of thezoomed in characters on the display [7].

Providing users with a keyboard layout which they are fa-miliar with allows for easier understandability and is morelikely to be accepted. This contrasts the keyboard optimisersolutions that were previous mentioned, namely the techniqueproposed by MacKenzie et al which optimises the keyboardinto four keys which would straight away make highly changeresistance users to reject the technique.

Oney et al. assert that their ZoomBoard technique solvesthe occlusion and fat finger problem and this assertion issupported by the conclusions made from user testing whichfound that users are easily able to reach approximately 10words per minute within a few attempts [7]. This wordsper minute rate is half as fast as the technique proposed byMacKenzie et al. Words per minute rate does not support theclaim of solving the fat finger and occlusion problem howeverit is necessary to allow the user to be able to type a decentWPM rate and 10 is sufficient enough.

One noticeable downside to the ZoomBoard is that a user isrequired to tap on the screen twice to enter a single characterwhich could build up user frustration. Although the WPMrate is manageable, over a long usage period a user could feelunbearably frustrated and this could lead to further researchon combining this technique with word prediction algorithmswhich could potentially increase the WPM rate and accuracywhilst limiting user frustration.

The techniques that were looked at up until this point requiredno additional components to provide a text input on a smartwatch by solving the fat finger and occlusion problems. Thenext section will review techniques which attempt to solve thesame problems but require special components which are notgenerally provided with touch-sensitive smart watches cur-rently in the market.

Figure 4. The prototypes with the two keyboard layouts. The multitaplayout is shown on the left and the linear keyboard is shown on the right[2]

Figure 5. Shown is the image of an old phone keypad with the commonlyused alphabet grouping.

TEXT INPUT TECHNIQUES WITH THE NEED OF SPECIALCOMPONENTSThis section aims to review literatures which propose noveltext input techniques which attempt to solve the fat finger andocclusion problems that plague smart watches. Techniquesthat will be discussed in this section differ from the other tech-niques discussed in the previous section simply by the fact thefollowing techniques involves a smart watch with additionalcomponents to aid text input.

Touch sensitive wristbandIt is no surprise that text input on a smart watch is very dif-ficult because of issues such as the fat finger problem wherea normal sized finger occludes multiple on display touch but-tons, due to the small screen size, thereby reducing the accu-racy of the touch [2]. Funk et al. propose two unique tech-niques which make use of touch-sensitive wrist band that at-tempts to tackle the fat finger problem for text input [2]. Theirfirst approach is laying out the 26 alphabets in groups similarto that a keypad of an older phone model (see Figure 4 andFigure 5).

This approach is similar to the keyboard optimisation tech-niques that have been addressed in the previous section,namely technique proposed by Yat Li et al, where the char-acters are grouped in an optimised fashion to improve effi-ciency. In the case of this technique however, the keyboard isoptimised in order to be placed a touch sensitive wrist bandas opposed to optimising for a good fit on the screen itself.

The second approach is an attempt to make good use of thetouch-sensitive wristbands vertical space but not a means toreduce the number of buttons unlike the keyboard optimisa-tion techniques that were discussed in the previous section.The user is required to slide over the 26 alphabet keys, alignedvertically on the wristband, and release their finger to a select

a character (see Figure 6).

This approach prevents the finger from occluding the screenwhilst providing text input without having to look at the wrist-band. This contrasts the first approach proposed by Funk etal, where the user would have to rotate their wrist in order tofind the correct key shifting his eyes off the screen [2].

Figure 6. This is what is displayed when the wristband vertical slidertechnique is used.

Funk et al claim that their two approaches solve the fat fingerand occlusion problems however they have lack of discussionon the inclusion of additional characters, such as tilde, excla-mation mark, or brackets, which would need to be placed allthe way around the wrist band. This lack of discussion is aresult of an experiment conducted on a group of participantsin a controlled environment.

This experiment concluded that participants found it harder toaccess the group of 26 alphabets situated at the bottom of thewrist band as they had to rotate their wrist and therefore hid-ing the display [2]. This paper provides a novel and plausiblesolution to the fat finger and occlusion problem however thispaper does not delve deeply into how more characters can besupported. Another noteworthy issue with this technique andalso techniques in the following sub-sections is that makingadditional components for the smart watch would need to begreatly justified.

Figure 7. Single-tap mode switching technique for touch interaction us-ing different areas of finger pads. (a) Different finger areas correspondto numbers vs operators. (b) The number 2 is entered by using a nor-mal tap. (c) The + operator is entered by tapping the key 3 with theright-side of the finger. The gray highlight indicates the on-screen keytouched, and the black highlight indicates the mode invoked. [3]

Smart watch with finger print scanningCertainly the most common manner of interaction with asmart watch is by tapping on the screen with the index fin-ger. Huang et al. suggest that it would be more efficient ifdifferent parts of a finger are recognised independently on asmart watch screen [3]. The other techniques have thus fareither optimised a keyboard by grouping multiple charactersonto the same button or providing off screen and on the wristband input. The technique proposed by Huang et al attempts

to make better use of the screen space provided for input bydistinguishing between the tip of a finger and the side of afinger [3]. They created a prototype of a smart watch whichuses finger print scanning to determine distinct parts of a fin-ger (see Figure 7).

Figure 8. The Facet prototype, consisting of six WIMM One devicesmounted in a bracelet such that they are removable [8]

Huang et al state that a user is able to input text in a novel andefficient manner on to a smart watch and make these claimsbased on the results obtained from usability tests. They testedwhether a user is able to differentiate between the distinctparts of the finger easily and with few errors [3]. Unlike thetests conducted for the touch-sensitive wrist band techniqueproposed by Fun et al, the test for the viability of Huang et alstechnique was done indirectly and therefore it is hard to con-clude whether this technique would be worth implementingwithout further research and testing.

One major issue with this technique is that a finger scan-ning module is required and these are not readily available onsmart watches which would pose a great development chal-lenge.

Multi-faceted smart watchSmart watches with sufficiently large screen sizes are nearimpossible to be accepted socially due to their clunky na-ture, even though they offer a nice means to input text onone screen and display input field in another. This solves thefat finger and occlusion problems however, would text entrybe any efficient if a smart watch has multiple screens?

Lyons et al. propose that their multi-segment smart watch,named Facet, would extend the input capabilities of the smartwatch and provide additional screen space to view more con-tent (see Figure 8) [5].

This technique is similar to the touch-sensitive wristbandtechnique proposed by Funk et al because they both offeran off-screen text input solution. Facet uses many differentscreens that go all the way around the hand which appearsto be too bulky for daily use. However, smart watches withdual screens have yet to be looked into and perhaps futureresearch could delve into this area as there is great potentialfor text input with this technique without the fat finger andocclusion problems.

Lyon et al claim that Facet is a solution worth looking at byproviding the readers with two user scenarios where Facet cancome in handy and could potentially fit right into the lives ofmost people. However, the scenarios seem to be pushing itsuse more than showing off its potential in the real world. The

size of the multi-faceted smart watch is quite bulky and couldget in the way of peoples daily lives which the paper ignoresto reveal in the user scenarios. The size contrasts the size ofthe touch-sensitive wristband smart watch [2] which is fairlythin and this means that the touch-sensitive wristband smartwatch is more likely to be accepted by the general public.

Having said that about the form factor, the potential of such adevice is that one screen can be dedicated to input and anotherfor display which solves the fat finger and occlusion problemsas stated earlier. The paper concludes that device has beenproposed to kick start research into multi-faceted devices andbecause this paper lacks any sort of usability testing it is quitedifficult to agree with its claims without conducting more re-search as stated.

FUTURE WORKDunlop et al suggest an optimised keyboard layout in theirpaper and their solution still allows occlusion to some degree.Future research into finding a better layout for their techniquewould help this field greatly [1].

From the works of Li et al, we found that a QWERTY opti-mised keyboard layout is slightly better in terms of averagewords per minute rate and therefore further research shouldhighly take into account a QWERTY setup [4].

Smart watches with dual screens seem to be potential can-didates for solving the ’fat finger’ and occlusion problem.However, care needs to be taken in order to make sure thedevice would be socially acceptable. Facet looks into multi-faceted screens however the device appears very bulky andthere would be less accepted by the public [5]. Dual screenhowever has greater potential to be made acceptable in termsof the form factor and still provide a solution for the fat fingerand occlusion problems.

CONCLUSIONThis review has looked at novel text input techniques whichattempt to solve the ’fat finger’ and occlusion problems thatplague smart watches. Native text input techniques tend toprovide the research area with somewhat of a solution to thestated problems by reducing screen space occupied by the in-put system.

The review has also looked at other techniques which requirespecial components in order to solve the stated problems. Thegreat thing about these techniques is that they completelysolve the ’fat finger’ and occlusion problems since the in-put happens outside the screen space. The downside to thismethod is that the techniques can not be implemented in toreadily available Smart watches currently in the market andcan often be too bulky for social acceptance.

A novel and effective means to solve the ’fat finger’ and oc-clusion problems is still unsolved and there is still massiveresearch potential in this field.

REFERENCES1. Dunlop, M. D., Komninos, A., and Durga, N. Towards

high quality text entry on smartwatches. In CHI ’14Extended Abstracts on Human Factors in ComputingSystems, CHI EA ’14 (2014), 2365–2370.http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2559206.2581319.

2. Funk, M., Sahami, A., Henze, N., and Schmidt, A. Usinga touch-sensitive wristband for text entry on smartwatches. In CHI ’14 Extended Abstracts on HumanFactors in Computing Systems, CHI EA ’14 (2014),2305–2310. http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2559206.2581143.

3. Huang, D.-Y., Tsai, M.-C., Tung, Y.-C., Tsai, M.-L., Yeh,Y.-T., Chan, L., Hung, Y.-P., and Chen, M. Y. Touchsense:Expanding touchscreen input vocabulary using differentareas of users’ finger pads. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems,CHI ’14 (2014), 189–192. http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2556288.2557258.

4. Li, F. C. Y., Guy, R. T., Yatani, K., and Truong, K. N. The1line keyboard: A qwerty layout in a single line. InProceedings of the 24th Annual ACM Symposium on UserInterface Software and Technology, UIST ’11 (2011),461–470. http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2047196.2047257.

5. Lyons, K., Nguyen, D., Ashbrook, D., and White, S.Facet: A multi-segment wrist worn system. InProceedings of the 25th Annual ACM Symposium on UserInterface Software and Technology, UIST ’12 (2012),123–130. http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2380116.2380134.

6. MacKenzie, I. S., Soukoreff, R. W., and Helga, J. 1thumb, 4 buttons, 20 words per minute: Design andevaluation of h4-writer. In Proceedings of the 24thAnnual ACM Symposium on User Interface Software andTechnology, UIST ’11 (2011), 471–480.http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2047196.2047258.

7. Oney, S., Harrison, C., Ogan, A., and Wiese, J.Zoomboard: A diminutive qwerty soft keyboard usingiterative zooming for ultra-small devices. In Proceedingsof the SIGCHI Conference on Human Factors inComputing Systems, CHI ’13 (2013), 2799–2802.http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2470654.2481387.

8. Xiao, R., Laput, G., and Harrison, C. Expanding the inputexpressivity of smartwatches with mechanical pan, twist,tilt and click. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, CHI ’14(2014), 193–196. http://doi.acm.org.ezproxy.auckland.ac.nz/10.1145/2556288.2557017.