MODELING VISUAL SEARCH TIME AND ITSAPPLICATION TO DESIGN VIRTUAL KEYBOARDS

Pradipta Kumar Saha

MODELING VISUAL SEARCH TIME AND ITSAPPLICATION TO DESIGN VIRTUAL KEYBOARDS

Thesis submitted to theIndian Institute of Technology Kharagpur

for award of the degree

of

Master of Science (by Research)

by

Pradipta Kumar Saha

Under the guidance of

Dr. Debasis Samanta

School of Information TechnologyIndian Institute of Technology Kharagpur

Kharagpur - 721 302, IndiaJuly 2013

c⃝2013 Pradipta Kumar Saha. All rights reserved.

CERTIFICATE

This is to certify that the thesis entitled Modeling Visual Search Time and itsApplication to Design Virtual Keyboards, submitted by Pradipta Kumar Sahato Indian Institute of Technology Kharagpur, is a record of bona fide research work undermy supervision and I consider it worthy of consideration for the award of the degree ofMaster of Science (by Research) of the Institute.

Date: 26/04/2013

Dr. Debasis SamantaAssociate ProfessorSchool of Information TechnologyIndian Institute of Technology KharagpurKharagpur - 721 302, India

DECLARATION

I certify that

a. The work contained in the thesis is original and has been done by myself underthe general supervision of my supervisor.

b. The work has not been submitted to any other Institute for any degree or diploma.

c. I have followed the guidelines provided by the Institute in writing the thesis.

d. I have conformed to the norms and guidelines given in the Ethical Code of Conductof the Institute.

e. Whenever I have used materials (data, theoretical analysis, and text) from othersources, I have given due credit to them by citing them in the text of the thesisand giving their details in the references.

f. Whenever I have quoted written materials from other sources, I have put themunder quotation marks and given due credit to the sources by citing them andgiving required details in the references.

Pradipta Kumar Saha

Dedicated to My Parents

ACKNOWLEDGMENT

During this period of my postgraduate study there are many people whose guidance,support, encouragement and sacrifice has made me indebted for my whole life. I takethis opportunity to express my sincere thanks and gratitude to all these people.

First and foremost I would like to express my deepest gratitude to my revered supervi-sor Dr. Debasis Samanta for his invaluable guidance, and his encouragement throughoutmy work. His guidance and support is far beyond duty. His constant motivation, sup-port and infectious enthusiasm have guided me towards the successful completion of mywork. My interactions with him have been of immense help in defining my research goalsand in identifying ways to achieve them. His encouraging words have often pushed meto put in my best possible efforts. Above all, the complete belief that he has entrustedupon me and has instilled a great sense of confidence and purpose in my mind, which Iam sure, will stand me in good stead throughout my career.

It gives me immense pleasure to thank the heads of the department Prof. J. Mukhopad-hyay and Prof. R. Mall for the world class infrastructure provided in the department tothe students. My sincere thanks to all of my departmental academic committee membersProf. A. Gupta, Prof. C. R. Mandal, Prof. S. Sural, Dr. S. K. Ghosh, Dr. K. S. Rao,Dr. S. Misra for their valuable suggestions during my research. I sincerely rememberthe support of office staffs Mithun Da, Soma Di, Malay Da, Vinod Da, Pratap Da andothers. I am also grateful to all members of School of Information Technology.

There is no way this thesis could ever have been completed without the inspiration,encouragement, criticism and patience of Sayan Da and Jayeeta.

I wish to convey my special thanks to my old friends Chandan, Bodhi and Aparnafor their constant support and help during the various stages of my work. I am greatlyindebted to many of my friends for their constant inspiration. The support of my labmates namely Sushantada, Debasish, Rajkumar, Col. Ranjit Singh, Prasenjit, Shobhanamadam, Ashalata madam, Shankar, Prasenjit, Barsha, Narendra, Jaswasi, Jainath, Gau-rang, Anant, Soumitri is invaluable. I would also like to express my thanks to Soumyajit,Barikda, Sajalda, Ranjan, Pushpitadi, Aditi, Soumya, Arindamda, Nirnay, Sudhamay,Gautamda, Subarao, Kanchan, Partha, Saptarshi, Subhomoy, Soumyadip and manymore. It is a great fun and source of ideas and energy to have friends like Sayan,Manoj, Pradipta, Soumalya, Santa, Jayeeta, Indira and many more during my stay at

vii

IIT Kharagpur. I also thanks all the family members of contemporary fellow neigh-bor, in particular Amit, Ashokeda, Kakali boudi and Anki for making the stay at IITKharagpur, ever memorable.

Nothing would have been possible without the moral support of my parents whohave been the pillars of strength in all my endeavors. I am always deeply indebtedto them for all that they have given me. I also thank all the other members of myfamily including my brother and sisters for their love, affection and timely help. Finally,to thank my wife Sayani, I really have no words to express my gratitude for all hersupport, encouragement, understanding and sacrifice, without which it would have beenimpossible for me to finish this work.

viii

Abstract

Performing user based evaluation of any user interface is practically an expensive andtedious job. To alleviate this issue, researchers advocate automatic design and evalua-tion procedure which includes modeling of different activities with respect to differentinterface design parameters. Visual search is one of such important activity. Visualsearch is a significant cognitive subtask of a sighted user in graphical user interfacebased interaction. Indeed, time required to perform visual search task is treated as animportant parameter while evaluating usability and user friendliness of an interface. Tothe best of our knowledge, work on modeling of visual search time is not reported. Wemay note that, visual search time depends on many visual search features such as size,shape, position etc. of the objects space of an interface. This thesis aims to address theabove mentioned research concern. More precisely, the main objective of our researchis to identify different visual search parameters and then to develop a computationalmodel to predict the visual search time for a given interface. So far the user interfaceis concerned, we limit our investigation to virtual keyboard, which is a graphical userinterface to compose texts. In the following, we summarize the major works carried outin this thesis.

Initially, we have studied visual features reported in different literatures which in-fluence visual search task. As impact of many of these features is subjective to userand beyond the scope of measuring them quantitatively, we have identified the visualfeatures which do not vary from user to user and also relevant in the context of a vir-tual keyboard interface. Next, we have conducted an empirical user evaluation on eachfeature to substantiate their degree of influences and come up with the features whichsignificantly influence the visual search time. Then, we have performed several user-based evaluations to accumulate knowledge about user performances with variation ofidentified features. To accomplish the modeling task, we have followed three modelingapproaches namely linear regression, non-linear regression and Support Vector Machine-based regression. Finally, the proposed model along with Fitts’-digraph model have beenused as two objectives to design an optimized virtual keyboard with respect to lesservisual search time and mouse movement time.

From our investigation, we observe that there are four visual search features in vir-tual keyboard interface which significantly influence the visual search time of a user.The computational model developed using SVM-based regression method outperformsother methods. It is also observed that considering visual search time into account, wecan develop a virtual keyboard with 12.37% improvement in text entry rate.

Keywords: visual search time, virtual keyboard, interface design, computational mod-eling, automatic design, design evaluation, human computer interaction

ix

Contents

Certificate i

Declaration iii

Dedication v

Acknowledgment vii

Abstract ix

Contents xi

List of Figures xv

List of Tables xvii

List of Symbols and Abbreviations xix

1 Introduction 11.1 Visual Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Visual Search Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Scope and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Related Work 92.1 Visual Search Features Identification . . . . . . . . . . . . . . . . . . . . . 102.2 Modeling Techniques for Prediction . . . . . . . . . . . . . . . . . . . . . . 12

2.2.1 Modeling of Visual Search Activity . . . . . . . . . . . . . . . . . . 132.2.2 Support Vector Regression-based Modeling . . . . . . . . . . . . . 15

2.3 Model-based Approaches to Design Virtual Keyboard . . . . . . . . . . . 17

xi

Contents

2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3 Identification and Analysis of Visual Search Features 253.1 Listing of Visual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2.2 Interface Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . . 33

3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.4 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4 Modeling of Visual Search Time 494.1 Training Data Set Generation . . . . . . . . . . . . . . . . . . . . . . . . . 504.2 Model to Predict Visual Search Time . . . . . . . . . . . . . . . . . . . . . 51

4.2.1 Linear Regression with Multiple Features . . . . . . . . . . . . . . 524.2.2 Non-linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . 544.2.3 Support Vector Regression . . . . . . . . . . . . . . . . . . . . . . 554.2.4 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3 Validation of the Proposed Model . . . . . . . . . . . . . . . . . . . . . . . 614.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5 Virtual Keyboard Layout Optimization 655.1 Proposed Virtual Keyboard Design Approach . . . . . . . . . . . . . . . . 66

5.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.1.2 Virtual Keyboard Design using Multiobjective Optimization . . . . 68

5.2 Movement Time Optimized Virtual Keyboard Design . . . . . . . . . . . . 725.3 Empirical Study to Evaluate Designs . . . . . . . . . . . . . . . . . . . . . 765.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

6 Summary and Conclusion 796.1 Contribution of Our Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.2 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.3 Future Scope of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Publications 85

xii

Contents

References 87

xiii

List of Figures

2.1 Some well known virtual keyboard layouts . . . . . . . . . . . . . . . . . . 21(a) Opti keyboard layout [66] . . . . . . . . . . . . . . . . . . . . . . . . 21(b) Metropolis keyboard layout [106] . . . . . . . . . . . . . . . . . . . . 21(c) GAG I keyboard layout [79] . . . . . . . . . . . . . . . . . . . . . . 21(d) GAG II keyboard layout [79] . . . . . . . . . . . . . . . . . . . . . . 21

2.2 Quasi-Qwerty keyboard layout [7] . . . . . . . . . . . . . . . . . . . . . . . 232.3 Sath keyboard layouts [20] . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

(a) Sath-Rectangular keyboard layout . . . . . . . . . . . . . . . . . . . 23(b) Sath-Trapezoidal keyboard layout . . . . . . . . . . . . . . . . . . . . 23

3.1 Virtual keyboard layouts used in experiments . . . . . . . . . . . . . . . . 31(a) Opti keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 31(b) Avro keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 31(c) iLipi-H keyboard layout with inflexion window . . . . . . . . . . . . 31

3.2 A sample layout with target object . . . . . . . . . . . . . . . . . . . . . . 343.3 Effects of different font sizes on visual search time . . . . . . . . . . . . . 36

(a) Avro keyboard with 12pt font size . . . . . . . . . . . . . . . . . . . 36(b) Visual search time for different font size . . . . . . . . . . . . . . . . 36

3.4 Effects of similar shaped characters on visual search time . . . . . . . . . 383.5 Effects of spacing between keys on visual search time . . . . . . . . . . . . 39

(a) Opti keyboard with 50% spacing between keys . . . . . . . . . . . . 39(b) Visual search time for different spacing between keys . . . . . . . . . 39

3.6 Effects of varying number of items on visual search time . . . . . . . . . . 40(a) iLiPi-H keyboard having 67 items . . . . . . . . . . . . . . . . . . . 40(b) Visual search time for different number of items . . . . . . . . . . . 40

3.7 Effects of search field size on visual search time . . . . . . . . . . . . . . . 41(a) Avro keyboard occupying 20% of screen area . . . . . . . . . . . . . 41

xv

List of Figures

(b) Visual search time for different search field size . . . . . . . . . . . . 413.8 Effects of different position of keys on visual search time . . . . . . . . . . 43

(a) iLiPi-H keyboard divided into 9 blocks . . . . . . . . . . . . . . . . 43(b) Result for different position of keys . . . . . . . . . . . . . . . . . . . 43

3.9 Effects of ordering of characters on visual search time . . . . . . . . . . . 443.10 Opti keyboard with two group . . . . . . . . . . . . . . . . . . . . . . . . 443.11 Effects of grouping and group size on visual search time . . . . . . . . . . 45

(a) Visual search time for different number of grouping . . . . . . . . . 45(b) Visual search time for varying group size . . . . . . . . . . . . . . . 45

4.1 Virtual keyboard layouts used for validation . . . . . . . . . . . . . . . . . 61(a) Fitaly keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 61(b) Guruji keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.1 Flowchart of virtual keyboard design using NSGA-II algorithm . . . . . . 705.2 Mouse movement and visual search time optimized keyboard layout . . . 725.3 Flowchart of virtual keyboard design using Genetic Algorithm . . . . . . . 745.4 Crossover operation in Genetic Algorithm . . . . . . . . . . . . . . . . . . 755.5 Keyboard layout minimizing mouse movement time . . . . . . . . . . . . . 75

xvi

List of Tables

3.1 Description of participants . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2 Similar character set of Bengali language . . . . . . . . . . . . . . . . . . . 373.3 Summary of statistical analysis for different features . . . . . . . . . . . . 47

4.1 Description of participants . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2 Summary of Collected Data . . . . . . . . . . . . . . . . . . . . . . . . . . 514.3 Different non-linear model with corresponding R2 . . . . . . . . . . . . . . 544.4 Different approaches with corresponding R2 . . . . . . . . . . . . . . . . . 604.5 Description of participants considered for validation . . . . . . . . . . . . 62

xvii

List of Symbols and Abbreviations

List of Symbols

p Pearson Correlation Coefficient

µ Mean

σ Standard Deviation

hθ(x) Hypothesis or Function

α Learning Rate

x Independent Variable

y Dependent Variable

x(i)j jth feature of ith training set

R2 Coefficient of Determination

D Training Dataset

ξ Slack Variable

ε Estimation Precision

αi Lagrange Multiplier

K Kernel

List of Abbreviations

AAM Area Activation Model

ANOVA Analysis of Variance

CPS Character Per Second

DF Degree of Freedom

EPIC Executive Process-Interactive Control

FD Fitts-Digraph

xix

List of Symbols and Abbreviations

FIT Feature Integration Theory

GA Genetic Algorithm

GOMS Goals, Operators, Methods and Selection

GS Guided Search

HCI Human Computer Interaction

KLM Keystroke Level Model

KS Key Size

MOGA Multi-objective Genetic Algorithm

MSE Mean Square Error

MT Movement Time

NC Number of Characters

NPGA Niched Pareto Genetic Algorithm

NSGA Non-dominated Sorting Genetic Algorithm

NSGA-II Non-dominated Sorting Genetic Algorithm - II

PAES Pareto-Archived Evolution Strategy

PC Position of Character

RBF Radial Basis Function

RT Reaction Time

SD Standard Deviation

SK Space between Keys

SPEA Strength Pareto Evolutionary Algorithm

SVM Support Vector Machine

SVR Support Vector Regression

TBS Target-Background Similarity

UCIE Understanding Cognitive Information Engineering

UI User Interface

VEGA Vector Evaluated Genetic Algorithm

VST Visual Search Time

WBGA Weight-based Genetic Algorithm

WPM Words Per Minute

xx

Chapter 1

Introduction

Of present day technologies, computer application has been spread in every sphere of life

and it has become a challenge to the user interface designer to design an interface which

has appeal to all category of users. As a consequence, interaction styles have changed

from command mode to direct manipulation mode [25]. With direct manipulation style of

interaction, a user can perform any task much more comfortably as it offers clear visibility

of intermediate and final results. While accessing the interface, the main concern of user

turns to find proper objects visually which are responsible for performing a task. As a

consequence, for a sighted user, visual search contemplates to be a significant cognitive

subtask which can be involved with performing any task in current user interfaces.

The objective of this thesis is to design a methodology to evaluate visual search tasks

automatically. More specifically, we plan to develop a computational model to predict

the average visual search time given an object space. In the domain of user interface

design, many object spaces are possible and each object space demands its own treatment

so far the perception task is concerned. Considering this, we limit the investigation only

to virtual keyboard [66], which is a graphical user interface to compose texts. The rest

of the chapter is organized as follows. Section 1.1 provides an overview of visual search

task. Details about visual search time and different features which influence the same

1

1. Introduction

are discussed in Section 1.2. The scope and objective of our work are presented in

Section 1.3. Finally, Section 1.4 presents the organization of this thesis.

1.1 Visual Search

The visual search processes that people use in Human Computer Interaction (HCI)

tasks have a substantial effect on the time and likelihood of finding the information

they required. Users encounter many challenges in finding the information they seek

when visually searching. Visual search is an intriguing human activity to study because

it requires complex interplay among three processes namely perceptual, cognitive and

motor [13]. Most important factor for visual search is eye movement. Information that

is perceived through eyes will vary as a function of the orientation of eyes. Visual acuity

is higher at the center of people’s field of vision known as fovea [4]. Cognitive processes

allow one to analyze the information perceived through the eyes. Motor process results in

actions generated by the cognitive process, that is, taking decisions once the information

is processed.

Visually searching for information is often fastest and most useful way of finding

information in a variety of user interfaces. Functionality such as web search or the Find

command present in many operating systems can be used to find items on a computer

screen quickly. However, there are many instances in which visual search is more useful.

Few of these scenarios are listed in the following.

• Searching among many similar results, where it is difficult to specify a search query

to locate the desired target (e.g. examining web search engine results).

• When an application does not include a ‘Find’ command (e.g. in video games).

• When the exact target is not known by the user (e.g. looking for items that match

some vague concept or goal).

2

1.2. Visual Search Time

In these cases, fast eye movements are necessary where many visual objects can be

evaluated by the user simultaneously, and target(s) can be located if exists. The visual

search process that people use has a substantial effect on the time and likelihood of

finding the information they seek. In fact, users encounter many challenges in finding

the information they seek when visually searching. In other words, visual search is

affected by many factors of the layout such as grouping, color, spacing, text size etc.

and also the strategies used like item-by-item, using labels or not, following the columns

or not etc. It has been observed from experiments that intensity of the object feature

often takes the decision of selecting target object from the distractor in the object space.

The more similar objects in the visual search space can be distinguished by more number

of features [93]. Human may first scan the object space, collect feature information from

object(s) in a parallel process then search for the required object with the special unique

feature in a serial process [98].

1.2 Visual Search Time

Visual search is at the heart of visual information searching. It is a subtask in each and

every visualization based interaction with computers. So the time required to perform a

visual search task is treated as an important parameter while evaluating usability, user

friendliness of an interface or any object space. Visual search time of a given object

space can be measured by either user-based evaluation or model-based evaluation.

The user-based evaluation requires to perform some tasks by users with different

perceptual and cognitive capabilities on an interface or an object space. Depending

on the users’ feedback, the average visual search time can be estimated and similarly

usability of the interface can be determined. This is a time consuming, tedious job and a

sufficient number of users are required for every interface to estimate visual search time

accurately. On the other hand, the model-based evaluation simulates users’ behavior

while using an interface. A mathematical model can be designed for predicting the

3

1. Introduction

performance of user interface in terms of visual search time, depending upon several

parameters of the user interface. Once a model is developed for a standard interface, it

can be used for measuring usability of the similar interfaces. In user-based evaluation,

users who are not familiar with the task may not get interest in accessing the interface

which causes poor results. In contrast, a computational model uses different features

to estimate average visual search time and thus provides both time and cost effective

solution as well as requires less users’ effort.

So, the model-based evaluation or the computational model can be considered as

User Interface (UI) prototyping tool that can produce quantitative predictions of how

users will behave, when the prototype is ultimately implemented. Thus, it provides a

rapid and inexpensive way to explore a large variety of UI ideas, compare them and

narrow down the options to a handful of designs to be empirically tested with users.

One can rapidly analyze competitor’s products as part of a competitive analysis and

compare new ideas with an existing version of the system to ensure that the new design

is better than the old one.

The most important contribution of computational cognitive models in the field of

HCI is that the models provide the science base that is needed for interface analysis

tools. Projects such as CogTool [46,90] and CORE/X-PERT [91] are at the forefront of

tools that utilize cognitive modeling to predict user interaction based on a description of

the interface and task. These tools provide theoretically-grounded predictions of human

performance in a range of tasks without requiring that the analyst, that is, the person

using cognitive models, be knowledgeable in cognitive, perceptual, and motored theories

embedded in the tool. Designers of application interfaces could use such tools to evaluate

their visual layouts, reducing the need for more expensive human user testing early in

the development cycle. Potential usability problems in interfaces could be identified

early, before time consuming user testing.

4

1.3. Scope and Objectives

A visual search time prediction model useful for specific application has been

proposed previously [37, 42]. But it is fixed for an object space and varies with only

the number of objects in the object space. The method has not considered any other

features of the objects in object space. So, the challenge still exists for addressing the

issues as well as developing a predictive model for calculating visual search time for

different interfaces.

1.3 Scope and Objectives

Estimating visual search time helps interface designers to evaluate their design with

minimum effort. But most of the work reported till date is mainly about describing

the visual search methodologies. This work proposes to address this limitation. We

propose to develop a computational model to predict the average visual search time

given an object space. In the domain of user interface design, many object spaces are

possible and each of them demands their own treatment so far the perception task is

concerned. Considering this, we limit the investigation only to virtual keyboard, which is

a graphical user interface to compose texts. Nowadays, modeling different tasks involved

in the context of text composition like messaging, chatting etc. draws more attention

with increasing use of PDAs, mobiles etc. Virtual keyboard contains different visual

features which affect text entry rate. Earlier, a model to predict visual search time has

been proposed by Hick and Hyman [37, 42]. This model only considers the number of

keys present in the keyboard and lacks in acquiring other visual search features like

shape, size, grouping, ordering etc. which also influence visual search time in finding a

key in the keyboard. With this perspective, in this dissertation, the plan of our work is

as follows.

• Identification and analysis of visual search features: All features of an

interface do not contribute equally for predicting the visual search time for it.

Some features vary with user, while some features are not applicable for a specific

5

1. Introduction

interface. Hence it is required to find out the set of features applicable for virtual

keyboard.

• Proposing a model to predict visual search time: After identifying the visual

search features, a computational model needs to be developed to predict visual

search time. This type of modeling requires knowledge about user performance on

various combination of identified feature values. So, several user-based experiments

are required to collect the knowledge to develop prediction model. The accuracy

of the model can be verified through users’ experiment.

• Application of the proposed model to design virtual keyboards: The

developed model then can be used to design an optimum virtual keyboard layout.

The layout can be optimized in terms of minimal mouse movement time as well as

minimal visual search time.

1.4 Organization of the Thesis

The thesis contains six chapters including this introductory chapter. This chapter

discusses about the motivation of the thesis work and main objectives of the work.

It gives a brief description of visual search methodologies and their types, importance

of visual search in interface design, different cognitive modeling and visual search time

modeling.

Chapter 2: Related Work

This chapter includes the state of the art for visual search task and provides an overview

of computational modeling. Also, we survey the existing model-based approaches of

virtual keyboard design.

Chapter 3: Identification and Analysis of Visual Search Features

In this chapter, we describe our approach to identify significant visual search features in

6

1.4. Organization of the Thesis

the context of virtual keyboard interface. This chapter includes user study on variation

of different visual search features and identification of significant features related to

virtual keyboard design parameters.

Chapter 4: Modeling of Visual Search Time

The proposed computational model to predict visual search time of virtual keyboard

interface, based on the identified features, is discussed in this chapter. This chapter also

includes details of data collection of user performance for different variation of multiple

visual search features, which is required to develop the model, performance of different

modeling approaches on the collected data and validation of the proposed model.

Chapter 5: Virtual Keyboard Layout Optimization

In this chapter, we describe our approach to design virtual keyboard layout based on

two objectives: minimization of mouse movement time and minimization of visual search

time. To judge the effectiveness of our optimized design, we design another virtual

keyboard with respect to minimum mouse movement only. The evaluation of these

designs through different user experiments are also discussed in this chapter.

Chapter 6: Summary and Conclusion

In this chapter, we summarize our research findings and major contributions of the

work. This thesis considers some assumptions and we discuss the threats to validity of

our claims subjective to these assumptions. Finally, we highlight extension of our work

and scope of future work.

7

Chapter 2

Related Work

In this chapter, a survey of literature related to the contributions made in this

dissertation is reported. This dissertation aims the modeling of visual search time and

application of the model in virtual keyboard design. Visual search is a perceptual task

of locating intended object visually from the object space. Thus, the search performance

as well as the search time required for the task depend on various features or factors

of the object space. So, first we discuss about various work on visual search feature

identification. Then, we include a survey on different modeling approaches. The state of

art scenarios in application of different mathematical models towards virtual keyboard

layout design are also illustrated in this chapter.

The rest of the chapter is organized as follows. Section 2.1 presents a survey of

state of art approaches in identification of different visual search influencing parameters.

Existing predictive models are discussed in Section 2.2. Section 2.3 describes several

work on model-based approaches to design virtual keyboard. Finally, the chapter is

summarized in Section 2.4.

9

2. Related Work

2.1 Visual Search Features Identification

Visual search task is known to be governed by the various features present in the search

space. Thus, identification of these features is required prior to modeling of the search

task or time. An outline of related literatures in the field of visual search task influencing

feature identification is reported in this section.

Feature Integration Theory (FIT) [93] states that people shift attention serially from

one object to the next, deciding for each whether it is the target or not. According to FIT,

attention must be directed serially to each stimulus in a display whenever conjunctions

of more than one separable feature are needed to distinguish the possible objects

presented. It has been assumed that the visual scene is initially coded along a number of

separable dimensions such as, color, orientation, spatial frequency, brightness, direction

of movement and features are registered early, automatically, and in parallel across the

visual field, while objects are identified separately and only at a later stage, which

requires focused attention. This process is said to be necessary when conjunctions of

object features (color, shape, size, orientation etc.) differentiate targets from distractors,

for example, searching for a red X among green X’s and red O’s (conjunction search).

According to FIT, attention is necessary for the correct perception of conjunctions,

although unattended features are also perceived next to conscious perception as without

any attention. However, these results could also be the result of inefficient parallel search

processes. This type of theories are supported by a variety of evidence presented a simple

parallel model that reproduced the finding of feature search times independent of number

of objects in the search display and conjunction searches times linearly dependent on

number of objects.

Guided Search (GS) [95,97,98] is a computational model of how visual features, such

as color and orientation, direct visual attention. Guided Search predicts that the order

in which objects are visually searched is affected by four scenarios, the “strength” of

objects’ visual features (e.g. their blueness, yellowness, steepness, and shallowness), the

10

2.1. Visual Search Features Identification

differences between objects, the spatial distance between objects, the similarity to the

target, and the distance of objects from the center of gaze (i.e. the eccentricity).

Active vision is the notion or collection of theory that asserts eye movements are

central to visual processes, including visual search [24]. It asserts that understanding

where and when the eyes move, and how information gathered during eye movements

is utilized, are critical for understanding vision and in particular visual search. The

reported literature reveals that no model of visual search provides answers to all of the

questions of active vision. However, every question of active vision is addressed by at

least one model.

According to Nagy et al. [71], several experiments are designed to investigate the

effects of target and distractor heterogeneity on the threshold for detection of a color

target in a search task. In their first two experiments, stimuli are chosen so that the

target and distractor stimuli varied along one Cardinal axis in color space, while the

target differed from distractors along another Cardinal axis. Varying stimuli along a

Cardinal axis other than the Cardinal axis that differentiates target and distractors can

impair performance in visual search tasks. Further experiments show that the presence

of heterogeneous distractors has little or no effect on thresholds when location or color

cues indicated that these stimuli are irrelevant to the task. Results suggest that the

effect of heterogeneity in these experiments is attentional in nature rather than sensory.

Neidar and Zeilensky [73] judge the effects of target-background similarity on visual

search. In their experiments, users search for targets among distractors under varying set

size and target-background similarity (TBS) conditions. Manual errors and reaction time

increased with TBS, although search slopes did not significantly differ. Eye movement

analysis reveal that the majority of fixations fell on discrete distractors rather than on

the target-similar background, even under high TBS conditions. These data suggest a

biased search process, salient patterns segmented from a background are preferred while

more target-similar unsegmented regions of the background are relatively neglected.

11

2. Related Work

Research suggests that neither visual nor verbal working memory have a major

impact on the fundamental visual search processes. Visual and verbal working memories

are limited capacity, temporary stores for visual and verbal information. These two

memories show little overlap in functionality [60]. Research has shown that when verbal

or visual working memory is occupied, visual search remains efficient [99]. When people

visually search for a shape while performing a task that is presumed to occupy visual

working memory, the rate at which people visually searched was unaffected [99]. A

similar result is found when visually searching for a word while verbal working memory

is filled [58, 59]. These findings do not mean that working memory does not affect

visual search tasks at all. In general, for each modality, people can recall four things

on average [62]. If the visual search task requires storing multiple items in memory

before search is terminated, limitations on working memory could require that the user

terminate search early or forget items for later use. However, the use of spatial memory,

especially for where one has previously fixated, does appear to affect visual search.

Research has shown that when spatial memory is occupied, visual search efficiency is

reduced [74]. A memory for previously fixated locations is also suggested by other

research. A study of the visual search in “Where’s Waldo?” scenes, in which a cartoon

figure is hidden within complex scenes, found that saccades tend to be directed away

from the locations of previous fixations [50].

2.2 Modeling Techniques for Prediction

One objective of our work is to develop a computational model to predict visual search

time. To the best of our literature survey, there is no prior work dealing with this

problem. However, there are a large volume of work to develop model related to visual

activities of human which are may not be so relevant to our work but have some

coherence. We also report few techniques proposed to develop predictive models in

different application domains.

12

2.2. Modeling Techniques for Prediction

2.2.1 Modeling of Visual Search Activity

When people use visual search, their eyes are moved and independent shifts of attention

are not used [23,24]. Since different information is available depending on the orientation

of the eyes [6, 24], the movements of the eyes as well as head and body movements

are important for models of visual search behavior. This is especially true due to the

increasing size of computer displays and the increasing ubiquity of computing interfaces.

A variety of models have been developed to predict visual search behavior. Some models

have been developed specifically to predict the performance in a narrow domain, such

as graph perception. Others have been developed to predict the effects of specific visual

features in a broad range of visual search tasks.

The Area Activation Model (AAM) [77] is also a computational model of how visual

features control visual attention. The characteristics of AAM and GS are common in

many ways, but differ in at least one important way. The AAM assumes that all objects

near the center of gaze are searched in parallel and GS assumes that objects are searched

serially.

Barbur, Forsyth, and Wooding [4] propose a computational model to predict eye

movements in visual search. The model uses a hierarchical set of rules to predict where

people’s gaze will be deployed. Like the AAM, Barbur, et al.’s model assumes that all

objects near the center of gaze are searched in parallel. It differs from the GS and AAM

in that eccentricity is the only visual feature that determines where the gaze moves next.

Understanding Cognitive Information Engineering (UCIE) is a computer model of

human reasoning about graphs and tables [61]. UCIE is based on GOMS (Goals,

Operators, Methods and Selection Rules [45], an engineering model for predicting task

execution time. UCIE extends the applicability of GOMS for visually searching task

with a model. The time to perceive objects, eye movements, and a limited memory for

information provide constraints for the simulation of how people scan graphs and tables

to answer questions about the graph or table.

13

2. Related Work

Executive Process-Interactive Control (EPIC) is a framework for building

computational models of tasks that lends itself well to building models of visual

search [49]. EPIC provides a set of perceptual, motor, and cognitive constraints based

on a variety of psychological literature. Models of visual search built within EPIC tend

to explain visual search as the product of cognitive strategies, perceptual constraints,

and motor constraints.

An eye tracking study, conducted to examine the search strategies of users, and

a revised model-based on the results of the eye tracking study have been done by

FleetWood and Bryne [27]. The revised model incorporates EMMA [80] and changes

in search strategy. Findings indicate key environmental influences of icon search,

particularly set size and icon quality, evaluate the vision module in the underlying

cognitive architecture, and provide some illumination on the strategies of users.

Computational cognitive models are computer programs that simulate aspects of

people’s perceptual, motor, and cognitive processes. Cognitive modeling is used in

two ways; in a post hoc fashion to help explain the behavior of people performing

a task and in a priori fashion to predict how people will perform a task. The

most important contribution of computational cognitive models to the field of Human

Computer Interaction (HCI) is that the modeling provides a science base that is

badly needed for predictive interface-analysis tools. Projects such as CogTool [46] and

CORE/X-PRT [91] are at the forefront of the development of tools that utilize cognitive

modeling to predict user interaction based on a description of the interface and task.

These tools provide theoretically grounded predictions of human performance for a range

of tasks without requiring that the analyst be knowledgeable in the cognitive, perceptual,

and motor theories embedded in the tool.

Designers of device and application interfaces may be able to utilize such tools to

evaluate their visual layouts, identify potential usability problems early in the design

cycle, and reduce the need for more human user testing early in the development cycle.

14

2.2. Modeling Techniques for Prediction

Predicting people’s visual interaction is one aspect of user behavior that research with

interface analysis tools is trying to improve. To this end, a recent version of CogTool [90]

incorporates modeling work based on an early summary of the work [35]. However,

interface analysis tools such as CogTool and CORE/X-PRT do not yet fully account

for human vision, as in where the eyes move and what they do and do not see. A

partial account of visual information processing is provided by EMMA [80], which is an

extension to the ACT-R [3] modeling framework underlying CogTool. EMMA provides a

simulation of the eyes including where the eyes move and how quickly visual information

is processed. But this falls short of a complete account of active vision; automated

interface analysis tools do not yet simulate active vision.

Building cognitive models to explain users’ behavior in a post hoc fashion has a rich

history. In explanatory modeling, human data is collected and models are built to explain

the observed behavior. Such explanatory cognitive models have been used to understand

web link navigation behavior [29], driving behavior [81], and time interval estimation [88].

Explanatory models are useful in their own right, to expand our understanding of user

behavior, but are also useful for informing a priori predictive models. For example, the

explanatory modeling of driving behavior [81] has been used to inform the development

of a predictive model of driver behavior while utilizing a cell phone [47].

2.2.2 Support Vector Regression-based Modeling

Support Vector Regression (SVR) has been introduced to solve regression and prediction

problems. SVR is a method of estimating function which establish relationship between

independent variables and dependent variable [94]. Presently, SVR is widely used in

various predictive task like financial forecasting [92], stock market prediction [102],

exchange rate prediction [39] etc.

Cao and Tay [11] deals with the application of SVR in financial time series forecasting.

This work analyzed the feasibility of applying SVR in financial forecasting by comparing

15

2. Related Work

it with the multilayer back-propagation neural network and the regularized radial basis

function (RBF) neural network. They have experimentally investigated variability in

performance of SVR with respect to the free parameters. This paper proposes the

concept of adaptive parameters by incorporating the non-stationarity of financial time

series into SVR. Authors have observed that, SVR with adaptive parameters can achieve

both higher generalization performance and use fewer support vectors than the standard

SVR in financial forecasting.

Yang et al. [102] used SVR to financial prediction problems. The financial data

are usually noisy and the associated risk is time-varying. Therefore, the SVR model

described in the paper, is an extension of the standard SVR which incorporates margins

adaptation. By varying the margins of the SVR, authors reflect the change in volatility

of the financial data. The effect of asymmetrical margins is analyzed to allow for the

reduction of the downside risk. The paper shows that the use of standard deviation to

calculate a variable margin gives a good predictive result.

The work by W. C. Hong [39] presents a hybrid support vector machine (SVM) model

to exploit the unique strength of the linear and nonlinear SVM models in forecasting

exchange rate. The parameters of both the linear and nonlinear SVM models are

determined by Genetic Algorithm. Authors have employed numerical examples from

existing literatures to compare the performance of the proposed model.

Travel time is a fundamental measure in transportation and accurate prediction is

crucial to the development of intelligent transportation systems and advanced traveler

information systems. Wu et al. [101] used SVR for travel time prediction and compared

its results to other baseline travel time prediction methods using real highway traffic

data. Since support vector machines have greater generalization ability and guarantee

global minima for given training data, SVR performed well for time series analysis.

Compared to other baseline predictors, the result showed that the SVR predictor can

significantly reduce both relative mean errors and root mean squared errors of predicted

16

2.3. Model-based Approaches to Design Virtual Keyboard

travel times.

2.3 Model-based Approaches to Design Virtual Keyboard

In digital devices, text can be entered usually by two ways, through QWERTY hardware

keyboard and virtual keyboard. One of our objectives is to design virtual keyboard

taking an estimation of visual search time into account. In this section, we first discuss

about various mathematical models used in keyboard layout design followed by a brief

description of design principle of some virtual keyboards. Also, we summarize existing

optimization approaches to design virtual keyboard, reported in recent literature.

In our review of existing literatures, we found two different performance evaluation

models for soft keyboards, namely KLM/GOMS models and Fitts-Digraph model.

KLM/GOMS Models: Keystroke Level Model (KLM) [12] and its successor Goals,

Operators, Methods and Selection (GOMS) rules [13] are two predictive user modeling

techniques widely used in HCI [44,45]. KLM/GOMS models allow designers to perform

quantitative analysis of system behavior from a description of the system, making it

possible to identify design problems or compare alternate designs. The work described

in [52–54] use KLM analysis to develop models for keyboard evaluation.

A KLM/GOMS analysis consists of the following stages.

• A prototype of the system is conceptualized or designed.

• A task is described using the corresponding modeling language. The description

consists of the steps to be taken to carry out the given task. For KLM analysis,

the description is a linear sequence of basic steps known as primitives. A hierarchy

of goals is used to describe the task in GOMS. Six primitives along with their

empirically estimated values are used in KLM/GOMS analysis.

• A quantitative analysis of the task description is performed using values of the

primitives to evaluate keyboard layouts.

17

2. Related Work

In typing through virtual keyboard, a task consists of inputting a string of characters.

However, performance of a user may vary depending on the nature such as number,

frequency of the characters present in the task. Since it is not possible to evaluate

user performance for all possible combinations, there is always a possibility of getting

wrong performance prediction because of task dependent approaches in KLM or GOMS.

Also in KLM/GOMS analysis techniques, it is necessary to construct task descriptions

for each design to compare among a set of alternate designs. However, construction of

task description is a tedious and time consuming process, as it needs to be performed

manually. For a large design space like keyboard layout design, it is impossible

to construct such task descriptions for all alternate designs. On the other hand,

performance models, those are easy to automate, will be more useful to evaluate and

compare among alternate designs for a large design space.

Fitts-Digraph Model: To address the limitations of KLM/GOMS based approach, two

qualities namely task independence and easy to automate are desirable in the models.

Another model having these characteristics is reported in the literature. The model

developed by Soukoreff and MacKenzie [86] is known as Fitts-Digraph model [64,67,105].

The Fitts-Digraph (FD) model [86] is developed primarily to evaluate soft keyboards used

on small sized mobile devices. In the model, it is assumed that a user selects character

keys from the keyboard interface with single finger or stylus movement. Following

components have been considered to develop the model.

• Visual search time: To select a character key from an interface, a user first needs to

locate the key on the interface. The time to locate a key is called the reaction time

or visual search time and is denoted by RT . In the FD model, RT is calculated

using the Hick-Hyman law [37, 42], as shown in Eqn. 2.1. In this equation, a and

b are empirically decided constants and N is the total number of character keys

18

2.3. Model-based Approaches to Design Virtual Keyboard

present on the interface.

RT = a + b × log2N (2.1)

• Movement time: In addition to locating a character key, the user also requires

finger or stylus movement to select the key. The movement time, that is, time to

make a finger or manual movement from one key to another, is calculated using the

Fitts’ law [26,63], shown in Eqn. 2.2. In the equation, MTij is the movement time

from the ith character to jth character, a′ and b′ are empirically decided constants,

dij is the Euclidean distance between the ith and jth keys and wj is the width of

the jth key.

MTij = a′ + b′ × log2(dij/wj + 1) (2.2)

• Digraph probability: In the FD model, the probability of occurrence of character

pairs or digraphs are considered. The digraph probability is calculated from a

text corpus by Eqn. 2.3, where fij is the digraph frequency and Pij is the digraph

probability of characters i and j. K is the total number of characters present in

the corpus.

Pij = fij/K∑

i=1

K∑j=1

fij (2.3)

Using Pij and MTij , mean movement time (MT ) for a given layout is calculated with

Eqn. 2.4, where N is the number of keys present on the interface.

MT =N∑

i=1

N∑j=1

(MTij × Pij) (2.4)

Using Eqn. 2.1 and Eqn. 2.4, text entry rate of a soft keyboard is calculated in characters

per second (CPS) as shown in Eqn. 2.5.

Text Entry Rate (CPS) = 1RT + MT

(2.5)

19

2. Related Work

Other than the models discussed above, researchers have followed several alternate

approaches to design various keyboard layouts. A brief description of those approaches

are discussed in the following.

Getschow et al. [31] introduced one algorithmic approach to virtual keyboard design.

In this article they have discussed about a simple assignment procedure called greedy

algorithm that place alphabets in the most easily accessible positions according to

each character frequency rank order. However, the greedy algorithm ignores many

arrangements that could be substantially better because it does not consider the letter

placement with respect to each other.

To improve the keyboard layout for single-finger typing Lewis et al. [56] developed

another layout. In this paper, they have introduced a network model of English character

to determine the layout design. Here, first a symmetrical matrix of the relative frequency

of unordered English-language digram is created and then a minimally connected network

generated by analyzing the matrix through Pathfinder network-definition program. The

target of this approach is to minimize movement time by reducing the distance between

strongly associated letters.

MacKenzie and Zhang [66] developed a model of the psychomotor act of key tapping

to predict user performance in text typing through virtual keyboards. This model has

several components such as, linguistic data, Fitts’ law, a shortest-path model, and a

key-repeat time measure. The model generates a theoretical text entry rate (in words

per minute) for any layout of virtual keyboard. It thus allowed to evaluate alternate

designs on paper before proceeding to an empirical evaluation. Following this model,

they have proposed one optimized virtual keyboard, Opti (Fig. 2.1(a)), where the key

arrangement of the keyboard is finalized by trial-and-error method.

Zhai et al. [106] minimized the energy, or tension, of a keyboard layout through a

well-known optimization method-the Metropolis algorithm, shown in Fig. 2.1(b). The

Metropolis algorithm is a Monte-Carlo method widely used in searching for the minimum

20

2.3. Model-based Approaches to Design Virtual Keyboard

energy state in statistical physics [69]. Considering the value of Fitts-Digraph model as

Fitts-Digraph energy, the problem of designing a high performance keyboard is equivalent

to searching for the structure of a molecule (the keyboard) at a stable low energy state

determined by the interactions among all the atoms (keys). Applying this approach,

they designed and implemented a software system that did a random walk in the virtual

keyboard design space. In each step of the walk, the algorithm picked a key and moved

it in a random direction by a random amount to reach a new configuration. The level of

Fitts’ energy in the new configuration is then evaluated. The Metropolis function decides

whether the new configuration is kept as the starting position for the next iteration or

not.

(a) Opti keyboard layout [66]

B K D G

C A N I M Q

F L E Space S Y X

J H T O P V

R U W Z

(b) Metropolis keyboard layout [106]

(c) GAG I keyboard layout [79] (d) GAG II keyboard layout [79]

Figure 2.1: Some well known virtual keyboard layouts

21

2. Related Work

Raynal et al. [79] use the concept of genetic algorithm [32] to solve the virtual

keyboard optimization problem in English language. They consider the average mouse

movement time as fitness function and try to maximize text entry speed only. They

took two existing layouts Metropolis and Opti, as inputs for the algorithm and obtain

the best solution or key arrangement are called GAG I and GAG II keyboard, as shown

in Fig. 2.1(c) and Fig. 2.1(d) correspondingly.

Majority of the works on virtual keyboard design have attempted to maximize text

entry rate by arranging the key positions. However, there exists few approaches to

design virtual keyboard considering optimization of multiple objectives, discussed in the

following.

Eggers et al. [21] design the keyboard by optimizing six ergonomic criteria, tapping

load distribution, number of keystrokes, hand alternation, finger alternation, finger

posture, and hit direction using Ant Colony optimization algorithm [17]. Deshwal et

al. [16] use Eggers criteria as the performance indexes and developed an optimal Hindi

keyboard using Genetic Algorithm. But those ergonomic criteria are not applicable to

virtual keyboard design.

Yin et al. [104] optimize ergonomic criteria and disambiguation/prediction

effectiveness simultaneously using Cyber Swam method [103] for keyboard designing.

Their keyboard also outperform the QWERTY and Dvorak [65] keyboard. Quasi-

Qwerty [7] keyboard layout, shown in Fig. 2.2, is designed for optimizing both average

motor movement time to enhance text entry rate and designing layout close to QWERTY

to reduce initial visual search time. The keyboard achieves 12% faster text entry rate

than QWERTY keyboard. Though Quasi-Qwerty is slower than Atomik [65], it requires

less initial text entry time.

Dunlop et al. [20] use multiobjective optimization to design keyboard layout for

touchscreen phone based on three design metrics: minimizing finger travel distance,

a new metric to maximize the quality of spell correction by reducing tap ambiguity

22

2.4. Summary

Figure 2.2: Quasi-Qwerty keyboard layout [7]

and maximizing familiarity through similarity function with the standard QWERTY

layout. Using the methodology, they design two keyboards namely Sath-Rectangular

(Fig. 2.3(a)) and Sath-Trapezoidal (Fig. 2.3(b)).

(a) Sath-Rectangular keyboard layout (b) Sath-Trapezoidal keyboard layout

Figure 2.3: Sath keyboard layouts [20]

2.4 Summary

In this chapter, we have discussed various approaches related with identifying different

parameters which influence visual search task of an interface. Proper identification of

those parameters helps designers to further find out the degree of influence of those

parameters on individual interface. These approaches also support designers to come

up with effective visual search task easing interaction techniques. Our survey reveals

that the visual search parameters do not affect every interface in the same way but vary

according to degree of influence. As an example, search strategy to find a key in virtual

keyboard not always be same but varies from one instance to other randomly. So, in case

23

2. Related Work

of virtual keyboard, fixed search strategy cannot be taken as a visual search influencing

parameters.

Many ways to quantify visual search task till date based on design parameters have

been reported. However, our literature survey does not find any work on modeling visual

search time based on virtual keyboard influencing design parameters. Moreover, there is

also an important need to identify the impact of each parameter on visual search time.

There exists significant work on model-based approaches in designing virtual

keyboards. We observe that Fitts’-Digraph model has been widely used for designing

as well as optimizing virtual keyboard layout. Two components are useful in developing

this model namely visual search time to find a key on a keyboard and mouse movement

time to move the pointer onto desired location. The model also needs unigram and

bigram frequencies extracted from corpus in corresponding language. Apart from these

strong mathematical models, researchers have followed several optimization approaches

to design various keyboard layouts. Single objective optimizations in context of virtual

keyboards usually produce solutions maintaining mouse movement minimized layouts.

Moreover, we also analyze several multiobjective virtual keyboard design principles

for English keyboard, which optimize text entry rate along with tap ambiguity etc.

However, none of the approaches have considered visual search time as an objective

to design virtual keyboard. In the next chapters, all these limitations in the existing

works have been addressed to develop visual search time prediction model and design a

multiobjective virtual keyboard to increase text entry rate and decrease time to search

a key. In order to develop the model, it is necessary to identify the virtual keyboard

design parameters which have significant influence on visual search. Identification of

visual search time influencing virtual keyboard design parameters is discussed in the

next chapter.

24

Chapter 3

Identification and Analysis of

Visual Search Features

Towards the automatic evaluation of visual search time of a user interface, we have

studied visual features reported in different literatures which influence visual search task.

All visual features not necessarily contribute equally in visual perception. Some features

may not be applicable for a particular interface as well as the effect of few features varies

from a user to another user. So, to model visual search task, a categorization of visual

features is required. We aim the categorization with respect to applicability on virtual

keyboard interface which is one of the popular text entry mechanisms. Further, the

features which may be applicable in virtual keyboard interface design may not influence

visual perception equally and also may depend on user. In order to judge the user

dependency and influence, several experiments with users have been carried out.

This chapter consists of five sections. Section 3.1 lists and describes various visual

search features reported in different literatures. Experiments and experimental details

to identify the significance of each features is presented in Section 3.2. The results of

these experiments and corresponding conclusion are discussed in Section 3.3 and 3.4,

respectively. Finally, Section 3.5 summarizes this chapter.

25

3. Identification and Analysis of Visual Search Features

3.1 Listing of Visual Features

Visual search activity is known to be governed by many visual search features. The

following visual search features, which user interface designers usually refer, are listed

below with a brief description of each.

• Size: The task of finding an object among distractors differs with the size of the

same [8, 14]. For example, if an interface contains objects of different sizes, then

visual search time is not necessarily same for all objects.

• Shape: User interface designers believe that shape of an object is a factor in visual

perception [8,14]. When a user searches for an object, it is usually compared with

objects in the search space [4, 76]. In other words, the target objects need to be

compared with a set of similar objects, where shape of the objects controls the

time of matching.

• Spacing: Placing of objects in a design space affects the visual search activity [22].

It is true that if objects are placed sparsely then it demands a different visual search

time than when the objects are placed densely.

• Orientation: Orientation of an object influences the visual search task and as a

result, visual search time varies with objects’ orientation [14]. Like, time to find a

straight line among straight lines with similar orientation is not same compared to

the time to locate the same straight line from a set of straight lines with different

orientation.

• Total number of items: Usually objects of an interface visible at a time are

compared randomly against the target till the target is located [4, 76]. Thus the

total number of items present in a visual search space controls the finding of an

object in the search space [37,41,42].

26

3.1. Listing of Visual Features

• Number of distractors: A distractor is an object which is not the target at

an instance. So, in a visual search space all different items except the target are

considered as distractors [8, 18,36].

• Types of distractors: The heterogeneity among distractors like variation by size,

shape, orientation etc. contributes moderately in visual search task [8, 19, 22, 27,

71,73,76,97].

• Size of a search field: User interface designers observe that size of a search field

is also a factor in visual perception [8, 14, 76]. If an interface have a significant

number of items, distributed evenly and visible at an instance then searching for

an item usually become random [10,36,96]. Although, search time does not solely

depend on size of the search space, but also varies in parallel with density of the

object within the unit space of the interface [68].

• Place of a target: Placing of target in a proper position within an interface

often helps user to find it in a short time [96]. Usually, interface designers and

psychologists observe that user’s focus is more concentrated for some portion or

zone of the interface [49,61,77]. So, finding any objects residing at or around those

areas often becomes easy.

• Ordering: It has been observed that ordering of objects affect visual search

task [2, 10]. Like, the time required to find an item in a ordered list differs from

the time required for an unordered list.

• Grouping: In any interface, the object can be found more quickly if it is placed

apart from the crowd [41]. Also, it is comparatively easy to find an individual

object from a group containing specific feature based objects rather than from the

total interface consisting of several constraints.

27

3. Identification and Analysis of Visual Search Features

• Size of a group: Size of a group indicates the number of items or objects in a

particular group. Visual search time to locate an object depends on the size of a

group [37,42].

• Labeling of a group: Labeling of groups within an interface also helps user to

identify them more specifically [2]. As an example, it would be easier to access the

virtual keyboard if labeling can be done in each group like vowel group, consonant

group, numeric group etc.

• Color of a text and background: Color property of an object may influence

visual search time of the object [73]. If a target is easily discriminable from its

distractor due to color difference then the search time required to find the target

visually become different [8, 14].

• Contrast of a background: Visual search time is also affected by background

contrast. Greater background contrast helps more to identify the target, that is,

visual search time becomes lesser [73]. For an example, effort required to read

a black colored text with white background and yellow colored text with white

background is not equal [71–73].

• Contrast of distractors: When there is a notable difference in contrast between

target and distractors then the time to identify the target differs [68, 71, 73]. For

example, from a set of similar shaped objects, a red colored object (target) can be

easily identified when other objects (distractor) are green colored.

• Ageing and memory: Older adults face more difficulty than younger adults

to identify and locate a target defined by a conjunction of features among

heterogeneous distractors [55]. It has been observed that with practice, people

can learn to find a target from randomized object set almost as quickly as from

ordered object set [10,41].

28

3.2. Experiments

• Search strategy: Visual search time directly depends on the search strategy

followed while searching, like top-down searching, bottom-up searching, item-by-

item searching, randomized searching etc. [8, 14].

3.2 Experiments

Since all visual features does not contribute equally in visual perception [93, 97], a

categorization of visual features is required. We aim the categorization in the context of

virtual keyboard interface. The preliminary analysis of those features reveals that some

features are not applicable in the context of virtual keyboard.

Orientation: In a virtual keyboard, orientation of all characters or keys present in the

keyboard layout is similar [65]. So, we should not consider this feature while modeling

visual search task in the context of virtual keyboard.

Search strategy: As keyboard contains a large number of keys as well as users get

acquainted after accessing the layout for some time, so it is not required for the user

to search all keys within the interface following any search strategy. The searching is

accomplished in random fashion usually [95,98].

Distractor type: Although, some keys like Backspace, Enter, Shift etc. are larger

in size than other keys in keyboard interface, they do not affect on visual search task

significantly. After being familiarize with the keyboard interface, users can find those

buttons quickly not because of their different size and shape than distractor, but from

their natural tendency (e.g. after certain time, user will guess the buttons correctly from

their fixed position) [95,98].

Varying spacing: We have observed that, in a virtual keyboard interface both the

horizontal distance and the vertical distance between keys remain same throughout the

keyboard layout. The variation of horizontal distance and vertical distance do not

consider as a good virtual keyboard design methodology. If varies, the design lacks

user comfort [65].

29

3. Identification and Analysis of Visual Search Features

Color : While accessing the graphical user interface, color feature retains impression

in user’s mind. In contrast, the feature behaves differently from user to user. So, the

color choice of background of virtual keyboard, button color, font color etc. may attract

one user but may distract another [73]. So, it is unlikely that a particular combination

of colors in the keyboard may satisfy all the users.

Further, the features which may be applicable in virtual keyboard interface design

may not influence visual perception equally and also may depend on user. In order to

identify the user dependency and influence, several experiments with users have been

carried out. In the following, we discuss about experimental setup, interfaces used to

perform the experiments, details of participants considered in experiment, experimental

procedure and outcome of the experiments.

3.2.1 Experimental Setup

All experiments have been conducted using 2.4GHz Pentium Core2Duo machine with

a 17" color monitor with 1280 × 1024 resolution. The developed interfaces for these

experiments are written in C# using Visual Studio 2008. All key press events are

recorded automatically and stored in a log file. The mouse positions are also stored in

separate log file using a separate window hook program written in C#. All experiments

are done in Windows 7 environment.

3.2.2 Interface Used

Three virtual keyboard interfaces have been used namely Opti, a virtual keyboard layout

for English language [66], Avro, a virtual keyboard layout for Bengali language developed

by OmricornLab [75] and iLipi-H, a virtual keyboard layout for composition of text in

Hindi proposed by [82]. Figure 3.1 shows the layout of each interface.

Opti (a frequency-based keyboard layout): It is one of the optimized virtual keyboard

layouts for the English language. Figure 3.1(a) shows the Opti layout as described by

30

3.2. Experiments

(a) Opti keyboard layout

50

61

72

83

94

a i u e o

k j

t h

Tab ---> <--- Backspace

ZWJ ZWNJ

S P A C E Enter

(b) Avro keyboard layout

(c) iLipi-H keyboard layout with inflexion window

Figure 3.1: Virtual keyboard layouts used in experiments

MacKenzie and Zhang. The keyboard layout was optimized for speed using trial and

error, Fitts’ law, and bi-gram frequencies of characters.

Avro (an alphabetical keyboard layout): This keyboard uses the alphabetical

arrangement of Bengali alphabets. The consonants are divided into two sub parts, as

shown in Fig. 3.1(b). All vowels are present in a row at the bottom of the layout.

31

3. Identification and Analysis of Visual Search Features

iLiPi-H (a multizonal, frequency and inflexion window based keyboard layout): In this

keyboard layout, Hindi characters are spatially arranged in layered zones depending on

their frequencies of occurrences (Fig. 3.1(c)). The high frequency characters are placed

in a central zone, the next higher frequency characters are placed in its outer zone

surrounded by the central zone and so on. In addition to this, inflexions are dynamically

appeared through an inflexion window for each consonant.

3.2.3 Participants

To evaluate the design with participants, we include 21 participants in our experiments.

Out of them, 12, 8 and 15 participants have tested the Bengali, Hindi and English

language keyboard interfaces, respectively. If a participant is familiar with multiple

language then the participant may have considered for multiple interface. Participants

are chosen from different educational backgrounds and with various level of computer

proficiency. The participants are having English and native language familiarity in terms

of reading and writing. However, some businessmen and housewives do not have any

exposure in English. There are 9 female and 12 male participants belong to different

age groups with average age is 27.52 years (SD = 3.61). The detail of the participants

is shown in Table 3.1.

Table 3.1: Description of participants

Participant’s profileNumber ofparticipantsOccupation Age

group Education Computerproficiency

Student 13–27

Post Graduate Level 1 5

Under Graduate Level 1 6

Higher Secondary Level 2 2

Office staff member 31–48Graduate Level 1 2

Higher Secondary Level 3 2

Business person 34–53Graduate Level 2 2

Higher Secondary Level 3 2

32

3.2. Experiments

We decide computer proficiency on the basis of expertise in four different tasks.

• Total length of time a participant has worked with computer.

• Number of application softwares known by participant.

• Length of daily work-time with computer.

• Number of messages typed in computer in their mother languages.

In view of the above, we categorize the participants at three different levels.

• Level 1: Having more than 3 years of experience with computer, know operating

systems, programming and application software.

• Level 2: Having 1−3 years experience and mainly familiar with application software

such as document processing, email and Internet browsing.

• Level 3: Having less than 1 year experience and poor in computer related task.

3.2.4 Experimental Procedure

In our experiments, four to seven different experimental sessions are carried out by each

participant. Each session includes several trials having two parts: stimulus presentation

(target) and layout presentation (interface). The goal of each trial is to locate the target

within the interface. An experimental trial begins by presenting a target followed by

a virtual keyboard interface till the goal is achieved. A screenshot of the layout with

target is shown in Fig. 3.2. We have used three different fonts namely Times New Roman,

Vrinda and Mangal to display English, Bengali and Hindi characters respectively, and

the font remain same throughout all experiments.

The target is displayed until the participant either hit the Spacebar or clicks the

mouse, indicating he/she is ready to proceed. Once the participant is ready, a virtual

keyboard interface is displayed. Participants are instructed to either press the Spacebar

33

3. Identification and Analysis of Visual Search Features

Figure 3.2: A sample layout with target object

or click the mouse again as soon as they locate the target within the interface. After

that, the participant is required to move the mouse to the identified target and click the

same. A trial is considered to be finished when these steps are completed. The visual

search time of each trial has noted programmatically and stored into log file. Next, a

fresh trial is started in the same way with a different target. A session continues till all

characters are considered in trials or the participant does not willing to continue further.

The cursor position is programmatically controlled and locked on the initial screen

that is where the target is displayed first. Cursor is programmatically kept hidden on

the interface until a participant finds the target within the interface, so he/she is not

getting distracted by the cursor position. For remaining of a trial, the mouse position

has not been controlled. It has been reported that a participant may get acquainted

with the interface while accessing over a long time [96]. This result lesser visual search

time compared to the situation where the participant is not familiar with the interface.

The difference becomes higher over the time. Although this effect depends solely on user

and affects intelligent users mostly, still we have tried to reduce this. In our experiments,

the chance of selecting any of the three interfaces for a particular trial is equal.

By analyzing the log files, we have calculated the center value for each variation of

a feature and plotted them into graph. Note that both median and mean can be used

to calculate the center value, but median reduces the average of the absolute deviations

34

3.3. Experimental Results

whereas mean makes it biased towards the extreme value. Hence, we have used median to

calculate the center value. We have also conducted analysis of variance1, that is, ANOVA

test to study the significance of each feature on visual search time. The ANOVA tests are

conducted using Statistical Package for the Social Sciences, also called SPSS tool [43].

3.3 Experimental Results

In this section, we discuss about the outcome of the experiments for each visual search

features.

Size: Size is a primitive property of all kind of objects. So, in a virtual keyboard

interface size may refers to size of the entire interface, size of a button or size of the text

appears on button, that is, font size of the text. Here, we have focused our experiments

on font size and kept other sizes or features unchanged throughout these experiments.

A set of experiments to observe the influence of size on visual search time has been

performed. For each experiment, a keyboard layout has been chosen, different font sizes

have been applied on that keyboard keeping all other features unaltered. The font size

is varied from 6pt to 16pt with an increment of 2pt.

From our experiments, we have observed that characters are almost unreadable for

font size less than 5pt, so 6pt is considered as minimum font size. Figure 3.3(a) shows

instance of Avro keyboard layout with 12pt font size. The analysis of experimental results

is shown graphically in Fig. 3.3(b). We have also calculated median of search time for

different font sizes and keyboards and observed that visual search time is higher at lower

font size. ANOVA test reveals that there is a significant difference between performance

of different font sizes (F5,217 = 18.12, p < 0.05).

Shape: Shape of an object is defined by the geometric properties of the object. In

context of virtual keyboard interface, the shape may refer to the virtual keyboard itself,

the buttons of the interface or the characters included in the interface. We have limited1Analysis of variance, http://en.wikipedia.org/wiki/Analysis_of_variance

35

3. Identification and Analysis of Visual Search Features

(a) Avro keyboard with 12pt font size

(b) Visual search time for different font size

Figure 3.3: Effects of different font sizes on visual search time

our experiments only to shape characteristics of the characters. To achieve this target,

first we have identified and grouped the similar character of a language. Here, we have

considered Bengali language for our experiments.

36

3.3. Experimental Results

Table 3.2 shows the similar character set of Bengali language. Later, when a user is

asked to find a character which belongs to a group of that set, one or more characters

from that group may present in the interface. We have analyzed the experimental results

and depicted the same in Fig. 3.4. From the analysis, it has been substantiated that the

presence of similar shaped characters in virtual keyboard interface affects visual search

time, although the effect is very less. The increment in visual search time due to presence

of four similar shaped characters over single character is 4.21%. ANOVA test indicates

that there is no significant difference among the performance of similar shape characters

on visual search time (F3,94 = 2.301, p > 0.05).

Table 3.2: Similar character set of Bengali language

Similar characters Base character

Group 1 A, Aa, t tGroup 2 U, Ŕ, D, Ĺ DGroup 3 E, Ţ, Č EGroup 4 O, Ů OGroup 5 k, b, r, C bGroup 6 T, Z, Ľ ZGroup 7 J, y, P J

Spacing: Spacing between the keys is also considered as an important visual feature.

In the context of a virtual keyboard interface, spacing between the keys refers to the

horizontal or vertical gap between the key buttons. The experiments are performed

on varying space between keys keeping other features constant. Also, for a particular

instance, we maintain equal distance between all keys. A set of experiments to observe

the influence of spacing on visual search time has been performed. In these experiments,

the spacing between the keys is varied from no spacing to 100% of the button width with

a step of 25%. Figure 3.5(a) shows Opti keyboard with 50% of button width as spacing

between keys. It has been observed that spacing between the keys affects visual search

time.

37

3. Identification and Analysis of Visual Search Features

Figure 3.4: Effects of similar shaped characters on visual search time

From the experiments, we have observed that visual search time is lesser when the

distance between two keys is around 25% of the key size. Figure 3.5(b) depicts results

collected from the experiments which are conducted on different conditions on three

keyboard interfaces. From the analysis of ANOVA, we have concluded that the mean of

performance of users for different spacing are significantly different (F4,205 = 15.643, p <

0.05).

Number of items: Many items present in the interface may distract the concentration

of user while searching for specific item and as a result, search time increases. To

study the effect of number of items present in an interface, the virtual keyboard

layouts are required to be modified to have different number of characters. In these

experiments, we have considered 7 different variations among number of items, that is,

6, 10, 15, 25, 40, 56, 67. An instance of iLiPi-H keyboard with 67 characters is shown in

Fig. 3.6(a).

It has been observed that number of items within the interface influences visual search

time. It is also observed that some users search the item within interface jumbled up

38

3.3. Experimental Results

(a) Opti keyboard with 50% spacing between keys

(b) Visual search time for different spacing between keys

Figure 3.5: Effects of spacing between keys on visual search time

with several objects efficiently but others may not perform the same. As a consequence,

the result may vary from user to user for unchanged features except number of items. We

have observed that usually visual search time increased with the increasing number of

items present in the interfaces. Experimental results in different scenario are plotted in a

graph which is shown in Fig. 3.6(b). The ANOVA test reveals that there is a significant

difference between the mean of visual search time as determined by different number of

39

3. Identification and Analysis of Visual Search Features

(a) iLiPi-H keyboard having 67 items

(b) Visual search time for different number of items

Figure 3.6: Effects of varying number of items on visual search time

items on keyboard (F6,266 = 41.673, p < 0.05).

Number of distractor: In any keyboard interface, number of distractors is almost

equivalent with number of items present in it. So, no other experiment has been

performed, as results would have been similar.

Search field size: The size of the area where user is intended to search the item may

affect the visual search time. To measure the effectiveness of search field size on visual

search task, we conduct experiments. In these experiments, search field size, that is,

40

3.3. Experimental Results

interface size has been varied from 10% to 50% of the screen area with an increment of

10%. Avro keyboard occupying 20% of screen is shown in Fig. 3.7(a).

(a) Avro keyboard occupying 20% of screen area

(b) Visual search time for different search field size

Figure 3.7: Effects of search field size on visual search time

The result signifies that, if all features remain unchanged except search field size,

visual search time increases while search field size is less than 20% or greater than 40%

41

3. Identification and Analysis of Visual Search Features

of the screen area. Figure 3.7(b) depicts the results collected from the experiments

which are conducted with different screen sizes on three keyboard interfaces. We have

performed ANOVA test on visual search time for different search field size. From the

analysis we have concluded that the mean of visual search time for different search field

size are not significantly different (F4,87 = 2.352, p > 0.05).

Position: Positioning of objects in the interface also influences visual search time.

Similarly, in a virtual keyboard interface, different key positions also affect the visual

search time. There are 30, 61 and 66 different key positions possible for Opti, Avro,

iLiPi-H keyboard interfaces, respectively. As it is not possible to observe the alteration

of visual search time for each position and accommodate all the results pictorially, we

divide the keyboard layout into 9 different blocks as shown in Fig. 3.8(a).

We have calculated visual search time for each block and plotted in a graph, shown in

Fig. 3.8(b). From the graph, we can observe that visual search time varies for different

position of characters. A reported measure analysis of variance reveals that, there is

a significant difference between mean of visual search time for nine different blocks

(F8,314 = 11.29, p < 0.05).

Ordering: Ordering of objects within interface is also considered to be an influential

factor in determining visual search time. We have considered alphabetical, frequency-

based and random ordering of keys in our experiments. The experimental result

establishes the fact that random ordering of the objects results in more searching time

than other orientations. We have noticed that different kinds of ordering influence users

much in finding keys from the interface. If all features are same, then it has been found

that alphabetic arrangement helps a user more in finding a key from the keyboard.

The other frequency-based arrangement takes less time than highest-valued random

arrangement. The observed effect of these different arrangements of keys on visual search

time is shown in Fig. 3.9. The ANOVA test on experimental results reveals that there

is a significant difference between the mean of visual search time for different ordering

42

3.3. Experimental Results

(a) iLiPi-H keyboard divided into 9 blocks

(b) Result for different position of keys

Figure 3.8: Effects of different position of keys on visual search time

of keys (F2,24 = 28.59, p < 0.05).

Grouping and group size: An object can be found out quickly in an interface if it

belongs within a particular group of objects and group size is not large. To study the

effect of grouping and group size on visual search time, we have modified keyboard

layouts maintaining similar type of characters, that is, consonant, vowel, numeral etc.

in a group. As an effect, the layout contains a maximum of 7 groups with varying group

size. Figure 3.10 shows an Opti keyboard layout modified to organize characters into

two groups with different group size.

43

3. Identification and Analysis of Visual Search Features

Figure 3.9: Effects of ordering of characters on visual search time

Figure 3.10: Opti keyboard with two group

The experimental result establishes the fact that a moderate number of groups, each

having a minimum number of objects, facilitate more in obtaining lesser visual search

time. The results are graphically depicted in Fig. 3.11(a) and Fig. 3.11(b). ANOVA

analysis reveals that, the mean of visual search time as determined for different grouping

and group size are not equal. The observed value of ANOVA for different group is

44

3.4. Observation

F6,30 = 12.49, p < 0.05 and grouping is F6,79 = 27.426, p < 0.05.

(a) Visual search time for different number of grouping

(b) Visual search time for varying group size

Figure 3.11: Effects of grouping and group size on visual search time

3.4 Observation

We have performed experiments on varying sizes of object. From the result, we can

observe the tendency of visual search time growth with respect to different text size

45

3. Identification and Analysis of Visual Search Features

and fixed fonts in three different languages. The outcome shows that for all keyboards,

visual search time is pretty high for small sized fonts (like 6pt). The visual search time

decreases with increasing font size up to a moderate value (12pt). But when the size

of object is high (14pt or more), the curve corresponding to each language keyboard

grows up. It means that, users get acquainted with text size belonging to certain range.

Beyond that, the visual search task for finding keys becomes time expensive for human.

From the effect of space between object in a virtual keyboard, it can be observed

that visual search time varies significantly with variation of space between object. Visual

search time is less when space between objects around 25% of the object size. The number

of items in a virtual keyboard also contributes in visual search time and corresponding

effect is also significant. We have observed that visual search time gets higher and

increases almost linearly when number of items in virtual keyboard raises. On the other

hand, proportion of screen area occupied by virtual keyboard does not affect significantly

in visual search time. Visual search time is almost constant irrespective of proportion of

screen area occupied. The statistical analysis of our observed data for different features

are summarized in Table 3.3. Here DFn, DFd and F value represent degree of freedom

between groups, degree of freedom within groups and ratio of mean square error between

groups and mean square error within groups, respectively.

Table 3.3 establishes the fact that there are 6 features, which contribute more in visual

search time in the context of virtual keyboard. The features are:

• Size of elements

• Space between elements

• Number of elements

• Position of elements

Thus, we would limit our investigation of developing a model which will compute average

visual search time of a virtual keyboard interface in terms of identified features only.

46

3.5. Summary

Table 3.3: Summary of statistical analysis for different features

Feature Name DFn DFd F value Significant(p ≤ 0.05)

Size of elements 5 217 18.12 Yes

Shape of elements 3 94 2.301 No

Space between elements 4 205 15.643 Yes

Number of elements 6 266 41.673 Yes

Search field size 4 87 2.352 No

Position of elements 8 314 11.29 Yes

Ordering of elements 2 24 28.59 No

Grouping of elements 6 30 12.49 No

Group size 6 79 27.426 No

3.5 Summary

This chapter presents the effect of different visual search features on visual search

time. The visual search features related to virtual keyboard design parameters, which

significantly influence visual search task are identified. To accomplish the task, first, we

have listed the features reported in various literatures which influence visual search time.

Next, we have performed several experiments of virtual keyboard interfaces with users of

different expertise level. From the statistical analysis of results of these experiments, we

have identified four features namely size, space, number and position of characters, which

significantly influence visual search task while composing text through the interface. The

identified features influence many other cognitive task performed in the context of virtual

keyboard. Further, these features would be helpful while developing a computational

model of visual search time for virtual keyboard, which is presented in the next chapter.

47

Chapter 4

Modeling of Visual Search Time

A computational model of visual search time estimates the average search time of a

keyboard layout from various keyboard design parameters. Thus, the model is helpful in

evaluation and automatic design keyboard layouts. Earlier, we have discussed about the

identification of virtual keyboard design parameters which make significant influence on

visual search in finding a key. In this chapter, we develop a predictive model of visual

search time based on those identified parameters. As user is an important concern

in assessing the interface, we perform several user-based evaluations to accumulate

data of user performances with variation of different features. These collected data

are then used to generate the model. The experimental setup, interfaces used and

experimental procedure we have followed are already discussed in Chapter 3. Then,

using the experimental data, we have built three predictive models based on three

different regression approaches namely Linear regression, Non-linear regression and

Support Vector Regression (SVR). We have analyzed the fitness of each modeling

approach against the gathered data. Analysis substantiates that SVR based approach

predicts with higher accuracy than other two approaches. In order to validate this

modeling approach, we have also conducted another set of user evaluation and judged

the performance by comparing model predicted value with user data.

This chapter consists of four sections. Section 4.1 details about experiments to

49

4. Modeling of Visual Search Time

collect knowledge about user performance on variation of different visual search features.

Different modeling approaches exercised to develop a visual search time predictive model

are discussed in Section 4.2. Then, Section 4.3 describes validation of our proposed

model. Finally, Section 4.4 summarizes this chapter.

4.1 Training Data Set Generation

In Chapter 3, we have identified four visual search features which significantly influence

the visual search task of virtual keyboard. We now vary only these features and

accumulate data involving a large users’ pool. These data constitute training data

set for the visual search time prediction model we have planned to develop. Our

approach to gather training data set is discussed in this section. We have followed

same experimental setup, interfaces and experimental procedure for these experiments,

as discussed previously in Chapter 3. However, alongwith the previously mentioned

participants we have included another set of participants for our experiments. Details

of the participants are given in Table 4.1.

Table 4.1: Description of participants

Participant’s profileNumber ofparticipantsOccupation Age

group Education Computerproficiency

Student 13–27

Post Graduate Level 1 5

Under Graduate Level 1 8

Higher Secondary Level 2 3

Secondary Level 2 2

Office staff member 31–48Graduate Level 1 4

Higher Secondary Level 3 3

Business person 34–53Graduate Level 2 4

Higher Secondary Level 3 3

Aged person 55–62Graduate Level 2 2

Under Graduate Level 3 3

Housewife 35–45Graduate Level 1 3

Secondary Level 3 2

50

4.2. Model to Predict Visual Search Time

All experimental data are stored programmatically into log files. Analyzing log

files, we have removed the erroneous data, that is, when the presented target and the

participant’s identified target are different. From the analysis we have also observed

that, sometimes participants get distracted due to environmental noise and take large

amount of time to locate the target. Thus data with visual search time greater than

µ + 2 × σ are filtered out from modeling. Here, µ and σ represent mean and standard

deviation, respectively. Summary of the collected data is tabulated in Table 4.2. Finally,

28734 data have been considered for the modeling task.

Table 4.2: Summary of Collected Data

Total collected data 30882

Mean (µ) 6.34

Standard deviation (σ) 3.81

Outlier data 2148

Data for modeling 28734

4.2 Model to Predict Visual Search Time

We have followed three different approaches toward developing predictive models of

visual search time. These approaches are linear regression, non-linear regression

and Support Vector Machine for regression (SVR). In linear regression with multiple

features, we have analyzed two different approaches; normal equation [70] and gradient

descent [70]. Whereas, non-linear regression has been analyzed through some of the

standard models [5]. SVR has been trained using the Radial Basis Function (RBF)

Kernel [94] and cross validation [94] has been performed. In depth description and

performance analysis of these approaches are given in the following.

51

4. Modeling of Visual Search Time

4.2.1 Linear Regression with Multiple Features

Linear regression attempts to model the relationship between two or more explanatory

variables and a response variable by fitting a linear equation to observed data [70].

Every value of the independent variable x is associated with a value of the dependent

variable y. Linear regression with multiple features is a generalization of linear regression

considering more than one independent variable. The hypothesis hθ(x) given by basic

model of linear regression is shown in Eqn. 4.1.

hθ(x) = θ0 + θ1x1 + θ2x2 + · · · + θnxn (4.1)

where x1, x2, . . . , xn are the multiple features or independent variable, θ0, θ1, θ2, . . . , θn

are the model parameters and hθ(x) is the hypothesis or function. In linear regression,

data are modeled using linear predictor functions, and model parameters are estimated

from the data.

Our training data contain four features. So, x1, x2, x3 and x4 represent the size of

objects, space between objects, number of objects and position of object in keyboard,

respectively. In our work, we have explored two different approaches of linear regression,

normal equation [70] and gradient descent [70]. The developed programs for both

approaches are executed using GNU Octave [78].

Normal Equation: The value of θ for the above hypothesis in normal equation

approach can be computed using Eqn 4.2.

θ = (XT X)−1XT y (4.2)

where X is a 28734 × 5 matrix having 1 in first column which represents the feature

vectors and y is a 28734 × 1 matrix representing visual search time for corresponding

feature vector of X. We compute the predicted visual search time based on this model.

The observed value of θ are θ0 = 6.93, θ1 = −6.19, θ2 = −12.83, θ3 = 17.54 and θ4 = 1.17

52

4.2. Model to Predict Visual Search Time

and the observed value of R2, coefficient of determination1, for this model is 0.39.

Gradient Descent: Gradient descent, also known as the steepest descent, is a first-

order optimization algorithm [70]. Here, the steps to be taken are proportional to the

negative of the gradient (or the approximate gradient) of the function at current point.

The objective of this approach is to minimize the error (J(θ)) which can be computed

using Eqn. 4.3.

J(θ) = 12m

m∑i=1

(hθ(x(i)) − y(i)

)2(4.3)

Here, m is number of data in training set, that is, 28734. x(i) represents ith training set,

hθ(x(i) is the predicted visual search time for x(i) and y(i) is the observed visual search

time for x(i). One way to do this is using the batch gradient descent algorithm. In this

approach, each θj is simultaneously updated as shown in Eqn. 4.4.

θj = θj − α1m

m∑i=1

(hθ(x(i)) − y(i)

)x

(i)j (simultaneously update θj for all j) (4.4)

where, x(i)j indicates jth feature in ith training set and α is the learning rate which

controls convergence of the algorithm. The higher value of α increases the convergence

rate of the algorithm but sometimes may create non-convergence situation. Contrary,

smaller value of α assures convergence of the algorithm but increases number of iteration

and in turn total computation time.

First, input data is normalized using z-score2 normalization. Then, we compute the

cost function (J(θ)) for different values of α. We observe that our problem converges

for α ≤ 0.1 with the training data. The observed value of θ are θ0 = 4.59, θ1 = −11.38,

θ2 = −14.08, θ3 = 18.93 and θ4 = 4.79 for α = 0.01. The computed R2 of gradient

descent approach for these values is 0.42.

1Coefficient of determination, http://en.wikipedia.org/wiki/Coefficient_of_determination2z-score indicates, by how many standard deviations an observation is above or below the

mean. It is derived by subtracting the mean from an individual observation and then dividingthe difference by the standard deviation.i.e. z = x−µ

σ , here µ is mean and σ is standard deviation.

53

4. Modeling of Visual Search Time

4.2.2 Non-linear Regression

Non-linear regression is a form of regression analysis in which observational data are

modeled by a function which is a non-linear combination of parameters and depends

on one or more independent variables [5]. Unlike linear regression, which is restricted

to estimating linear models, non-linear regression can estimate models with non-linear

relationship between independent and dependent variables.

In order to compute non-linear regression one need to specify a non-linear model

which states the relationship between independent and dependent variables. The initial

values of model parameters are also need to be specified. Here, we have computed with

five standard models. The computations are performed using SPSS tool [43]. Details of

each model along with R2 are shown in Table 4.3. In this table, b1, b2, b3, b4 represent the

various model parameters and x represents the combined value of independent variables.

Here, we have followed the linear regression model to combine independent variables into

a single value.

Table 4.3: Different non-linear model with corresponding R2

Model Definition R2

Asymptotic Regression [5] f(x) = b1 − b2 × b3x 0.49

Gauss [5] f(x) = b1 × (1 − b3 × exp−b2x2) 0.53

Log-Modified [5] f(x) = (b1 + b3x)b2 0.59

Ratio of Quadratics [5] f(x) = (b1+b2x+b3x2)(b4x2) 0.68

Richards [5] f(x) = b1

(1+b3×exp−b2x)1

b40.62

From the Table 4.3, we observe that Ratio of Quadratics model performs better than

others with our input dataset. The values of constants for this models are b1 = 479.15,

b2 = 276.47, b3 = 92.54 and b4 = 22.68.

54

4.2. Model to Predict Visual Search Time

4.2.3 Support Vector Regression

Support Vector Machine (SVM) was developed by Vladimir Naumovich Vapnik [94]

to solve the classification problem. Recently, SVM have been successfully extended to

regression and density estimation problems [9]. SVM based regression or Support Vector

Regression (SVR) is a method to estimate a function that maps from an input vector to

a real number based on training data. Margin maximization and kernel trick are used

in SVR for nonlinear mapping [85].

In SVR, the training dataset (D) for regression is represented as follows,

D = {(X1, y1), (X2, y2), ..., (Xm, ym)} (4.5)

where Xi is a n-dimensional vector of independent variables and yi is the real number,

that is, dependent variable or observation for each Xi. The SVR function F (X) makes

a mapping from an input vector Xi to the target yi as shown in Eqn. 4.6.

F (X) ⇒ w · X + b (4.6)

where w is the weight vector and b is the bias. The goal is to estimate these

parameters w and b of the function that give the best fit of the data. A SVR function

F (X) approximates all pairs (Xi, yi) while maintaining the differences between estimated

values and real values under ε precision, that is, yi−w·Xi−b ≤ ε or w·Xi+b−yi ≤ ε [85].

Thus, the margin of this estimation is identified as,

margin = 1∥w∥

(4.7)

Consequently, by minimizing ∥w∥2 to maximize the margin, the training of SVR

55

4. Modeling of Visual Search Time

becomes a constrained optimization problem as follows.

minimize : L(w) = 12

∥w∥ (4.8)

subject to : yi − w · Xi − b ≤ ε

w · Xi + b − yi ≤ ε

This solution assumes that the input data do not contain any errors. To allow some

errors to deal with noise in the training data, the soft margin SVR uses slack variables

ξ and ξ̂. Then, the optimization problem can be revised as shown in Eqn. 4.9.

minimize : L(w, ξ) = 12

∥w∥ + C∑

i

(ξ2i + ξ̂2

i ), C > 0 (4.9)

subject to : yi − w · Xi − b ≤ ε + ξi, ∀(Xi, yi) ∈ D

w · Xi + b − yi ≤ ε + ξ̂i, ∀(Xi, yi) ∈ D

ξi, ξ̂i > 0

The constant C > 0 is the trade-off parameter between the margin size and the amount

of errors. The slack variables ξ and ξ̂ deal with infeasible constraints of the optimization

problem by imposing the penalty to the excess deviations which are larger than ε. To

solve the optimization problem, we can construct a Lagrange function from the objective

function with Lagrange multipliers [85] as follows,

minimize: L(w, ξ) = 12

∥w∥2 + C∑

i

(ξi + ξ̂i) −∑

i

(ηiξi + η̂iξ̂i)

−∑

i

αi(ε + ηi − yi + w · Xi + b)

−∑

i

α̂i(ε + η̂i − yi + w · Xi + b) (4.10)

subjected to: η, η̂i ≥ 0 and α, α̂i ≥ 0

where ηi, η̂i, αi, α̂i are the Lagrange multipliers. The optimization problem with

56

4.2. Model to Predict Visual Search Time

inequality constraints can be changed to following dual optimization problem.

maximize: L (α) =∑

i

yi (αi − α̂i) − ε∑

i

(αi + α̂i)

−12∑

i

∑j

(αi − α̂i) (αi − α̂i) XiXj (4.11)

subjected to:∑

i

(αi − α̂i) = 0, 0 ≤ α, α̂i ≤ C

Accordingly, the SVR function F(X) becomes the following function.

F (X) ⇒∑

i

(αi − α̂i) XiX + b (4.12)

Here, Xi is the ith training data and X represents the set of all training data. This

function can map the training vectors to target real values with allowing some errors

but it does not support the nonlinear mapping between independent and dependent

variables [85]. So, the kernel is applied by replacing the inner product of two vectors

Xi, Xj with a kernel function K(Xi, Xj) [94]. The transformed feature space is usually

high dimensional, and the SVR function in this space becomes nonlinear in the original

input space. Using the kernel function K, the inner product in the transformed feature

space can be computed as fast as the inner product Xi · Xj in the original input space.

The linear optimization function of Eqn. 4.11 is changed by using kernel function as

shown in Eqn. 4.13.

maximize: L (α) =∑

i

yi (αi − α̂i) − ε∑

i

(αi + α̂i)

−12∑

i

∑j

(αi − α̂i) (αi − α̂i) K (Xi, Xj) (4.13)

subjected to:∑

i

(αi − α̂i) = 0

α̂i ≥ 0, αi ≥ 0, 0 ≤ α, α̂i ≤ C

57

4. Modeling of Visual Search Time

Finally, the SVR function F(X) becomes the following using the kernel function [85].

F (X) ⇒∑

i

(α̂i − αi) K (Xi, X) + b (4.14)

The basic steps in SVR are consists of the following.

• Scaling of collected data.

• Selection of kernel.

• Cross validation to find out best value of the parameter(s).

• Train with entire training set using best value(s).

• Testing of developed model.

Data scaling: Scaling of data before using those in SVR is important [94]. The main

advantage of scaling is that it reduces the dominance of higher numeric values on smaller

values. It also helps to reduce numerical complexities during computation. Kernel values

usually depend on the inner products of feature vectors. Thus, large attribute values

might cause numerical complexities. It is preferred to use same scaling method to scale

both training and testing data. We use z-score normalization to scale the collected data.

Kernel selection: In our work, we use Gaussian Radial Basis Function (RBF) as SVR

kernel. The RBF kernel nonlinearly maps samples into a higher dimensional space. The

RBF kernel is represented as

K(Xi, Xj) = eγ∥Xi−Xj∥2 (4.15)

So it, unlike the linear kernel, can handle the case when the relation between class labels

and attributes is nonlinear. Further, the linear kernel is a special case of RBF since the

linear kernel with a penalty parameter C̃ has the same performance as the RBF kernel

with some parameters (C, γ). The polynomial kernel has more hyperparameters, which

inuences the complexity of model, than the RBF kernel.

58

4.2. Model to Predict Visual Search Time

Cross validation: There are two parameters for SVR with RBF kernel; C and γ. It

is not known beforehand which C and γ values are best for a given problem. Thus,

parameter search is required to identify the best values of C and γ so that the model

can accurately predict unknown data. Note that, it is not useful to achieve high training

accuracy. Thus, the strategy is to separate the data set into two parts, of which one

is considered as unknown or testing set. The prediction accuracy obtained from the

unknown set more precisely reflects the performance on an independent data set. This

procedure is known as cross-validation.

The cross-validation procedure can prevent the overfitting problem. In k−fold cross-

validation, we first divide the training set into k subsets of equal size. Sequentially, one

subset is tested using the model trained on the remaining k − 1 subsets. Thus, each

instance of the whole training set is predicted once so the cross-validation accuracy is

the percentage of data which are correctly predicted.

To find the best values of C and γ through cross validation, grid search is performed.

We test with various pairs of C and γ values and then choose the one with the best

cross-validation accuracy. The values of C varies from 2−5 to 2−15 with step 22 and γ

varies from 2−15 to 23 with step 22. We select 1000 data, as sample, form the collected

data to decide best values of C and γ. We perform 5 − fold cross validation to choose

the best C and γ. We obtained best values of C and γ as 2.4 and 0.02, respectively.

Training with SVR: The collected data are divided into two parts. We use 80% of

collected data for training with SVR. The best values of C and γ, required for training,

are already decided through cross validation as mentioned earlier. Now, we use the 80%

training data to train the SVR and generate the final model.

Testing of the model: We have developed a model through SVR with RBF kernel

using 80% of the collected data. Testing of the developed model is required to judge the

accuracy of the model. Thus, we have tested model with remaining 20% of the collected

59

4. Modeling of Visual Search Time

data. The mean square error computed with test data for this approach is 1.17. The

computed R2, using Eqn. 4.16 [70], of this approach is 0.91. Here, V ar(y) indicates

variance of y.

R2 = 1 − MSE

V ar(y)(4.16)

4.2.4 Observation

We have developed three visual search time predictive model with three approaches

namely linear regression, nonlinear regression and Support Vector Regression. We have

tested the developed model and computed coefficient of determination that is R2, for

each approach as shown in Table 4.4.

Table 4.4: Different approaches with corresponding R2

Approach Definition R2

LinearRegression

Normal Equation θ = (XT × X)−1 × XT × y 0.39

Gradient Descent J(θ) = 12m

∑m

i=1

(hθ(x(i)) − y(i)

)20.42

Non-linearRegression

Asymptotic Regression f(x) = b1 − b2 × b3x 0.49

Gauss f(x) = b1 × (1 − b3 × exp−b2x2 ) 0.53

Log-Modified f(x) = (b1 + b3x)b2 0.59

Ratio of Quadratics f(x) = (b1+b2x+b3x2)(b4x2) 0.68

Richards f(x) = b1

(1+b3×exp−b2x)1

b40.62

Support VectorRegression

Regression usingRBF kernel

F (x) ⇒∑

i(α̂i − αi)K(xi, x) + b

0.91K(xi, xj) = eγ∥xi−xj ∥2

From the Table 4.4 we can observe that Support Vector Regression based modeling

approach performs better than other modeling approaches for our collected data.

60

4.3. Validation of the Proposed Model

4.3 Validation of the Proposed Model

We have experimented with 3 different interfaces and 42 participants to collect data

for modeling. We have modeled the collected data with three different approaches and

observed that SVR performs better than other with R2 as 0.91. However, the accuracy of

this approach can better be established if similar type of result are observed for different

interfaces and participants. In order to achieve this, an empirical study has been carried

out to determine the efficacy of the proposed model. In the study, performances predicted

by the proposed models for a set of virtual keyboards are compared with the observed

user performances. The experimental setup we have followed in the empirical study is

same as discussed earlier. We have used two new interfaces namely Fitaly, a virtual

keyboard for English and Guruji, a Hindi virtual keyboard layout attached with Guruji

search engine for these experiments.

Fitaly (a frequency based keyboard layout): The Fitaly one-finger keyboard [105] is

designed to optimize mouse movements during the text entry with one finger, a stylus or

a pen. Figure 4.1(a) shows the interface of the Fitaly keyboard. This keyboard’s name

is taken from the letter sequence along the second row of keys.

(a) Fitaly keyboard layout

(b) Guruji keyboard layout

Figure 4.1: Virtual keyboard layouts used for validation

Guruji (an alphabetical keyboard layout): A virtual keyboard is a part of Guruji

search engine interface [33], which allows users to enter queries using mouse as the input

61

4. Modeling of Visual Search Time

device. The keyboard layout in Hindi is shown in Fig. 4.1(b). The first row comprises

of Hindi vowels completely, in alphabetical order. The rest of the keyboard layout is

divided into two parts. The first part comprises of first twenty five consonants in order.

Second part is made up of the remaining consonants.

Here, we include 11 new participants in our experiments for evaluating the proposed

model. These participants are not considered in earlier experiments. The details of the

participants are shown in Table 4.5.

Table 4.5: Description of participants considered for validation

Participant’s profileNumber ofparticipantsOccupation Age

group Education Computerproficiency

Student 13–29Post Graduate Level 1 3

Under Graduate Level 1 1

Higher Secondary Level 2 2

Office staff member 31–48Graduate Level 1 2

Higher Secondary Level 3 1

Housewife 35–45 Graduate Level 1 2

We have observed visual search time for different combination of feature values from

user experiments. The corresponding visual search time for individual combination is

also calculated from the proposed model. The observed mean square error and R2

between predicted value and user performance are found as 1.31 and 0.88, respectively.

4.4 Summary

In the context of virtual keyboard interface, developing a visual search time prediction

model remains a serious problem. This chapter addresses the problem and proposes a

computational model to predict visual search time of a virtual keyboard layout from

different keyboard layout design parameter. Prior to modeling, we have performed

some experiments with user to collect knowledge about user performance on variation

of different visual search features. Then we have exercised three different modeling

62

4.4. Summary

approaches namely linear regression, non-linear regression and support vector machine

for regression. By analyzing these model performances we observe that support

vector machine for regression based modeling approach performs better than other two

approaches. Finally, to validate the developed model, we perform another set of user

based experiments with new participants and interfaces and compared user performance

with model predicted values. The proposed model is not only useful to predict visual

search time of a virtual keyboard layout, it can also be applied to design an efficient

and visual search time minimized virtual keyboard layouts. In the next chapter, we

discuss our approach to design optimum virtual keyboard taking our visual search time

prediction model into account.

63

Chapter 5

Virtual Keyboard Layout

Optimization

It is evident that visual search time to find a key greatly influence the text entry rate while

composing text through virtual keyboard. Previously, researchers have concentrated

on enhancing text entry rate by optimizing mouse movement only. However, the

development of a virtual keyboard with minimum mouse movement time as well as visual

search time incurred is desirable. In order to account visual search time in the design, a

predictive model has been developed using SVR approach, as described in Chapter 4. It

is difficult to evaluate designs on the basis of visual search time calculated by proposed

model alone, as the text entry rate significantly depends on mouse movement time

also. Hence, in order to arrive at a design methodology for virtual keyboard design,

both these measures has to be taken into account. In this chapter, we propose an

approach to design a virtual keyboard optimizing both mouse movement time and visual

search time. To judge the effectiveness of our multiobjective design approach, we design

another virtual keyboard using genetic algorithm (GA) considering mouse movement

minimization only. To inspect the efficacy of the proposed multiobjective optimized

design, we have evaluated both keyboard layouts with users and calculated text entry

65

5. Virtual Keyboard Layout Optimization

rate. The achieved text entry rate through optimized keyboard is higher than other

keyboard’s text entry rate.

This chapter consists of five sections. Our proposed approach of designing virtual

keyboard is discussed in Section 5.1. Section 5.2 describes the alternate traditional

design approach using GA. An empirical study has been carried out to show the validity

of the proposed design approach, discussed in Section 5.3. Finally, Section 5.4 contains

the summary of the chapter contents.

5.1 Proposed Virtual Keyboard Design Approach

In this work, we propose an approach to design a Bengali virtual keyboard which achieves

higher text entry rate from proper combination of different virtual keyboard design

parameters. Moreover, we consider the single tap virtual keyboard only, defining the

following assumptions:

1. All keys are of equal size and square shape.

2. Equal amount of horizontal and vertical spaces in between all adjacent keys.

3. Each key contains only one character.

4. Keyboard layout remains static over the time.

We design a multiobjective optimized Bengali virtual keyboard layout based on two

optimality metrics; mouse movement minimization and visual search time minimization.

The average movement time is measured from Fitts-digraph model [105] and visual

search time is computed by our proposed model discussed in previous chapter.

5.1.1 Problem Statement

The text entry rate in character per second (CPS), performance measurement of a virtual

keyboard, is a function of both average movement time (MT ) and visual search time

66

5.1. Proposed Virtual Keyboard Design Approach

(V ST ). So, the problem statement of maximization of text entry rate with respect to

both objectives is shown in Eqn. 5.1.

max CPS = f(MT, V ST

)(5.1)

Mean Movement Time (MT ): Mean Movement Time MT according to Fitts-digraph

model [86,105] is calculated by summing up movement time (MTij) between all digram

multiplied by corresponding digram probability (Pij), as represent in Eqn. 5.2. Here N

is the number of character present in keyboard layout.

MT =N∑

i=1

N∑j=1

(MTij × Pij) (5.2)

Movement time (MTij) from ith key to jth key is calculated from Fitts’s law [63] as shown

in Eqn. 5.3, where a and b are empirically determined constants, Dij is the Euclidean

distance between the center of both keys and Wj is width of the target key (jth key).

MTij = a + b log2

(Dij

Wj+ 1

)(5.3)

Pij , the digraph probability of occurrence of jth character after ith character, is calculated

using Eqn 5.4. Here, we analyze the Bengali Wikipedia corpus, to compute digram

probabilities of occurrences.

Pij = fij

N∑i=1

N∑j=1

fij

(5.4)

Visual Search Time (VST:) Our proposed visual search time model described in

Chapter 4 is used as a measurement of VST, which is expressed in terms of virtual

keyboard design parameters. In other words, V ST is a function of four design parameters

namely key size (KS), space between keys (SK), number of characters in the layout

67

5. Virtual Keyboard Layout Optimization

(NC) and position of the character (PC), that is,

V ST = F (KS, SK, NC, PC) (5.5)

and it is computed by combining the equations shown in Eqn. 4.14 and 4.15.

Here, we consider only square shaped keys and in the design space, font size of any

jth key is varied from 8pt to 14pt, with an increment of 2pt. Please note that in a

particular keyboard we use same font size for all keys of the keyboard. We consider

equal amount of horizontal and vertical space for any two adjacent keys in a particular

keyboard interface. In our design space, space between any two adjacent keys namely

ith key to jth key is varied 20% to 35% of the key size, with an increment of 5%. The

number of character remains same as 53, for Bengali, through out all designs and the

position of character is determined from the layout.

5.1.2 Virtual Keyboard Design using Multiobjective Optimization

There are many approaches to solve a multiobjective problem using GA like

Vector Evaluated Genetic Algorithm (VEGA) [83], Multiobjective Genetic Algorithm

(MOGA) [28], Niched Pareto Genetic Algorithm (NPGA) [40], Weight-based Genetic

Algorithm (WBGA) [34], Non-dominated Sorting Genetic Algorithm (NSGA) [87],

improved NSGA (NSGA-II) [15], Strength Pareto Evolutionary Algorithm (SPEA) [107],

Pareto-Archived Evolution Strategy (PAES) [51] etc. In our work, we use NSGA-

II algorithm, which is treated as a fast (computational complexity O(MN2); M is

number of objectives and N is number of initial population) and elitist multiobjective

GA proposed by Dev et al. [15]. It has been reported that, compared to other elitist

multiobjective GAs (PAES, SPEA etc.), it has better diversity preservation [15,48]. So,

it can compete with them in the context of converging in the true Pareto-optimal front.

Moreover, traditional approaches (MOGA, NSGA) use the concept of fitness sharing

by niching. The main problem with sharing is that it requires the specification of a

68

5.1. Proposed Virtual Keyboard Design Approach

sharing parameter and performance of the sharing function method depends largely on

the chosen sharing parameter. NSGA-II replaces the sharing function approach with a

crowded-comparison approach that eliminates difficulties. Detailed procedure of NSGA-

II is explained below.

NSGA-II algorithm follows the usual crossover and mutation operators, but selection

operator work differently from simple GA [15]. Selection is done with the help

of fast non-dominated sorting, crowded distance estimation and crowded-comparison

operator [15]. We optimize key arrangement and other virtual keyboard design

parameters by employing NSGA-II algorithm, to minimize mouse movement time along

with minimization of visual search time. Each step of our virtual keyboard design

approach using NSGA-II is illustrated in Fig 5.1.

Here, we generate 20 (N) chromosomes of each 57 bits in initial population (P ).

Initial 53 bits of a chromosome use real encoding to represent the key arrangement. The

last 4 bits of a chromosome encoded in binary form, where each of 2 bits represent key

size and space between keys, respectively.

Here, single point substring crossover technique [38] with randomly selected crossover

point in between 20 to 30 is applied in the real coded portion. Similarly, for binary

portion, we also use the single point binary crossover technique.

For mutation of real coded portion, we randomly select two point and swap each

other. Similarly, for binary portion each bit of the chromosome is mutated 0 to 1 and

vice-verse with mutation probability 0.1. The objective functions are calculated from

Fitts-digraph model [105] and our proposed model discussed in the previous chapter.

We perform ranking and crowding distance operations as described in NSGA-II

algorithm [15] to select individual for next generation. In fast non-dominated sorting,

rank is assigned to each solution according to its non-domination level [15]. Any solution

of a particular rank is not better with respect to other solutions of the same rank.

After ranking, we calculate crowding distance for each solution of all non-domination

69

5. Virtual Keyboard Layout Optimization

Figure 5.1: Flowchart of virtual keyboard design using NSGA-II algorithm

levels to get the density estimation of solutions. The crowding distance is calculated

by measuring the average distance of two nearest solution on either side for each of the

70

5.1. Proposed Virtual Keyboard Design Approach

objectives. Finally, the overall crowding distance of each solution is computed as the

sum of distance values corresponding to each objective. We use crowded comparison

operator [15] to select the solutions toward a uniformly spread-out Pareto-optimal front.

Here, if two solutions are belong to different non-domination ranks, then we select the

solution with lower rank. Also, we select the solution with lesser crowded distance, if

both solutions are in the same front.

We select 20 individuals by performing ranking and crowding distance operations

and consider them as initial population for next generation. We execute the process

iteratively up to 100 generations to find the set of optimal solutions. Pareto-optimal set

of 20 non-dominated solutions are obtained at the end of 100 generations.

Then, a Fuzzy-based approach [1] is applied to choose the single best compromise

solution from the non-dominated solutions. The ith objective function of a solution in

the non-dominated set, Fi, is represented by a membership function µi as defined in

Eqn. 5.6.

µi =

1 Fi ≤ F mini

F maxi −Fi

F maxi −F min

iF max

i < Fi < F mini

0 Fi ≥ F maxi

(5.6)

Here, the maximum and minimum values of ith objective function is denoted by F imax

and F imin, respectively. For each non-dominated solution j, the normalized membership

function µj is calculated as shown in Eqn. 5.7, where n is the number of objective

functions considered and m is the number of non-dominated solutions. Here, the value

of m and n are 2 and 20 respectively.

µj =

n∑i=1

µji

m∑j=1

n∑i=1

µji

(5.7)

71

5. Virtual Keyboard Layout Optimization

The solution having the maximum value of µj is the best compromise solution for all

objectives. Here, we use the design parameter values obtained from the best compromise

solution to design our proposed multiobjective optimized virtual keyboard. Here, values

of key size and space between keys are 12pt and 30%, respectively.

The implementation has been done using MATLAB tool in Windows 7 environment

in a PC having Pentium Core2Duo processor with 2.4 GHz clock speed. The keyboard

layout designed using the obtained feature values is depicted in Fig. 5.2.

e a

iu

o

s

Figure 5.2: Mouse movement and visual search time optimized keyboard layout

5.2 Movement Time Optimized Virtual Keyboard Design

To measure the effectiveness of optimized keyboard design, we design another keyboard

layout considering mouse movement time minimization only. We apply Genetic

Algorithm (GA) to arrange the characters in such a way that it leads to minimum mouse

movement time. Mean mouse movement time (MT ), calculated from Fitts-digraph

model [105], is used as fitness function to minimize mouse movement time (Eqn. 5.2).

We optimize key arrangement of virtual keyboard layout to minimize mouse

movement time using GA. We kept other design parameters like size, spacing etc. as

obtained from our NSGA-II based approach. As Space character is extensively used

in text composition, we place it at the central position of the keyboard having double

72

5.2. Movement Time Optimized Virtual Keyboard Design

width of any character button and exclude it from considering in key arrangement. The

arrangement of characters on keys is targeted to possess minimum number of mouse

movements. We apply a genetic algorithm-based method to find an optimal arrangement

of characters on keys. The flowchart of the approach is depicted in Fig. 5.3.

We follow the real coded ordered GA [100] to solve the above mentioned problem. We

define a chromosome which is of 53 characters length. To decide an initial candidate, we

choose a random arrangement of characters. We decide 20 such random arrangements

for each candidate and consider as the initial population in our work.

After generating the initial population, we calculate the cost values based on the

objective function for each candidate and arrange them in ascending order. We consider

rank-based selection method [84] to generate offspring for the next generation. We

divide the candidate parents into two groups; one group contains 10 individuals with

higher cost values and the rest contains in another group. Ten individuals of first

group are considered as population of next generation. However, to decide remaining 10

population, we randomly take two parents from the two different groups and perform

mating between them.

We follow the substring crossover technique [38] which prefers that a part of the

first parent should be copied in the offspring and the rest should be taken in the same

order as they appear in second parent. Let us consider any two chromosomes A and B

corresponding to two candidate parents in the current population. We consider P1 and

P2, as the crossover points in the chromosomes. The crossover mechanism copies a part

of the length |P2 − P1| from chromosome A and paste it into the child chromosome C in

between P1 and P2, both inclusive. For the rest of the parts in chromosome C, we fill

with characters in the order as they appear in the parent chromosome B. We perform

crossover operation with crossover probability 0.9. The crossover procedure we have

followed is illustrated in Fig. 5.4.

Repeating the above procedure, we construct offspring of strength 20, the population

73

5. Virtual Keyboard Layout Optimization

Figure 5.3: Flowchart of virtual keyboard design using Genetic Algorithm

size we have decided in each iteration. Next, we consider a mutation operation which

consists of swapping the locations of two characters with respect to a zone. We have

74

5.2. Movement Time Optimized Virtual Keyboard Design

……. …….

……. …….

……. …….

A

B

C

P1 P2

Figure 5.4: Crossover operation in Genetic Algorithm

carried out the mutations in the resulting chromosomes with mutation probability 0.1.

The process is executed iteratively up to 100 generations for finding the optimal

solution. After 100 iteration, chromosome corresponding to the highest rank value

is selected as the solution. We design a keyboard layout following the optimization

approach which is shown in Fig. 5.5.

e o

a

u

is

Figure 5.5: Keyboard layout minimizing mouse movement time

75

5. Virtual Keyboard Layout Optimization

5.3 Empirical Study to Evaluate Designs

We have performed an empirical study to ensure the effectiveness of our design

approaches. In this study, we compare user performance in terms of text entry rate

during text composition with those designed keyboard layouts.

We have considered 10 new participants of varying educational background having

Level 1 and Level 2 computer proficiency, as discussed previously in Chapter 3, for the

experiments. We use a text corpus of 50 Bengali sentences considering the occurrence

of almost all characters of Bengali. This corpus is created by selecting several portion

of Bengali wikipedia page.

In each experiment, each user is given with a keyboard interface to type the selected

text corpus freely. During experiments, the composed text through each keyboard are

stored into log files separately for each participant. Then, by analyzing the log files, we

calculate text entry rate in terms of words per minute as shown in Eqn. 5.8.

Text Entry Rate (WPM) = |T | − 1S

× 60w

(5.8)

Here, |T | is the length of text entered by user that is the number of characters entered.

S represents the time taken for entering text by a user, in seconds. It is measured from

the entry of the first character to the last. w denotes the average length of words for a

language, 5.11 for Bengali.

The text entry rate achieved from keyboard layout designed through NSGA-II based

approach is 6.72 words per minute. However, observed text entry rate from keyboard

layout designed through GA-based approach is 5.98 words per minute. So, the achieved

text entry rate of NSGA-II based approach is 12.37% higher than the keyboard layout

designed through GA-based approach.

76

5.4. Summary

5.4 Summary

In this chapter, we propose the design of a multiobjective optimized Bengali virtual

keyboard interface using NSGA-II to minimize mouse movement time as well as visual

search time. To judge the efficacy of the optimized keyboard, we design another alternate

virtual keyboard through traditional approach that is using GA, minimizing mouse

movement only. Then, an empirical study has been carried out with the existing virtual

keyboards on the basis of text entry rate. The study reveals that our proposed keyboard

performs better than other with respect to text entry rate.

77

Chapter 6

Summary and Conclusion

The main objective of our research is to develop a computational model for predicting

visual search time of a virtual keyboard layout, as well as we achieve some improvement

in keyboard layout design. The outcomes of our research are discussed in Chapter 3 to

Chapter 5. This chapter summarizes the major contributions of our work and future

scope of extending the research.

This dissertation contributes in modeling visual search time and demonstrates its

application toward designing virtual keyboard. Virtual keyboard, a popular text

composition interface contains different visual features which significantly affect the text

entry rate. Existing Hick-Hyman Law [37,42] to predict visual search time considers only

the number of keys present in the keyboard and lacks in acquiring other visual search

features like shape, size, grouping, ordering etc. This work addresses this limitation.

We develop a computational model to predict the average visual search time given an

object space. In the domain of user interface design, many object spaces are possible

and each of them demands their own treatment so far the perception task is concerned.

Considering this, we limit the investigation only to virtual keyboard which is a graphical

user interface to compose texts.

User interface designers advocate many visual features to be incorporated in designs

so that a user can interact with the interfaces in a better way. Impacts of many of those

79

6. Summary and Conclusion

features are subjective to users and beyond the scope of measuring them quantitatively.

It is therefore an issue to identify the visual features which are users’ specific and which

are quantitatively measurable or analyzable. In this work, we propose a methodology to

identify all such features. Our approach includes listing of visual features which might be

important in visual perception and then identification of the significant visual features

considering degree of influence. We use three virtual keyboard interfaces namely Opti, a

frequency based layout for English, Avro, an alphabetical layout for Bengali and iLipi-H

a multizonal, frequency and inflexion window based layout for composing text in Hindi,

to perform this experiment. To identify visual search time parameters which significantly

influence the performance of text entry rate, we have conducted several experiments

with 32 users; the required data are logged into file and statistically analyzed through

ANOVA test. This work establishes the fact that there are 4 out of 18 features, which

contribute more in visual search time in context of virtual keyboard. The features are

size of elements, space between elements, number of elements, and position of elements.

We study several mathematical models based on the identified features to estimate

visual search time of a virtual keyboard. To accomplish this task, we have gathered user

accomplished visual search times on different combination of feature values through

experiment. The proposed modeling is carried out by three different approaches

namely linear regression, non-linear regression and, support vector regression. In

linear regression with multiple features, we have analyzed two different approaches,

normal equation and gradient descent. Non-linear regression has been analyzed through

5 standard models among which Ratio of Quadratics model performs better than

others with our test data. Finally, visual search time model based on support vector

regression has been developed. This model is able to predict visual search time for a

virtual keyboard with different combination of design parameter values. The models

are validated with both in domain and out domain data on the basis of statistical

performance metric like mean square error (MSE) and R2. The experimental results

80

6.1. Contribution of Our Work

conclude that the proposed model performs better than linear and nonlinear models.

The accuracy of our proposed model can better be judged by comparing its performance

with different interfaces and participants. So, an empirical study has been carried out

to measure the efficacy of the proposed model. In this study, performances predicted

by the proposed model for a set of virtual keyboards are compared with the observed

user performances. Here, we have used two new layouts namely Fitaly, an English

one-finger keyboard optimizing mouse movements during the text and Guruji, a Hindi

keyboard layout attached with Guruji search engine for the experiments. We include

15 participants to evaluate the proposed model. The observed mean square error

and computed R2 between predicted value and user performance are 1.31 and 0.88,

respectively.

We design a multi objective optimized virtual keyboard using NSGA-II to minimize

mouse movement time as well as visual search time. The mouse movement time and

visual search time are measured using Fitts-digraph model and our proposed visual

search time model, respectively. To quantify the effectiveness of the proposed keyboard,

we design another virtual keyboard by employing GA to minimize mouse movement time

only. We evaluate the proposed keyboard layout with 10 different users and achieve

text entry rate of 6.72 words per minute compared to 5.98 words per minute for GA

based approach. In other words, the achieved text entry rate is 12.37% higher than the

keyboard layout designed through traditional approach.

6.1 Contribution of Our Work

Several contributions have been made in the domain of developing a visual search time

predictive model of virtual keyboards and designing multiobjective optimized virtual

keyboard. These contributions are listed below.

Visual search features identification: Several visual feature influencing design

parameters, reported in various literatures, are listed. Then, empirical study with

81

6. Summary and Conclusion

users and ANOVA tests have been carried out on those parameters which establish

that there are four parameters which have significant influence on visual search time.

The parameters are size of elements, space between elements, number of elements and

position of elements.

Visual search time model: A computational model is developed to predict visual

search time of a virtual keyboard based on the identified features. We have exercised

various mathematical approaches to develop the predictive model. We find that support

vector regression based model outperform over other models. The experiments with users

conclude that the observed mean square error and computed R2 between predicted value

and user performance result are 1.31 and 0.88, respectively.

Keyboard layout optimization: We design a virtual keyboard based on

multiobjective optimization using NSGA-II algorithm based on two optimality metrics,

minimizing mouse movement time as well as visual search time influenced by design

parameters. The mouse movement time and visual search time are measured using Fitts-

digraph model and our proposed visual search time model, respectively. The proposed

approach achieves 12.37% higher text entry rate than the keyboard layout designed

through traditional approach.

6.2 Threats to Validity

Relating to our experiments and experimental results, we would like to point out their

validity and limitations.

External validity: We have followed several user-based experiment to identify

significant visual search features as well as generate training data set. For the

experiments, we have considered 10 to 44 users with various educational backgrounds.

Further it is assumed that every users perform experiments maintaining high

concentration level. The results of the user evaluation is therefore subject to the

82

6.2. Threats to Validity

limitation on number of users involved.

Internal validity: We have not considered few visual search features in the process

of identifying significant visual search features. However, experiment can be extended

by considering features like search strategy, background-foreground color etc. In the

present work, our experiments have been restricted to desktop environments, only. It

would be another interesting matter to validate the results with small display devices

such as cell phone, iPod etc. We have considered three keyboard interfaces for our

experiments. Some other keyboard layouts like iLeap [89], Lipik [57], Gate2Home [30]

etc. can considered into evaluation.

Construct validity: The visual search time predictive model is developed through

Support Vector Regression. However, other soft computing approaches such as Simulated

Annealing (SA), Ant Colony Optimization (ACO), Particle Swarm Optimization (PSO),

Artificial Neural Network - Genetic Algorithm (ANN-GA) can be considered for

modeling. The optimal arrangement of character placing in the layout is governed

by the factors: digraph probability and NSGA-II algorithm. We have considered the

Wikipedia corpus to calculate digraph probabilities for languages aimed in this work.

It has been verified that size of the corpus are comparable to the size of other sources

such as daily newspapers. As an alternative to NSGA-II, multiobjective optimization

problem solving could be carried out with optimization methodology such as Niched

Pareto Genetic Algorithm (NPGA), Strength Pareto Evolutionary Algorithm (SPEA),

Pareto-Archived Evolution Strategy (PAES) etc. We have chosen NSGA-II because of

its better fitting with the objective function, higher convergence rate and hence faster

computation.

83

6. Summary and Conclusion

6.3 Future Scope of Work

• Proposed visual search time model can be extended for mobile devices. It will help

in developing virtual keyboard for small display devices.

• Specialized virtual keyboard can be designed for physically challenged persons.

• The proposed modeling approach can be extended to other type of interfaces such

as menu driven interface, gesture-based interface, tangible interface etc.

84

Publications out of this work

• P. K. Saha, D. Samanta, S. Sarcar and M. K. Sharma. Analysis of Visual SearchFeatures. International Journal of Human Factors Modelling and Simulation,InderScience, Vol. 3, No. 1, pages 66 − 89, 2012.

• P. K. Saha and D. Samanta. A Computational Model of Visual SearchTime, Human-centric Computing and Information Sciences, SpringerOpen (UnderRevision).

• M. K. Sharma, D. Samanta, P. K. Saha and S. Sarcar. Visual Clue: An Approachto Predict and Highlight Next Character. In Proceedings of the 4th InternationalConference on Intelligent Human Computer Interaction (IHCI), Kharagpur, India,December 27 − 29, 2012. (IEEE Xplore)

• D. Samanta, S. Ghosh, S. Dey, M. K. Sharma, S. Sarcar, P. K. Saha and S.Maiti. Development of Multimodal User Interface to Internet for Common People.In Proceedings of the 4th International Conference on Intelligent Human ComputerInteraction (IHCI), Kharagpur, India, December 27 − 29, 2012. (IEEE Xplore)

• M. K. Sharma, S. Dey, P. K. Saha, and D. Samanta. Parameters Effecting thePredictive Virtual Keyboard. In Proceedings of the IEEE Students’ TechnologySymposium, pages 268 − 275, Kharagpur, India, April 3 − 4, 2010. (IEEE Xplore)

• S. Sarcar, S. Ghosh, P. K. Saha, and D. Samanta. Virtual Keyboard Design:State of the Arts and Research Issues. In Proceedings of the IEEE Students’Technology Symposium, pages 289 − 299, Kharagpur, India, April 3 − 4, 2010.(IEEE Xplore)

85

References

[1] M. A. Abido. Multiobjective Evolutionary Algorithms for Electric Power DispatchProblem. IEEE Transactions on Evolutionary Computation, 10(3):315 – 329, 2006.

[2] D. Anderson, E. Anderson, N. Lesh, J. Marks, B. Mirtich, D. Ratajczak, andK. Ryall. Human-Guided Simple Search. In Proceedings of the 17th NationalConference on Artificial Intelligence, pages 209–216, Austin, Texas, Jul 31–Aug 22000.

[3] J. R. Anderson, M. Matessa, and C. Lebiere. ACT-R: A Theory of Higher LevelCognition and Its Relation to Visual Attention. Human Computer Interaction,12(4):439 – 462, 1997.

[4] J. L. Barbur, P. M. Forsyth, and D. S. Wooding. Visual Search, chapter EyeMovements and Search Performance, pages 253–264. Taylor & Francis, Inc.,Bristol, PA, USA, 1993.

[5] D. M. Bates and D. G. Watts. Nonlinear Regression Analysis and Its Applications.John Wiley & Sons, 2007.

[6] J. H. Bertera and K. Rayner. Eye Movements and The Span of Effective Stimulusin Visual Search. Perception & Psychophysics, 62(3):576 – 585, 2000.

[7] X. Bi, B. A. Smith, and S. Zhai. Quasi-qwerty Soft Keyboard Optimization. InProceedings of Human Factors in Computing Systems, pages 283 – 286, Atlanta,USA, 2010.

[8] A. B. Bond. The Bead Game: Response Strategies in Free Assortment. HumanFactors: The Journal of the Human Factors and Ergonomics Society, 24(1):101–110, 1982.

[9] C. J. C. Burges. A Tutorial on Support Vector Machines for Pattern Recognition.Data Mining and Knowledge Discovery, 2:121–167, 1998.

87

References

[10] M. D. Byrne, J. R. Anderson, S. Douglass, and M. Matessa. Eye Tracking theVisual Search of Click-Down Menus. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, CHI ’99, pages 402–409, Pittsburgh,United States, May 15–20, 1999.

[11] L. J. Cao and F. E. H. Tay. Support Vector Machine with Adaptive Parametersin Financial Time Series Forecasting. IEEE Transactions on Neural Networks,14(6):1506–1518, 2003.

[12] S. K. Card, T. P. Moran, and A. Newell. The Keystroke-Level Model for UserPerformance Time with Interactive Systems. Communication of ACM, 23(7):396–410, 1980.

[13] S. K. Card, T. P. Moran, and A. Newell. The Psychology of Human-ComputerInteraction. Lawrence Erlbaum Associates, Hillsdale, USA, 1983.

[14] A. Clark. Some Logical Features of Feature Integration. Neuronal Coding ofPerceptual Systems, 9(2):3–20, 2001.

[15] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A Fast and ElitistMultiobjective Genetic Algorithm : NSGA-II. IEEE Transactions on EvolutionaryComputation, 6(2):182–197, 2002.

[16] S. Deshwal and K. Deb. Design of an Optimal Hindi Keyboard for Convenient andEfficient Use. Technical Report, Indian Institute of Technology, Kanpur, 2003.

[17] M. Dorigo and L. M. Gambardella. Ant System : Optimization by a Colony ofCooperating Agents. IEEE Transactions on Systems Man Cybernetics, Part B,26(2):29 – 41, 1996.

[18] B. A. Dosher. Models of Visual Search: Finding a Face in a Crowd. An Invitationto Cognitive Science: Methods, Models and Conceptual Issues, 4(1):455–521, 1998.

[19] J. Duncan and G. W. Humphreys. Visual Search and Stimulus Similarity.Psychological Review, 96(3):433, 1989.

[20] M. D. Dunlop and J. Levine. Multidimensional Pareto Optimization ofTouchscreen Keyboards for Speed, Familiarity and Improved Spell Checking. InProceedings of Human Factors in Computing Systems, pages 2669 – 2678, Austin,USA, 2012.

88

References

[21] J. Eggers, D. Feillet, S. Kehl, M. O. Wagner, and B. Yannou. Optimization ofthe Keyboard Arrangement Problem using an Ant Colony Algorithm. EuropeanJournal of Operational Research, 148(3):672 – 686, 2003.

[22] S. P. Everett and M. D. Byrne. Unintended Effects: Varying Icon Spacing ChangesUsers’ Visual Search Strategy. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’04, pages 695–702, Vienna, Austria, Apr 24–29, 2004.

[23] J. M. Findlay and I. D. Gilchrist. Eye Guidance in Reading and Scene Perception,chapter Eye Guidance and Visual Search, pages 295 – 312. Elsevier, Amsterdam,Netherlands, 1998.

[24] J. M. Findlay and I. D. Gilchrist. Active Vision: The Psychology of Looking andSeeing. Oxford University Press, Oxford, UK, 2003.

[25] The first GUIs. Xerox 8010 Star Information System. [Online] Available:http://www.catb.org/~esr/writings/taouu/html/ch02s05.html. accessed onDecember 2011.

[26] P. M. Fitts. The Information Capacity of the Human Motor System in Controllingthe Amplitude of Movement. Journal of Experimental Psychology, 47(6):381 – 391,1954.

[27] M. D. Fleetwood and M. D. Byrne. Modeling the Visual Search of Displays:A Revised ACT-R Model of Icon Search Based on Eye-Tracking Data. Human-Computer Interaction, 21(2):153–197, 2006.

[28] C. M. Fonseca and P. J. Fleming. Genetic Algorithms for MultiobjectiveOptimization: Formulation, Discussion and Generalization. In Proceedings of the5th International Conference on Genetic Algorithms, pages 416 – 423, California,USA, 1993.

[29] W. Fu and P. Pirolli. SNIF-ACT: A Cognitive Model of User Navigation on theWorld Wide Web. Human Computer Interaction, 22(4):355 – 412, 2007.

[30] Gate2Home. [Online] Available: http://gate2home.com/Bengali-Keyboard#INBengaliInscript. accessed on August 2012.

[31] C. O. Getschow, M. J. Rosen, and Goodenough-Trepagnier. A SystematicApproach to Design a Minimum Distance Alphabetical Keyboard. In Proceedings

89

References

of 9th Annual Conference of Rehabilitation Engineering Society of North America,Minneapolis, USA, 1986.

[32] D. E. Goldberg and J. H. Holland. Genetic Algorithms and Machine Learning.Machine Learning, 3(2):95 – 99, 2005.

[33] Guruji. Indian Internet Search Engine. [Online] Available: http://www.guruji.com/hi/index.html. accessed on February 2012.

[34] P. Hajela and C. Y. Lin. Genetic Search Strategies in Multicriterion OptimalDesign. Structural and Multidisciplinary Optimization, 4(2):99 – 107, 1992.

[35] T. Halverson and A. J. Hornof. A Minimal Model of For Predicting Visual Searchin Human-Computer Interaction. In Conference on Human Factors in ComputingSystems, pages 431 – 434, CA, USA, 2007.

[36] S. A. Herd and R. C. O’Reilly. Serial Visual Search from a Parallel Model. VisionResearch, 45(24):2987–2992, 2005.

[37] W. E. Hick. On the Rate of Gain of Information. The Quarterly Journal ofExperimental Psychology, 4(1):11–26, 1952.

[38] T. P. Hong, H. S. Wang, W. Y. Lin, and W. Y. Lee. Evolution of AppropriateCrossover and Mutation Operators in a Genetic Process. Applied Intelligence,16(1):7 – 17, 2001.

[39] W. C. Hong. A Hybrid Support Vector Machine Regression for Exchange RatePrediction. International Journal of Information and Management Sciences,17(2):19–32, 2006.

[40] J. Horn, N. Nafploitis, and D. E. Goldberg. A Niched Pareto Genetic Algorithmfor Multiobjective Optimization. In Proceedings of the 1st IEEE Conference onEvolutionary Computation, pages 82 – 87, New Jersey, USA, 1994.

[41] A. J. Hornof. Visual Search and Mouse-Pointing in Labeled Versus UnlabeledTwo-Dimensional Visual Hierarchies. ACM Transactions on Computer-HumanInteraction (TOCHI), 8(3):171–197, 2001.

[42] R. Hyman. Stimulus Information as a Determinant of Reaction Time. Journal ofExperimental Psychology: General, 45(3):188–196, 1953.

90

References

[43] IBM. SPSS Statistics. [Online], Available: http://www-01.ibm.com/software/analytics/spss/products/statistics/. accessed on June 2012.

[44] B. E. John and D. E. Kieras. The GOMS Family of User Interface AnalysisTechniques: Comparison and Contrast. ACM Transaction on Computer HumanInteraction, 3(4):320–351, 1996.

[45] B. E. John and D. E. Kieras. Using GOMS for User Interface Design andEvaluation: Which Technique? ACM Transaction on Computer HumanInteraction, 3(4):287–319, 1996.

[46] B. E. John and D. D. Salvucci. Multipurpose Prototypes for Assessing UserInterfaces in Pervasive Computing Systems. IEEE Pervasive Computing, 4(4):27–34, Dec 2005.

[47] B. E. John, D. D. Salvucci, P. Centgraf, and K. Prevas. Predictive HumanPerformance Modeling Made Easy. In Proceedings of the International Conferenceon Cognitive Modeling, pages 455 – 462, Pittsburgh, Pennsylvania, 2004.

[48] V. Khare, X. Yao, and K. Deb. Performance Scaling of Multiobjective EvolutionaryAlgorithms. In Proceedings of the 2nd International Conference on EvolutionaryMulti-Criterion Optimization, pages 376 – 390, Faro, Portugal, 2003.

[49] D. E. Kieras and D. E. Meyer. An Overview of the EPIC Architecture forCognition and Performance with Application to Human-Computer Interaction.Human Computer Interaction, 12(4):391–438, Dec 1997.

[50] R. M. Klein and W. J. MacInnes. Inhibition of Return is a Foraging Facilitator inVisual Search. Psychological Science, 10(4):346 – 352, 1999.

[51] J. D. Knowles and D. W. Corne. Approximating the Nondominated Front usingthe Pareto Archived Evolution Strategy. Evolutionary Computation, 8(2):149 – 72,2000.

[52] H. H. Koester and S. P. Levine. Modeling the Speed of Text Entry with a WordPrediction Interface. IEEE Transactions on Rehabilitation Engnieering, 2(3):177–187, 1994.

[53] H. H. Koester and S. P. Levine. Keystroke Level Models for User Performance withWord Prediction. Augmentative and Alternative Communication, 13(4):239–257,1997.

91

References

[54] H. H. Koester and S. P. Levine. Model Simulation of User Performance with WordPrediction. Augmentative and Alternative Communication, 14(1):25–36, 1998.

[55] A. F. Kramer, W. R. Boot, J. S. McCarley, M. S. Peterson, A. Colcombe, andC. T. Scialfa. Aging, Memory and Visual Search. Acta Psychologica, 122(3):288–304, 2006.

[56] J. R. Lewis, P. J. Kennedy, and M. J. LaLomia. Development of a Digram-basedTyping Key Layout for Single-Finger/Stylus Input. In Proceedings of 43rd AnnualMeeting of The Human Factors and Ergonomics Society, pages 415–419, 1999.

[57] Lipik. A Predictive Text Input System. [Online] Available: http://www.lipik.in.accessed on August 2012.

[58] G. D. Logan. Attention in Character Classification Tasks: Evidence for TheAutomaticity of Component Stages. Journal of Experimental Psychology: General,107(1):32 – 63, 1978.

[59] G. D. Logan. On The Use of Concurrent Memory Load to Measure Attentionand Automaticity. Journal of Experimental Psychology: Human Perception &Performance, 5(2):189 – 207, 1979.

[60] R. H. Logie. Visuo-spatial Working Memory. Lawrence Erlbaum, Hove, UK, 1995.

[61] G. L. Lohse. A Cognitive Model for Understanding Graphical Perception. HumanComputer Interaction, 8(4):353–388, Dec 1993.

[62] S. J. Luck and E. K. Vogel. The Capacity of Visual Working Memory for Featuresand Conjunctions. Nature, 390(6657):279 – 281, 1997.

[63] I. S. MacKenzie. Fitts Law as a Research and Design Tool in Human ComputerInteraction. Human Computer Interaction, 7(1):91–139, 1992.

[64] I. S. MacKenzie and R. W. Soukoreff. Text Entry for Mobile Computing: Modelsand Methods, Theory and Practice. Human Computer Interaction, 17(2-3):147–198, 2002.

[65] I. S. MacKenzie and K. Tanaka-Ishii. Text Entry Systems: Mobility, Accessibility,Universality. Morgan Kaufmann Inc., San Francisco, CA, USA, 2007.

[66] I. S. MacKenzie and S. X. Zhang. The Design and Evaluation of a High-performance Soft Keyboard. In Proceedings of the SIGCHI Conference on Human

92

References

Factors in Computing Systems, CHI ’99, pages 25–31, Pittsburgh, United States,May 15–20, 1999.

[67] I. S. MacKenzie, S. X. Zhang, and R. W. Soukoreff. Text Entry using SoftKeyboards. Behavior & Information Technology, 18(4):235–244, 1999.

[68] E. McSorley and J. M. Findlay. Visual Search in Depth. Vision Research,41(26):3487–3496, 2001.

[69] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller.Equation of State Calculations by Fast Computing Machines. The Journal ofChemical Physics, 21(6):1087–1092, 1953.

[70] D. C. Montgomery and G. C. Runger. Applied Statistics and Probability forEngineers. John Wiley & Sons, 5th edition, 2010.

[71] A. L. Nagy, K. E. Neriani, and T. L. Young. Effects of Target and DistractorHeterogeneity on Search for a Color Target. Vision Research, 45(14):1885–1899,2005.

[72] R. Näsänen and H. Ojanpää. Effect of Image Contrast and Sharpness on VisualSearch for Computer Icons. Displays, 24(3):137–144, 2003.

[73] M. B. Neider and G. J. Zelinsky. Searching for Camouflaged Targets Effects ofTarget-Background Similarity on Visual Search. Vision Research, 46(14):2217–2235, 2006.

[74] S. Oh and M. Kim. The Role of Spatial Working Memory in Visual SearchEfficiency. Psychonomic Bulletin & Review, 11(2):275 – 281, 2004.

[75] OmicronLab. Avro Keyboard. [Online], Available: http://www.omicronlab.com/avro-keyboard.html, 2003. accessed on December 2012.

[76] J. Palmer, P. Verghese, and M. Pavel. The Psychophysics of Visual Search. VisionResearch, 40(10):1227–1268, 2000.

[77] M. Pomplun, E. M. Reingold, and J. Shen. Area Activation: A ComputationalModel of Saccadic Selectivity in Visual Search. Cognitive Science, 27(2):299–312,2003.

[78] GNU Project. GNU Octave. [Online], Available: http://www.gnu.org/software/octave/. accessed on June 2012.

93

References

[79] M. Raynal and N. Vigouroux. Genetic Algorithm to Generate Optimized SoftKeyboard. In Proceedings of Human Factors in Computing Systems - ExtendedAbstracts, pages 1729 – 1732, Portland, USA, 2005.

[80] D. D. Salvucci. An Integrated Model of Eye Movements and Visual Encoding.Cognitive Systems Research, 1(4):201 – 220, 2001.

[81] D. D. Salvucci. Predicting the Effects of In-Car Interface Use on DriverPerformance: An Integrated Model Approach. International Journal of Human-Computer Studies, 55(1):82 – 107, 2001.

[82] D. Samanta, S. Sarcar, and S. Ghosh. An Approach to DesignVirtual Keyboards for Text Composition in Indian Languages.International Journal of Human-Computer Interaction, [Online] Available:http://dx.doi.org/10.1080/10447318.2012.728483, 2012.

[83] J. D. Schaffer. Multiple Objective Optimization with Vector Evaluated GeneticAlgorithms. In Proceedings of the 1st International Conference on GeneticAlgorithms, pages 93 – 100, New Jersey, USA, 1985.

[84] R. Sivaraj and T. Ravichandran. A Review of Selection Methods in GeneticAlgorithm. International Journal of Engineering Science and Technology,3(5):3792 – 3797, 2011.

[85] A. J. Smola and B. Schölkopf. A Tutorial on Support Vector Regression. Statisticsand Computing, 14(3):199–222, 2004.

[86] W. Soukoreff and I. S. MacKenzie. Theorectical Upper and Lower Bounds onTyping Speed using a Stylus and Soft Keyboard. Behaviour & InformationTechnology, 14(6):370–379, 1995.

[87] N. Srinivas and K. Deb. Muiltiobjective Optimization using Nondominated Sortingin Genetic Algorithms. Evolutionary Computation, 2(3):221 – 248, 1994.

[88] N. A. Taatgen, H. V. Rijn, and J. R. Anderson. An Integrated Theory ofProspective Time Interval Estimation: The Role of Cognition, Attention andLearning. Psychological Review, 114(3):577 – 598, 2007.

[89] TDIL. iLeap - Multilingual Word Processor. [Online] Available: http://www.cdac.in/html/gist/products/ileap.asp. accessed on June 2011.

94

References

[90] L. Teo and B. E. John. Towards a Tool for Predicting Goal-directed ExploratoryBehavior. Human Factors and Ergonomics Society Annual Meeting Proceedings,52(5):950–954, 2008.

[91] I. Tollinger, R. L. Lewis, M. McCurdy, P. Tollinger, A. Vera, A. Howes, andL. Pelton. Supporting Efficient Development of Cognitive Models at Multiple SkillLevels: Exploring Recent Advances in Constraint-based Modeling. In Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems, CHI ’05,pages 411–420, Portland, USA, Apr 2–7 2005.

[92] T. B. Trafalis. Support Vector Machine for Regression and Applications toFinancial Forecasting. In Proceedings of the International Joint Conference onNeural Networks, pages 348–353, Norman, USA, 2000.

[93] A. M. Treisman and G. Gelade. A Feature-Integration Theory of Attention.Cognitive Psychology, 12(1):97–136, 1980.

[94] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 2nd edition,2000.

[95] J. M. Wolfe. Guided Search 2.0: A Revised Model of Visual Search. PsychonomicBulletin & Review, 1(2):202–238, 1994.

[96] J M. Wolfe. What can 1 Million Trials Tell Us About Visual Search? PsychologicalScience, 9(1):33–39, 1998.

[97] J. M. Wolfe. Guided Search 4.0: Current Progress with a Model of Visual Search.Integrated Models of Cognitive Systems, 25(3):99–119, 2007.

[98] J. M. Wolfe, K. R. Cave, and S. L. Franzel. Guided Search: An Alternative to theFeature Integration Model for Visual Search. Journal of Experimental Psychology:Human Perception and Performance, 15(3):419–433, 1989.

[99] G. F. Woodman, E. K. Vogel, and S. J. Luck. Visual Search Remains Efficientwhen Visual Working Memory is Full. Psychological Science, 12(3):219 – 224, 2001.

[100] A. H. Wright. Genetic Algorithms for Real Parameter Optimization. Foundationsof Genetic Algorithms, 1:205 – 218, 1991.

[101] C. H. Wu, J. M. Ho, and D. T. Lee. Travel Time Prediction with Support VectorRegression. IEEE Transactions on Intelligent Transportation Systems, 5(4):276–281, 2004.

95

References

[102] H. Yang, L. Chan, and I. King. Intelligent Data Engineering and AutomatedLearning, chapter Support Vector Machine Regression for Volatile Stock MarketPrediction, pages 391–396. Springer Berlin Heidelberg, 2002.

[103] P. Y. Yin, F. Glover, M. Laguna, and J. X. Zhu. Cyber Swarm Algorithms- Improving Particle Swarm Optimization using Adaptive Memory Strategies.European Journal of Operational Research, 201(2):377 – 389, 2010.

[104] P. Y. Yin and E. P. Su. Cyber Swarm Optimization for General KeyboardArrangement Problem. International Journal of Industrial Ergonomics, 41(1):43– 52, 2011.

[105] S. Zhai, M. Hunter, and B. A. Smith. Performance Optimization of VirtualKeyboards. Human Computer Interaction, 17(2):229–269, 2002.

[106] S. Zhai, P. Kristensson, and B. A. Smith. In Search of Effective Text InputInterfaces for off the Desktop Computing. Interacting with Computers, 17(3):229–250, 2005.

[107] E. Zitzler and L. Thiele. Multiobjective Evolutionary Algorithms: A ComparativeCase Study and the Strength Pareto Approach. IEEE Transactions onEvolutionary Computation, 3(4):257 – 271, 1999.

96