July 24, 2005 UM 2005, Edinburgh, UK 1

User Control over User Adaptation:

A Case Study

Xiaoyan Peng and Daniel L. Silver

Jodrey School of Computer ScienceAcadia University

Wolfville, Nova Scotia Canada B4P 2R6danny.silver@acadiau.ca

2

Introduction UAI = HCI + UM

Pro: Potential for more usable software interfaces Con: Complexity of system, user uncertainty

Controllability – degree of human control over the system remains a key factor:

Some researchers prefer maximum control Others suggest control can lead to distraction [1]

An adaptive intelligent email client is the application of choice:

Focus on predicting the priority of incoming email messages based on a learned UM

The predicted priorities can be used to filter low priority “Spam” email and identify high priority messages quickly

3

Background Prior work on UM for spam filtering [4] has focused on

performance of the learning algorithms Less attention given to the usability of the UAI based

system

Lack of a user’s perspective has led to significant barrier against UAI technology [2]

For example: automatic placement of legitimate email into a spam folder can be unacceptable to some users [5]

UAI can frustrate good HCI design – the interface may be perceived as a moving target that at times does not meet the expectations of the user [6]

We present a theoretical model of the relationship between the expectations of a user and the changing state of a UAI

4

Theory of User Interaction Expectation and Adaptation

Figure 1. Adaptation viewed as movement through an HCI state space

Space of HCI states

s

R

P

R2 R3R’

NowPast

Future

5

Theory of User Interaction Expectation and Adaptation Consider a space of HCI states as shown in Figure 1

Interaction states are topologically organized Similar states are proximal to each other

System adaptation is trajectory, P, through the space

Each point represents the system’s state of interaction with the user at a particular time

A user has a region of interaction expectation, R Preferably R contains the systems current state of

interaction, s |R|, is the number of interaction states within R If |R| = 1, then no variation from s will be tolerated by the

user; this user is very conservative in terms of adaptation If |R| = n then there are n states that will be acceptable to

the user; this user is more accepting of adaptation

6

Theory of User Interaction Expectation and Adaptation Ideally, as the system interface adapts, the

user shifts her R so as to centre it on the new s

This transition is not always in concert: If the system adapts too quickly, the user is left

behind at R2 If the system adapts too slowly, the user may

assume an interaction state too far in advance of the current s, at R3

In either case the user will not be satisfied with the system and task performance will suffer

The worst case is R’, a region of interaction space through which adaptation will never pass - the user is continually dissatisfied

7

Theory of User Interaction Expectation and Adaptation A UAI should provide the user with control over

adaptation: We advocate that user control should be exercised over the deployment of user models rather than their development

Model deployment requires minimal knowledge of the UM subsystem

Example: A user model for predicting incoming email priority

can be developed using information retrieval and machine learning methods [4]

Control over spam filtering can be provided by adjustable cut-off values that determine when the predicted priority of a message classifies it as legitimate or Spam

8

The Intelligent Email Client We have created an intelligent email client

and developed an intuitive user interface for controlling adaptation (see Figure 3)

The user model is trained to assign a value between 0 and 1 to each incoming email message

0 is lowest priority, 1 is highest priority The default priority (no user model) is 0.5

The user model is developed with a BP ANN system using email training examples from the current spam and legitimate email folders (spam messages are assign priority 0, all others 1)

9

The Intelligent Email Client

Figure 3. Major interface of the e-mail client

10

User Control over Adaptation There are two priority cut-off values, Suspect

and Spam, controlled by two GUI sliders (Figure 4):

Email with a priority value lower than the spam cut-off will be placed in the Spam folder.

Email with a priority value equal to or higher than the suspect cut-off will be filed into the Inbox folder.

Email with a priority equal to or higher than the spam cut-off and lower than the suspect cut-off will be put in a Suspect folder.

11

User Control over Adaptation

Figure 4. Interface of cut-off controlling function

12

User Control over Adaptation

Using the GUI sliders, a new or conservative user can select cut-off values that curtail the UAI’s automated classification of legitimate and spam messages (thus reducing risk)

A more experienced user can establish cut-offs that give the UAI greater freedom to classify email messages

As the cut-offs are adjusted, the system automatically reallocates the messages to the Inbox, Suspect and Spam folders

Provides immediate feedback to the user on their choice of cut-offs

Adaptation of the systems interaction state can be kept within the user’s current region of interaction expectation.

The approach will direct the most important legitimate email to the Inbox folder

The Suspect folder can be cleaned up periodically, sorting legitimate and spam email - it is this process that provides data for improving the user model.

13

Empirical Study Objective: To demonstrate that user control over

appropriate aspects of UAI can improve the usability and user satisfaction

Scenario: Each subject pretends to be a secretary for a professor. He or she must classify the incoming email (initially received in either the Inbox, Suspect or Spam folder) by moving the messages into one of six relevant folders including the Spam folder.

Performance of the email UAI is recorded in terms of: FP (false positives) = legitimate emails placed in the Spam

folder FN (false negatives) = spam email placed in the Inbox folder Overall error = FP + FN

Usability of the system is based on user surveys, post trial

14

Empirical Study - Method Three variants of the system were tried by

each subject and compared as per [2]: Variant N - no user model; subjects must manually

sort the Inbox email to legitimate & Spam folders Variant F – user model with fixed cut-off values;

user model automatically sorts email to the Inbox (legitimate), Suspect and Spam folders based fixed cut-off values set to optimally performing values (as determined by preliminary trials)

Variant A - user model with adjustable cut-off values; user model automatically sorts email to the Inbox (legitimate), Suspect and Spam folders based adjustable spam and suspect cut-off values. Subjects have control over the deployment of user model.

15

Empirical Study - Method A within-subject experimental design selected:

28 subjects selected from the university campus (ages 18-38)

Subjects expected to vary considerably in their use of the system and tolerance to adaptation

Wanted to collect comparative results/comments over all 3 variants

Overcoming experimental bias: Each subject used all three variants of the system in one of

two possible orders: variant N, F, then A or variant N, A, then F

A different subset of emails was used for each variant to prevent subjects from memorizing the content of messages.

Each subject was provided the same working environment. The instructions were provided by a power point

presentation with minimal input by a researcher

16

Empirical Study - Method Procedure:

Email data used was collected from a professor at Acadia over a 5 month timeframe in 2003 [4].

Three different subsets of 100 emails used for each of the three variants of the system: 50 legitimate and 50 spam messages

Variant N was always first - the subject was asked to manually sort the 100 emails to their respective folders without assistance of the user model

This acted as training data for developing a user model for predicting message priority for variants A and F

The FP and FN statistics were recorded for each subject and used to determine each variant’s performance

17

Empirical Study – Results 92.86% of subjects found the email client easier to use

after the UM was developed

Figure 6 shows the difference between the fixed cut-off and adjustable cut-off variants of the UAI for both orders of exp.

The fixed cut-off variant performed better on average with fewer overall misclassifications (p = 0.01, paired, two-tailed T-test)

However, 82.14% of subjects preferred the ability to adjust the Spam and Suspect cut-offs

78.57% felt that the cut-off adjustment increased the accuracy of email classifications

18

Figure 6. Misclassifications of variant A and F. Order of system variants used is defined by order of N, F and A in bar labels

Empirical Study – Results

19

Figure 7. Summary of survey results: preference for user control versus performance.

Empirical Study – Results

Subject ID: Brackets represents a subject who tried variant F before A; otherwise a subject who tried variant A before F.

20

Empirical Study – Results Those subjects above the line represent fewer

misclassifications under variant A than variant F The distance between an ID and the line indicates the

difference in performance between the two variants Examples:

Subject 15 preferred variant A, had 51 misclassifications with variant A, 50 with variant F; a difference of 1

Subject 14 preferred variant A, had 49 misclassifications with variant A, 5 with variant F; a difference of 44

Subject 10 preferred variant F, had 11 misclassifications with variant A, 22 with variant F; a difference of -11

21

Empirical Study – Results 67.9% of the subjects (19/28) preferred variant

A over F Subject 14 was an extreme case of where a user

preferred control even though it reduced system performance

This shows a strong desire to remain in control

Typical responses for those who preferred adjustable cut-offs were:

“It helps me to control how the emails will be separated”

“It is good to add user’s point view to the system” “I like the feeling of control”

22

Empirical Study – Results Majority of subjects liked adaptation as long as

they felt in control: 95.65% of subjects who liked cut-off adjustment,

preferred FN (spam email messages classified as legitimate)

Adjustable cut-offs allow the user to err on the side of FN classifications even if this reduces overall performance

Of the subjects who responded “do not know” or

“disagree” to cut-off adjustment: 80% preferred FP (legitimate emails classified as spam) The fixed default cut-off values worked well for that

purpose, the subjects recognized this and preferred it

23

Conclusions and Future Work Theory: Users will be satisfied with a UAI provided the

interaction state of the system is maintained within the current region of user expectation

If the system’s interaction falls outside of this region of expectation then user satisfaction and performance will degrade

Solution: Give the user control over aspects of adaptation that limit changes in interaction state.

In the case of the email client, the user controls the cut-off at which emails are considered legitimate or spam.

The results of an empirical study using 28 subjects demonstrated that performance and user satisfaction is improved with user control over adaptation

User satisfaction is higher even when control leads to reduced performance (greater numbers of email misclassifications)

We are currently working on a related problem of automatically classifying emails to one of several category folders

24

References1. Kay, J.: Learner control, Proceedings of User Modeling & User-

Adapted Interaction (2001)2. Jameson A., and Schwarzkopf, E.: Pros and Cons of

Controllability: An Empirical Study, Adaptive Hypermedia and Adaptive Web-Based Systems: Proceedings of AH (2002)

3. Crawford, E., Kay, J., and McCreath E.: An Intelligent Interface for Sorting Electronic Mail, IUI’02, San Francisco, California, USA (2002) 13-16

4. Fu, C.: User Modelling for an Adaptive System: an Intelligent Email Client, Master Thesis, Jodrey School of Computer Science, Acadia University, Wolfville, Canada (2003)

5. Crawford, E., Kay, J., and McCreath, E.: Automatic Induction of Rules for Email Classification, Proceedings of 6th Australian Document Computing Symposium (2001)

6. Cranor, L., F.: Designing a Privacy Preference Specification Interface: A Case Study, Proceedings of the Workshop on HCI and Security Systems, CHI2003, Florida (2003)

25

Introduction Objective: To improve usability and

performance of User Adapted Interfaces (UAI) UAI = HCI + UM

Pro: Potential for better software interfaces Con: Complexity of interface, user uncertainty

Controllability – degree of human control over the system remains a key factor:

Some researchers prefer maximum control Others suggest control can lead to distraction [1] There is a deficiency of evidence from users

perspective on adaptation and controllability [2]

26

Introduction An adaptive intelligent email client is the

application of choice: Offers lots of functionality Example data is readily available Knowledgeable test subjects available

Focus on predicting the priority of incoming email messages based on a learned UM

The predicted priorities can be used to: Filter low priority “spam” email Identify high priority messages quickly

27

Synopsis Problem: The mix of automation and user

control is a major issue for a successful UAI

Theory: A UAI will be acceptable as long as the state of system interaction is within the current region of user interaction expectation

Application: Spam filtering via a learned user model that predicts user’s sense of email priority

Solution: The user is provided with control over email filtering via a novel method of adjusting priority cut-offs

28

The Intelligent Email Client

Figure 2. System architecture of the intelligent email client

29

Empirical Study - SubjectSurvey