Probabilistic User interfaces
Roderick Murray-SmithDepartment of Computing Science,
University of Glasgow &Hamilton Institute, NUI Maynooth
[email protected]://www.dcs.gla.ac.uk/~rod
With John Williamson, Parisa Eslambolchilar, Andy Crossan, Steve Strachan,
Vuokko Lantz & Stephen Brewster
Overview
1. Dynamics & Statistics in Interaction
2. Uncertain, Dynamic Feedback mechanisms
3. Demos1. Hex Entry: Intelligent
adaptation of handling qualities during interaction
2. Pointing without a pointer: Control models in interaction design
4. Conclusions
Dynamics & Statistics in HCI?
• Why introduce dynamics – is that not harder?– We can only control what we can perceive.– Dependent on feedback, so upper limits on the
speed of change of display.– Dynamics allows us to slip in ‘intelligence’ which
couldn’t be done with a static interaction technique
• Why uncertain interaction?– Uncertainty in user’s mind about what to do next,
and system uncertain about user’s intentions.– With mobile devices, interaction with the user is now
continuous instead of discrete, and input devices are noisier.
Feedback & Inference
• A model of user interaction in a closed-loop system involving uncertainty.
• Feedback of the results of the inference process is provided to the users which they can then compare with their goals.
• Inference may often be about user’s beliefs, desires or intentions…
Display and control
• Display augmentation– Improve input to human to simplify control
task
• Control augmentation– Change the effective dynamics between
control input and system output
The display is to provide the user with information needed to exercise control. i.e. predict consequences of control alternatives, evaluate status and plan control actions.
Ambiguous displays
• Used in psychophysics experiments (e.g Körding & Wolpert 2004)• Transfer idea to user interface design. If the system is uncertain
about inputs or user intentions, present data in an appropriately ambiguous fashion.
• Does it regularise user behaviour & improve usability appropriately?
• Pattern recognition and displays are interdependent and should be developed together
Mobile & Acceleration Sensing
• HP IPAQ• Xsens accelerometer
– 3 DOF linear accelerometer
– Samples up to ~100 Hz– Weight ~10.35g
• Potential for one handed / screen free interaction
• Mobile devices used in many contexts, subject to varying levels of disturbance– Ideal testbed for
probabilistic interaction– Small screen– Vibrotactile/audio
feedback
Audio displays - Granular Synthesis
• A structured approach to probabilistic audio
• Quantum theory of sound
• Accumulate short grains from waveforms sources
• Select grains according to some probability distribution
Sonification of Probability Distributions
• Straightforward sonification of probabilistic models – Associate distributions with collections of source waveforms– Continuous distributions can be sonified via sampling from a
parametric synthesis algorithm
• Produces a smooth texture representing the changing probabilities
Feedback for gesture recognition
• Mapping from an input trajectory to an audio display via a number of gesture recognition models.
• Each gesture is associated with a model and the output probabilities are fed to the synthesis algorithm.
• Can be combined with direct sonification of gesture movements.• Users can explore functionality
– Feedback from the goals, depends on accuracy of gesture & estimated skill of user– User behaviour can be ‘shaped’, starting with simple, blurred gestures and progressing
to sharper, more complex expression.
Outline of the gesture recognition and sonification system
Parametric model of gestures
(Dynamical motor primitives, Locally
Weighted Learning)
Model parameters fitted to the current
observations
Gesture recognition
engine
Audio feedback
generation
Audio source 1
Audio source 2
Audio source 3
…
Audio source N
Audio feedback
generation
Acceleration measuremen
ts of the phone’s
movements
Sonification of the recognition result and its
confidence (Granular Synthesis)
Sonification of the performed
gesture (Granular Synthesis)
A posteriori probabilities for different gestures
Feedback for gesture recognition
• Benefits of feedback– Feedback on how the gesture recognition engine is
performing, i.e. recognition result and confidence – Gives the user insight into the pattern recognition
mechanics.– Feedback on how the user is performing, i.e. sonification of
the users actions • Coupling of gesture recognition and feedback generation
– Simple, parametric representation for the observed gesture data
– Gesture model parameters can act as pattern features of the recognition engine
– Recognition engine produces a posteriori probabilities for each gesture class
– Parametrized feedback generation on the basis of gesture feature vectors or classification results, e.g. Granular Synthesis of audio/vibro sources
Haptic Targeting• Spatially & Time
Varying Vector Field• Directional Grains
using Vector Summation
• Highlights Areas of High Uncertainty
Quickening/Predictive displays
• Augmentation of a display with predictive information– “Experience indicates that, by using a properly designed predictor instrument, a novice can in 10
minutes or less learn to operate a complex and difficult control system as well as or better than even the most highly skilled operator using standard indicators”, from Kelley, C.R. “Manual and Automatic Control” 1968
• Standard technique in manual control systems– e.g quickening of helicopter displays, Showing derivatives of current state
• Quicken the probabilistic audio display– Add predictions of change of probability to the display, e.g. if derivative of probability is
increasing, decrease if derivative is decreasing…
– Allows users to determine when they are moving towards regions of high probability; aids in targeting of modes
• Models such as Gaussian processes allow derivative uncertainty to be included.
td
dpkpv
i
in
ii
1
Demo: Nonlinear dynamics & Monte-Carlo simulations
Feedback conclusions
• Provided examples of granular synthesis for sonifying probabilistic interfaces– with quickening, & Monte Carlo predictions can
help improve interfaces to a continuously controlled environment which involves uncertainty
– can be extended to force- and vibrotactile feedback
– Helps users learn gestures for mobile devices– Allows flexibility to give feedback about different
orders of derivatives, applications in rehabilitation engineering.
Hex Entry: Intelligent adaptation of handling qualities during
interaction• Flexibility brought by dynamic models allows
intelligent interaction, – handling qualities of the dynamics of the interface are
adapted depending on current inferred user goals. – actions require less effort, equivalent to a lower bit rate
in communication terms, the more likely the system’s interpretations of user intentions.
• Serves as example of continuous interaction system, – with gestures, augmented control and potential for
audio feedback• Predictions of future trajectories
– Could be linked to sound as MC samples• Quickening via velocity & acceleration info
Hex: The Aims
• A continuous input system – all entry is one single smooth sequence
• Incorporating a probabilistic model to represent uncertainty and increase performance
• But with a structure that can be learned – gestures must be repeatable– Support transition from
novice (tightly closed control loop) to expert (open-loop, learned behaviour)
Current most probable word
Future path
Cursor
Current entry
Example Words
“Hello” “GIST” “Hexagons”
Augmented control
• System provides augmented, nonlinear control– Don’t perform actions for user - help user reproduce ideal
behaviour themselves– Adapts to context, changing properties of the control system– Minimises effect of disturbances and errors
• Goal is that although initial use is very dependent on feedback, user learns open-loop gesture-like behaviour.
Nonlinear dynamics
• Vector field adapts to current context– In this case ‘Q’ has been
chosen
• Handling qualities improved appropriately– Makes it easy to get to ‘U’
Current mostprobable word
Future path
CursorCurrententry
Semantic Pointing (Blanch, Guiard, Beaudouin-Lafon 2004.)
• Motor space and Display space have different properties
• Control-Display ratio adapted depending on proximity of target
Predictive control & Word Autocomplete
• Also show top k most probable paths – fitting a cubic spline through hexagon
centres
Hex Conclusions: From control to gestures
• Progress from feedback control to open-loop gesturing– With audio/tactile feedback, can be less reliant on
screen– Progression to higher-order control– Provides new users with a way to gradually learn
system.• Gestures can be used for simple common tasks
(Autocomplete, delete etc)• Dynamic representation allows ‘intelligent
systems’ make life easier for the user.• Current system would need a lot of
development before being a natural text-entry system (only ca. 17 words per minute)
The Selection Problem
• How can we determine user intention?• Evaluate probability distribution over potential goals• Closed-loop interaction
– Goals are negotiated with the system in continuous time– Continuous feedback on user state with respect to goals
• Selection for devices for which pointing is non-intuitive
Perceptual Control Theory
• How can we determine intention?• Perceptual Control Theory – Powers, et al.
– Fundamental hypothesis: Humans act to control their perceptions
– Test for this control behaviour
• Hypothesis: Identify intentions via correlations between input and known disturbance patterns
System
Feedback
Control loopGoal
An Agent Perspective
• Reformulate selection in terms of agents– Each goal or item is considered an independent agent
• Agents probe user – “experiment”, look for response– Evaluate probabilities p(selectedi)– Akin to MCMC Sampling from users mind!
Example: Movement
a
ei fp
1
a
e
1a
e
If no control,
If controlled,
Hence, we have:
Agent Disturbance User Control Result
Test: Compare distribution of histories over some time window
Interpretations• Can be seen as control, damping,
imitation, gesture recognition or excitation
– Control/damping– Imitation of the motion of the
disturbance (pursuit task)– Gesture recognition with dynamically
created gestures– User excites “modes” of the object,
inducing a meaningful disturbance in the object
Feedback• Real-time feedback on potential goals
– Visual example in demo– Mapping entropy to audio dissonance
Log P(i)
Entropy
Sampling from user’s mind
• E.g. Bubbles (Schyns et al 2003)
• Adapt idea for use with fisheye interfaces and continuous interaction instead of discrete accept/reject.
PCT Conclusions
• Probabilistic selection method suitable for non-conventional sensing and feedback systems, supporting real-time feedback on progress towards potential goals, incorporating models from manual control theory to optimize performance
• Much scope for extending and applying these general ideas to practical interfaces– More sophisticated user models– Disturbance/experiment design– Feedback design…
Outlook
• Dynamics allow intelligence to be sandwiched into an interface– ‘look-and-feel’ of an interface, ‘noisy channel’, or in
control terms, the ‘adaptive handling qualities’?• Augment the display or the control?• Adapt ambiguity in display to context
– E.g. walking, in bus, at desk• Exciting overlap between:
– human motor control, – Statistics– manual control/dynamic systems– human-computer interaction
Other fun things…