View
251
Download
0
Category
Preview:
Citation preview
Yury Puzis, Yevgen Borodin, I.V. Ramakrishnan
Complexities of Practical Web Automation
Stony Brook University2015
NSF Grant No. IIS-1218570
Contents
Goal: help design practical web automation tools by sharing observational experience
❖ Human-Computer Interaction Perspective
❖ Technical Perspective
❖ Example: Automation Assistant
❖ Conclusion
Why Web Automation?
❖ Problem: non-visual browsing is hard
❖ It is hard (or impossible) to find relevant information and easy to become overwhelmed by what is irrelevant
❖ There are many shortcuts (gestures) to learn and hard (or impossible) to accomplish non-trivial tasks
❖ Web automation has the potential to enable visually impaired users to breeze through Web browsing tasks that beforehand were slow, hard, or even impossible to achieve
Observation
User Environment Web Automation Tool
Browsing Actions Events
User Environment Web Automation Tool
Events Automation Instructions
Automation
Maximizing Trust
❖ Gaining and maintaining user trust is the cornerstone of web automation: even a few disasters is a big problem
❖ The user needs to know and influence what will happen (review, parameterize, choose) and what has happened (review, revert, recover) at all times
❖ Failure is inevitable and has to be graceful: terminate automation, ignore failed action, take corrective action, or suggest the user to take corrective action
Minimizing End-To-End Cost❖ Cognitive load and operation time must be end-to-end
lower when using automation than otherwise
❖ Web automation costs: managing creation, execution and consequences of automation; context switching
❖ Screen-Reader and browser costs: many and are well known, including the need to plan complex sequences actions by memory or execute exhaustive search and guess, guess, guess
❖ In conflict with the need to maximize trust
Dealing with Uncertainty
Goal: automate user intent without resorting to handcrafting scripts (programming), interpret environment reaction
Problem: we can only guess
❖ Semantics of user browsing actions
❖ Semantics of environment events
❖ Semantics of webpage elements
Making Observations
❖ Goal: make meaningful observations from events
❖ Problem: browsing actions can trigger multiple (including cascading) events, and there are different types of events: e.g., shortcut press -> JavaScript call -> DOM mutation -> virtual cursor movement
❖ Problem: over time, an event may change its semantics (same event - different results) or implementation (different event - same results)
Addressing Webpage Elements
❖ Goal: identify target webpage element
❖ Problem: most addressing approaches are designed to query DOM for elements at the specified address, but we need to query DOM for address of the specified element
❖ Solutions: sloppy programming, machine learning, etc. but no unbreakable approaches exist
Detecting Action Completion
❖ Goal: wait for action to complete (succeed or fail) before continuing to interact with the user & the environment
❖ Problem: no standard way to specify action completion; cascading, asynchronous and scheduled JavaScript events make things harder
❖ Solutions: listen to all relevant JavaScript events through callback functions; timeout; wait for predefined DOM mutations / value changes (success or failure)
Example: Automation Assistant❖ Observes everything the user is doing (no macros)
❖ Guides the user through browsing tasks step-by-step
❖ suggests several alternative browsing actions based on user’s prior actions
❖ automates only one action at a time
❖ each set of suggestions is explicitly requested, each action is explicitly chosen, each outcome is reviewed
❖ No context switch between automation and screen-reading
Puzis Y., Borodin Y., Puzis R., Ramakrishnan I. V., Predictive Web Automation Assistant for people with vision impairments. WWW '13.
Conclusion❖ There are some successes but automation is not there yet
❖ The biggest technical challenge is uncertainty which stems from lack standardization
❖ The biggest HCI challenges are building trust and keeping things “cheap”
❖ The HCI aspect of this talk is, to a large extent, applicable to all automation tools, not just web automation. It is also applicable to all users not just the visually impaired users (think handheld, wrist devices)
Recommended