Embed Size (px)
Citation preview
Innovations in Engineering
Operationally Relevant Metrics for Analyst Test Beds:
Evaluating How Analysts Make Use of
their Tools to Inform Tool Development, Integration, and Adaptation
Joshua Poore, Ph.D., Laura Mariano, Jana Schwartz, Ph.D. {jpoore, lmariano, jana}@draper.com
2/11/15
2
Monolithic suites require training and embedded support to align capability
with current needs.
Monolithic Software Suites
Palantir Analyst Notebook (i2)
3
Application suites are more agile to emergent needs, but applications are
agnostic to one another; not clear how to vertically integrate them for users.
Application Suites
4
Canon is to collect interpretable data at a cost to ecological validity.
Traditional Human Signals
ECG GSR
PPG
Eye Tracker
Analyst
5
Software is the new ecology; we can collect data from it.
Time-stamped, labeled activity logs give sequential user & system activity.
Software is a Sensor
6
Software Instrumentation is a process for logging user- and software-initiated functions (activities), and labeling them.
Making Software a Sensor
7
Instrumentation Made Easy
http://draperlaboratory.github.io/user-ale/
3
1
2
8
Usage in Analytic Software
adjusting x/y
plotting parameters
entering filter text
Ensemble modeling demonstrates how users integrate activities across a tool.
Tool-Related Task Difficulty.
…Predicts … Integration Across a Tool…
9
Software logs adds context for understanding user engagement and
experience.
Coherence in HCI
0
10
20
30
40
50
60
% V
aria
tion
in P
ost-
Task
Eng
agem
ent
Acco
unte
d Fo
r
Independent Variables and Additive Step-Wise Regression Coefficients
R^2 Step 2
R^2 Step 1
10
DARPA Open Catalog: http://www.darpa.mil/opencatalog/index.html
Draper Open Source Products: http://draperlaboratory.github.io/
Participate in the DARPA XDATA Online
Experiment! Contact: [email protected]; (617) 258-4023
Some of this work supported by AFRL Contract No. FA8750-12-C-0293.
Acknowledgements
{jpoore, lmariano, jana}@draper.com
