Context Aware Paper- based Review Instrument: A User Study Presented by: Dipti Shah, Hoda Homayouni, Daniel Belcher

Context Aware Paper-Context Aware Paper-based Review Instrument: based Review Instrument: A User StudyA User Study

Presented by: Dipti Shah, Hoda Presented by: Dipti Shah, Hoda Homayouni, Daniel BelcherHomayouni, Daniel Belcher

AgendaAgenda

Introduction of CAPRIIntroduction of CAPRI Related Works and MotivationRelated Works and Motivation Experimental DesignExperimental Design DiscussionsDiscussions

ObjectiveObjective

Evaluating CAPRI (in terms of Evaluating CAPRI (in terms of efficiency and perceptual efficiency and perceptual measures) by comparing it to measures) by comparing it to two other review methodstwo other review methods

Traditional Review SystemTraditional Review System

SortingSorting Memory workMemory work Errors Errors TimeTime CollaborationCollaboration

What is CAPRI ?What is CAPRI ?

Context Aware Paper Based Context Aware Paper Based Review InstrumentReview Instrument

Tangible and augmented user Tangible and augmented user interface for architectural interface for architectural document review document review

CAPRI SystemCAPRI System

Table topTable top PlanPlan PapersPapers ProjectorProjector CameraCamera Computer with a Computer with a

software software program (Java)program (Java)

CAPRI ProcessCAPRI Process

Input InterfaceInput Interface Medium Medium

DetectorDetector Context Context

InterpreterInterpreter Output InterfaceOutput Interface

Input InterfaceInput Interface

Menu SelectionMenu Selection

Grab and MoveGrab and Move

Related WorksRelated Works

CAPRI is an CAPRI is an augmentedaugmented and and tangibletangible environment for environment for facilitating facilitating documentdocument reviewreview using design objects that are using design objects that are the fundamental computational the fundamental computational elements. elements.

Document Review SystemsDocument Review Systems

Document review is a Document review is a cognitively challenging process.cognitively challenging process.

The “hypertext-like” aspect of The “hypertext-like” aspect of CAPRI decreases the reviewers’ CAPRI decreases the reviewers’ cognitive load. It also saves the cognitive load. It also saves the reviewer time and effort in reviewer time and effort in sorting and presenting the sorting and presenting the relevant details.relevant details.


Architectural documents are Architectural documents are often reviewed by different often reviewed by different people. Thus, the issue of people. Thus, the issue of authority always exist in design authority always exist in design review process.review process.

CAPRI affords diverse teams to CAPRI affords diverse teams to directly and intuitively interact directly and intuitively interact with the design document.with the design document.


Related WorksRelated Works JANUSJANUS is based on a model of a is based on a model of a

computer supported design that computer supported design that utilizes a knowledge-based utilizes a knowledge-based approach to link a graphic approach to link a graphic construction system to construction system to hypertext.hypertext.

JANUSJANUS

Tangible User InterfacesTangible User Interfaces

Tangible User Interfaces (TUI) Tangible User Interfaces (TUI) are interfaces that couple digital are interfaces that couple digital information to physical objects information to physical objects and environments. and environments.

By using socially accepted media, By using socially accepted media, CAPRI relies on natural CAPRI relies on natural behaviors to support a complex behaviors to support a complex task.task.

Tangible User InterfacesTangible User Interfaces

Related WorksRelated Works The The metaDESKmetaDESK

system system focuses on the focuses on the use of tangible use of tangible objects as objects as driving driving elements of elements of HCI.HCI.

Augmented Reality InterfacesAugmented Reality Interfaces

Augmented Reality (AR) is an Augmented Reality (AR) is an emerging platform for emerging platform for interacting with computers, interacting with computers, contextualizing virtual objects in contextualizing virtual objects in a physical world.a physical world.

CAPRI is an interface that has CAPRI is an interface that has almost no learning curve. almost no learning curve. Unifying the tasks space, it Unifying the tasks space, it encourages focus on the design encourages focus on the design discussion instead of the discussion instead of the mechanics of interactions.mechanics of interactions.

Augmented Reality InterfacesAugmented Reality Interfaces

Related WorksRelated Works ATIATI is an interface is an interface

device for distributed device for distributed design review that design review that fuses input and output fuses input and output that can be handled, that can be handled, passed back and forth, passed back and forth, inspected visually, inspected visually, positioned relative to positioned relative to other objects and other objects and even assembled or even assembled or disassembled disassembled

with them. with them.

MotivationMotivation

Predictions based upon the CAPRI Predictions based upon the CAPRI claims:claims:

More efficient (less time, better More efficient (less time, better performance, less errors)performance, less errors)

Less cognitive loadLess cognitive load Higher level of collaborationHigher level of collaboration

Experimental Design OverviewExperimental Design Overview

Three Conditions:Three Conditions: Condition 1: Paper-Based Condition (PB – 1)Condition 1: Paper-Based Condition (PB – 1) Condition 2: Computer-Supported Condition (CS -2)Condition 2: Computer-Supported Condition (CS -2) Condition 3: CAPRI Condition (CP – 3)Condition 3: CAPRI Condition (CP – 3)

45 Tasks (15 per condition)45 Tasks (15 per condition)

10-15 Subject (projected) = 100-150 data points per condition.10-15 Subject (projected) = 100-150 data points per condition.

Data recorded:Data recorded: Performance measures (time and error) recorded.Performance measures (time and error) recorded. Perceptual measures (post-experiment questionnaire) recorded.Perceptual measures (post-experiment questionnaire) recorded.

Analysis:Analysis: Compatible with three-way ANOVA design.Compatible with three-way ANOVA design.

Within and between conditions.Within and between conditions. Within and between subjects.Within and between subjects. Within and between tasks.Within and between tasks.

Paper-Based Condition (PB – 1)Paper-Based Condition (PB – 1)

Similar to “traditional paper-based” document review.Similar to “traditional paper-based” document review. Subjects may browse paper-based plots in any order.Subjects may browse paper-based plots in any order. Uses A1 sized plotted drawing set (plan and section).Uses A1 sized plotted drawing set (plan and section). Details are on Standard (8Details are on Standard (8½×11") paper.½×11") paper.

Computer-Supported Condition (CS – 2)Computer-Supported Condition (CS – 2)

A mix of “paper-based” and digital references.A mix of “paper-based” and digital references. Uses the same A1 sized plotted drawing set (plan and Uses the same A1 sized plotted drawing set (plan and

section).section). Details are on a computer monitor.Details are on a computer monitor. Subjects may browse details by clicking on a detail Subjects may browse details by clicking on a detail

thumbnail to enlarge in a main browser window.thumbnail to enlarge in a main browser window.

CAPRI Condition (CP – 3)CAPRI Condition (CP – 3)

A mix of “paper-based” and digital references, but with A mix of “paper-based” and digital references, but with automated linking and detail sorting.automated linking and detail sorting.

Uses the same A1 sized plotted drawing set (plan and Uses the same A1 sized plotted drawing set (plan and section).section).

Relevant details are projected on Standard (8Relevant details are projected on Standard (8½×11") paper½×11") paper.. Subjects may browse details by overlying paper on detail Subjects may browse details by overlying paper on detail

bugs.bugs.

ProcedureProcedure

Consent formsConsent forms InstructionsInstructions Training phaseTraining phase First condition (randomized)First condition (randomized) InstructionsInstructions Training PhaseTraining Phase Second condition (randomized)Second condition (randomized) InstructionsInstructions Training PhaseTraining Phase Third condition (randomized)Third condition (randomized) Post-experiment QuestionnairePost-experiment Questionnaire Debriefing (optional)Debriefing (optional) Participants paidParticipants paid

Pre-experiment Consent FormPre-experiment Consent Form

PURPOSE OF THE STUDYPURPOSE OF THE STUDY The study is to test different methods of architectural document The study is to test different methods of architectural document

review. With the help of participants, we will assess the review. With the help of participants, we will assess the effectiveness and usability of three different tools of document effectiveness and usability of three different tools of document review. review.

STUDY PROCEDURESSTUDY PROCEDURES The study involves sitting, reading and executing certain tasks. The study involves sitting, reading and executing certain tasks.

Participant will be given a learning phase in which to understand Participant will be given a learning phase in which to understand the task, then 45 discrete questions related to architectural the task, then 45 discrete questions related to architectural drawings will be presented. drawings will be presented.

[…][…]

RISKS, STRESS, OR DISCOMFORTRISKS, STRESS, OR DISCOMFORT This study involves sitting for more than an hour, which might This study involves sitting for more than an hour, which might

cause discomfort to some subjects. There is also a possibility that cause discomfort to some subjects. There is also a possibility that a participant may experience mild eye-strain while looking at the a participant may experience mild eye-strain while looking at the computer screen. computer screen.

If any of the above symptoms occur, and the participant does not If any of the above symptoms occur, and the participant does not wish to continue, they may alert the attending researcher and wish to continue, they may alert the attending researcher and immediately stop the experiment without penalty.immediately stop the experiment without penalty.

WRITTEN PERMISSION MUST BE GAINED BEFORE EXPERIMENT.WRITTEN PERMISSION MUST BE GAINED BEFORE EXPERIMENT.

InstructionsInstructionsExample from CAPRI condition:Example from CAPRI condition:““In this segment, you are asked to find answers to a In this segment, you are asked to find answers to a

set of questions relating to an architectural drawing set of questions relating to an architectural drawing set. The relevant details necessary to answer each set. The relevant details necessary to answer each of the questions are contained in the drawing set. of the questions are contained in the drawing set. Your task is to read the questions and find the Your task is to read the questions and find the answer.” answer.”

““In this segment, you will use a standard paper In this segment, you will use a standard paper drawing and pieces of standard paper to query the drawing and pieces of standard paper to query the drawing for detail. When the upper right-hand drawing for detail. When the upper right-hand corner of a piece of paper is placed on a bug, the corner of a piece of paper is placed on a bug, the relevant detail is projected from above onto the relevant detail is projected from above onto the page…”page…”

““Please answer all the questions as quickly and Please answer all the questions as quickly and accurately as possible.”accurately as possible.”

Sample TasksSample Tasks

““What is the distance between the 1What is the distance between the 1stst step and the end of the hand-rail on step and the end of the hand-rail on stair 3B?”stair 3B?”

““What kind of sign is in between the What kind of sign is in between the two doors in the east side of the two doors in the east side of the lobby?”lobby?”

““What is the width of central aisle of What is the width of central aisle of the auditorium?”the auditorium?”

““On the southern wall of the On the southern wall of the vestibule seminar room, what is the vestibule seminar room, what is the width of the whiteboard?”width of the whiteboard?”

Controls - LearningControls - Learning COUNTERBALANCED CONDITION ORDERSCOUNTERBALANCED CONDITION ORDERS

First ConditionFirst Condition Second ConditionSecond Condition Third ConditionThird Condition

Subject 01Subject 01 22 33 11















KEYKEY

1 = Paper-Based (PB)1 = Paper-Based (PB)2 = Computer-2 = Computer-

Supported (CS)Supported (CS) 3 = CAPRI (CP)3 = CAPRI (CP)

Data Collected: PerformanceData Collected: PerformanceTwo performance measures:Two performance measures: Time (in milliseconds)Time (in milliseconds) Error (correct/incorrect)Error (correct/incorrect)

Measurement Systems:Measurement Systems: Time: Participants press a key to start the Time: Participants press a key to start the

time trial and again when finished. time trial and again when finished. Subjects may pause the experiment at Subjects may pause the experiment at any time, but not continue with tasks.any time, but not continue with tasks.

Error: Participants record their answers to Error: Participants record their answers to questions in pencil on a worksheet. All questions in pencil on a worksheet. All errors are hand-checked and recorded. errors are hand-checked and recorded.

Data Collected: PerceptualData Collected: Perceptual The perceptual measures were recorded in the form of a The perceptual measures were recorded in the form of a

subjective post-experiment questionnaire.subjective post-experiment questionnaire.

All questions are standard Likert-scale rankings.All questions are standard Likert-scale rankings.

Forced choice, no center (6-level scale)Forced choice, no center (6-level scale)

4.) “Locating a detail bug in the plan drawing is an easy task.”4.) “Locating a detail bug in the plan drawing is an easy task.” 1 21 2 3 3 4 4 5 5 6 6

1 = Completely disagree1 = Completely disagree

2 = Disagree2 = Disagree

3 = Somewhat disagree3 = Somewhat disagree

Unstructured comment space will be provided at the end of Unstructured comment space will be provided at the end of the post-experiment questionnaire to record any notes about the post-experiment questionnaire to record any notes about their experience or feedback about the interfaces. their experience or feedback about the interfaces.

4 = Somewhat agree4 = Somewhat agree5 = Agree5 = Agree6 = Completely agree6 = Completely agree

Analysis Format (Sample Data)Analysis Format (Sample Data)

Mean Response Time

0

5

10

15

20

25

Paper Based Computer Supported CAPRI

Experimental Condition

Res

po

nse

Tim

e (s

eco

nd

s)

Analysis Format (Sample Data)Analysis Format (Sample Data)

"Locating a Detail Bug in the Plan Drawing is an easy task."

0

1

2

3

4

5

6

7

CompletelyDisagree

Disagree SomewhatDisagree

SomewhatAgree

Agree CompletelyAgree

Participant Responses

Res

pons

es

ChallengesChallenges

Statistical Comparison Statistical Comparison Architectural data setsArchitectural data sets Research MethodologyResearch Methodology Human Subjects issuesHuman Subjects issues CAPRI interface issuesCAPRI interface issues Confound it!Confound it!

Documents

Context Aware Paper- based Review Instrument: A User Study Presented by: Dipti Shah, Hoda Homayouni, Daniel Belcher