S C O T T I ES Y S T E M AT I C C O M M U N I C AT I O N O B J E C T I V E S

A N D T E L E C O M M U N I C AT I O N S T E C H N O L O G Y

I N V E S T I G AT I O N S A N D E VA L U AT I O N S

R A Y T H E O N T E C H N O L O G I E S R E S E A R C H C E N T E R ( R T R C ) : P E G G Y W U , A H M A D O S M A N , C A L E B H A R D E R

I O W A S T A T E U N I V E R S I T Y ( I S U ) :

S T E P H E N G I L B E R T , J I M O L I V E R ,

A N G E L I C A J A S P E R , M A R I E L L E M A C H A C E K , M I N D Y H O O V E R , R A C H E L D I A N I S K A , C H A R L E S P E A S L E Y , K A I T L Y N

O U V E R S O N , N I C H O L A S W I L S O N , Y A S H V A R D H A N , K E N N E D Y C O O K , M A D I S S E N L A W R E N C E ,

H A Y L E E L A W R E N C E , M A T T G R E I N E R , J O S E P H R O Z E L L , K Y L E G O B E N , A A D I T Y A P I N J A R K A R , A N D R E W

D E I C K , R O X A N N A A R R E O L A , H A N N A H W E L L I K , H A R R I SS E A B O L D , C A R L E Y H A U S , L E I L A N I H A M M E L

This document contains no export controlled technical data.

R ESEAR C H QU ESTION :

TR AVEL VS. TELEC OMMU TE? W H AT AR E

TH E IMPAC TS ON C OMMU N IC ATION

PER FOR MAN C E?

2

Trust

Rapport

Engagement and Co Presence

Conflict Management

Collective Efficacy and Workload

Shared Mental Model

Shared Situation Awareness

C OMMU N IC AT ION OBJEC T IVES MOD EL (C OM)

3

Context Framework

Task Related (Requirements)

Time

Space

Symmetry

Task Focus

Artificiality

Scenario*

Social Factors

Social Distance

Power Difference

Number of Participants

Technology Affordances

Agency

Person Space

Shared Space

Work Space

FluidityEmbodiment

(Avatars)Media Richness (Environment)

MU LT ID IMEN SION AL C ON T EXT S

FOR C OMMU N IC AT ION S FR AMEW OR K (MC C F)

MCCF describes the situational contexts (tasks

and social factors) and the technology features

that can affect communication effectiveness

Will this work for

my use case?

It depends...

Context Framework

Task Related (Requirements)

Time

Space

Symmetry

Task Focus

Artificiality

Scenario*

Social Factors

Social Distance

Power Difference

Number of Participants

Technology Affordances

Agency

Person Space

Shared Space

Work Space

FluidityEmbodiment

(Avatars)Media Richness (Environment)

MU LT ID IMEN SION AL C ON T EXT S

FOR C OMMU N IC AT ION S FR AMEW OR K (MC C F)

MCCF describes the situational contexts (tasks

and social factors) and the technology features

that can affect communication effectiveness

• Identified 30 relevant studies and 1 meta

study

• Many studies were evaluations of their

specific features/technologies.

• Conflicting results (e.g. increased fidelity

has mixed effects on

performance (Kock, 2005; Mütterlein et al.,

2018))

• Some had more generalizable

findings: e.g. Pan & Steed (2016) found

subjects more frequently seeked advice

(trust) from robot & video over avatar, Jo

et al., (2017) found co-presence to be

higher in more realistic backgrounds and

preference for higher fidelity avatars

Agency

The ability to control perspective

and move about the environmentGreater Degrees of Freedom equals higher Agency.

Embodiment

Ability to represent the visual and auditory characteristics of the human operator.A high embodiment value therefore equals greater capability for the human to use the technology to represent

and exhibit gestures and body language.

T EC H N OL OGY AFFOR D AN C ES

D ATA AN ALYSIS C ON ST R UC TS:

IN D EPEN D EN T VAR IABL ES

Construct Definition

Agency

The extent to which a participant can make modifications to the three different

functional spaces and modify his/her own perspective of the world and artifacts within

it. High denotes complete control, medium represents limited control where the

participant may have limited ability to modify some objects or attributes within the

digital environments, and low represents very limited abilities where the participant

may only be able to direct their own attention within a confined perspective.

Embodiment

Representation of the human operator, which includes visual and audio. Ranges from

simple representations (e.g. text, icons, and caricature) to photoreal 3D digital avatars

and animations. Higher values represent greater capability to replicate environmental

context independent gestures and body language (e.g. nodding, shrugging). In our

definition gestures that rely on environmental context fall under Media Richness (e.g.

gaze or gestures directed at an object).

T EC H N OL OGY AFFOR D AN C ES BY C ON D IT ION

Condition Description Agency Embodiment

Face to

FacePhysically colocated High High

Telecon

Software

Zoom and

WebcamLow

Medium

(partial

representation)

XR

VR Headset, WebcamMedium

Low – Medium

(view of hands or

avatar)

Connect to the power source (+,-)

TASK PR OC ED U R E OVERVIEW

will light up based

on the sensor input

Select the proper circuit board

Replace LED with an RGB LED

Find and properly

connect Soft Potentiometer

Scenario 1 Hypotheses Across 3 conditions

Agency of Apprentice Embodiment of Instructor

Low

(Telecon)

Medium

(XR)

High

(F2F)

Low

(XR)

Medium (Telecon) High

(F2F)

Trust T-H1: Trust increases with agency of A. Since A

can control viewing perspective, Instructor is less

likely to deceive or cause distrust (defined as -'ve of

trust per Marsh & Dibben, 2005)

T-H2: Trust increases with realism of

avatar (Pan & Steed, 2017; Liew et al., 2017;

Campellone, & Kring, 2013)

Shared Situation

Awareness (SA) /

Mental Model

SA-H1: Shared SA increases with increased

agency. Since A can change perspectives, less time

is spent on explicit descriptions of what can

otherwise be gleaned naturally. This affords more

time to for participants to synchronize their

mutual understanding of the situation or task.

SA-H2: Shared SA increases with increased

embodiment of both participants. Increased

embodiments can provide more cues, allowing

partners to more easily identify whether

attention is being paid to the correct objects of

interest and take corrective measure more

quickly.

Engagement / Co-

Presence

E-H1: Co-P increases with increased agency as it

affords sustained attention which is a component of

engagement (Peters, Castellano, & de Freitas,

2009)

E-H2: Co-presence increases with increased

embodiment because it affords more cues of

the participant's emotional or attentional

involvement.

EXPER IMEN TAL D ESIGN S

MCCF = IV to manipulateC

OM

= D

V to

me

asure

Scenario 1 Hypotheses Across 3 conditions

Agency of Apprentice Embodiment of Instructor

Low

(Telecon)

Medium

(XR)

High

(F2F)

Low

(XR)

Medium (Telecon) High

(F2F)

Trust T-H1: Trust increases with agency of A. Since A

can control viewing perspective, Instructor is less

likely to deceive or cause distrust (defined as -'ve of

trust per Marsh & Dibben, 2005)

T-H2: Trust increases with realism of

avatar (Pan & Steed, 2017; Liew et al., 2017;

Campellone, & Kring, 2013)

Shared Situation

Awareness (SA) /

Mental Model

SA-H1: Shared SA increases with increased

agency. Since A can change perspectives, less time

is spent on explicit descriptions of what can

otherwise be gleaned naturally. This affords more

time to for participants to synchronize their

mutual understanding of the situation or task.

SA-H2: Shared SA increases with increased

embodiment of both participants. Increased

embodiments can provide more cues, allowing

partners to more easily identify whether

attention is being paid to the correct objects of

interest and take corrective measure more

quickly.

Engagement / Co-

Presence

E-H1: Co-P increases with increased agency as it

affords sustained attention which is a component of

engagement (Peters, Castellano, & de Freitas,

2009)

E-H2: Co-presence increases with increased

embodiment because it affords more cues of

the participant's emotional or attentional

involvement.

EXPER IMEN TAL D ESIGN S

T EST BED - C IR C U IT W OR L D

• Virtual room mimics the spatial setup of the lab space

• Circuits function similarly to real world counterpart

• Includes spatial, semantic and procedural memory components

Introduction to Circuit World

Expert Maintenance

Training

COM and UX Surveys

Knowledge Quiz

MaintenanceProcedure

Assessment

Delayed Knowledge

Retention Test

EXPER IMEN T PR OC ED U R E Circuit World (CW) Task

Telecon

XR

F2F*

EXPER IMEN T PR OC ED U R E Circuit World (CW) Task

Telecon

XR

F2F*

MOD IFIED ST IMU L I - C IR C U IT W OR L D

Task Performance Metrics

• Knowledge Quiz Scores

• Task accuracy (retrieval and assembly)

• Navigational information

• Task and subtask completion times

& Repeat ~2 weeks later

Example Quiz Questions

D ATA C OL L EC T ION STAT U S: PR OC ED U RAL

Surv ey

• Demographics

• Subject – Trainer Interaction

• Knowledge retention

Testbed Data

• Task success and Time stamped Task Completion

• Subject Interactions with Circuit

Video/Audio Data

• Gaze

• Facial Expression

• Transcripts

Boris Experimenter Coded Behav ioral Data

Time Budget

Phase 1: First contact Training and Performance15-minute set up

50-minute experiment30-minute debrief

Phase 2: Follow-on Knowledge Retention after

time lapse15-minute set up

15-minute test30-minute debrief

• Participant Management

o Additional participant screening for technical requirements (I.e., network bandwidth, computer requirements, ensuring VR equipment before scheduling)

o Additional coordination due to remote study staff (need two study staff per subject)

o Recruiting subjects remotely (using both Prolific and email)

o Prolific eased participant recruitment but added complexity requiring reminders from their prolific

account to their personal email addresses which initially caused more no-shows

o Troubleshooting software downloads (Zoom, testbed), and coordinating testbed updates

• Remote Management Tool Challenges

o Additional Testbed testing and development to accommodate different internet connect quality of subjects and research staff

o Zoom license structure provided different video resolution and speech transcript capabilities.

o Zoom also forced cloud storage requiring modifications to IRB protocol

o Computer security protocols and additional coordination of data management due to remote researchers using their own machines

P IV O T F R O M L A B - B A S E D TO R E M OTE PA R TIC IPA N TS :

L O G IS TIC S C H A L L E N G E S

Undergraduate Research Assistant (URA) Training

• 14 URAs

• Individual script memorization

• Script practice within small groups

EXPER IMEN T PR EPAR AT ION S

Online Prolific Total

Screener 51 494 545

Eligible & Invited for Zoom Part 2 (Informed Consent, Pre-Survey, Calendly)

29 226 255

Eligible for VR Part 2(Informed Consent, Pre-Survey, Calendly)

3 61 64

Completed Visits 1 & 2 3 5 8

Completed Visit 1, Scheduled Visit 2 3 1 4

Scheduled Visit 1 1 6 7

No Shows (additional) 2 4 6

D ATA C OL L EC T ION STAT U S:

R EC R U IT MEN T & AT T R IT ION ( A S O F 4 / 3 0 / 2 0 2 1 )Z

oom

Only

Zoom

Only

Online (N = 5) Prolific (N = 3)

Age 28.20 (SD = 7.56, 21-39) 41.67 (SD = 6.43, 37-49)

Gender (%Male) 60 66.7

% Student (Majors)60 0

Chemistry, PoliSCi, Software Eng --

% Non-Student (Professions)

40 100

Research Engineer, Software EngineerArt handler, Customer Service,

Engineer

Education Level

High school graduate 20 --

Some college/2-year degree -- 33.3

4-year degree 40 33.3

Graduate Degree 40 33.3

Computer Expertise (1-5)

4.40 (SD = .89)

"Moderate-Expert - I am able to teach

others 'expert' features."

3.00 (SD = 0)

"Moderate - I am able to complete

daily computing tasks."

D ATA C OL L EC T ION STAT U S:

D EMOGR APH IC IN FOR MAT ION ( A S O F 4 / 3 0 / 2 0 2 1 )

D ESC R IPT IVE STAT IST IC S OF SU RVEY D ATA

Sample Items Response Range Avg (SD) (N=8) Online (N=5)

Virtual Embodiment

"I felt like I was controlling the movements of the virtual body."

1 = Strongly Disagree5 = Strongly Agree

2.42 (SD = 1.23) 3.37 (SD = 1.20)

Co-Presence"In the training, to what extent did you have the sense of the 'trainer being together with you?"

1 = Not at all7 = Very Much

6.00 (SD = 1.37) 4.89 (SD = 1.71)

Engagement"When I was building the circuit, I lost track of the

world around me1 = Strongly Disagree

5 = Strongly Agree3.95 (SD = 1.07) 3.83 (SD = 1.21)

RapportRate interaction with trainer:

Positive – Negative; Cooperative - Uncooperative1 = Better Rapport9 = Worse Rapport

2.62 (SD = 1.42) 2.45 (SD = 1.42)

Usability: Perceived Ease

of Use

"My interaction with this system was clear and understandable."

"Using this system would improve my performance in training procedures."

1 = Strongly Disagree7 = Strongly Agree

4.21 (SD = 2.32) 5.78 (SD = 1.46)

Usability: Task Experience

"How difficult was it to apply the trained content in performance and implementation?"

1 = Very Difficult5 = Very Easy

3.83 (SD = .76) 4.30 (SD = 1.05)

D ESC R IPT IVE STAT IST IC S OF SU RVEY D ATA

Sample Items Response Range Prolific (N=3) Online (N = 5)

Immediate Performance

"How confident would you feel about your ability to perform the circuitry task again, now that the study is

complete?"

1 = Not confident at all5 = Very confident

4.80 (SD = .45) 2.67 (SD = 1.53)

Future Performance

"How confident are you that in one to two weeks you will be able to remember how to complete the task

that you were trained on?"

0 = Not confident at all100 = Extremely confident

67 (SD = 46.04) 36.67 (SD = 43.68)

Trust: Ability "I feel very confident about the trainer's skil ls." 1 = Strongly Disagree

5 = Strongly Agree

4.83 (SD = .29) 4.53 (SD = .73)

Trust: Integrity "The trainer tries hard to be fair in dealing with others." 1 = Strongly Disagree

5 = Strongly Agree

3.06 (SD = .25) 3.90 (SD = .89)

Trust: Benev olence "The trainer would be very concerned about my welfare." 1 = Strongly Disagree

5 = Strongly Agree

2.80 (SD = 1.59) 4.04 (SD = 1.15)

S C E NA R I O 1 – C I R C UI T W O R L D

SOFT W AR E EN GIN EER IN G

VIRT U AL C IR C U IT S IMU L ATOR

Desktop version

• Extensively tested:

• Networking capabilities and robustness

• Circuit simulation

• UX

• Data Collection

VR Integration

T EST BED EN GIN EER IN G

• Oculus Rift CV1

• Oculus Rift S

• HTC Vive

• HTC Vive Pro

• HTC Vive Cosmos

• Valve Index

• Oculus Quest +

Link

• Oculus Quest 2 +

Link

D E P L O YIN G E X P E R IM E N TS A N D A C Q U IR IN G

B E H AV IO R A L D ATA R E M O TE LY

Unity Experimental Framework (UXF)

Monitor performance and capture behavioral

data in real-time

Take notes, update text banner with

task info, restart trials easily in a Researcher UI

Conditions and variables stored with

participant list and backed up

Prevents processing issues and sickness with

multithread system

D ATA AN ALYSIS

Task

Performance

Number of errors (Count)

Part acquisition (Time & Proportion)

Time to completion

Knowledge Quiz Score

Trust (Self-report)

Rapport (Self-report)

Situational Awareness(Self-report)

Engagement (Observation)

Communication

Objectives User Experience

NASA-TLX(Self-report)

UTAUT (Self-report)

NPS (Self-report)

At what level of effectiveness is the replacement of

travel with virtual alternatives a viable solution?

D ATA AN ALYSIS D ESIGN

Between Subjects

• Phase 1 collection method

• More variability in conditions (b/c only one task)

• Allows F2F data collection later

Within Subjects

• Comparing training decay

• Mixed design (within subjects in a single condition)

• Analyzed with Repeated Measures MANOVA

D ATA AN ALYSIS C ON ST R UC TS:

D EPEN D EN T VAR IABL ES

Construct Definition

Co-Presence A feeling of connection between two people

EngagementDirecting one’s attention, acknowledging other participants, and demonstrating a readiness to interact

with other participants, whether positively or negatively

Knowledge How well the participant retains learned knowledge about the circuit board and its functions

Mental Workload The cost incurred by human operators to achieve a specific level of performance

Performance How accurately and efficiently a participant completes the circuit board tasks

Rapport The quality of relationship and interactions between the participant and the trainer

Shared Situational

Awareness

The extent to which two or more people have a commonly understood mental model of a situation

(I.e., what is currently happening and what is going to happen)

Trust The amount of trust the participant feels for the trainer

Usability The perceived ease of use of the Circuit World system and tasks

Construct Survey Testbed Video/Audio Behavioral (Boris)

Co-Presence - Co-Presence Questionnaire - Lexical and semantic similarity.

- Sentiment Analysis

Gesturing to an object

Engagement - User Engagement Scale: Focused

Attention

- Eye gaze Verbal affirmation, leaning

forward, head nod, visible

hands, crossed arms, questions

Knowledge - Soft Potentiometer Knowledge Quiz - Intent Classifier

Mental Workload - NASA TLX Raw (Communication)

- NASA TLX Raw (Training System)

Performance - Performance Confidence items - If and when participant

completes necessary circuit tasks

- Participant looks at points of

interest (VR training session)

- Linguistic Style Matching

Rapport - Bernieri Adapted Scale

- Student-Instructor Rapport Scale

- Synchrony in Facial Expressions

- Sentiment Analysis (positive

emotion)

- Lexical and semantic similarity.

Socializing, mirroring, asking for

assistance, in-group language,

out-group language

Shared Situational

Awareness

- 3D-SART - Intent Classifier Jargon

Trust - Trust Scale (Ability, Integrity, Benevolence,

Trustworthiness)

- Sentiment Analysis

Usability - System Usability Scale

- Net Promotor Scale

D ATA MAPPIN G

ST R U C TU RAL EQU AT ION MOD EL IN G

Performance

Knowledge Quiz

Usability

Co-Presence

Engagement

Shared SA

Trust

Rapport

Mental Workload

Performance

User Experience

Communication

Objectives

Agency

Embodiment

Q UANTI FY I NG THE USE O F HI G H EM BO DI MENT AFFO RDANCES

IN TE R P E R S O N A L E YE G A Z E C L A S S IF IC ATIO N

The teleconferencing condition provides a

view of the trainer

Use of this affordance may impact

communication outcomes

A clustering algorithm can discern

moments of visual attention to the video

feed (Tran et al., 2020)

EYE GAZE P IPEL IN E

FAC IAL EXPR ESSION S IN ZOOM AN D XR

C ON D IT ION

Mouth region reliably conveys the

following emotions:

Happiness (84%)

Neutral (72%)

Anger (62%)

FAC E D ETECTION U N DER

OC CLUSION

FAC IAL SYN C H R ON Y P IPEL IN E

FAU

FDM

BEH AVOR IAL D ATA - FU T U R E D IR EC T ION S

Motion Energy Analysis

Synchronous facial

expressions and global

motion signify Rapport (Ramseyer, 2020; Riehle et al.,

2017; Yokotani et al., 2019)

Neural network to label emotion

from text transcripts (Ghosal et al.,

2019; Majumder et al., 2019) Frustrated

(Negative)

Neutral

(Neutral)

Excited

(Positive)

0-50 frames 51-100 frames

Participant

Trainer

Expression

Synchrony

Dialogical Emotion Analysis

N L P AN ALYSIS

Text File Generated

To be ingested by

NLP Pipeline

T R AVEL R EPL AC EMEN T T H R ESH OL D

T R AVEL R EPL AC EMEN T T H R ESH OL D

T R AVEL R EPL AC EMEN T T H R ESH OL D

• Ouverson, Kaitlyn, Angelica Jasper, Stephen Gilbert, Nick WIlson, and

Peggy Wu (2021). Adaptive Moderated Research: Lessons Learned in Redesigning a Moderated Virtual Reality Collaboration Study. In

proceedings of accepted for the 2021 ACM CHI conference. Remote

XR Research Workshop. May 8-13, 2021. Yokohama, Japan.

• Rozell et al. (2021). Circuit World: A Multiplayer Virtual Environment for Researching Engineering Learning. In proceedings of IEEE VR

2021.March 23-April 7. Lisbon, Portugal.

• Peasley, C., Dianiska, R., Oldham, E., Wilson, N., Gilbert, S., Wu, P.,

Israelsen, B., & Oliver, J. (2020). Evaluating Metrics for Standardized Benchmarking of Remote Presence Systems. IEEE VR Workshop VR

in VR, Virtual Meeting.https://arxiv.org/abs/2105.01772

• Dianiska, R. E., Peasley, C. J., Wilson, N., Barnett, N., Hammel, L.,

Purdy, B., Wu, P., Shirtcliff, E., Oliver, J. H., & Gilbert, S. B. (2020). Do You Need to Travel? Mapping Face-to-Face Communication Objectives

to Technology Affordances. In Proceedings of the Human Factors &

Ergonomics Society (HFES) Annual Meeting.

• Completed manuscript to be submitted to Journal of Computer Supported Collaborative Work (CSCW).

PU BL IC AT ION S

V I R T UA L C O L L A BO RA T I O N – XR A ND D E S K T O P US E R S

T EC H N OL OGY TO MAR KET

Benefits:

• Travel /scheduling for larger teams (30+)

• Time delays due to creating mockups

• Physical space for larger teams

Needs:

• Backchannel communications

• Ease of navigation in virtual space

• Getting/directing attention of individuals

T EC H N OL OGY 2 MAR KET

45

Collins Super Diamond Seats

• Design for Manufacture and Assembly (DFMA)

Q& A

46