What Did I Miss?

Kori Inkpen John C. TangRajesh Hegde

Zhengyou ZhangSasa JunuzovicChris BrooksUniv. Saskatchewan

In-Meeting Review using Multimodal Accelerated Instant Replay (AIR) Conferencing

GoalMake meetings (and play) better with telepresence technology

Meetings are important

Do you ever ….

Partial Meeting Review95% 91%

73%

30%

Verizon Conferencing Report - 2003

What did I just miss?

DVR Functionality -> Live Conference

What makes it challenging?

Previous ResearchMeeting Capture

Cutler et al., 2002; Jian, Kim & Li, 2003; Ranjan, Birnholtz, & Balakrishnan, 2008

SummariesVideo Skimming & Summarization - Smith & Kanade, 1995; He, Gupta & Grudin, 1999; Christel, 2006; Money & Agius, 2008;

Indexing Recorded MeetingsMoran et al., 1997

Time-CompressionOmoigui et al., 1999; Orr, 1971; Wildemuth et al., 2003

Real-time CatchupTucker et al., 2010

Prototype System

AIR Conferencing

AIR Conferencing System 1/2

LIVE

AIR Conferencing System 2/2

PLAYBACK

AIR Conferencing - User Study #1

Explore the benefits (and challenges) of Accelerated Instant Replay for meetings

Experimental Design

18 Participants (1 female)Three conditions

Background InterviewsTask: Live Status Meeting

3-way VideoconferenceEach person gets a chance to presentEach person gets interrupted 4 times and has to catch up

Recall assessment (quiz on facts + explanations)

Transcript-Only Muted Replay Full Replay

AIR – Interview Results

All occasionally miss parts of meetings

Current catch-up strategies (most indicated they do not want to disrupt the meeting)

often just listen and try to deduce what they missed, possibly interrupt with a question if they are confusedwait for a recap by the presenterbrowse meeting material to figure out what was misseddiscreetly ask someone what was missed (whisper or IM)

AIR – Interview ResultsUseful to replay what was missed during a videoconference?

Yes (10), maybe (7), no (1). “Often you miss critical conversations when you step out or are interrupted during a meeting and then you try to play catch-up during the rest of the meeting. Getting to know what was covered and who said it and the body language would put me back into the meeting very quickly.”

“It would depend mostly upon the importance of the meeting, followed by the duration of how much I missed, and finally, on how discreetly I could review the video”

AIR – Recall Results

Full Replay enabled users to recall as much as they did when they were not interrupted.

Recall TestFacts Explanations

Baseline 78% 43%

Transcript only 45% 16%

Muted Replay 40% 13%

Full Replay 80% 49%

AIR – Preference Results

Current System

Perfect Speech-to-

TextTranscript Only 2 7

Muted Replay 0 2

Full Replay 16 6

–

Full Replay:“it was fast, easy to concentrate and auto catch-up”“easiest to follow; leads most smoothly into rejoining live”“seems like the only way to catch up in a focused way”“fast audio was really cool …. I feel like I can cheat in time with fast forward”

AIR – Divided Attention Results

Very Difficult

Somewhat Difficult

Neither Easy nor Difficult

Somewhat Easy

Very Easy

6 8 2 2 0

Past + Present:“I can listen to audio while watching the live slide show and transcript”“you can listen to current conversations and read transcripts”

New Questions

1. How important were the additional modalities (over just audio alone)?

2. How successful would the system be with perfect speech-to-text transcription?

3. How can we better support multitasking?

AIR Conferencing - User Study #2

Examining individual modalities and impact of perfect speech-to-text

transcription

Prototype System

Videos

SharedDesktop

Speech-to-text Transcript

Experimental Design

58 Participants (25 female)Pre-recorded status meeting Mixed Design

Everyone did audio only + 1 other condition

Focused review (no divided attention)

Modalities

Transcript only

Audio only

Audio + Transcrip

t

Audio + Video

Audio + Shared

Workspace

Audio + All

(full replay)

User study #2 – Recall resultsRECALL Fact Explanation Identification

Live 93% 77% 84%

Audio Only 74% 50% 61%

Enhanced Audio 83%1 62%1 72%1

1 Significantly better than audio only but worse than live

User study #2 – Recall results

RECALL Fact Explanation Identification

Live 93% 77% 84%

Audio Only 74% 50% 61%

Audio + All 89%1 70%1 81%1

Audio + Workspace 90%1 65% 72%

Audio + Transcript 83% 67%2 66%

Audio + Video 70% 48% 68%

1 Significantly better than audio only and no difference with live

2 Significantly better than audio only but worse than live

User study #2 – Recall results

RECALL Fact Explanation Identification

Live 93% 77% 84%

Audio Only 74% 50% 61%

Audio + All 89%1 70%1 81%1

Audio + Workspace 90%1 65% 72%

Audio + Transcript 83% 67%2 66%

Audio + Video 70% 48% 68%

1 Significantly better than audio only and no difference with live

2 Significantly better than audio only but worse than live

Users were significantly more confident with their answers in

the Audio + All condition (p<.05).

User study #2 – Recall results

RECALL Fact Explanation Identification

Live 93% 77% 84%

Audio Only 74% 50% 61%

Audio + All 89%1 70%1 81%1

Audio + Workspace 90%1 65% 72%

Audio + Transcript 83% 67%2 66%

Audio + Video 70% 48% 68%

1 Significantly better than audio only and no difference with live

2 Significantly better than audio only but worse than live

“I used the audio to see who was talking, and every time I missed something, I had the transcript which kept a recording of everything and I could just look back at it.”

“Being able to see who was talking during catch-up helped to associate a face, name, and voice with the answers given.”

User study #2 – Transcript Only

RECALL Fact Explanation Identification

Audio Only 71% 50% 56%

Transcript Only 77%1 52% 43%2

1 Significantly better than audio only

2Significantly worse than audio only

User study #2 – Transcript Only

RECALL Fact Explanation Identification

Audio Only 71% 50% 56%

Transcript Only 77%1 52% 43%2

1 Significantly better than audio only

2Significantly worse than audio only

Transcript alone was way too fast to really understand what was going on. I could kind of skim, and I got some information but I felt like I was just bouncing along.

AIR Conferencing Conclusions

Enhanced audio catch-up is superior to audio-only review.

Users preferred, felt more confident with, and performed better with enhanced-audio than audio-only review. As good as live!Results consistent between two studies

Audio + All showed the strongest benefitAudio + Shared Workspace improved recall of factsAudio + Transcript improved recall of explanations

Speech-to-Text transcript not as beneficial as expected

AIR Conferencing Conclusions

Cost/benefit tradeoff: importance of information vs. overhead Ensure seamless use so as to not detract from the meeting

Next Steps:Enhanced timeline indexMulti-taskingAIR for face-to-face meetings

(with mobile phones/laptops & headsets)

AIR Vision

Replaying life should be “a given”But sometimes participation in real-time is important, so we need to enable catch-up

Thank you!

Questions?