Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Dynamics of Collective Performancein Collaboration Networks
Amelkin V.†, Askarisichani O.†, Kim Y.J.‡, Singh, A.K.†, Malone T.W.‡
†University of California, Santa Barbara, Department of Computer Science
{victor,omid55,ambuj}@cs.ucsb.edu
‡Massachusetts Institute of Technology, Sloan School of Management
{youngji,malone}@mit.edu
INSNA, XXXVI Sunbelt Social Networks Conference, 2016
1 / 18
Introduction – Motivation
…
• Small teams of humans working on a sequence of simple tasks.
• Questions:
. Why do some teams perform well while others do not?
. What are the key factors affecting team performance?
. How to accurately predict team performance?
. How does team performance evolve?
. What is the impact of collaboration patterns on team performance?
2 / 18
Introduction – Dataset – Teams
• Dataset
. 68 teams, 4 members each, diverse population found through ads1,2
• Team features (select)
. social sensitivity3
. gender balance
. personality traits (“Big Five”)
. verbal / nonverbal communication
1David Engel et al. “Reading the mind in the eyes or reading between the lines? Theory ofmind predicts collective intelligence equally well online and face-to-face”. In: PloS one 9.12(2014), e115212.
2Anita W Woolley et al. “Evidence for a collective intelligence factor in the performance ofhuman groups”. In: Science 330.6004 (2010), pp. 686–688.
3Simon Baron-Cohen et al. “Reading the Mind in the Eyes test revised version: A study withnormal adults, and adults with Asperger syndrome or high-functioning autism”. In: Journal ofchild psychology and psychiatry 42.2 (2001), pp. 241–251.
3 / 18
Introduction – Dataset – Teams
• Dataset
. 68 teams, 4 members each, diverse population found through ads1,2
• Team features (select)
. social sensitivity3
. gender balance
. personality traits (“Big Five”)
. verbal / nonverbal communication
(reflective)
(disapproving)
Social sensitivity quantified viaReading the Mind in the Eyes test.
1Engel et al., “Reading the mind in the eyes or reading between the lines? Theory of mindpredicts collective intelligence equally well online and face-to-face”.
2Woolley et al., “Evidence for a collective intelligence factor in the performance of humangroups”.
3Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.
3 / 18
Introduction – Dataset – Tasks
• Tasks
Typing(Text)
Typing(Numbers)
Matrixsolving
Detection(Images)
Detection(Words)
Unscramblewords
Memory(Video)
Memory(Images)
SudokuBrainstorm
(Words)
Brainstorm(Brick)
Brainstorm(Equations)
Judgement(Slogans)
Judgement(Pictures)
Judgemet(Pages)
• Task features
. category: generating, choosing, executing, remembering, sensing;
. verbality
4 / 18
Introduction – Dataset – Tasks
• Tasks
0 0.5 10
2
4
Typing Text
0 0.5 10
2
4
Typing Numbers
0 0.5 10
2
4
Matrix Solving
0 0.5 10
2
4
Detection Images
0 0.5 10
2
4
Detection Words
0 0.5 10
2
4
Unscramble Words
0 0.5 10
2
4
Memory Video
0 0.5 10
2
4
Memory Images
0 0.5 10
2
4
Sudoku
0 0.5 10
2
4
Brainstorm Words
0 0.5 10
2
4
Brainstorm Brick
0 0.5 10
2
4
Brainstorm Equations
0 0.5 10
2
4
Judgement Slogans
0 0.5 10
2
4
Judgement Pictures
0 0.5 10
2
4
Judgement Pages
Figure: Task scores (scaled to [0, 1])
• Task features
. category: generating, choosing, executing, remembering, sensing;
. verbality
4 / 18
Introduction – Dataset – Team communication
• Some teams only communicate through text chat.
• For these teams, we have communication logs.
Time Sender Message Sentiment9:33:45 daniel thanks tim !! +0.8
9:33:46 liza type in your own lines +0.3
9:39:50 tim tao, look at the rules +0.3...
......
...9:47:53 tim this is seriously subjective −0.6
9:48:11 liza I think they all suck honestly −0.8
9:49:48 tim when is this going to end? 0.0
9:50:21 tao one is nonsense −0.6
• Logs used to build network representation of teams.
5 / 18
Introduction – Problem statement
Team features Observed team behavior Observed performance
Predicted / explained team performance
6 / 18
Predicting team performance from similar tasks
• Natural Question: How a team will perform on a given task?
• Natural Answer: Similarly to how it performed on similar tasks.
• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.
Task similarity (task features)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.2
0.4
0.6
0.8
1Task similarity (scores)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).
• To predict team performance, we need to go beyond task similarity.
7 / 18
Predicting team performance from similar tasks
• Natural Question: How a team will perform on a given task?
• Natural Answer: Similarly to how it performed on similar tasks.
• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.
Task similarity (task features)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.2
0.4
0.6
0.8
1Task similarity (scores)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).
• To predict team performance, we need to go beyond task similarity.
7 / 18
Predicting team performance from similar tasks
• Natural Question: How a team will perform on a given task?
• Natural Answer: Similarly to how it performed on similar tasks.
• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.
Task similarity (task features)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.2
0.4
0.6
0.8
1Task similarity (scores)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).
• To predict team performance, we need to go beyond task similarity.
7 / 18
Predicting team performance from similar tasks
• Natural Question: How a team will perform on a given task?
• Natural Answer: Similarly to how it performed on similar tasks.
• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.
Task similarity (task features)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.2
0.4
0.6
0.8
1Task similarity (scores)
Typing TextTyping N
umbers
Matrix Solving
Detection Im
ages
Detection W
ords
Unscram
ble Words
Mem
ory Video
Mem
ory Images
SudokuBrainstorm
Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).
• To predict team performance, we need to go beyond task similarity.
7 / 18
Team features vs. Team performance
• Goal: use a team’s static features (e.g., gender balance, mean socialsensitivity, mean age) to predict its performance on each task.
• Methods:
. Correlation analysis
. Team performance modeling
- Linear / non-linear regularized regression
- Regression in latent space
- Support Vector Regression
- Gaussian Process
- REP Trees
results not reported
8 / 18
Team features vs. Team performance – Correlation
Correlation: Team features vs. Scores (p-val < 0.05)
Mean Age
Mind in the Eyes
# Females
Big-5 Sympathetic
Big-5 ExtravertBig-5 D
ependableTeam
TalksTeam
Cohesion
Big-5 Calm
Big-5 Open to N
ew
Typing Text
Typing Numbers
Matrix Solving
Detection Images
Detection Words
Unscramble Words
Memory Video
Memory Images
Sudoku
Brainstorm Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages -0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Team Feature ρ, [min,max] max p-valueMean Age [−0.53,−0.25] 0.04Mind in the Eyes [+0.24,+0.48] 0.05# Females [+0.26,+0.38] 0.03
Table: Top team features significantly correlated with task scores.
9 / 18
Team features vs. Team performance – Regression
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Typing TextRMSE = 0.20
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Typing NumbersRMSE = 0.20
0.2 0.4 0.6 0.8 1
0.2
0.4
0.6
0.8
1
Matrix SolvingRMSE = 0.20
0 0.5 10
0.2
0.4
0.6
0.8
1
Detection ImagesRMSE = 0.24
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Detection WordsRMSE = 0.27
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Unscramble WordsRMSE = 0.23
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Memory VideoRMSE = 0.27
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
1
Memory ImagesRMSE = 0.24
0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
SudokuRMSE = 0.22
0 0.2 0.4 0.60
0.2
0.4
0.6
Brainstorm WordsRMSE = 0.13
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Brainstorm BrickRMSE = 0.18
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Brainstorm EquationsRMSE = 0.20
0.4 0.6 0.8 1
0.4
0.6
0.8
1
Judgement SlogansRMSE = 0.20
0.2 0.4 0.6 0.8 1
0.2
0.4
0.6
0.8
1
Judgement PicturesRMSE = 0.22
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Judgement PagesRMSE = 0.21
Figure: True vs. predicted task scores (linear regression, 80% LASSO + 20% ridge)
10 / 18
Team features vs. Team performance – Regression
Team feature significance
# Females
Big-5 ExtravertM
ean AgeM
ind in the EyesBig-5 O
pen to New
Big-5 Sympathetic
Big-5 Dependable
Team Talks
Big-5 Calm
Team C
ohesion
Typing Text
Typing Numbers
Matrix Solving
Detection Images
Detection Words
Unscramble Words
Memory Video
Memory Images
Sudoku
Brainstorm Words
Brainstorm Brick
Brainstorm Equations
Judgement Slogans
Judgement Pictures
Judgement Pages-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Team Feature min |βi| max |βi| mean |βi|# Females 0.19 0.89 0.47Big-5 Extravert 0.07 0.46 0.24Mean Age 0.04 0.36 0.18Mind in the Eyes 0.01 1.00 0.14
Table: Top team features emphasized in the best linear regression models. Sorted by mean |βi|,where βi is the model’s coefficient corresponding to the team feature.
10 / 18
Team features vs. Team performance dynamics
• Can we explain the dynamics of team performance in terms of teamfeatures?
0
.5
1Team 3
0
.5
1Team 10
0
.5
1Team 64
0
.5
1Team 67
0
.5
1Team 48
0
.5
1Team 5
0
.5
1Team 59
0
.5
1Team 4
0
.5
1Team 46
0
.5
1Team 9
0
.5
1Team 40
0
.5
1Team 54
0
.5
1Team 20
0
.5
1Team 55
0
.5
1Team 15
0
.5
1Team 18
0
.5
1Team 58
0
.5
1Team 33
0
.5
1Team 28
0
.5
1Team 21
0
.5
1Team 1
0
.5
1Team 26
0
.5
1Team 41
0
.5
1Team 68
0
.5
1Team 23
0
.5
1Team 14
0
.5
1Team 7
0
.5
1Team 65
0
.5
1Team 36
0
.5
1Team 32
0
.5
1Team 62
0
.5
1Team 66
0
.5
1Team 16
0
.5
1Team 19
0
.5
1Team 63
0
.5
1Team 11
0
.5
1Team 6
0
.5
1Team 17
0
.5
1Team 47
0
.5
1Team 34
0
.5
1Team 52
0
.5
1Team 56
0
.5
1Team 57
0
.5
1Team 43
0
.5
1Team 51
0
.5
1Team 2
0
.5
1Team 60
0
.5
1Team 39
0
.5
1Team 8
0
.5
1Team 13
0
.5
1Team 61
0
.5
1Team 30
0
.5
1Team 53
0
.5
1Team 25
0
.5
1Team 45
0
.5
1Team 31
0
.5
1Team 42
0
.5
1Team 22
0
.5
1Team 49
0
.5
1Team 50
0
.5
1Team 35
0
.5
1Team 38
0
.5
1Team 24
0
.5
1Team 44
0
.5
1Team 27
0
.5
1Team 29
0
.5
1Team 37
0
.5
1Team 12
Figure: Series of 15 task scores for each of 68 teams. Teams are ordered by their overallperformance, from low (top left) to high (bottom right) performers.
11 / 18
Team features vs. Team performance dynamics
Correlation: Team features vs. Score dynamics features
CI
# Females
Mind in the Eyes
Team C
ohesionBig-5 ExtravertBig-5 Sym
patheticBig-5 D
ependableBig-5 C
almBig-5 O
pen to New
Mean Age
first scorelast score
median scoremean score
mean wavelet scorescore std
score variancescore skewness
score kurtosisbest-fit line, Y(first)best-fit line, Y(last)
best-fit line, Y(middle)best-fit line slope (tan)
best-fit line slope (angle)last score - first score
# below-median score changes# above-median score changes
# score increases# score decreases
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
• Teams with a higher proportion of females (ρ = 0.30) and socialperceptiveness (ρ = 0.53) perform better overall.
• Better-performing teams start doing so early (first task ρ = 0.88) andperform well consistently (slope ρ = −0.42, skewness ρ = −0.50).
• Older teams perform worse on the first few tasks (ρ = −0.55).
11 / 18
Observed performance dynamics vs. Team performanceTypingText(training)
TypingNumbers(training)
MatrixSolving(training)
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
DetectionImagesRMSE = 0.24
0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
DetectionWordsRMSE = 0.25
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
UnscrambleWordsRMSE = 0.22
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
MemoryVideoRMSE = 0.27
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
1
MemoryImagesRMSE = 0.23
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
SudokuRMSE = 0.22
0.1 0.2 0.3 0.4 0.50
0.2
0.4
BrainstormWordsRMSE = 0.14
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
BrainstormBrickRMSE = 0.17
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
BrainstormEquationsRMSE = 0.18
0.4 0.6 0.8 1
0.4
0.6
0.8
1
JudgementSlogansRMSE = 0.19
0.4 0.6 0.8
0.4
0.6
0.8
JudgementPicturesRMSE = 0.21
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
JudgementPagesRMSE = 0.20
Figure: True vs. predicted task scores; linear regression using features derived from the past teamperformance dynamics.
Features RMSEs for task score predictions
Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21
Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20
• Performance prediction based on (static) team features is as goodas prediction based on the observed performance dynamics.
12 / 18
Observed performance dynamics vs. Team performanceTypingText(training)
TypingNumbers(training)
MatrixSolving(training)
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
DetectionImagesRMSE = 0.24
0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
DetectionWordsRMSE = 0.25
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
UnscrambleWordsRMSE = 0.22
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
MemoryVideoRMSE = 0.27
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
1
MemoryImagesRMSE = 0.23
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
SudokuRMSE = 0.22
0.1 0.2 0.3 0.4 0.50
0.2
0.4
BrainstormWordsRMSE = 0.14
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
BrainstormBrickRMSE = 0.17
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
BrainstormEquationsRMSE = 0.18
0.4 0.6 0.8 1
0.4
0.6
0.8
1
JudgementSlogansRMSE = 0.19
0.4 0.6 0.8
0.4
0.6
0.8
JudgementPicturesRMSE = 0.21
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
JudgementPagesRMSE = 0.20
Figure: True vs. predicted task scores; linear regression using features derived from the past teamperformance dynamics.
Features RMSEs for task score predictions
Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21
Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20
• Performance prediction based on (static) team features is as goodas prediction based on the observed performance dynamics.
12 / 18
Communication Logs for Team Performance Prediction
• For chatting teams, we have team member communication logs:
Time Sender Message Sentiment9:33:45 daniel thanks tim !! +0.89:33:46 liza type in your own lines +0.3
.
.
....
.
.
....
• Goal: connect team performance with features derived from logs.
• General log-based features: (select)
. # of turns in conversation
. time delays between chat messages
. mean # words/characters produced per team member
. message sentiment
13 / 18
Communication Logs for Team Performance Prediction
• General log-based features: (select)
. # of turns in conversation
. time delays between chat messages
. mean # words/characters produced per team member
. message sentiment
• Network-level features:
. A team is viewed as a network; team members – nodes.
. Edge weight wij = how often member i responds4 to member j.
. Features extracted from the network:- node degree
- edge density
- clustering coefficients
- betweenness centrality
- max/mean shortest path lengths
- · · ·4The likelihood that one message is a response to another message is computed based on their
temporal order and proximity.13 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.
.
.
....
.
.
.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah
nick kate
jeral greg
14 / 18
Communication Logs to Networks
• Networks derived from logs:
nick kate
jeral greg
Dense, Weighted
nick kate
jeral greg
Sparse, Unweighted
14 / 18
Communication Logs to Networks
• Networks derived from logs:
nick kate
jeral greg
Dense, Weighted
nick kate
jeral greg
Sparse, Unweighted
• Typical weighted adjacency matrices:
0
0.1
0.2
0.3
0.4
Low-performing team
0
0.1
0.2
0.3
High-performing team
14 / 18
Communication Logs vs. Team performance – CorrelationCorrelation: Log-based features vs. Scores (p-val < 0.05)
pwn-nturns
pwn-delays-m
ean
pwn-delays-m
edian
sn25-cluscoeff-mean
pwn-outdeg-m
ean-m-std
pwn-delays-std
sn25-density
sn25-outdegree-mean
sn25-indegree-mean
pwn-sent-total-std
sn25-nsccsn50-nccpw
n-indegree-std
pwn-indeg-m
ean-m-std
sn50-outdegree-mean
sn50-indegree-mean
pwn-nw
ords-total
sn50-density
pwn-nw
ords-per-mem
ber-mean
pwn-delays-m
ax90
pwn-delays-m
ax80
pwn-nchars-total
pwn-nchars-per-m
ember-m
ean
pwn-delays-m
ax70
sn50-avg-path-len
sn50-centrality-mean
sn50-diameter
pwn-outdegree-m
ean
pwn-indegree-m
ean
pwn-outdegree-std
sn50-cluscoeff-mean
sn50-nsccsn50-indegree-std
sn25-nccsn75-nccpw
n-sent-total-mean
pwn-sent-nn-std
sn25-avg-path-len
pwn-sent-nn-m
ean
pwn-sent-neg-std
sn75-outdegree-mean
sn75-indegree-mean
sn50-outdegree-std
pwn-sent-neg-m
ean
sn75-avg-path-len
pwn-reciprocity-m
ean
pwn-reciprocity-std
sn25-outdegree-std
sn25-indegree-std
sn25-diameter
sn25-centrality-mean
sn75-density
sn75-outdegree-std
sn75-indegree-std
sn75-diameter
sn75-nsccsn75-centrality-m
ean
sn75-cluscoeff-mean
pwn-nchars-per-m
ember-std
pwn-nw
ords-per-mem
ber-std
pwn-sent-pos-m
ean
pwn-sent-pos-std
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages -0.5
0
0.5
Team Feature ρ, [min,max] max p-value# of turns in conversation [ 0.37, 0.37] 0.03Median delay between chat messages [−0.44,−0.35] 0.05Mean clustering coefficient [ 0.36, 0.54] 0.04Node degree, mean− std [ 0.35, 0.49] 0.04Edge density [ 0.42, 0.52] 0.01Message sentiment variance [−0.52,−0.39] 0.02# connected components [−0.48,−0.35] 0.04# words typed [ 0.35, 0.41] 0.05Mean path length [ 0.39, 0.50] 0.02Mean betweenness [ 0.41, 0.45] 0.05
Table: Top 10 (out of 63) log-based team features significantly correlated with task scores.
15 / 18
Communication Logs vs. Team performance – Regression
0.2 0.4 0.60
0.2
0.4
0.6
Typing TextRMSE = 0.21
0 0.2 0.4 0.60
0.2
0.4
0.6
Typing NumbersRMSE = 0.17
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Matrix SolvingRMSE = 0.27
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Detection ImagesRMSE = 0.24
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Detection WordsRMSE = 0.27
0 0.5 10
0.2
0.4
0.6
0.8
1
Unscramble WordsRMSE = 0.24
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Memory VideoRMSE = 0.27
0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Memory ImagesRMSE = 0.24
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
1
SudokuRMSE = 0.25
0 0.2 0.4 0.6
0.1
0.2
0.3
0.4
0.5
0.6
Brainstorm WordsRMSE = 0.18
0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
Brainstorm BrickRMSE = 0.20
0.2 0.4 0.6 0.8
0.2
0.4
0.6
0.8
Brainstorm EquationsRMSE = 0.19
0.4 0.6 0.8 1
0.4
0.6
0.8
Judgement SlogansRMSE = 0.23
0.2 0.4 0.6 0.8 1
0.2
0.4
0.6
0.8
Judgement PicturesRMSE = 0.26
0 0.5 10
0.2
0.4
0.6
0.8
1
Judgement PagesRMSE = 0.23
Figure: True vs. predicted task scores
16 / 18
Communication Logs vs. Team performance – Regression
Log-based feature significance (model coefficients)
pwn-outdeg-m
ean-m-std
sn25-cluscoeff-mean
sn50-centrality-mean
sn25-densitysn50-densitypw
n-nturnssn50-nccpw
n-nwords-total
sn25-nsccpw
n-delays-mean
pwn-delays-std
sn50-diameter
sn50-avg-path-len
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Team Feature min |βi| max |βi| mean |βi|Node degree, mean− std 0.15 1.00 0.62Mean clustering coefficient 0.47 0.70 0.58Mean betweenness 0.14 0.66 0.31# connected components −0.34 −0.26 −0.30
Table: Top 4 team features emphasized in the best linear regression models. Sorted bymean |βi|, where βi is the model’s coefficient corresponding to the team feature.
16 / 18
Communication Logs vs. Team performance – Regression
Log-based feature significance (model coefficients)
pwn-outdeg-m
ean-m-std
sn25-cluscoeff-mean
sn50-centrality-mean
sn25-densitysn50-densitypw
n-nturnssn50-nccpw
n-nwords-total
sn25-nsccpw
n-delays-mean
pwn-delays-std
sn50-diameter
sn50-avg-path-len
Typing TextTyping Numbers
Matrix SolvingDetection ImagesDetection Words
Unscramble WordsMemory Video
Memory ImagesSudoku
Brainstorm WordsBrainstorm Brick
Brainstorm EquationsJudgement SlogansJudgement Pictures
Judgement Pages -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Features RMSEs for task score predictions
Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21
Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20
Log-based .21 .17 .27 .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20
16 / 18
Conclusion
• Team performance prediction – Summary
. Team performance prediction via task similarity is intuitive,but does not work.
. Performance prediction via regression on static team features worksreasonably well. Key positive factors – number of females and socialsensitivity.
. Performance prediction based on a team’s performance history worksslightly better (although the history’s often being unavailable).
. Predictions based on log-data are of the same quality, yet,have different model interpretation.
- positive factors: # of turns in conversion, node degrees and clusteringcoefficients, edge density
- negative factors: communication delays, network disconnectedness, messagesentiment variance.
17 / 18
Conclusion
• Recipe for a better-performing team
. Team members are socially apt, as reflected in their social sensitivity.
. The team is gender-balanced∗.∗Number of females is likely emphasized by the predictive models
because of its positive connection with social sensitivity5
. Team members enthusiastically interact and
equally contribute to the discussion.
5Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.
17 / 18
Conclusion
• Recipe for a better-performing team
. Team members are socially apt, as reflected in their social sensitivity.
. The team is gender-balanced∗.∗Number of females is likely emphasized by the predictive models
because of its positive connection with social sensitivity5
. Team members enthusiastically interact and
equally contribute to the discussion.
• Future Work
. Capturing actual communication/collaboration between teammembers.
. Varying task sequence composition and, particularly, task order.
. Tracking team members’ emotions.
. Studying effect of particular collaboration patterns on teamperformance dynamics.
5Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.
17 / 18
Questions
∼ Thanks ∼
18 / 18