Creating Dynamic Social Network Models from Sensor Data

Tanzeem ChoudhuryTanzeem ChoudhuryIntel Research / Affiliate Faculty CSEIntel Research / Affiliate Faculty CSE

Dieter Fox Dieter Fox Henry KautzHenry Kautz

CSECSEJames KittsJames Kitts

SociologySociology

What are we doing?Why are we doing it?

How are we doing it?

Social Network AnalysisSocial Network Analysis

Work across the social & physical sciences is increasingly studying the structure of human interactiono 1967 – Stanley Milgram – 6 degrees of separation

o 1973 – Mark Granovetter – strength of weak ties

o 1977 –International Network for Social Network Analysis

o 1992 – Ronald Burt – structural holes: the social structure of competition

o 1998 – Watts & Strogatz – small world graphs

Social NetworksSocial Networks

Social networks are naturally represented and analyzed as graphs

Example Network PropertiesExample Network Properties

Degree of a nodeEigenvector centrality

o global importance of a node

Average clustering coefficiento degree to which graph decomposes into

cliques 

Structural holes o opportunities for gain by bridging

disconnected subgraphs

ApplicationsApplications

Many practical applicationso Business – discovering organizational

bottlenecks

o Health – modeling spread of communicable diseases

o Architecture & urban planning – designing spaces that support human interaction

o Education – understanding impact of peer group on educational advancement

Much recent theory on finding random graph models that fit empirical data

The Data ProblemThe Data Problem

Traditionally data comes from manual surveys of people’s recollectionso Very hard to gather

o Questionable accuracy

o Few published data sets

o Almost no longitudinal (dynamic) data

1990’s – social network studies based on electronic communication

Social Network Analysis of Social Network Analysis of EmailEmail

Science, 6 Jan 2006

Limits of E-DataLimits of E-Data

Email data is cheap and accurate, but misseso Face-to-face speech – the vast

majority of human interaction, especially complex communication

o The physical context of communication – useless for studying the relationship between environment and interaction

Within a Floor

Within a Building

Within a Site

Between Sites

0 20 40 60 80

Proportion of Contacts

Face-to-FaceTelephone

High Complexity Information

• Can we gather data on face to face communication automatically?

Research GoalResearch Goal

Demonstrate that we can… Model social network dynamics by gathering

large amounts of rich face-to-face interaction data automatically o using wearable sensors

o combined with statistical machine learning techniques

Find simple and robust measures derived from sensor datao that are indicative of people’s roles and relationships

o that capture the connections between physical environment and network dynamics

Questions we want to Questions we want to investigate:investigate:

Changes in social networks over time:o How do interaction patterns dynamically relate to

structural position in the network?

o Why do people sharing relationships tend to be similar?

o Can one predict formation or break-up of communities?

Effect of location on social networkso What are the spatio-temporal distributions of

interactions?

o How do locations serve as hubs and bridges?

o Can we predict the popularity of a particular location?

SupportSupport

Human and Social Dynamics – one of five new priority areas for NSFo $800K award to UW / Intel / Georgia Tech

team

o Intel at no-cost

Intel Research donating hardware and internships

Leveraging work on sensors & localization from other NSF & DARPA projects

ProcedureProcedure

Test groupo 32 first-year incoming CSE graduate students

o Units worn 5 working days each month

o Collect data over one year

Units record o Wi-Fi signal strength, to determine location

o Audio features adequate to determine when conversation is occurring

Subjects answer short monthly surveyo Selective ground truth on # of interactions

o Research interests

All data stored securelyo Indexed by code number assigned to each subject

PrivacyPrivacy

UW Human Subjects Division approved procedures after 6 months of review and revisions

Major concern was privacy, addressed byo Procedure for recording audio features

without recording conversational content

o Procedures for handling data afterwards

Data CollectionData Collection

Intel Multi-Modal Sensor Board

Real-time audio feature

extraction

audiofeatures

WiFistrength

Coded

Database

codeidentifier

Data CollectionData Collection

Multi-sensor board sends sensor data stream to iPAQ

iPAQ computes audio features and WiFi node identifiers and signal strength

iPAQ writes audio and WiFi features to SD card

Each day, subject uploads data using his or her code number to the coded data base

Older ProcedureOlder Procedure

Because the real-time feature extraction software was not finished in time, the Autumn 2005 data collections used a different process (also approved)o Raw data was encrypted on the SD card

o The upload program simultaneously unencrypted and extracted features

o Only the features were uploaded

Speech DetectionSpeech Detection

From the audio signal, we want to extract features that can be used to determineo Speech segments

o Number of different participants (but not identity of participants)

o Turn-taking style

o Rate of conversation (fast versus slow speech)

But the features must not allow the audio to be reconstructed!

Speech ProductionSpeech Production

vocal tractfilter

Fundamental frequency (F0/pitch) and formant frequencies (F1, F2 …) are the most important components for speech synthesis

The source-filter Model

Speech ProductionSpeech Production Voiced sounds: Fundamental frequency (i.e.

harmonic structure) and energy in lower frequency component

Un-voiced sounds: No fundamental frequency and energy focused in higher frequencies

Our approach: Detect speech by reliably detecting voiced regions

We do not extract or store any formant information. At least three formants are required to produce intelligible speech*

* 1. Donovan, R. (1996). Trainable Speech Synthesis. PhD Thesis. Cambridge University 2. O’Saughnessy, D. (1987). Speech Communication – Human and Machine, Addison-Wesley.

Goal: Reliably Detect Voiced Goal: Reliably Detect Voiced Chunks in Audio StreamChunks in Audio Stream

Speech Features ComputedSpeech Features Computed

1.Spectral entropy

2.Relative spectral entropy

3.Total energy

4.Energy below 2kHz (low frequencies)

5.Autocorrelation peak values and number of peaks

6.High order MEL frequency cepstral coefficients

Features used: AutocorrelationFeatures used: Autocorrelation

Autocorrelation of (a) un-voiced frame and (b) voiced frame.

Voiced chunks have higher non-initial autocorrelation peak and fewer number of peaks

(a) (b)

Features used: Spectral EntropyFeatures used: Spectral Entropy

Spectral entropy: 3.74Spectral entropy: 4.21

FFT magnitude of (a) un-voiced frame and (b) voiced frame.

Voiced chunks have lower entropy than un-voiced chunks, because voiced chunks have more structure

Features used: EnergyFeatures used: Energy

Energy in voiced chunks is concentrated in the lower frequencies

Higher order MEL cepstral coefficients contain pitch (F0) information. The lower order coefficients are NOT stored

Segmenting Speech RegionsSegmenting Speech Regions

Attributes Useful for Inferring Attributes Useful for Inferring InteractionInteraction

Attributes that can be reliably extracted from sensors:o Total number of interactions between people

o Conversation styles – e.g. turn-taking, energy-level

o Location where interactions take place – e.g. office, lobby etc.

o Daily schedule of individuals – e.g. early birds, late nighters

LocationsLocations

Wi-Fi signal strength can be used to determine the approximate location of each speech evento 5 meter accuracy

o Location computation done off-line

Raw locations are converted to nodes in a coarse topological map before further analysis

Topological Location MapTopological Location Map

Nodes in map are identified by area typeso Hallway

o Breakout area

o Meeting room

o Faculty office

o Student office

Detected conversations are associated with their area type

Social Network ModelSocial Network Model

Nodeso Subjects (wearing sensors, have given

consent)

o Public places (e.g., particular break out area)

o Regions of private locations (e.g., hallway of faculty offices)

o Instances of conversations

Edgeso Between subjects and conversations

o Between places or regions and conversations

Non-instrumented SubjectsNon-instrumented Subjects

We may recruit additional subjects who do not wear sensors

Such subjects would allow us to infer information about their behavior indirectly, and to appear (coded) as a node in our network modelo E.g., based on their particular office locations

Only people who have provided written consent appear as entities in our network models

Disabling Sensor UnitsDisabling Sensor Units

As a courtesy, subjects will disable their units in particular classrooms or offices

Access to the DataAccess to the Data

Publications about this project will include summary statistics about the social network, e.g.:o Clustering coefficient

o Motifs (temporal patterns)

We will not release the actual grapho This is prohibited by our HSD approval

We welcome additional collaborators