1

2007 Pittsburgh Brain Activity Interpretation Competition Interpreting subject-driven actions and sensory experience in a

rigorously characterized virtual world Overview Welcome

This is a short auditory graphical overview of the

competition.

For details see, please see our web page at:

http://www.braincompetition.org

Speakers: Walter Schneider & Greg Siegle of the U. of Pittsburgh

Call in speakers:

• Emanuele Olivetti; ITC-IRST, Italy

• Greg Stephens; Princeton U., USA

• Alexis Battle; Stanford U., USA

© 2007 University of Pittsburgh

2

WebCast Features

• May need to turn off pop-up blockers

• Submit Questions (Live only)

• Answer Survey

• Enlarge Slide Window

• Skip Slides (Download Only)

3

Goals of Competition

• Advance the understanding of how the brain represents and manipulates information dynamically during real-world behaviors.

• Show off the “power” of brain imaging

• Enable scientists from many disciplines and nations, including nonimagers, to develop new brain interpretation methodologies from the same data

• Provide focus, data, and educational materials to expand research in brain interpretation

• Give top groups visibility

4

Overview of competition• Who can compete:

– People from many disciplines, nations, and types of positions (students, faculty, scientists, engineers)

– Individuals or groups

– Cross disciplinary teams

– Classes

(Note: only one cash prize per institution)

5

Overview of what to do

• You will be examining the brain activity and feature ratings of 3 people operating in a virtual reality world

• Using fMRI you will predict what individuals perceive and how they act and feel in a novel Virtual Reality world involving searching for and collecting objects, interpreting changing instructions, and avoiding a threatening dog

• Develop classifier systems such that for Run1 and Run2 you can predict the related feature rating data

• Apply that classifier to the Run3 brain activity data to predict the feature vectors produced during Run3

6

Brain Activity and Eye Movement Data Collection

• Data was collected at the Brain Imaging Research Center in Pittsburgh http://www.birc.pitt.edu/

• Brain activity data on 3 subjects during 3 VR runs

• Eye movement data was collected during the runs

Auditory

7

Task: Anthropologist Search of VR World

Interactive environment with subject action providing:

– Perception based on normal interaction in a complex environment

– Differential executive tasking

– Complex multiple object search

– Interactions with objects

– Threat processing via a snarling dog

– Reward processing with money on the line

– Objective eye movement based feature processing

Anthropologist visiting a neighborhood collecting artifacts, taking pictures of piercing and avoiding a dog.

8

Anthropologist Search Environment

Anthropologist visiting a neighborhood collecting artifacts, taking pictures of piercing and avoiding a dog. Visiting multiple streets and interiors.

9

Anthropologist Search

• 1) Collect a sequence of fruits in order (apple, grapes banana pineapple), ignore vegetables

Four tasks called in on a cell phone.

• 2) Collect toy weapons (ignoring other objects such as tools)

10

Anthropologist Search• 3) Take pictures of the people in the neighborhood with

piercing.

• 4) Stay away from stray dog that growls before possible attack and costs real money every time you’re bitten.

11

Rich Graphic Environment

VR2 world copyright Psychology Software Tools Inc.

12

High Quality VR Graphics

13

Sample Video of a Run

14

Task Design• Train subjects ~ 4 hours in the environment before magnet

run.

• Signs in the environment provide information as to where objects are

• In magnet, 3 runs of ~20 minutes per subject

• Provide subject money at the beginning that they can loose Pay subjects bonus to visit all areas

• After run subject rates continuous arousal, emotional valence (positive/negative) and discrete emotions (happy, sad, annoyed/angry, fearful/anxious, neutral).

15

Scanning

• Similar to last year: 3T Allegra 30 slices 1.75s TR, reverse EPI full head coverage

• Concurrent eye tracking

• 3 subjects

• Full structural data will be made available

16

Data Coding• Behavioral ratings

– Three 1 hr sessions rating sequentially arousal, valence, discrete emotions

Subject and expert rating data (2 + 6) x860(1.75 s)

17

Objective Eye Fixation Coding• Eye movement traces

– Replay world in object colorized mode– Overlay eye fixation on image– Eye movement track (expect 80% good

data) overlaid on world– Identify fixations (200ms minimal

movement)– Analyze fixation frame to identify

object contact.

AObject = Σ pixelsf(ecentricityx,y)

AFeature = AObject(Feature) /Σall AObject

f(ecentricityx,y)

Eye Object Contact

0 1 2 3 4 5 6

Time (s)

Face

Fruit

Dog

Face

Fruit

DOG

18

Motion, Location, Action Coding• Code velocity, rotation

• Interior/exterior

• Actions Grab object

Location & Action Data

0 50 100 150 200 250

Time (s)

Motion

Interior/ext

Bar

Grab Object

Motion

Interior/Exterior

Bar

GrabObject

19

Required Feature Vectors Required Features

Code Feature Description Rating Type

R1 Arousal How much does what is going on in the scene affect how calm the subject is (subjective rating) Subjective

R2 Valence How positive or negative is the environment Subjective

R3 Music Degree to which subject heard music in the environment Computed

R4 Hits Times when subject correctly picked up fruit or weapon or took picture of a pierced person Computed

R5 SearchPeople Times when subject searched for pierced people Computed

R6 SearchWeapons Times when subject searched for weapons Computed

R7 SearchFruit Times when subject searched for fruits Computed

R8 Instructions Times when task instructions were presented Computed

R9 Dog Times when dog was seen or heard by subject Computed

R10 Faces Times when subject looked at faces of a pierced or unpierced person Computed

R11 FruitsVegetables Times when subject looked at fruits or vegetables Computed

R12 WeaponsTools Times when subject looked at weapons or tools Computed

R13 InteriorExterior Times when subject was inside a building (1=subject was inside, 0=subject was outdoors) Computed

R14 Velocity Times when subject was moving but not interacting with an object Computed

20

Some Brief Background fMRI

• You will be looking at fMRI data that low frequency filters the data

Feature Ratings

21

time [s]0.5÷2 4 10

stimulus

0-0.5

15

% s

igna

l cha

nge

fMRI Data Brain Activity DataBOLD -Blood Oxygenation Level Dependent contrast

Neural pathway Hemodynamics MR Signal

22

Feature HRFFull Run Feature Vectors

0 200 400 600 800 1000 1200 1400

Time (seconds)

Act

ivit

y

Cell

Fruit

Photo

Correct

Vel

200 Seconds Feature Vectors

360 410 460 510 560

Time (seconds)

Ac

tiv

ity

Cell

Fruit

Photo

Correct

Vel

23

Example Activation of Cell Phone

24

Diagram of VR Runs

Brain activation data 34x64x64x704 (1.75s)

Feature Vectors Computed & Subjective 14x704(1.75 s)

after hemodynamic lag

25

Basic Analysis Step

Reduce Dimensionationality

of Data

Do PostPost Process

Clean-Up

Preprocess data (spatial & temporal

filtering)

Score Fit

VR Run1

Load VR Run1 Sub1

4D Functional Data

Regress 2 D activation to

Feature Hemodynamics

26

Example Linear Prediction Approach

=

For a linear system you can solve for betas by taking inverse

100 highest r voxels

To predict each feature calculate the betas to linearly predict feature strength from Activation Table (ROI,Time)

n – number of time points

k – number brain areas

Note this approach is meant as an illustration only and can be done as an exercise to learn to work with the data

BETA

0 0.2 0.4 0.6 0.8

1

2

3

4

5

6

7

8

FEATURE(Time)

27

Developing/training techniques

• Use the data from Run1 & Run2 to develop the ability to go from the brain activation data to the feature data

VR Run1 VR Run2

28

Prediction of rating data for VR Run3Do not know what VR events occurred

? ? ?? ?

Brain activation data 34x64x64x=704 (1.75s)

Use techniques to predict rating data

Subject and expert rating data 23x704(1.75 s) after hemodynamic lag

29

Using matlab to process the data

• Load our provided data files into matlab

>> load sub1_run1_baseregs.mat>> load sub1_run1_fmri.mat>> load sub1_run1_vr_mask.mat>> whos Name Size Bytes Class

baseregs_conv_run1 1x704 5632 double array baseregs_run1 1x704 5632 double array epi_run1 35499x704 199930368 double array featurenames_run1 1x1 78 cell array wholebrain_run1 64x64x34 1114112 double array

0 200 400 600300

350

400

450

500

0 100 200 300 400 500 600 700

-2

0

2

30

Regression in matlab

Detrendfor ct=1:size(epi_run1,1)

epi_run1(ct,:)=detrend(epi_run1(ct,:));

end

Regress% get zero order relationshipsfor ct=1:size(epi_run1,1)

xy=corrcoef(baseregs_run1(1,:),epi_run1(ct,:));

corrs(ct)=xy(1,2);

end

% find the best voxels (I.e., with correlation >.25)

[inds]=find(abs(corrs>.25));

voxtouse=epi_run1(inds,:);

% simultaneously regress all the voxels against the feature vector

[Rsq,B,B0,Ypred]=mreg(voxtouse',baseregs_conv_run1');

0 100 200 300 400 500 600 700 800-100

0

100

0 0.5 1 1.5 2 2.5 3 3.5 4

x 104

-0.5

0

0.5

R2=.69!!0 100 200 300 400 500 600 700 800

-0.5

0

0.5

1

1.5Regressor

Prediction

0 200 400 600 800300

350

400

450

500

Voxel #

correlation

0 100 200 300 400 500 600 700 800-0.5

0

0.5

1

1.5Regressor

Prediction

31

Plot the activations that were predictive

% make a brainimage by overlaying the predictions on the wholebrain image[goodvox]=find(wholebrain_run1);predbrain=zeros(size(wholebrain_run1));for ct=1:length(goodvox) predbrain(goodvox(ct))=corrs(ct).*(corrs(ct)>.25);end

% plot a few slices to see the activationsfor ct=11:15 subplot(1,5,ct-11+1); pcolor((wholebrain_run1(:,:,ct)'+5.*predbrain(:,:,ct)')); shading interp; colormap bone; view(-180,90); axis off;end

32

Or, use the MVPA toolkit fromhttp://www.csbmb.princeton.edu/mvpa/

• Works with our matlab data

• Creates subject record in which it’s easy to reference features

• Allows detrending, whole brain regressions, etc. from within a toolkit

• Has the whole process the 2nd place group used last year!

33

Example functional data sets

DICOM slices

We provide multiple formats to minimize the start up time and allow those without specific brain imaging experience to get the benefits from experts on the preprocessing stages.

For example, some computer science data-mining students that may enter the competition may have no brain imaging experience. Given that, we will make available the data preprocessed.

DICOM, Analyze or MatLab formats provided

Analyze Volumes MatLab Matrixes

34

What we’ll score

• Top 3 scores:– Those with the highest average correlation of

predictions to run 3 feature vectors. – There are 14 extra credit features – we’ll average in

your top 5 – so you can tune your exploration to the features you’re interested in.

• Special Cognitive Neuroscience Prize:– The group with the best “story” that informs cognitive

neuroscience.– E.g., how functional connectivity leads to features or

how the dog interferes with search behaviors.

35

Submission

• Submit Predicted Run3 data to competition before May 21, 2007

– Note you are only allowed to submit the Run3 predictions 3 times

– Test out the methods on Run1 and Run2 data sets

• Submission enabled April 16

36

Keep this a Collegial Competition• You might work with multiple people at your site

• We will be providing resources (e.g., readings, additional formats, routines) that people wish to share

• If you have a processing step you would like others to try and comment on contribute them

• We will be doing special events (web conferences) if requested

37

Keep in Contact

• We will provide posting on the discussion board and major notices via email

• We will be posting updates on procedures and corrections of documents

38

Hope To See You In Chicago

• We will present awards June 14, 2007 at the Organization for Human Brain Mapping in Chicago, Illinois, USA

• http://www.humanbrainmapping.org/

• We have space for posters and an hour long workshop

39

Competition Board of Scientists

G. Siegle & W. Schneider (University of Pittsburgh – coordinating site)

A. Bartels (Max Planck Institute for Biological Cybernetics)

E. Formisano & R. Goebel (Maastricht University)

J. Haxby & G. Stephens (Princeton University)

U. Hasson (New York University & Weizmann Institute)

T. Mitchell (Carnegie Mellon University)

T. Nichols (University of Michigan)

A. Battle (Stanford University)

E. Olivetti, (ITC-IRST; Italy)

40

Credit & Contact

• For questions, e-mail ebc@pitt.edu

• Experience Based Cognition team members: – PI: Walter Schneider & Greg Siegle

• Pittsburgh Technical staff: Kate Fissell, Lena Gemmer, Kevin Jarbo, Dan Jones, Lori Koerbel, Kyung Hwa Lee, Adrienne McGrail, Maureen McHugo, Sudhir Pathak, David Pfendt , Melissa Thomas

• Psychology Software Tools Inc. for VR worlds Kyle Brauch, Tom Yothers

41

Comments from the Winners

Diego Sona(1), Greg Siegle(2), Emanuele Olivetti(1), Sriharsha Veeramachaneni(1), Alexis Battle(3), Greg Stephens(4), Walter Schneider(2)

42

Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks

Alexis Battle, Gal Chechik, Daphne Koller

Stanford University AI http://ai.stanford.edu

43

Predicting Base Features with SupervoxelsTop (left-to-right): Ken Norman, Denis Chigirev, Matt Weber, Shannon

Hughes, Eugene Brevdo, Melissa Carroll. Bottom (left-to-right): Christopher Moore, Greg Detre, Greg Stephens

Center for the Study of Brain, Mind and Behavior:http://www.csbmb.princeton.edu

44

TGaussian process regression and recurrent neural networks for fMRI image classification

Emanuele Olivetti, Sriharsha Veeramachaneni, Diego Sona

ITC/IRST (Center for Scientific and Technological Research) is a public research Institute located in Trento in northern Italy http://sra.itc.it/

45

Answering Questions

For competition details see, http://www.BrainCompetition.org

46