Download pdf - Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Transcript
Page 1: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Algonauts Workshop — July 19, 2019 — MIT

Deep generative networks as models of the visual system

Thomas NaselarisDepartment of Neuroscience

Medical University of South Carolina

(MUSC)

Charleston, SC

Page 2: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

t = 0

while dead==False:

thought[t] = f(thought[:t],

world[:t],

plans[:t])

if thought[t] is fatal:

dead = True

else:

t += 1

infer the human algorithm

behavior

brain

world

Page 3: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

What should the human (visual) algorithm do?

Arbitrary queries over representations

Page 4: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Does the dog have pointy ears?What is there?

A dog is there.

”clamped”

Page 5: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

me

nta

l

ima

ge

ryvis

ion

Page 6: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

me

nta

l

ima

ge

ryvis

ion

Breedlove, St-Yves, Naselaris et al., in rev.

Page 7: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

HOW TO TEST NETWORK AGAINST HUMAN BRAINS?

Page 8: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

“ababie”

Cue Picture

An experiment:

Breedlove, St-Yves, Naselaris et al., in rev.

Page 9: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 10: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 11: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 12: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Imagine objects

Page 13: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 14: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 15: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Breedlove, St-Yves, Naselaris et al., in rev.

Page 16: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

correlationcorrelation

vEM iEM

Visual encoding model

(vEM) predicting voxel-

wise brain activity during

visual task

Imagery encoding model

(iEM) predicting voxel-

wise brain activity during

imagery task

Prediction accuracy maps for visual and imagery encoding models

Page 17: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Tu

nin

g

Spatial Frequency

Tuning to seen and imagined spatial frequencies

Breedlove, St-Yves, Naselaris et al., in rev.

Page 18: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

larger RF

Receptive fields for seen and imagined stimuli

Breedlove, St-Yves, Naselaris et al., in rev.

Page 19: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

RF eccentricity shift

RF size shift

Receptive fields for seen and imagined stimuli

Breedlove, St-Yves, Naselaris et al., in rev.

Page 20: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

A DEEP GENERATIVE MODEL CAN

PREDICT DIFFERENCES IN

ENCODING OF SEEN AND MENTAL

IMAGES

Page 21: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

BUT IS THERE A DEEP GENERATIVE

MODEL THAT CAN ACCURATELY

PREDICT ACTIVITY DURING VISION OF

NATURAL SCENES?

Page 22: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Han et al, Neuroimage 2019

A generative model A discriminative model

A DCNN-based

encoding model

yields more accurate

predictions of brain

activity in all visual

areas than an

encoding model

based on a state-of-

the-art deep

generative network.

Page 23: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

SO IS THAT A “NO” ON THE GENERATIVE MODEL

IDEA?

PERHAPS THE “RIGHT” GENERATIVE MODEL IS HARD

TO LEARN FROM IMAGE DATA ALONE.

MIGHT WE INFER IT DIRECTLY FROM BRAIN

RESPONSES?

Page 24: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models
Page 25: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

IT’S NOT YET CLEAR IF THIS WILL WORK.

BUT IT’S CLEAR THAT MORE DATA REALLY HELPS

Page 26: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

DCNN- vs. Gabor-based

encoding models, ~5K data

samples from the (incomplete)

NSD

DCNN- vs. Gabor-

based encoding models,

~1.5K data samples

from vim-1

Page 27: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

DCNN- vs. Gabor-based

encoding models, ~5K data

samples from the (incomplete)

NSD

DCNN- vs. Gabor-based

encoding models, ~5K data

samples from the (incomplete)

NSD

Data-driven vs. DCNN-based

encoding models, ~5K data

samples from the (incomplete)

NSD

Page 28: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

TAKE-HOME

THE VISUAL SYSTEM CAN POSE

AND ANSWER MANY DIFFERENT

QUERIES. SO SHOULD OUR

MODELS.

A DEEP GENERATIVE MODEL CAN

PREDICT DIFFERENCES IN

ENCODING OF SEEN AND MENTAL

IMAGES…

Page 29: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

TAKE-HOME

…BUT CANNOT PREDICT RESPONSES TO

NATURAL SCENES AS ACCURATELY AS

MODELS BASED ON A DISCRIMINATIVE

NETWORK.

WE NEED BETTER THEORY. AND MORE

DATA.

MORE DATA IS ON THE WAY.

Page 30: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

NIH R01 EY023384

BRAIN N00531701

NSF IIS-1822683

Page 31: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

NSD Collaborators

Page 32: Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

https://ccneuro.org/2019/

https://ccneuro.org/2019/

Recommended

Deep Generative Models for Spatial Networks
Deep Generative Models for Spatial Networks Documents
Generative Deep Neural Networks for Dialoguemausam/courses/col864/spring2017/slides/15-h… · Generative Deep Neural Networks for Dialogue Presented By Shantanu Kumar Adapted from
Generative Deep Neural Networks for Dialoguemausam/courses/col864/spring2017/slides/15-h… · Generative Deep Neural Networks for Dialogue Presented By Shantanu Kumar Adapted from Documents
10. Generative Adversarial Networks EE-559 { Deep learningAdvances in generative models that can sample but not estimate ... { Deep learning / 10. Generative Adversarial Networks 18
10. Generative Adversarial Networks EE-559 { Deep learningAdvances in generative models that can sample but not estimate ... { Deep learning / 10. Generative Adversarial Networks 18 Documents
DAGAN: Deep De-Aliasing Generative Adversarial Networks ... · conditional Generative Adversarial Networks-based model (DAGAN)-based model is proposed to reconstruct CS-MRI. In our
DAGAN: Deep De-Aliasing Generative Adversarial Networks ... · conditional Generative Adversarial Networks-based model (DAGAN)-based model is proposed to reconstruct CS-MRI. In our Documents
Deep Learning Generative Models in Wireless Networks · 2019-03-29 · Deep Learning Generative Models in Wireless Networks Wireless AI Innovation @ Verizon (WAIV) ... Enable detection
Deep Learning Generative Models in Wireless Networks · 2019-03-29 · Deep Learning Generative Models in Wireless Networks Wireless AI Innovation @ Verizon (WAIV) ... Enable detection Documents
Unsupervised Image Reconstruction using Deep Generative ...web.stanford.edu/~ekcole/sedona.pdfUnsupervised Image Reconstruction using Deep Generative Adversarial Networks Elizabeth
Unsupervised Image Reconstruction using Deep Generative ...web.stanford.edu/~ekcole/sedona.pdfUnsupervised Image Reconstruction using Deep Generative Adversarial Networks Elizabeth Documents
Generative Deep Neural Networks for Inverse Materials
Generative Deep Neural Networks for Inverse Materials Documents
Lecture 9: Unsupervised, Generative & Adversarial Networks · PDF fileuva deep learning course –efstratios gavves unsupervised, generative & adversarial
Lecture 9: Unsupervised, Generative & Adversarial Networks · PDF fileuva deep learning course –efstratios gavves unsupervised, generative & adversarial Documents
Deep Generative Models - microsoft.com€¦ · •Auxiliary deep generative networks [Maaløe et al., 2016] •Inverse autoregressive flow (IAF) [Kingma et al., NIPS 2016] •Householder
Deep Generative Models - microsoft.com€¦ · •Auxiliary deep generative networks [Maaløe et al., 2016] •Inverse autoregressive flow (IAF) [Kingma et al., NIPS 2016] •Householder Documents
Generative Adversarial Networks (GANs) - Ian Goodfellotions. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to gen-erate highly compelling images of
Generative Adversarial Networks (GANs) - Ian Goodfellotions. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to gen-erate highly compelling images of Documents
A Deep Convolutional Generative Adversarial Networks … · In this paper, we present a semi-supervised learning method that is based on the standard deep convolutional generative
A Deep Convolutional Generative Adversarial Networks … · In this paper, we present a semi-supervised learning method that is based on the standard deep convolutional generative Documents
Deep Generative Models with Learnable Knowledge Constraintspapers.nips.cc/paper/8250-deep-generative-models-with-learnable... · Deep Generative Models with Learnable Knowledge Constraints
Deep Generative Models with Learnable Knowledge Constraintspapers.nips.cc/paper/8250-deep-generative-models-with-learnable... · Deep Generative Models with Learnable Knowledge Constraints Documents
Watch Your Up-Convolution: CNN Based Generative Deep Neural … · 2020-06-29 · Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral
Watch Your Up-Convolution: CNN Based Generative Deep Neural … · 2020-06-29 · Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Documents
Deep generative learning_icml_part1
Deep generative learning_icml_part1 Science
ODE2VAE: Deep generative second order ODEs with ......ODE2VAE: Deep generative second order ODEs with Bayesian neural networks C¸ agatay Yıldız˘ 1, Markus Heinonen1,2, Harri Lahdesm¨
ODE2VAE: Deep generative second order ODEs with ......ODE2VAE: Deep generative second order ODEs with Bayesian neural networks C¸ agatay Yıldız˘ 1, Markus Heinonen1,2, Harri Lahdesm¨ Documents
DEEP-LEARNED GENERATIVE …people.cs.umass.edu/~hbhuang/files/thesis_lowres.pdfDEEP-LEARNED GENERATIVE REPRESENTATIONS OF 3D ... DEEP-LEARNED GENERATIVE REPRESENTATIONS OF 3D SHAPE
DEEP-LEARNED GENERATIVE …people.cs.umass.edu/~hbhuang/files/thesis_lowres.pdfDEEP-LEARNED GENERATIVE REPRESENTATIONS OF 3D ... DEEP-LEARNED GENERATIVE REPRESENTATIONS OF 3D SHAPE Documents
Deep Generative Stochastic Networks Trainable by Backprop · Deep Generative Stochastic Networks Trainable by Backprop shallower models without them (Table 1). 6 – Dependency networks:
Deep Generative Stochastic Networks Trainable by Backprop · Deep Generative Stochastic Networks Trainable by Backprop shallower models without them (Table 1). 6 – Dependency networks: Documents
O U DEEP GENERATIVE MODELS...Another emerging family of deep generative models is the Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), in which a discriminator is
O U DEEP GENERATIVE MODELS...Another emerging family of deep generative models is the Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), in which a discriminator is Documents