Lost in MemoriesInteracting With Large Photo

Collections on PDAs

Susumu Harada, Mor Naaman*, Yee Jiun Song,

QianYing Wang, Andreas PaepckeDigital Library Project

Stanford University

JCDL 2004 2

Motivation

• Small devices ubiquitous

• Storage, bandwidth: cheap

• The new photo wallet: PDA/Cellphone

• Can have access to all my photos

• Co-present sharing

JCDL 2004 3

Challenges

• Acquisition rate

• Small screen

• Find photos now!

• In addition…

!

JCDL 2004 4

Personal Photo Collections

• Searching/browsing very difficult

• Little discernible structure to photo collections

Warning: repeat!

JCDL 2004 5

• Content-based retrieval– Basic, primitive (far from semantic)

• Manual labeling– Improved, yet cumbersome

• Visual methods for fast scanning (Zoom)– Don’t scale well, utilize unavailable screen space

Managing Personal Photos

Warning: repeat!

JCDL 2004 6

Our Approach

• Absolutely no human effort required

• Utilize automatically-captured, easy-to-get metadata, like time [and location]

Warning: repeat!

JCDL 2004

Oy Vey!

Hmm…

I can do this! [JCDL 2002]

Big Mama

JCDL 2004 8

Outline

• Timeline Browsing– What are “Events”– Timeline user interface

• Time clustering (detecting events)

• Experiment

JCDL 2004 9

Existing PDAPhoto Browsers

• Require manual organization

• Even then, is it good enough?

JCDL 2004 10

Instead

• Use natural notion of time and event:– Photos that were taken in the same

occasion and context.

JCDL 2004 11

Events: Time-based Structure

Birthday

Europe

July 4th

Feb

Jul

Sep

Shoot bursts

2003

…

JCDL 2004 12

Events

Birthday

Europe

July 4th

Feb

Jul

Sep

2pm

5pm6pm

Feb 12

CakeGifts

Sub events

13

Timeline PDA Opening Game

Automaticallycreated

“meaningful”events

“Drill in”: Click on any box, or use timeline

Number ofPhotos

Stack icon

Time range

Back button

ScrollNegative space

Go to Thumbnail view

Dates

15

Zoom in time

16

Flipping Through a

Pile in Place

17

That photo of the DB group members from WWW2003…

18

That photo of the DB group members from WWW2003…

JCDL 2004 19

Outline

• Timeline Browsing

• Time clustering (detecting events)

• Experiment

JCDL 2004 20

Event Detection in Personal Collections

• Graham et al (our project). Time as essence for photo browsing through personal digital libraries. JCDL 2002.

• U. Gargi. Time-based analysis and event clustering. HP Tech Report (2003).

• Platt et al. Phototoc: Automatic clustering for browsing personal photographs. MSR Tech Report (2003).

• Stent and Loui. Using event segmentation to improve indexing of consumer photographs. SIGIR 2001.

• Cooper et al. Automatically Organizing Digital Photographs Using Time and Content IEEE Image Processing (2004).

• More…

JCDL 2004 21

Events According to Graham (et al)

Time

1 day, 11 hours

– Detect threshold gaps (6-24 hours)– Within each segment

• Find outliers• Split at outliers

– Repeat recursively

PDA: Merge at any level – limit to 10 events

10, 15, 13, 26, 400, 21, 55,…

JCDL 2004 22

Event Tree

Days

Months

Years

JCDL 2004 23

Outline

• Timeline Browsing

• Time clustering (events)

• Experiment

JCDL 2004 24

Experiment: Basic vs. Timeline

TL thumbTL time view

Basic

Timeline

• 15 subjects

• Average 1200 personal photos

JCDL 2004 25

Tasks

• Search: find randomly chosen photo in own collection as quickly as possible

• Browsing: build a collage of– Friends– Family– Trip– Special events

“Make time span and set of occasions broad”

JCDL 2004 26

The Interface or the Organization?

• Controlled using an additional condition

• Use automatic organization (events) with basic interface

JCDL 2004 27

• Even with basic interface, “automatic” matches “manual”

• After learning, Timeline improves search time by 29% over basic.

• Timeline success rate better

• Least backtracking in Timeline (after learning)

Results

JCDL 2004 28

0

10

20

30

40

50

60

70

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69

Time (seconds)

Ful l P hoto

T humbnai l

Day

Week

Month

Y ear

Time (sec)

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61

Nu

mb

er

of

Su

bje

ct-T

ria

ls 70

60

50

40

30

20

10

YearMonthWeekDayThumbsFull

View Dwelling Times

JCDL 2004 29

• At least as good as manual organization

• Novel interface beneficial

• Learning effects considerable (even better)

Conclusions

JCDL 2004 30

• Geo-photos, of course

• Utilizing “off screen space” (classify, share)

• Data collection

Current Work

JCDL 2004 31

Off Screen Space

JCDL 2004 32

Thank You!

More details:

Proceedings

Google: Mor Naaman

mor@cs.stanford.edu

http://www-db.stanford.edu/~mor/

JCDL 2004 33

Additional Slides