Working With Big Visual Cultural Data - 2015

softwarestudies.com

Analyzing Big Visual Data: theory, methods, examplesLev Manovich / [email protected] Professor of Computer Science, The Graduate Center, City University of New York (CUNY)Director, Software Studies Initiative www.softwarestudies.com

Summer 2015

1

http://www.softwarestudies.com

softwarestudies.com 2

All projects included in these slides were created by members of Software Studies Initiative and our collaborators between 2008 and 2015.

Please refer to project descriptions on softwarestudies.com for credits and further details.

http://softwarestudies.com


Research at Software Studies Initiative


BACKGROUND:MODERN ART, STATISTICS, SCIENCE,DATA SCIENCE,DATA VISUALIZATION


1. TWO MODERN ABSTRACTIONS: STATISTISTICAL GRAPHS (1800-) ABSTRACT ART (1900-)

softwarestudies.com 6statistical graphs, early 1990s


Piet Mondrian, 1909-1912


2. MODERN ART VS. MODERN SCIENCE (1600-1960):ART: showing general types though the concrete (people, landscape, etc.) SCIENCE: modeling / explaining the regular; concerned with general laws (example: linear regression: y = xB + e)


Example of how art represents general through the particular (Aleksander Deineka)

softwarestudies.com 10example of scientific method: modeling the general


3. VISUALIZATION WITHOUT REDUCTION?how to combine concrete and abstract? can we create visualizations that show patterns but do not use aggregation and abstraction?


One possible “visualization without reduction” method: in our lab we create visualizations show all images in a dataset without reducing them to points, bars, etc. By sorting the images in different ways we can see patterns.


SOFTWARE STUDIES INITIATIVE GOALS AND METHODS, 2007-

1. LOOKING AT EVERYTHING AT ONCE 2. SEEING CONTINUOS CHANGES

3. THINKING WITHOUT CATEGORIES?

4. VISUALIZING THE SOCIAL


1. LOOKING AT EVERYTHING AT ONCE

- using the complete data (or at least a larger sample that represents the phenomenon well)

- more inclusive cultural history - seeing what has been excluded - mapping contemporary cultural fields


We were invited by MoMA to analyze their whole photo collection and contribute to the OBJECT : PHOTO exhibition book


Seeing the museum collection: 20,000 photographs from MoMA,1844-1989.

Organized by year (top to bottom). Each bar shows photographs from a particular year.


closeup, 1925-1929


Visualization of 5000 paintings of French Impressionist artists

x and y - first two dimensions of PCA using 200 features

the familiar impressionist paintings (see closeup on next slide) turn to be only %10-20 of their whole creative output

softwarestudies.com 19closeup


Visualizing time-based media and user experiences (films, animations, TV programs, playing a video game):

example: visualizations of films by Dziga Vertov - using one frame from every shot (collaboration with Austrian Film Museum)

softwarestudies.com 21“Kino Pravda” (1921)

softwarestudies.com 22“The Eleventh Year” (1928)

softwarestudies.com 23“A Man with a Movie Camera” (1929)

softwarestudies.com 24“Six Part of the World”


Kingdom Hearts gameplay: 62.5 hours, 27 sessions over 20 days (left to right, top to bottom).


2. SEEING CONTINUOS CHANGES- visualizing cultural and stylistic changes in time

- seeing continuos historical change (instead of discrete periods / stages)


Mark Rothko, 393 paintings,1927-1970. X - year. Y - brightness mean.


Animating artistic development: 128 paintings by Piet Mondrian. Animated PCAvisualization using60 features. Images that are visually similar in some ways appear closely together.


4535 Time covers 1923-2009.

Organized by date, left to right, top to bottom.

Every pattern we observe is continuous, with changes taking places over years or decades.


4535 Time covers 1923-2009.

closeup: 1920s


4535 Time covers 1923-2009.

closeup: 1990s-2000s


4535 Time covers 1923-2009 (left to right). Each cover is represented by a single vertical line.

softwarestudies.com

Image plots of 4535 Time covers, 1923-2009. X-axis = date; Y-axis = saturation mean.

33

softwarestudies.com 34closeup

softwarestudies.com 35covers that have highest saturation (1960s)


Exhibition of our visualizations including Time covers, Graphic Design Museum Breda



3. THINKING WITHOUT CATEGORIES?

- from categories to continuos descriptions - computer describes properties of media using continuos variables (example: RGB color values) - instead of using a small number of categories, we extract hundreds or thousands of features from every object - - do features determine what we can see in the data?

softwarestudies.com

1 million manga pages x - standard deviation y - entropy


Closeups of the bottom left corner and top corner (previous slide). Entropy feature sorts all pages according to low detail/no texture/flat - high detail/texture/3D dimension. Visualization reveals continuos variation on this dimension. This example suggests that our standard concept of "style" may not be appropriate when looking at particular characteristics of big cultural samples (because "style assumes presence of distinct characteristics, not continuos variation across a whole dimension).


1 million manga pages plotted as points x - standard deviation y - entropy Some plot areas are densely filled in, while others are almost empty. Why manga visual language developed in this way? Visualization of a large number of samples allows us to map a cultural fields to see what is typical and what is rare, and what kind of clusters (if they exist) are present in this field.


single short manga series (>1000 pages).

Does this manga series has a coherent style on the two analyzed dimensions?


4. VISUALIZING THE SOCIAL (using visual social media)

- creating portraits of society though social media data

- using social media as lens into society - interactive interfaces for exploring

large visual social media - analysis of contemporary popular

digital photography

softwarestudies.com 44Phototrails project, 2013: analysis of 2.3 million Instagram photos collected in 13 global cities


Example of data aggregation - reducing 2.3M photos to 13 data points (one point per city)


Another plot of cities differences (using only color features)


Comparing San Francisco and Tokyo using 50K image samples. Photos are organized by average brightness (distance to plot center) and average hue (angle).

softwarestudies.com 48Comparing NYC and Tokyo using 50K image samples shared over few days (organized by upload date/time.)

softwarestudies.com 49closeup of the visualization from previous slide


Example of data aggregation: comparing Memorial and National Days in Israel (2012). Each plot shows locations of all Instagram images with geo locations shared on that day. Time of day is represented by colors (green-yellow-red). If a user shared photos within a small time interval, they are connected by lines.


Instead of aggregating the data for all users, we plot locations of photos by each user separately.

Plots show locations of all Instagram photos by top 289 users in Tel Aviv for 3 months in 2012. Each user photos are in a separate plot.


closeup of the visualization (previous slide)


Selfiecity project, 2014: analysis of 3200 Instagram selfie shared in 5 global cities. http://selfiecity.net

http://selfiecity.net

softwarestudies.com 54One of the visualizations from Selfiecity


Screenshot from interactive app selfiexploratory: http://selfiecity.net/selfiexploratory/

http://selfiecity.net/selfiexploratory/


softwarestudies.com 57SelfieSaoPaolo project, 2014. http://manovich.net/index.php/exhibitions/selfiesaopaulo

http://manovich.net/index.php/exhibitions/selfiesaopaulo


SelfieSaoPaolo project, 2014. Different views of the animated projection.

softwarestudies.com 59On Broadway project, 2015. http://http://on-broadway.nyc/

http://http://on-broadway.nyc/

softwarestudies.com 60On Broadway is an interactive installation shown at New York Public Library, 12/2004-1/2016

softwarestudies.com 61Artists team in front of On Broadway installation

softwarestudies.com 62Interface uses familiar multi-touch gestures to navigate Broadway street in Manhattan (21 km, 40M data points)


Zoomed out view: all of Broadway is visible (21 km)


Zoomed in view. User can move along Broadway in 30 meter intervals


Video showing interaction with On Broadway interactive application on a 46-inch touch screen


Video showing interaction with On Broadway interactive application on a 46-inch touch screen

The Exceptional & The Everyday project, 2014


The Exceptional & The Everyday project, 2014 http://www.the-everyday.net/

The visualization shows 13,208 Instagram images shared by 6,165 people in the center of Kiev during 2014 Ukrainian revolution ( February 17 - February 22, 2014). The photos are organized chronologically (left to right, top to bottom). The right column shows summary of the events from Wikipedia page about the revolution.

A single condensed narrative history (Wikipedia text) vs. visual experiences of thousands of people (Instagram)? The second is potentially richer - but also more difficult to interpet.

Can we narrate history without aggregation and summarization? History as timelines of million of people?

http://www.the-everyday.net/


our projects, papers, free tools: www.softwarestudies.com contact: [email protected]

http://www.softwarestudies.com

Documents

Working With Big Visual Cultural Data - 2015