Computational Media

2005.02.15 SLIDE 1IS146 – SPRING 2005

Computational Media

Prof. Marc Davis & Prof. Peter Lyman

UC Berkeley SIMS

Tuesday and Thursday 2:00 pm – 3:30 pm

Spring 2005http://www.sims.berkeley.edu/academics/courses/is146/s05/

IS146:

Foundations of New Media

2005.02.15 SLIDE 2IS146 – SPRING 2005

Lecture Overview

• Assignment Check In– Assignment 3: Documenting Artifact Usage

• Review of Last Time– Computation: Programmability

• Today– Computational Media

• Preview of Next Time– New Media On The Go and In The Home

2005.02.15 SLIDE 3IS146 – SPRING 2005

Lecture Overview





2005.02.15 SLIDE 4IS146 – SPRING 2005

Lecture Overview





2005.02.15 SLIDE 5IS146 – SPRING 2005

Programming Concepts

• Basic programming constructs– Parameters– Loops– Procedural abstraction– Subroutines– Conditionals

2005.02.15 SLIDE 6IS146 – SPRING 2005

Making a “C”

• to c– params [height]– make halfheight :height/2– left 90– forward :height– right 90– forward :halfheight – right 180– forward :halfheight– left 90– forward :height – left 90– forward :halfheight– end

2005.02.15 SLIDE 7IS146 – SPRING 2005

Making an “A”

• to a– params [height]– make halfheight :height/2– left 90– forward :height– right 90– forward :halfheight – right 90– forward :halfheight – right 90– forward :halfheight– right 180– forward :halfheight– right 90– forward :halfheight– left 90– end

2005.02.15 SLIDE 8IS146 – SPRING 2005

Making an “M”

• to m– params [height]– make diagonal (:height/2)*7/5– left 90– forward :height– right 135– forward :diagonal – left 90– forward :diagonal – right 135– forward :height– left 90– end

2005.02.15 SLIDE 9IS146 – SPRING 2005

Making an “R”

• to r– params [height]– make halfheight :height/2– make diagonal :halfheight*7/5– left 90– forward :height– right 90– forward :halfheight – right 90– forward :halfheight – right 90– forward :halfheight – left 135– forward :diagonal– left 45– end

2005.02.15 SLIDE 10IS146 – SPRING 2005

Making “MARC”

• to marc– params [height kerning]– m :height– space :kerning– a :height– space :kerning– r :height– space :kerning– c :height– end

2005.02.15 SLIDE 11IS146 – SPRING 2005

Making a Circle of “MARC”

• to marccircle– params [letterheight letterkerning]– make marcnamewidth

((:letterheight*5/2)+(3*:letterkerning))– repeat 360/:letterheight – [marc :letterheight :letterkerning– hopback :marcnamewidth – right :letterheight– ]– end

2005.02.15 SLIDE 12IS146 – SPRING 2005

Conditionally Making “MARC” Circles• to marccirclecond

– params [letterheight letterkerning circletightness]– make marcnamewidth

((:letterheight*5/2)+(3*:letterkerning))– ifelse (:circletightness=0)– [make rotation :letterheight] – [make rotation :letterkerning]– repeat 360/:rotation [marc :letterheight :letterkerning– hopback :marcnamewidth – right :rotation– ]– end

2005.02.15 SLIDE 13IS146 – SPRING 2005

Devin Blong on Papert

• Papert repeats the idea that when children are taught to program, they are more self directed and active. – “By contrast, when a child learns to program, the

process of learning is transformed. It becomes more active and self directed. In particular, the knowledge is acquired for a recognizable personal purpose […].The new knowledge is a source of power and is experienced as such from the moment it begins to form in a child’s mind.”

• This is juxtaposed with the idea that programming is a normal, rather than strange and foreign skill for a child to learn. Why then is this knowledge more powerful than other types of knowledge?

2005.02.15 SLIDE 14IS146 – SPRING 2005

Devin Blong on Papert

• What good is a language if it is not spoken?• How does the programming process of

constantly debugging relate to communication metaphors?

• What are some of the self-imposed barriers that keep technology from moving forward today (e.g., QWERTY) ?

• How does lacking a “vocabulary” in a particular area affect your understanding and learning in that area?

2005.02.15 SLIDE 15IS146 – SPRING 2005

Trevor Newhouse on Papert

• Papert writes of turtle geometry as being mathetic in nature, or knowledgeable about learning. If the logo turtle can be characterized by such a term, is there a better one to describe a developing child’s mind. If logo, or computer programming is the logically definitive way to systematically learn, how then can we account for human instinctual preference and our version of “once removed” learning. Can a computer be programmed with these devices? Has it already?

2005.02.15 SLIDE 16IS146 – SPRING 2005


• Is syntonic learning deemed as enjoyable because, in the case of programming, it asks you to break apart what you already know, eliminate and clutter, and then build it back up cleanly? Could it be that there is some relaxing quality in this kind of, Cartesian spring cleaning?

2005.02.15 SLIDE 17IS146 – SPRING 2005


• The conversation Papert highlights between two kids programming a flower utilizes a great deal of “repeat” commands and storing. Psychologically, and anthropologically speaking, is this ability of a computer to instantly copy work that took a man hours or days to create ultimately a good thing? Are there enough sci-fi movies out there to give us pause?

2005.02.15 SLIDE 18IS146 – SPRING 2005


• Papert describes juggling using computation certainty, however in real life juggling there exist myriad environmental variables that can affect performance. Weather, hangovers, noise levels, slippery fingers… such factors act as unforeseen variables going into the total juggling experience. Are there such variables in computer programming?

2005.02.15 SLIDE 19IS146 – SPRING 2005

Lecture Overview





2005.02.15 SLIDE 20IS146 – SPRING 2005

Lecture Overview





2005.02.15 SLIDE 21IS146 – SPRING 2005

What New Media Is Not Defined By

• Digitized analog media

• Media displayed on a computer

• Random access media

• Necessarily having less information than analog media

• Necessarily being able to be copied without generation loss

• Being “interactive”

2005.02.15 SLIDE 22IS146 – SPRING 2005

Manovich on New Media

• Numerical representation– Can be described formally (mathematically)– Can be manipulated algorithmically (programmability)

• Modularity– Constructed out of substitutable components

• Automation– Automation of media creation, manipulation, and access– Low level (bits) and high level (semes) automation

• Variability– Media objects can have potentially infinite versions– Media database, separation of data and interface,

customization/personalization, branching-type interactivity, hypermedia (links), periodic updates, scalability (e.g., resolution)

• Transcoding– … Media => Data => Media …

2005.02.15 SLIDE 23IS146 – SPRING 2005

AutoBuddy Example

2005.02.15 SLIDE 24IS146 – SPRING 2005

• Movies change from being static data to programs

• Shots are inputs to a program that computes new media based on content representation and functional dependency (US Patents 6,243,087 & 5,969,716)

Central Idea: Movies as Programs

Parser

Parser

Producer

Media

Media

Media

ContentRepresentation

ContentRepresentation

2005.02.15 SLIDE 25IS146 – SPRING 2005

AutoBuddy: A Computed Movie

• Driver and Gunner play “Zone Raiders” a first-person driving shooter videogame

• The Gunner can only shoot in the direction the Driver drives

• The Driver cannot shoot• The Driver and Gunner can

talk to and hear each other over headphones, but cannot not see each other

• They both hear and see the same videogame output

• AutoBuddy films the Driver and Gunner with digital video cameras

• AutoBuddy computes a “buddy driving movie” from edited according to the patterns of conversation and game play between the Driver and Gunner

Computerw/ video game

TV

Camera

DriverTV

Gunner

Camera

AutoBuddySoftware

Buddy Driving Movie

2005.02.15 SLIDE 26IS146 – SPRING 2005

How AutoBuddy Makes the Movie

3 Digital Movies (QuickTime)

Synchronize & Crop

Create Shots

Dialog-based Cutting

Add Credits

• Synchronize movies (AudioStreams)• Find beginning and end of game

• Create 3 new movies: - Driver in car - Gunner in car - Both in car• Add parametric special effects

• Driver, Gunner, and Video Game

• Cut between 3 new movies and game video based on who is talking• Cutting rules for continuity editing

• Insert stills of Driver, Gunner

2005.02.15 SLIDE 27IS146 – SPRING 2005

AutoBuddy Dialog-Based Cutting

• AutoBuddy analyzes the Driver and Gunner audio to determine who is speaking at each point in movie

• Produces a stream of speech events with durations and values (Driver, Gunner, both, or neither)

GunnerDriver Pause Both Gunner Pause

time

Gunner

2005.02.15 SLIDE 28IS146 – SPRING 2005

Dialog-Based Cutting

• AutoBuddy uses a set of cutting rules that cut between shots based on patterns of speech events

• Example: if Driver speaks and then Gunner speaks, show Driver and cut to Gunner slightly before Gunner starts to speak

• Example: if there is a long pause between Driver and Gunner speaking, cut to the game video

Input Speech Events:

Output Video Cuts:

Driver Pause

Driver Gunner

Gunner

2005.02.15 SLIDE 29IS146 – SPRING 2005

AutoBuddy Composite Shots

• Driver, Gunner, and Both shots are multi-layer composites

• View out the car rear window is generated video games rear view mirror image– Flipped, scaled, smoothed, and placed

• Back of car is a static image from a 3D model• Images of Driver/Gunner are generated by

background subtraction• Front of car is a static image from a 3D model

2005.02.15 SLIDE 30IS146 – SPRING 2005

AutoBuddy Special Effects

• Car is shaken based on “gas pedal”– Gas pedal parsed from acceleration indicator in game

video– Car and people are shaken 90 degrees out of phase

• Gunfire art added to frames based on game audio– Audio Streams used to detect gunfire in game audio– Able to detect gunfire even when other audio effects

present• Explosions

– Game video analyzed to determine when explosions happen

– Images are lightened and rumbled during explosions

2005.02.15 SLIDE 31IS146 – SPRING 2005

Computation for Designing Artifacts

• Four computational ideas/techniques from Carlo Sequin– Procedural generation– Parameterization– Optimization– Evolutionary power

2005.02.15 SLIDE 32IS146 – SPRING 2005

Procedural Generation

• Rather than creating artifacts directly, the user may design a generating program that will then generate the desired artifact

• The empowering aspect of this approach is that the generating procedure will not just create the one artifact originally desired, but, with minor variations to the program, it can produce many different artifacts that may all fit a specified set of constraints or usage

2005.02.15 SLIDE 33IS146 – SPRING 2005

Parameterization

• For classes of frequently needed artifacts, the procedural generation mentioned above can be captured in a robust and more general program that contains a modest number of parameters that can be easily adjusted by non-programming users

• A judicious selection and coupling of such parameters can enhance the likelihood that any arbitrary combinations of parameters still produce a meaningful output, although it may be far from desirable or optimal with respect to some specific application

• However, the ease of modifying the parameter values and previewing the expected outcome, would allow even novice users to achieve better than average results obtained by un-aided users

2005.02.15 SLIDE 34IS146 – SPRING 2005

Optimization

• Given that the tedium of creating individual artifacts can be greatly reduced by procedural generation, users can explore a far larger space of possibilities than they could if they had to craft each artifact individually

• This allows them to home in on a more optimal solution than they could by building a few prototypes

• If the constraints and goal functions are well understood, then the generating program may contain its own evaluation loop that allows it to explore many options on its own and gradually converge towards a local optimum

2005.02.15 SLIDE 35IS146 – SPRING 2005

Evolutionary Power

• The ease of exploration afforded by the use of procedural generation permits an informed user to more clearly see and locate the boundaries of the paradigm captured in a generating program

• By making these boundaries more visible, it also becomes more obvious to ask what lies beyond

• Often such questions can be answered with a modest re-programming effort that enlarges the scope of the generator

2005.02.15 SLIDE 36IS146 – SPRING 2005

Steven Lybeck on Benjamin

2005.02.15 SLIDE 37IS146 – SPRING 2005

Mark Martell on Manovich

• Why does Manovich carefully avoid using "digital media" and "new media" interchangeably? Are they, in fact, interchangeable? Is one a subset of the other? Can you think of examples of non-digital new media?

2005.02.15 SLIDE 38IS146 – SPRING 2005


• In discussing the principle of Variability, Manovich asserts that: "Every hypertext reader gets her own version of the complete text by selecting a particular path through it… New media objects assure users that their choices [thoughts and desires] are unique, rather than preprogrammed and shared with others." Yet this assertion seems to overlook the non-linear and unique consumption patterns of "old media" such as magazines and newspapers (who reads either linearly from front to back?), visual art work and photographs (does everyone begin and end at the same points on a photo?), even radio and television (consider how the individual often flips his attention back and forth between programs and other activities or thoughts). Furthermore, web logs may reveal just a few patterns of site consumption among visitors – and these may, in fact, be deliberate. Webmasters may want visitors to consume the site in a particular path order and may engineer the site to achieve this end. Does Manovich exaggerate in making a case for the principle that new media produces unique consumption patterns – and if so, why?

2005.02.15 SLIDE 39IS146 – SPRING 2005


• Manovich also goes on to refute several popular notions of unique properties of media. He critically examines notions that:– New media alone allows for random access. He

refutes this point by claiming that Edison developed a random access media format (which was never mass produced). Does his refutation hold water in light of the overwhelming prevalence of non-random access storage (video, photos, film)?

– Digitally-encoded media can be copied endlessly without degradation. Manovich notes, however, that the overwhelming majority of distribution of digital media is done in compressed, lossy format. Is his argument valid?

2005.02.15 SLIDE 40IS146 – SPRING 2005

Lecture Overview





2005.02.15 SLIDE 41IS146 – SPRING 2005

Readings for Next Time

• Paul du Gay, Stuart Hall, Linda Janes, Hugh Mackay, and Keith Negus. Doing Cultural Studies: The Story of the Sony Walkman, London: Sage Publications Ltd, 1997, p. 7-41. – Discussion Questions

• Will Avla

• Hugh Mackay. Consumption and Everyday Life, London: Sage Publications Ltd, 1997, p. 259-309. – Discussion Questions

• Alex

2005.02.15 SLIDE 42IS146 – SPRING 2005

Readings for Next Time

• Paul du Gay, Stuart Hall, Linda Janes, Hugh Mackay, and Keith Negus. Doing Cultural Studies: The Story of the Sony Walkman, London: Sage Publications Ltd, 1997, p. 7-41.– Discussion Questions

• Willian Avila

• Hugh Mackay. Consumption and Everyday Life, London: Sage Publications Ltd, 1997, p. 259-309.– Discussion Questions

• Allen Lew

2005.02.15 SLIDE 43IS146 – SPRING 2005

For Next Time

Assignment 3: Observing

Artifact Usage DUE

Documents

Computational Media