First Courses Workshop Day 2 Mark Guzdial College of Computing Georgia Institute of Technology...

First Courses WorkshopDay 2

Mark Guzdial College of Computing

Georgia Institute of Technologyguzdial@cc.gatech.edu

http://www.cc.gatech.edu/~mark.guzdialhttp://www.georgiacomputes.org

Workshop Plan-Day 2

9-10:00 am: Explanation of Media Computation class structure.

10:00-10:15: Break 10:15-12:00: Movies in Media Computation:

Implementing video special effects 12:00-1:00: Lunch 1-2:30: Tackling a homework assignment in Media

Computation. Making a movie. 2:30-2:45: Break 2:45-4:15: Introducing computing through robotics in

Python 4:15-5:30: Tackling a homework assignment in robotics.

Follow the light.

Introduction to Media Computation in Python

Python as the programming language Huge issue

Use in commercial contexts authenticates the choice IL&M, Google, Nextel, etc.

Minimal syntax

Looks like other programming languages Potential for transfer

Rough overview of Syllabus

Defining and executing functions Pictures

Psychophysics, data structures, defining functions, for loops, if conditionals

Bitmap vs. vector notations Sounds

Psychophysics, data structures, defining functions, for loops, if conditionals

Sampled sounds vs. synthesized, MP3 vs. MIDI Text

Converting between media, generating HTML, database, and networking

A little trees (directories) and hash tables (database) Movies Then, Computer Science topics (last 1/3 class)

def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor)

def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0)

def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance))

Examples:

Some Computer Science Topics inter-mixed We talk about algorithms across

media Sampling a picture (to scale it) is

the samealgorithm as sampling a sound (to shift frequency)

Blending two pictures (fading one into the other) and two sounds is the same algorithm.

We talk about representations and mappings (Goedel) From samples to numbers (and into

Excel), through a mapping to pixel colors

We talk about design and debugging But they mostly don’t hear us

Computer Science Topicsas solutions to their problems

“Why is PhotoShop so much faster?” Compiling vs. interpreting Machine language and how the computer works

“Writing programs is hard! Are there ways to make it easier? Or at least shorter?” Object-oriented programming Functional programming and recursion

“Movie-manipulating programs take a long time to execute. Why? How fast/slow can programs be?” Algorithmic complexity

Introduction to Java Math operators, printing results, data types, casting,

relational operators, Strings, variables

Introduction to Programming Creating and naming objects

Using a turtle and a world

Creating new Turtle methods Draw simple shapes Using parameters

Syllabus

Syllabus - Continued

Modifying Pictures using Loops One-dimensional

arrays Use for-each, while,

and for loops to Increase/decrease

colors, fake a sunset, lighten and darken, create a negative, and grayscale

Introduction to Media Computation in Java

Syllabus - Continued

Modifying Pixels in a Matrix Two-dimensional

arrays Nested loops Copying, mirroring,

blending, rotation, scaling

Syllabus - Continued

Conditionally Modifying Pixels Boolean expressions Using && and || Replacing a color,

reducing red-eye, edge detection, sepia-toned, posterize, highlight extremes, blurring, background subtractions, chromakey

Syllabus - Continued

Drawing on Pictures Using existing Java

classes Inheritance Interfaces Drawing simple

shapes, drawing text, general copy, general scale, shearing, gradient paint, general blending, clipping

Syllabus - Continued

Modifying all Samples in a Sound 1D arrays Loops Conditional execution Change volume,

normalizing a sound (make it as loud as possible), force to extremes

Syllabus - Continued

Modifying Samples using Ranges Loops Clipping, splicing,

reversing, mirroring

Syllabus - Continued

Combining and Creating Sounds Class and private

methods Composing sounds,

blending sounds, changing frequencies, and creating echoes

Creating sounds Sine Waves, Square

Waves, Triangle Waves MP3 and MIDI

Syllabus - Continued

Creating Classes Identifying objects and classes Defining a class Overloading constructors Creating and initializing an array Creating getters and setters Creating a main method Javadoc comments

Reusing a class via inheritance ConfusedTurtle

ConfusedTurtleTurtle

Syllabus - Continued

Creating and Modifying Text String methods Reading from and writing to

files Handling Exceptions

Creating a form letter Modifying programs Getting text from networks Creating random sentences Using text to shift between

media

Syllabus - Continued

Making Text for the Web Throwing exceptions,

“unnamed” package, HashMap, Generics, and Iterators

Generating HTML Create a web page from a

directory Create a web page from

other web pages Databases Creating a web page from

a database

Syllabus - Continued

Encoding, Manipulating, and Creating Movies Frame-based animations

with simple shapes and text

Special effects – fade out, fake sunset, and chromakey

Syllabus - Continued

Speed What makes programs

fast? Compilers and Interpreters Writing a graphics

interpreter and compiler Searching Algorithms that can’t be

written What makes computers

fast? Clock rates, Storage, Display

Syllabus - Continued

Javascript Syntax User Interfaces Multimedia

Modules and Movies

Using external features in Python Making movies

Implementing digital video special effects

Processing many files at once

Sometimes you want to process a bunch of files without knowing the name of every one. Examples: Putting a title on every picture in a directory. Creating an index page that references every picture

and sound in a directory. Processing every frame in a video, where each frame

is stored as a JPEG frame.

Adding new capabilities: Modules

What we need to do is to add capabilities to Python that we haven’t seen so far.

We do this by importing external modules. A module is a file with a bunch of additional functions and

objects defined within it. Some kind of module capability exists in virtually every

programming language.

By importing the module, we make the module’s capabilities available to our program. Literally, we are evaluating the module, as if we’d typed them into

our file.

Python’s Standard Library

Python has an extensive library of modules that come with it.

The Python standard library includes modules that allow us to access the Internet, deal with time, generate random numbers, and…access files in a directory.

Accessing pieces of a module

We access the additional capabilities of a module using dot notation, after we import the module.

How do you know what pieces are there? Check the documentation. Python comes with a Library Guide. There are books like Python Standard Library that

describe the modules and provide examples.

The OS Module

The OS module offers a number of powerful capabilities for dealing with files, e.g., renaming files, finding out when a file was last modified, and so on.

We start accessing the OS module by typing: import os

The function that knows about directories is listdir(), used as os.listdir() listdir takes a path to a directory as input.

Using os.listdir

>>> import os>>> print getMediaPath("barbara.jpg")C:\Documents and Settings\Mark Guzdial\My Documents\

mediasources\barbara.jpg>>> print getMediaPath("pics")Note: There is no file at C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\picsC:\Documents and Settings\Mark Guzdial\My Documents\

mediasources\pics>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg',

'students7.jpg', 'students8.jpg']

Writing a program to title pictures

We’ll input a directory We’ll use os.listdir() to get each filename in the

directory We’ll open the file as a picture. We’ll title it. We’ll save it out as “titled-” and the filename.

Titling Pictures

import os

def titleDirectory(dir):

for file in os.listdir(dir):

picture = makePicture(file)

addText(picture,10,10,"This is from CS1315 Spring 2003")

writePictureTo(picture,"titled-"+file)

Okay, that didn’t work

>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")

makePicture(filename): There is no file at students1.jpg

An error occurred attempting to pass an argument to a function.

Why not?

Is there a file where we tried to open the picture? Actually, no. Look at the output of os.listdir() again

>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")

['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg']

The strings in the list are just the base names No paths

Creating paths

If the directory string is in the placeholder variable dir, then dir+file is the full pathname, right?

Close—you still need a path delimiter, like “/” But it’s different for each platform! Python gives us a notation that works: “//” is as a path

delimiter for any platform. So: dir+”//”+file

A Working Titling Program

import os

def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

Showing it work>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")Processing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students1.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students2.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students5.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students6.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students7.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students8.jpg>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg', 'titled-

students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']

Inserting a copyright on pictures

What if you want to make sure you’ve got JPEG files?import os

def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file if file.endswith(".jpg"): picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

Say, if thumbs.db is there

>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")Processing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students1.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students2.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students5.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students6.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students7.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//students8.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics//Thumbs.db>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg',

'Thumbs.db', 'titled-students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']

Another interesting module: Random

>>> import random>>> for i in range(1,10):... print random.random()... 0.82113693141939280.63542667797032460.94600601635201590.9046156965596840.335004644632541870.081249821269405940.07114813768070150.72552173073460480.2920541211845866

Randomly choosing words from a list

>>> for i in range(1,5):

... print random.choice(["Here", "is", "a", "list", "of", "words", "in","random","order"])

...

list

a

Here

list

Randomly generating language

Given a list of nouns,verbs that agree in tense and number,and object phrases that all match the verb,

We can randomly take one from each to make sentences.

Random sentence generator

import random

def sentence(): nouns = ["Mark", "Adam", "Angela", "Larry", "Jose", "Matt", "Jim"] verbs = ["runs", "skips", "sings", "leaps", "jumps", "climbs", "argues",

"giggles"] phrases = ["in a tree", "over a log", "very loudly", "around the bush",

"while reading the Technique"] phrases = phrases + ["very badly", "while skipping","instead of

grading", "while typing on the CoWeb."] print random.choice(nouns), random.choice(verbs),

random.choice(phrases)

Running the sentence generator

>>> sentence()Jose leaps while reading the Technique>>> sentence()Jim skips while typing on the CoWeb.>>> sentence()Matt sings very loudly>>> sentence()Adam sings in a tree>>> sentence()Adam sings around the bush>>> sentence()Angela runs while typing on the CoWeb.>>> sentence()Angela sings around the bush>>> sentence()Jose runs very badly

How much smarter can we make this?

Can we have different kinds of lists so that, depending on the noun selected, picks the right verb list to get a match in tense and number?

How about reading input from the user, picking out key words, then generating an “appropriate response”?

if input.find(“mother”) <> -1: print “Tell me more about your mother…”

Joseph Weizenbaum’s “Eliza”

Created a program that acted like a Rogerian therapist. Echoing back to the user whatever they said, as a

question. It had rules that triggered on key words in the user’s

statements. It had a little memory of what it had said before.

People really believed it was a real therapist! Convinced Weizenbaum of the dangers of computing.

Session with the “Doctor”

>>>My mother bothers me.

Tell me something about your family.

>>>My father was a caterpillar.

You seem to dwell on your family.

>>>My job isn't good either.

Is it because of your plans that you say your job is not good either?

Digital Video Effect: Background subtraction Let’s say that you have a picture of someone,

and a picture of the same place (same background) without the someone there, could you subtract out the background and leave the picture of the person?

Maybe even change the background? Let’s take that as our problem!

Person (Katie) and Background

The core of the algorithm:

if distance(personColor,bgColor) > someValue:

bgcolor = getColor(getPixel(newBg,x,y))

setColor(getPixel(person,x,y), bgcolor)

What else do we need? We need to get all these variables set up

We need to input a person picture, a background (background without person), and a new background.

We need a loop where x and y are the right values We have to figure out personColor and bgColor

Simplifying a little, and specifying a little

def swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) > 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)

Trying it with a jungle background

What happened?

It looks like we reversed the swap If the distance is great, we want to KEEP the pixel. If the distance is small (it’s basically the same thing),

we want to get the NEW pixel.

Reversing the swap

def swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)

Better!

But why isn’t it alot better?

We’ve got places where we got pixels swapped that we didn’t want to swap See Katie’s shirt stripes

We’ve got places where we want pixels swapped, but didn’t get them swapped See where Katie made a

shadow

How could we make it better?

What could we change in the program? We could change the threshold “someValue” If we increase it, we get fewer pixels matching

That won’t help with the shadow If we decrease it, we get more pixels matching

That won’t help with the stripe

What could we change in the pictures? Take them in better light, less shadow Make sure that the person isn’t wearing clothes near

the background colors.

Another way: Chromakey Have a background of a known

color Some color that won’t be on

the person you want to mask out

Pure green or pure blue is most often used

I used my son’s blue bedsheet This is how the weather people

seem to be in front of a map—they’re actually in front of a blue sheet.

Chromakey recipe

def chromakey(source,bg): # source should have something in front of blue, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

Can also do this with getPixels()

def chromakey2(source,bg): # source should have something in front of blue, bg is the new background for p in getPixels(source): # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

Example results

Just trying the obvious thing for Red

def chromakey2(source,bg): # source should have something in front of red, bg is the new background for p in getPixels(source): if getRed(p) > (getGreen(p) + getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

Doesn’t always work as you expect

Let’s try that with green

def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness + blueness if getGreen(p) > getBlue(p) + getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

The same definition of green doesn’t work

What happened?

The towel isn’t just green The green of the towel has lots of blue and red in it.

Use MediaTools to figure out a new rule that makes sense.

Tweaking Chromakey

def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness AND blueness if getGreen(p) > getBlue(p) and getGreen(p) > getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

That looks better

Movies, animations, and video…oh my! We’re going to refer generically to captured

(recorded) motion as “movies.” This includes motion entirely generated by graphical

drawings, which are normally called animations. This also includes motion generated by some kind of

photographic process, normally called video.

Psychophysics of Movies:Persistence of Vision What makes movies work is yet another limitation of our

visual system: Persistence of vision

We do not see every change that happens in the world around us.

Instead, our eye retains an image (i.e., tells the brain “This is the latest! Yup, this is still the latest!”) for a brief period of time. If this were not the case, you would be aware of every time that

your eye blinks because the world would “go away” for a moment.

16 frames and it’s motion

If you see 16 separate pictures in one second, and these pictures are logically sequenced, That is, #2 could logically follow from the scene in #1. 16 pictures of completely different things doesn’t work,

You will perceive the pictures as being in motion. 16 frames per second (fps), 16 pictures in a

second, is the lower bound for the sensation of motion.

Beyond 16 fps

Early silent pictures were 16 fps. Motion picture standards shifted to 24 fps to make sound

smoother. Videocameras (digital video) captures 30 fps How high can we go?

Air force experiments suggest that pilots can recognize a blurb of light in 1/200th of a second!

Video game players say that they can discern a difference between 30 fps and 60 fps.

Bottomlines: Generate at least 16 fps and you provide a sense of motion. If you want to process video, you’re going to have 30 fps to

process (unless it’s been modified elsewhere for you.)

Processing movies

Our frames are going to be JPEG pictures. One JPEG file per frame.

So, if we’re going to be processing movies, we’re going to generating or processing sequences of JPEG files.

Using MediaTools

To generate a series of frame pictures in a folder from an MPEG file.

To play a folder of frame pictures and to save it as a JMV file. (JPEG Movie format.)

To play JMV or MPEG movies.

MPEG? QuickTime? AVI? JMV?

MPEG, QuickTime, and AVI are compressed movie formats. They don’t record every frame. Rather, they record some key frames, and then store data about

what parts of the screen change on intervening frames. MPEG is an international standard, from the same people who

invented JPEG. AVI is a Microsoft standard. QuickTime is an Apple standard.

JMV is a file consisting of JPEG frames in an array. All frames represented

Generating other kinds of movies

Other tools can generate other kinds of movie formats.

QuickTime Pro (http://www.apple.com/quicktime) can read a sequence of JPEG images and produce MPEG, AVI, or QuickTime movies.

ImageMagick (open source toolkit) can also read a sequence of JPEG images and produce MPEG movies.

Why do we compress movies?

Do the math: One second of 640x480 pixels at 30 fps 30 (frames) * 640 * 480 (pixels) = 9,216,000 pixels With 3 bytes of color per pixel, that’s 27,648,000

bytes or 27 megabytes of information per second. For a 90 minute feature movie (short), that’s 90 * 60 *

27,648,000 = 149,299,200,000 bytes (149 gigabytes)

A DVD stores 6.47 gigabytes of data. So even on a DVD, the movie is compressed.

MPEG movie = MPEG frames plus MP3 soundtrack An MPEG movie is actually a series of MPEG

frames composed with an MP3 soundtrack. It’s literally two files stuck together in one.

We’re not going to deal with sound movies for now. The real challenge in doing movie processing is

generating and manipulating frames.

Get the frames in order

Many tools (including os.listdir()) can process frames in order if the order is specified.

We specify the order by encoding the number of the frame into the name. If you put in leading zeroes so that everything is the

same length, the order is alphabetical as well as numerical.

Simple Motiondef movingRectangle(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

A Few Frames

frame00.jpg frame02.jpg frame50.jpg

The trick here is all mathematics

if frame < 50: #Less than 50, move down

# Generate new positions each frame number

addRectFilled(canvas,frame*10,frame*5, 50,50,red)

if frame >= 50: #Greater than 50, move up

addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red)

When frame = 1, addRectFilled(canvas,10,5,50,50,red)

When frame = 2, addRectFilled(canvas,20,10,50,50,red)

When frame = 49, addRectFilled(canvas,490,285,50,50,r

ed) When frame = 50,

50-(50-50) = 50 addRectFilled(canvas,500,250,50,50,r

ed) When frame = 51,

50-(51-50)=50-1=49 addRectFilled(canvas,490,285,50,50,r

ed) When frame = 99,

50-(99-50)=50-49=1 addRectFilled(canvas,10,5,50,50,red)

Important cool thing: You can draw past the end of the picture!

addText, addRect, and the rest of the drawing tools will work even if you go beyond the edge of the drawing. Drawings will clip what can’t be seen in them, so you

don’t get an array out of bounds error. This is a big deal, because it means that you don’t

have to do complicated math to see when you’re past the end of the drawing.

But only for the drawing functions. If you set pixels, you’re still on your own to stay in range.

Making a tickertapedef tickertape(directory,string): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) #Start at right, and move left addText(canvas,600-(frame*10),100,string) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

tickertape(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\tickertape","Welcome to CS1315!")

Can we move more than one thing at once? Sure!

def movingRectangle2(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Let's have one just moving around addRectFilled(canvas,100+ int(10 * sin(frame)),4*frame+int(10* cos(frame)),50,50,blue) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

addRectFilled(canvas,100+ int(10 * sin(frame)), 4*frame+int(10* cos(frame)),50,50,blue)

What’s going on here? Remember that both sine and cosine vary between

+1 and -1. Int(10*sin(frame)) will vary between -10 and +10 With cosine controlling y and sine controlling x, should

create circular motion frame=1

x is 108, y is 9

frame=2 x is 109, y is 4

Frames from two motions at once

Moving something else:Remember this?

def copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas

To move Barb’s face around, we have to do this for each frame, moving the target each time.

Note: This will take a while

On my fastest computer, it takes over a minute. Think about it: Why? We’ll get to it next week.

You might to know where you are. Print isn’t too useful here.

Print doesn’t print until the program is done. There is a JES function called printNow() that takes a string and

will print it immediately. That way, we can see what frame number we’re at.

Most important thing to know: DID WE GET PAST FRAME ONE? (Very important to know as code gets slower!)

Moving Barb’s head

def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

moveahead(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbshead")

My, isn’t that gorey!

Can’t we make it easier to read? Can we just deal with the parts that we care

about? Maybe we could use sub-functions?

At least for the writing out of the frame.

Using subfunctions

def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color =

getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,canvas)

def writeFrame(num,directory,framepict): # Have to deal with single digit vs.

double digit frame numbers differently framenum=str(num) if num < 10:

writePictureTo(framepict,directory+"//frame0"+framenum+".jpg")

if num >= 10:

writePictureTo(framepict,directory+"//frame"+framenum+".jpg")

What if we have over 100 frames?

def writeFrame(num,directory,framepict): # Have to deal with single digit vs. double digit frame numbers

differently framenum=str(num) if num < 10: writePictureTo(framepict,directory+"//frame00"+framenum+".jpg") if num >= 10 and num<100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg") if num >= 100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg")

This will work with moveahead() and other functions—it’s generally useful

Using real photographs

Of course, we can use any real photographs we want.

We can use any of the techniques we’ve learned previously for manipulating the photographs.

Even more, we can use the techniques in new ways to explore a range of effects.

Slowly making it (very) sunset

Remember this code?def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7) What if we applied this to create frames of a movie, but

slowly increased the sunset effect?

SlowSunset

def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the

loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas)

def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99)

Not showing you writeFrame() because you know how that works.

Just one canvas repeatedly being manipulated

SlowSunset frames

Fading by background subtraction

Remember background subtraction?def swapbg(person, bg, newbg,threshold): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < threshold: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)

One change here is that the threshold is now an input.

Use the frame number as the threshold

def slowfadeout(directory): bg = makePicture(getMediaPath("wall.jpg")) jungle = makePicture(getMediaPath("jungle2.jpg")) for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("wall-two-people.jpg"))

#outside the loop! printNow("Frame number: "+str(frame)) swapbg(canvas,bg,jungle,frame) # Now, write out the frame writeFrame(frame,directory,canvas)

SlowFadeout

Dealing with real video

We really can’t deal with real video. Dealing with each frame takes a lot of processing. If you were going to process each frame as fast as it

was coming in (or going out), you’d have 1/30th of a second to process each frame!

We cheat by Saving each frame as a JPEG image Processing the JPEG images Convert the frames back to a movie

The original kid-in-bg-seq movie

Let’s have Mommy “watching”

We’ll paste Barb’s head into each frame. We’ll use os.listdir to process all the frames of

the kid sequence.

MommyWatchingdef mommywatching(directory): kiddir=r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\kid-in-bg-seq" barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) frame = 0 for file in os.listdir(kiddir): if file.endswith(".jpg"): frame = frame + 1 printNow("Frame number: "+str(frame)) framepic = makePicture(kiddir+"//"+file) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3, frame*3+((200-45)/2) ): sourceY = 25 for targetY in range(frame*3, frame*3+((200-25)/2) ): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(framepic,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,framepic)

We process each frame, and copy Mommy’s head to the frame, just like we animated in a line before onto a blank canvas.

MommyWatching

Lightening a picture

I took some video of a puppet show in black light.

Very hard to see the puppets. Your eye can pick them up, but the camera can’t. Recall earlier discussion: Your eye can detect

luminance changes that no media can replicate.

Dark-fish2 sequence

How I did the processing

First try, lighten every pixel. Didn’t work. Made all the black whiter as well as the colors No improvement in contrast

Second try, explore under MediaTools first Black parts are really black Lighter parts have really low number values So:

Look for any pixel less black than black (threshold=8) Lighten it a couple values

Lightenfish

import osdef lightenFish(directory): framenum = 0 for framefile in os.listdir(getMediaPath("dark-fish2")): framenum = framenum + 1 printNow("Frame: "+str(framenum)) if framefile.endswith(".jpg"): frame=makePicture(getMediaPath("dark-fish2")

+"//"+framefile) for p in getPixels(frame): color = getColor(p) if distance(color,black)>8: color=makeLighter(color) color=makeLighter(color) setColor(p,color) writeFrame(framenum,directory,frame)

Original sequence again

Same frames after lightening

That took forever!

On a 500 Mhz G4, this took almost a minute per frame to process! Over 160 frames

I started it at 3 pm, checked to make sure that the first few frames looked good, went to a meeting, and collected the result after 4:30

Why?

Why does movie processing take so long? Why does sound processing seem to go so fast?

Why can Photoshop do these things faster than we can in Python?

What makes software fast, or slow? See next lecture…

CS1 Robotics with Python

Microsoft Research has funded the Institute for Personal Robotics in Education Tucker Balch, Directing

Monica Sweat, Developing GT’s CS1 Joint between Bryn Mawr and

Georgia Tech Doug Blank (Developing Myro) Deepak Kumar (Teaching their first

CS1 with robots) http://www.roboteducation.org

Curriculum Goals

Make entry into computer science more accessible Revamp CS1 & CS2 curricula Use personal robots as a “vehicle” Fresh approach to introducing computing Influence robot design based on past experience Influence software development

Pilot CS1 in Spring 2007

Offered at Bryn Mawr (Deepak Kumar) and GaTech (Monica Sweat)

Use the Parallax Scribbler robot Wireless bluetooth interface designed by

GaTech enables Scribbler interface Myro software running under Dr. Python (IDE) to

control Scribbler Course will provide additional data on curriculum

design

Curriculum Development Strategy

Step One: Figure out something that students will want to do.

Step Two: Teach the CS and Robotics to make that happen.

Goals: Relevance Motivation Context for Transferrable Learning

Example CS1 Exercises: Early Focus on variables, arguments, functions, giving

commands to a computer, sequencing commands

Personalize your robot (use colors, pens, stickers, etc.). Give it a personality, and a specific “move” Then experiment with difference basic robot movement

behaviors (go forward, backward, spin, turn, etc.) and learn how to upload and play a tune. Design a series of steps for your robot to perform a dance while playing a tune.

Test the sensitivity and range of IR sensors for light detection, obstacle detection. The latter will vary depending on the reflectance of the object being sensed... Put these results in Excel to introduce ideas of computational

science: Gathering data from sensors and analyzing it.

Example CS1 Exercises: Later Iteration Focus on iterating, sequencing within an iteration, limiting iteration

Experiment with simple behaviors like going forward for a while and then backward. Repeat these behaviors several times. Does the robot track the same space or are the movements shifting its space? A good way to test these would be to insert a pen and let the robot draw its path each time. This will help observe and get used to variations in your robots motor behaviors.

Write a loop to input a sound (could be on laptop instead of robot) then play it back. Do this in a group of robots. Make a sound, allowing the group behavior and dynamics to create a cacophony.

Write programs for the robot to draw a square, a circle, a figure-8, a 5 point star...and other shapes...a spiral?

Example CS1 Exercises: Powerful Representation and Interpretation Using some other data structure or sensed data as a command to drive

other behavior

Create a file with commands like "left on 30 \n right on 30 \n beep" Write a program to read these commands and execute them – in other words, define a mini robot language and write an interpreter for it. Gives us the opportunity to address string processing, a typical CS1 topic.

Have one robot "call out" audio commands (perhaps specific tones, via the speaker), and have other robots "hear" the commands (via microphones) and execute the commands.

Follow a line of symbols/fiducials, where the symbols (colors or scan codes) can be interpreted as music. Start the robot on the line, and as it encounters the markings, it plays a note. The spacing is the timing of the songs. Put one after another, and they play in a round. Have two go together (on two lines) and they play a harmony. The robot could also write the music with a pen. First robot writes the song, second robot plays it.

Parallax Scribbler Robot

Low cost (~$80, ~$60 in bulk) and features

3 photosensors IR sensors in front IR on chasis Servo controlled motors 3 Programmable LEDs Speaker Battery operated (6 AAs) Serial interface

Assessment Methods

Within term: Initial, midterm, final surveys.

Focus group or individual student interviews. Across term:

Tracking – do they stay in CS?How well do they do?

Installing Myro on Windows

1. Download the latest Myro from http://wiki.roboteducation.org/Windows_Setup

2. Save it, open it, and expand it.

3. Double-click on install.bat

Setting up Bluetooth

Testing Myro

Testing the Robot

In other words:

initialize(“com5”)

Controlling the robot’s motors

Most Myro robot movement functions control the speed (specifically, amount of power) of the motor. motors(1.0,0.5) # Gives full power to motor Left,

# half power to motor Right forward(1.0) #Both motors, full speed ahead

forward(1.0,0.25) #Full forward, for 0.25 seconds backward(0.5) #Backwards, half speed turnLeft (0.1) #Turn left, slowly

turnLeft(0.1,1) #Turn left slowly, for 1 second turnRight(0.25) # Turn right, slightly more quickly stop() #STOP EVERYTHING

same as motors(0,0)

Defining a robot function: Yoyo

def yoyo():forward(1)

wait(1) # Wait one second

backward(1)

wait(1)

stop()

Can just type this in, and then execute it as:

yoyo()

Or can enter it into a module (filename)

Parameterizing our YoYo

def yoyo1(speed):forward(speed)

wait(1) # Wait one second

backward(speed)

wait(1)

stop()

Further parameterizing

def yoyo2(speed, wait):forward(speed)

wait(wait) # Wait a bit

backward(speed)

wait(wait)

stop()

Can do something forever

from myro import *initialize(“com5”)def yoyo2(speed, wait):

forward(speed)wait(wait) # Wait one

secondbackward(speed)wait(wait)stop()

def wiggle(speed,waitTime):rotate(speed)wait(waitTime)rotate(-speed)wait(waitTime)stop()

def dance():while True:

yoyo2(1.0,1)wiggle(1.0,0.2)yoyo2(0.5,0.5)wiggle(0.5,0.5)

Use Control-C to stop this.

BUT WAIT, before you try that…

What are all the ways that we can limit time?for seconds in timer(5):

turnRight(0.1,1)

while timeRemaining(5):turnLeft(0.1,1)

time_to_run = 25 start_time = currentTime() while (start_time + time_to_run > currentTime() ) :

forward(0.5,1)

So, trying our dance:

def dance():while timeRemaining(5):

yoyo2(1.0,1)

wiggle(1.0,0.2)

yoyo2(0.5,0.5)

wiggle(0.5,0.5)

Remember that you still need those two functions in your code!

Using the robot as an object

Robot can be controlled using global functions,or as methods to robot objects.

Review

from myro import *

robot = Scribbler(“com4”)

robot.turnRight(0.25); wait(0.25); robot.stop()

robot.turnRight(0.25,0.25)

robot.forward(0.25); wait(0.25); robot.stop()

robot.forward(0.25,0.25)

def function():

while True:

while timeRemaining(n):

for seconds in timer(5):

Reading the Light Sensors

robot.getLight(0)

robot.getLight(1)

robot.getLight(2)

>>> robot.getLight()

[657, 1453, 1025]

Light sensors

Which one is which?

Exercise: Playing blind man’s bluff 0,1,2: left, right, center? Do the values go up with darkness or with light?

Modeling Animals

How do animals sense light? Why do moths move to the light? How do they know which way to turn to

get there? Does it matter if you see vs. smell?

Let’s model light-seeking behavior

Choosing action depending on senses

New statement: if Allows us to test conditions (logical expressions)

and choose actions based on those conditions. if (some logical test goes here) :

The actions go on the line below, indented.if robot.getLight(0) > 800:

robot.turnLeft(0.25,0.1)

Blocks

Just like the lines after def and while, all the lines indented the same after if are part of the same block

A block is defined in Python by indentation. Lines at the same level of indentation are in the same block.

A block is used to define the body of a function and the loop.

A block is used to define the actions of a successful if test.

Where’s the light?

We can compare against a value, but what we really care about is the relative light values.

if robot.getLight(0) > robot.getLight(2):

print "LEFT!"

Signaling a Turn

def signalingTurn(): left = 0 right = 2 while timeRemaining(10): if robot.getLight(left) < robot.getLight(right): print “Left!" if robot.getLight(right) < robot.getLight(left): print "Right!"

signalingTurn()

Audibly signaling a turn

def signalingTurn(): left = 0 right = 2 while timeRemaining(10): if robot.getLight(left) < robot.getLight(right): robot.beep(0.25,400) if robot.getLight(right) < robot.getLight(left): robot.beep(0.25,800)

signalingTurn()

Making Music

beep(1,400) #Plays at frequency 400 Hz # for 1 second

computer.beep(0.5,440)# Has the computer beep at A-above-middle C

# (440 Hz) for ½ second

Can they play a duet?Try it!

Myro Song Format

Myro has a more music-like format that it supports:s = makeSong(“c 1; d .5; d#4 1/2”)

# C in the fifth octave for a whole note

# D in the fifth octave for a 1/2 note

# D-sharp in fourth octave for a 1/2 note

robot.playSong(s,0.75) # Play this for ¾ second

More Speech Options

computer.speak(message) - computer will say the text message

computer.stopSpeaking() - stop talking computer.setVoice(name) - set the voice to a named

voice computer.getVoice() - get the name of the current voice computer.getVoices() - get alternative set of named

voices computer.playSpeech(filename) - play a speech file

(wav file) computer.saveSpeech(message, filename) - save

speech to a wav file

Exercises

Option #1: Follow the Light Write a function that will turn the robot toward

the light, much as an insect might follow the light. Can you turn based on the amount of light?

Option #2: Make the Robot Dance Write a program to play (or sing or give a

speech) and “dance” your Scribbler Use beep() and playSong to make music Use movements like yoyo and wiggle