Making and Manipulating Media for Learning Mark Guzdial College of Computing/GVU Georgia Institute...

Making and Manipulating Media for Learning

Mark Guzdial

College of Computing/GVU

Georgia Institute of Technology

Story

The Dynabook model for what a computer is for. Learning through manipulating media.

30 years of literature suggests that it’s the right idea.

Two examples of trying it: Learning gravity by simulating it in elementary school Learning programming by manipulating media as a

context.

Dynabook to Personal Computer

The Personal Computer* as we know it today was invented in pursuit of the Dynabook

(*And object-oriented programming, too!)

Alan Kay’s Dynabook (1972)

Alan Kay sees the Computer as Man’s first metamedium A medium that can represent any other media: Animation,

graphics, sound, photography, etc. Programming is yet another medium

The Dynabook is a (yet mythical) computer for creative metamedia exploration and reading Handheld, wireless network connection Writing (typing), drawing and painting, sound recording, music

composition and synthesis One goal: End-user programming. But WHY?

Prototype Dynabook(Xerox PARC Learning Research Group)

A Dynabook is for Learning

The Dynabook offers a new way to learn new kinds of things…and perhaps old things in better ways Knowledge representation: Think about your own thinking

(Papert, 1980) Programming (Kay & Goldberg, 1977)

But need a system for creative expression In a time when “windows” were made of glass, and “mice”

were undesirable rodents

Smalltalk-72

For the Dynabook, WIMP was invented: overlapping

Windows Icons Menus mouse Pointer

What ~30 Years of Learning Sciencestell us about the Dynabook model

Since the 1970’s, Learning Scientists (Resarchers in Cognitive Science, Education, AI, etc.) have studied learning Do people learn from watching multimedia? Do people learn from creating multimedia? How well do people learn programming?

Do media make a difference in learning? Not obvious

“Equivalenced” media show little difference in learning benefit (Clark, 1983)

But it may be that equivalencing is the problem (Kozma, 1991) Mixing media may make a big difference

The case of animation Some studies show animations improve learning (Mayer, 1988) Most show animation has no benefit in learning (Stasko, Badre,

Lewis, 1993), and can even hinder learning (Palmiter, Elkerton, Baggett, 1991)

Do people learn from creating media?

Yes, and it’s a clearer case Children building educational software for other kids

learn fractions and science (Harel, 1988; Kafai & Harel, 1990)

Students building multimedia end up with higher grades (Hay et al., 1994)

“Multimedia literacy” may be a component of expertise For example, expert chemists constantly shift between

media and representations (Kozma et al., 1996)

How well do people learn programming? Really, really badly Freshmen and Sophomores at Yale couldn’t handle error

conditions in loops in 1982 (Soloway et al.) Freshmen and Sophomores in 3 countries and 4 institutions

can’t build calculators in 2001 (McCracken et al., 2001) Failure/withdrawl rate in CS1 is double-digits in most

institutions Percentage of women and minorities in CS is falling

Bottom line: Not a bad idea!

30 years of research says that the Dynabook is a pretty good model. Creation of media is a real “killer app” for

learning with computers. Programming is much harder to teach than we

thought,but maybe more of a Dynabook focus would help there, too!

Two Stories

Using Squeak to teach elementary school science students about gravity…by simulating it!

Using a media computation context to learn programming by non-majors.

Squeak

Smalltalk-80 running on modern machines (over 30 platforms) http://www.squeak.org Supporting wide range of media: Flash, MIDI, AIFF/WAV, MPEG Open-source

Alan Kay, Dan Ingalls, Ted Kaehler from original Xerox PARC LRG Apple to Disney to Viewpoints Research

http://www.squeakland.org

Breaking the Lines

Squeak Books

Example of Squeak scripting: Building the car

Testing Gravity, by timing it

Segmenting the video, measuring the acceleration

Duplicate the measurement

Simulate the Reality

Computer science is more important than Calculus In 1961, Alan Perlis argued

that computer science should be part of a liberal education. Explicitly, he argued that all

students should learn to program.

Calculus is about rates, and that’s important to many.

Computer science is about process, which is important to everyone

How close are we to being able to teach everyone CS?

Not very CS1 is one of the most despised courses for non-majors

At many departments, CS retention rates are lower than the rest of campus At Georgia Tech: 65% for 1995 cohort, vs. 73% for

Engineeering

Drop-out rates near 50% at many institutions Female enrollment in CS has been dropping nationally

Why?

Several recent studies and books claim that CS instruction tends to dissuade anyone but white males “Tedious,”

“taught without application relevance,”“boring,” “lacking creativity,” “asocial”

The best uses for computing technologies will come from others

Thomas Edison vs. D.W. Griffith Suggestion: D.W. Griffith knew things that Edison didn’t.

If we want computing technologies to become useful, they have to get out of our hands.

It can’t be just through applications. Computer science will never have the potential

that it might, if future practitioners hate our introductory course!

The Challenges

We need to motivate CS, potential CS, and non-CS students to care about computing

We need to make it social, creative, relevant,exciting, and not tedious

Which is how many of us already see Computing, but that’s not getting communicated

Our Attempt: Introduction to Media Computation A course for non-CS and non-Engineering majors

International Affairs, Literature, Public Policy, Architecture, Management, Biology, etc.

120 students this semester,planning 400-600 in the Fall 2/3 female in this semester’s CS1315

Focus: Learning programming and CS concepts within the context of media manipulation and creation

Language: Python (Jython)

Motivating the Computing

As professionals, these students will often the use the computer as a communications medium.

All media are going digital,and digital media are manipulated with software.

Knowing how to program, then, is a communications skill.

def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor)

def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0)

def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance))

def chromakey(source,bg): for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y))) return source

Use a loop!Our first picture recipe

def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Used like this:>>> file="/Users/guzdial/mediasources/barbara.jpg">>> picture=makePicture(file)>>> show(picture)>>> decreaseRed(picture)>>> repaint(picture)

original

Recipe to Increase the Volume

def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2)

Using it:>>> f="/Users/guzdial/mediasources/gettysburg10.wav">>> s=makeSound(f)>>> increaseVolume(s)>>> play(s)>>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")

A Sunset-generating function

How do we turn this beach scene into a sunset?

What happens at sunset? Tried increasing the red,

but that failed.

New Theory: As the sun sets, less blue and green is visible, which makes things look more red.

A Sunset-generation Functiondef makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

SlowSunset

def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas)

def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99)

Not showing you writeFrame() because you know how that works.

Just one canvas repeatedly being manipulated

SlowSunset frames

Introducing IF:Making Barb a redhead

def turnRed(): brown = makeColor(57,16,8) file = r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for px in getPixels(picture): color = getColor(px) if distance(color,brown)<50.0: redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture) Original:

Generalizing Algorithms

We talk about algorithm complexity later in the course, after the media is done.

We talk about different approaches to the same problem, where the criteria might be aesthetics or correctness, instead of speed or size For example, generating greyscale

During the media, we point out similar themes in different functions.

Scaling the picture downdef copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas

Scaling the picture updef copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

Recipe for halving the frequency of a sound

def half(filename): source = makeSound(filename) target = makeSound(filename)

sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex,

getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5

play(target) return target

This is how a sampling synthesizer works!

Here are the pieces that do it

Compare these two

def half(filename): source = makeSound(filename) target = makeSound(filename)

sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex,

getSampleValueAt( source, int(sourceIndex)))

sourceIndex = sourceIndex + 0.5

play(target) return target

def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(

getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

Both of them are sampling

Both of them have three parts: Set up the objects Loop over samples or pixels and copy from one place to

another To decrease the frequency or the size, we take each

sample/pixel twice In both cases, we do that by incrementing the index by

0.5 and taking the integer of the index Finishing up and returning the result

Using your personal pictures

And messin’ with them

Data-first Computing

Real users come to a user with data that they care about, then they (unwillingly) learn the computer to manipulate their data as they need.

Introduction to Media Computation works the same. We use pictures of students in class demonstrations. Students do use their own pictures as starting points for

manipulations. They started doing this in the second week

How often do students use concepts from the second week of CS1 on their own data?

How does that change the students’ relationship to the material?

Rough overview of Syllabus Defining and executing functions Pictures

Psychophysics, data structures, defining functions, for loops, if conditionals

Sounds Psychophysics, data structures, defining functions, for

loops, if conditionals Text

Converting between media, generating HTML, saving text from database and using in HTML, text searching Web pages (e.g., for data like the temperature)

Movies Then, Computer Science

Computer science as a solution to their problems “Why is PhotoShop so much faster?”

Compiling vs. interpreting Machine language and how the computer works

“Movie-manipulating programs take a long time to execute. Why?” Algorithmic complexity

“Writing programs is hard! Are there ways to make it easier? Or at least shorter?” Functional programming and recursion Object-oriented programming

Assignments encourage creativity

For several homeworks, the task is to manipulate media in some way, but we don’t care what media For example, creating a collage or building an animation

Encouraging homework results to be posted to CoWeb (collaborative website) in galleries

First Homework assignment

Homework 1:

Write a program named hw1 to accept a picture as input, and change its pixels as follows:

• Set the green component to 125% of its current value • Decrease the blue by 25% • Decrease the red by 75%

Solutions shared in the CoWeb

Grade distribution

Much better than anticipated.

Homework #3: Make a collage with images that you modify by code only—any images you want

Grades on Homework #3

Assessment results so-far

Of the 120 students who started, only two dropped the course.

97% of the students on a midterm survey answered Yes to “Are you learning to program?” Compared with 88% in our traditional CS1

What do you like about the class?

“I like the feeling when I finally get something to work.” “Very applicable to everyday life.” ‘I dreaded CS, but ALL of the topics thus far have been

applicable to my future career (& personal) plans- there isn't anything I don't like about this class!!!”

“When I finally get a program to work like I want it to.” “The professor answers questions in class and online and is

concerned about our success in the class. He also seems to understand that most of us are not engineers and most likely won't do straight programming in the future- just the way of thinking is important.”

What have you learned that you found interesting or surprising? “The most useful things I have learned are the

basics about computers and pictures/sound. I think when we learn HTML- that will be interesting and useful to real life applications.”

“Just general concepts about programming. It's pretty logical, sort of like in math, so it's understandable.”

“Programming is fun and ANYONE can do it!”

The Power of the Dynabook

The Dynabook has given us the desktop user interface and object-oriented programming.

But it also offers its original purpose: A model for using the computer for learning. Allowing students to learn from the real world by

manipulating media from it and simulating it. Motivating students to learn hard things in interesting

ways.

Acknowledgements

Kim Rose for the Squeakers DVD Course materials development: Jason Ergle, Claire

Bailey, David Raines, Joshua Sklare, Adam Wilson, Andrea Forte, Mark Richman, Matt Wallace, Alisa Bandlow.

Assessment: Andrea Forte, Rachel Fithian, Lauren Rich Thanks to Bob McMath and the Al West Fund, to GVU

and CoC, and the National Science Foundation

For further information

Squeak: http://www.squeak.org Viewpoints Research:

http://www.viewpointsresearch.org Course CoWeb:

http://coweb.cc.gatech.edu/cs1315 Where we planned the course:

http://coweb.cc.gatech.edu/mediaComp-plan guzdial@cc.gatech.edu