Portland Cyber Portland Cyber TheatreTheatreMarek Perkowski

Graduate Seminar, Friday, May 7, 2004

Part 1Part 1

Portland Cyber Portland Cyber TheatreTheatreFunded by Korea Institute of Science and Technology,

Daejeon, Republic of Korea and Intel Corporation, Hillsboro

Martin Lukac, Jacob Biamonte, Stefan Gebauer, Robert Klug, Myron Machado, Normen Giesecke, Atsumu Iseno, Tsutomu Sasao, Michele Folgheraiter, Uland Wong, Mikhail Pivtoraiko, Jeff Morris, Chris Motch, Nicola Sian Rees, Ping Hang Cheung, Randall Borck, Ray Schmelzer, Steve Mbah,Yu Xia, Akashdeep Aulakh, Tyler Nguyen, Jeff Allen, Kevin Xue, Kevin Chen, Ben Xue

ContributorsContributors

A robot with many bodies, A robot with many bodies, many personalities, quantum, many personalities, quantum, fuzzy and logic brains fuzzy and logic brains oror how how to Use of Machine Learning to Use of Machine Learning

based on Constructive based on Constructive Induction in Dialogs with Induction in Dialogs with

Robotic HeadsRobotic Heads

Or more scientifically

Talking RobotsTalking Robots• Many talking toys exist,

but they are still very primitive

• Actors for robot theatre, agents for advertisement, education and entertainment.

• Designing inexpensive natural size humanoid caricature and realistic robot heads

We concentrate on Machine Learning techniques used to teach robots behaviors, natural language dialogs and facial gestures.

Dog.com from Japan

Work in progress

Robot with a Personality?Robot with a Personality?• Future robots will interact

closely with non-sophisticated users, children and elderly, so the question arises, how they should look like?

• If human face for a robot, then what kind of a face?

• Handsome or average, realistic or simplified, normal size or enlarged?

• Why is Kismet so successful? •We believe that a robot that will interact with humans should have some kind of “personality” and Kismet so far is the only robot with “personality”.

•The famous example of a robot head is Kismet from MIT.

Professor Perky – the robot Professor Perky – the robot built by us in Japanbuilt by us in Japan

A robot of Martin Lukac: A robot of Martin Lukac: Cynthi-A singsCynthi-A sings

Robot face should be friendly and funny

The Muppets of Jim Henson are hard to match examples of puppet artistry and animation perfection.

We are interested in robot’s personality as expressed by its:– behavior,

– facial gestures,

– emotions,

– learned speech patterns.

Behavior, Dialog Behavior, Dialog and Learningand Learning

• Robot activity as a mapping of the sensed environment and internal states to behaviors and new internal states (emotions, energy levels, etc).

• Our goal is to uniformly integrate verbal and non-verbal robot behaviors.

Words communicate only about 35 % of the information transmitted from a sender to a receiver in a human-to-human communication.

The remaining information is included in para-language.

Emotions, thoughts, decision and intentions of a speaker can be recognized earlier than they are verbalized.

Fig. 1. Learning Behaviors as Mappings fromFig. 1. Learning Behaviors as Mappings fromenvironment’s features to interaction proceduresenvironment’s features to interaction procedures

AutomaticAutomaticsoftwaresoftwareconstructionconstructionfrom examplesfrom examples

(decision tree, (decision tree, bibi--decomposition,decomposition,AshenhurstAshenhurst, DNF), DNF)

Speech frommicrophones

Image featuresfrom cameras

Sonars and othersensors

Emotions andknowledge memory

probability Verbal responsegeneration (textresponse and TTS).Stored sounds

Headmovementsand facialemotionsgeneration

Neck and shouldersmovement generation

Robot Head ConstructionRobot Head Construction

Furby head with new controlFurby head with new control JonasJonas

We animate various kinds of humanoid heads with from 4 to 20 DOF, looking for comical and entertaining values.

Robot Head ConstructionRobot Head Construction

SkeletonSkeleton Alien

We use inexpensive servos from Hitec and Futaba, plastic, playwood and aluminum.

The robots are either PC-interfaced, use simple micro-controllers such as Basic Stamp, or are radio controlled from a PC or by the user.

AdamAdamMarvin the Crazy RobotMarvin the Crazy Robot

Technical Construction Technical Construction DetailsDetails

MaxMax

Image processing and pattern recognition uses software developed at PSU, CMU and Intel (public domain software available on WWW). Software is in Visual C++, Visual Basic, Lisp and Prolog.

BUG (Big Ugly Robot)BUG (Big Ugly Robot)

Visual Feedback and Learning based on Visual Feedback and Learning based on Constructive InductionConstructive Induction

Mister ButcherMister Butcher

4 degree of freedom neck

Latex skin from Hollywood

Virginia WoolfVirginia Woolf

heads equipped with microphones, USB cameras, sonars heads equipped with microphones, USB cameras, sonars and CDS light sensorsand CDS light sensors

Professor Perky Professor Perky

1 dollar latex skin 1 dollar latex skin from Chinafrom China

• We compared several commercial speech systems from Microsoft, Sensory and Fonix. •Based on experiences in highly noisy environments and with a variety of speakers, we selected Fonix for both ASR and TTS for Professor Perky and Maria robots.

• We use microphone array from Andrea Electronics.

Professor Perky with automated Professor Perky with automated speech recognition (ASR) and speech recognition (ASR) and text-to-speech (TTS) capabilitiestext-to-speech (TTS) capabilities

MariaMaria

20 DOF

Construction Construction details of Mariadetails of Maria

location of location of controlling controlling rodsrods

location location of head of head servosservos

location location of remote of remote servosservosCustom

designed skin

skull

Animation of eyes and eyelidsAnimation of eyes and eyelids

Frown of eyes and eyelidsFrown of eyes and eyelids

Smile and Jaw ActionSmile and Jaw Action

High School students help us to build High School students help us to build robot headsrobot heads

The newest robot of Jake Biamonte and Jeff The newest robot of Jake Biamonte and Jeff Allen has many sensors on the faceAllen has many sensors on the face

The back of the new robot head

The hands do not move. The new, movelable hands are under The hands do not move. The new, movelable hands are under design by Michele Folgheraiter, our collaborator from Italy.design by Michele Folgheraiter, our collaborator from Italy.

The new The new robot is robot is mobile.mobile.

The new robot is controlled by a

micro-controlled and a laptop. It

will move randomly and talk

about PSU greatness. See

demo soon.

Laptop and its screen

I am I am friendly friendly and and beautifulbeautiful

I can turn I can turn my body my body left and left and right right towards towards the the speaker speaker or away or away from himfrom him

Figure 2: Advanced construction of the robotic head, with servos inside the head, added are eyes and jaws. Also can be seen wires for future movements of cheeks, lips, eyebrows, chins and front.

Figure 1: Empty skull of the robot

Figure 3: Opened back head of the robot showing implacement of servos controlling the lower jaw, eyes, eye brows, cheeks and the chin.

Figure 4: Latex skin on the robotic head

New neck mechanism constructed by Jake and two Jeffs.

New neck mechanism constructed by Jake and two Jeffs.

On a grant from Korea Institute of

Advanced Studies we are

building a theatre of traditional

korean masks

Hahoe Hahoe RobotsRobots

Hahoe RobotsHahoe Robots

Hahoe RobotsHahoe Robots

The Hahoe Theatre has 12 robots, automatically controlled stage, lights and smoke generators. Help needed to design more heads and program scripts.

The Hahoe robots will sing and

dance.

A long neck and springs.

Mechanisms of Braitenberg’s Vehicles for Quantum vehicles explained next

Please wait for change of slides file to Part 2