PERCEPTUAL INTELLIGENCE

SEMINAR REPORT

submitted by

AMITH K P

EPAHECS009

for the award of the degree

of

Bachelor of Technology

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

GOVERNMENT ENGINEERING COLLEGE SREEKRISHNAPURAM

PALAKKAD

December 2010

CERTIFICATE

This is to certify that the mini seminar report entitled PERCEPTUAL

INTELLIGENCE submitted by AMITH K P, to the Department Of Com-

puter Science and Engineering, Government Engineering College, Sreekr-

ishnapuram, Palakkad-679513, in partial fulfilment of the requirement for the

award of B.Tech Degree in Computer Science and Engineering is a bonafide record of

the work carried out by him during year 2010.

Dr P C Reghu Raj Dr P C Reghu Raj

Staff Co-ordinator Head of the Department

Place: Sreekrishnapuram

Date: 24-10-2010

Acknowledgement

It stands to reason that the completion of a seminar of this scope

needs the support of many people. I take this opportunity to express my

boundless thanks and commitment to each and every one,who helped me in

successful completion of my seminar. I am so happy to acknowledge the help

of all the individuals to fulfil my attempt.

First and foremost I wish to express wholehearted indebtedness to

God Almighty for his gracious constant care and magnanimity showered bliss-

fully over me during this endeavour.

I thank to Dr P C Reghu Raj, Head of Department of Computer

Science and Engineering, Govt. Engineering College Sreekrishnapuram, for

providing and availing me all the required facilities for undertaking the semi-

nar in a systematic way .I express my heartfelt gratitude to him for working

as a seminar coordinator and guide ,who corrected me and gave valuable sug-

gestions.

Gratitude is extended to all teaching and non teaching staffs of De-

partment of Computer Science and Engineering, Govt Engineering College

Sreekrishnapuram for the sincere directions imparted and the cooperation in

connection with the project.

I am also thankful to my parents for the support given in connection

with the project. Gratitude may be extended to all well-wishers and my

friends who supported us to complete the seminar in time.

ii

Table of Contents

List of Figures v

Abstract 1

1 Introduction 2

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Why Perceptual Intelligence ? . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Organisation of the Report. . . . . . . . . . . . . . . . . . . . . . . . 3

2 Perceptual Intelligence 4

2.1 Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 The filters that make up perception are as follows: . . . . . . . 4

2.2 Perceptual User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Information flow in Perceptual User Interfaces . . . . . . . . . 7

2.2.1.1 Perceptive User Interface/Perceptual Interface . . . 7

2.2.1.2 Multimodal Interfaces . . . . . . . . . . . . . . . . . 10

2.2.1.3 Multimedia Interfaces . . . . . . . . . . . . . . . . . 13

2.3 Perceptual Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.4 Perceptual Intelligent Systems . . . . . . . . . . . . . . . . . . . . . 21

2.4.1 Gesture Recognition System . . . . . . . . . . . . . . . . . . . 21

2.4.1.1 Gesture Recognition . . . . . . . . . . . . . . . . . . 21

2.4.1.2 Working of Gesture Recognition System . . . . . . . 22

2.4.1.3 Challenges of Gesture Recognition . . . . . . . . . . 26

2.4.2 Speech Recognition System . . . . . . . . . . . . . . . . . . . 27

2.4.2.1 Types of Speech Recognition . . . . . . . . . . . . . 29

2.4.2.2 Working of Speech Recognition Systems . . . . . . . 30

2.4.2.3 Performance of speech recognition systems . . . . . . 32

2.4.3 Nouse Perceptual Vision Interface . . . . . . . . . . . . . . . . 33

2.4.3.1 Computer Control Actions . . . . . . . . . . . . . . . 33

2.4.3.2 Tools Used In Nouse Perceptual Vision Interface . . 34

2.4.3.3 Working Of Nouse Perceptual Vision Interface . . . . 35

3 Applications 41

3.1 SMART ROOMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.2 SMART CLOTHES . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

iv

List of Figures

2.1 Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Perceptual User Interface . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Information flow in Perceptual User Interface . . . . . . . . . . . . . . 8

2.4 Human Computer Interaction . . . . . . . . . . . . . . . . . . . . . . 9

2.5 Perceptual Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.6 Phantom input/output device . . . . . . . . . . . . . . . . . . . . . . 13

2.7 Different Medias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.8 User cancels video by pressing Cancel push button . . . . . . . . . . 15

2.9 User adjusts sound volume with slider and turns off sound with Sound

off push button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.10 Suggested sizes for touch areas and visual targets . . . . . . . . . . . 19

2.11 Background subtracted mask . . . . . . . . . . . . . . . . . . . . . . . 24

2.12 image illustrating body center and arm algorithim . . . . . . . . . . . 25

2.13 Selecting Shift, selecting H . . . . . . . . . . . . . . . . . . . . . . . . 39

3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Abstract

Human computer interaction has not changed fundamentally for nearly two

decades. Most users interact with computers by typing, clicking and pointing. Now

most of the research works are concentrating on interaction techniques that com-

bine an understanding of natural human capabilities with computer I/O devices and

machine perception and reasoning.

Perceptual Intelligence is the knowledge and understanding that everything we

experience (especially thoughts and feelings) are defined by our perception. Its im-

portant to realize that this is an active, not passive, process and therefore we have

the ability to control it or change it. Computers need to share our perceptual en-

vironment before they can be really helpful. They need to be situated in the same

world that we are; they need to know much more than just the text of our words.

They also need to know who we are, see our expressions and gestures, and hear the

tone and emphasis of our voice.

CHAPTER 1

Introduction

Inanimate things are coming to our life. That is the simple objects that surround

us are gaining sensors,computational powers, and actuators.Consequently,desks and

doors, TVs and telephones, cars and trains,eyeglasses and shoes, and even the shirts

on our backs are changing from static, inanimate objects into adaptive, reactive sys-

tems that can be more friendly, useful, and efficient. Or, of course,these new systems

could be even more difficult to use than current systems;it depends how we design

the interface between the world of humans and the world of this new generation of

machines.

1.1 Motivation

The main problem with todays systems are,they are both deaf and blind.They

mostly experience the world around them through a slow serial line to a keyboard

and mouse. Even multimedia computers, which can handle signals like sound and

image,do so only as a transport device that knows nothing. Hence these objects are

still static and inanimate.

To change inanimate objects like offices,houses, cars, or glasses into smart, active

help-mates,they need some kind of Intelligence.This kind of intelligence what they

need here is Perceptual Intelligence. Perceptual Intelligence is paying attention to

people and the surrounding situation in the same way another person would, thus

allowing these new devices to learn to adapt their behavior to suit us,rather than

adapting to them as we do today.

1.2 Why Perceptual Intelligence ?

The problem with current computers is they are incredibly isolated. If

you imagine yourself living in a closed, dark, soundproof box with only a telegraph

connection to the outside world, you can get some sense of how difficult it is for

computers to act intelligently or be helpful.They exist in a world almost completely

disconnected from ours, so how can they know what they should do in order to be

helpful?

Computers need to share our perceptual environment before they can be really

helpful.They need to be situated in the same world that we are; they need to know

much more than just the text of our words of the signals content.Once the computer

has the perceptual ability to know who, what, when, where, and why, by understand-

ing,learning and interacting with the physical world sufficent for the computer to

determine a good course of action.If the systems have the ability to learn perception,

they can act in a smart way.Perceptual intelligence is actually a learned skill.

1.3 Organisation of the Report.

Each chapter begins with a brief introduction to its content. Chapter 2 gives a brief

background information on the various functions of the system and discusses other

related work in this area. Chapter 3 describes the Perceptual Intelligence in detail.

Chapter 4 discusses the Applications of Perceptual Intelligence. Chapter 5 includes

the conclusion and future scope.

3

CHAPTER 2

Perceptual Intelligence

2.1 Perception

Perception is the process of receiving information about and making sense

of the world around us. It involves deciding which information to notice, how to

categorize this information and how to interpret it within the framework of existing

knowledge.A process by which individuals organize and interpret their sensory im-

pressions in order to give meaning to their environment.For instance ,as shown in Fig

3.1,in perception ,process firstly recieve stimuli and select them accordingly.Now after

selecting,it organises and interprets proceeding to give response.

Perception is the end result of a thought that begins its journey with the senses.

We see, hear, physically feel, smell or taste an event. After the event is experienced

it must then go through various filters before our brains decipher what exactly has

happened and how we feel about it. Even though this process can seem instantaneous,

it still always happens.

2.1.1 The filters that make up perception are as follows:

• What we know about the subject or event.I saw an orange and knew it was

editable.

• What our previous experience (and/or knowledge) with the subject or event

was. Last time I ate an orange I peeled it first (knowledge to peel an or-

ange before eating it) and it was sweet. Our previous experience forms our

expectations.

Figure 2.1: Perception

• Our current emotional state. How we are feeling at the time of the event does

affect how we will feel after the event. I was in a bad mood when I ate the

orange and it angered me that it was sour and not sweet (my expectation).

• In the end my intellectual and emotional perception regarding the eating of an

orange was an unpleasant experience. Depending on how strong that experience

was, determines how I will feel next time I eat an orange. For example, if I got

violently sick after eating an orange, the next time I see an orange, I probably

wont want to eat it. If I had a pleasant experience eating an orange, the next

time I see an orange, Ill likely want to eat it.

Even though emotions seemly occur as a result of an experience, they are actually

the result of a complicated process. This process involves interpreting action and

thought and then assigning meaning to it. The mind attaches meaning with prejudice

as the information goes through the perceptual filters we mentioned above.

Our perceptual filters also determine truth, logic along with meaning though they

dont always do this accurately. Only when we become aware that a bad feeling could

5

be an indication of a misunderstanding (error in perception).we can begin to make

adjustments to our filters and change the emotional outcome.

When left alone and untrained, the mind chooses emotions and reactions based

on a ”survival” program which does not take into account that we are civilized beings

its only concerned with survival.

A good portion of this program is faulty because the filters have created distor-

tions, deletions and generalizations which alter perception. For example,jumping to

a conclusion about ”all” or ”none” of something based on one experience.The uncon-

scious tends to think in absolutes and supports ”one time” learnings from experience

(this is the survival aspect of learning).

2.2 Perceptual User Interfaces

A perceptual interface is one that allows a computer user to interact with the

computer without having to use the normal keyboard and mouse. These interfaces are

realised by giving the computer the capability of interpreting the user’s movements

or voice commands.

Perceptual Interfaces are concerned with extending human computer interaction

to use all modalities of human perception. All current research efforts are focused at

including vision, audition, and touch in the process. The goal of perceptual reality is

to create virtual and augmented versions of the world, that are perceptually identical

to the human with the real world. The goal of creating perceptual user interfaces

is to allow humans to have natural means of interacting with computers, appliances

and devices using voice, sounds, gestures, and touch.

Perceptual User interfaces (PUI) are characterised by interaction techniques that

combine an understanding of natural human capabilities with computer I/O devices

and machine perception and reasoning. They seek to make the user interface more

natural and compelling by taking advantage of the ways in which people naturally in-

teract with each other and with the world-both verbally and nonverbally. Devices and

6

Figure 2.2: Perceptual User Interface

sensors should be transparent and passive if possible, and machines should perceive

relevant human communication channels as well as generate output that is naturally

understood. This is expected to require integration at multiple levels of technologies

such as speed and sound recognition and generation, computer vision, graphical ani-

mation and visualization, language understanding, touch based sensing and feedback

learning, user modelling and dialogue management.

2.2.1 Information flow in Perceptual User Interfaces

PUI integrates perceptive, multimodal, and multimedia interfaces to bring

our human capabilities to bear on creating more natural and intuitive interfaces.

2.2.1.1 Perceptive User Interface/Perceptual Interface

A perceptive user interface is one that adds human-like perceptual capabil-

ities to the computer, for example, making the computer aware of what the user is

7

Figure 2.3: Information flow in Perceptual User Interface

saying or what the users face, body and hands are doing. These interfaces provide in-

put to the computer while leveraging human communication and motor skills.Unlike

traditional passive interfaces that wait for users to enter commands before taking any

action, perceptual interfaces actively sense and perceive the world and take actions

based on goals and knowledge at various levels. Perceptual interfaces move beyond

the limited modalities and channels available with a keyboard, mouse, and monitor,

to take advantage of a wider range of modalities, either sequentially or in parallel.

The general model for perceptual interfaces is that of human-to-human commu-

nication. While this is not universally accepted in the Human Computer Interaction

community as the ultimate interface model,there are several practical and intuitive

reasons why it makes sense to pursue this goal. Human interaction is natural and

in many ways effortless; beyond an early age, people do not need to learn special

techniques or commands to communicate with one another.

Figure 1.2 depicts natural interaction between people and, similarly, between

8

Figure 2.4: Human Computer Interaction

Figure 2.5: Perceptual Interface

9

humans and computers. Perceptual interfaces can potentially effect improvements in

the human factors objectives mentioned earlier in the section, as they can be easy

to learn and efficient to use,they can reduce error rates by giving users multiple and

redundant ways to communicate, and they can be very satisfying and compelling for

users.

2.2.1.2 Multimodal Interfaces

A multimodal user interface is closely related emphasizing human commu-

nication skills.It is a system that combines two or more input modalities in a coor-

dinated manner.Humans interact with the world by way of information being sent

and received, primarily through the five major senses of sight, hearing, touch, taste,

and smell.A modality refers to a particular sense. A communication channel is a

course or pathway through which information is transmitted. In typical HCI usage, a

channel describes the interaction technique that utilizes a particular combination of

user and computer communication i.e., the user output/computer input pair or the

computer output/user input pair . This can be based on a particular device, such as

the keyboard channel or the mouse channel, or on a particular action, such as spoken

language, written language, or dynamic gestures. In this view, the following are all

channels: text (which may use multiple modalities when typing in text or reading

text on a monitor), sound, speech recognition, images/video, and mouse pointing and

clicking.

So are multimodal interfaces or multi-modality or multi-channel. Certainly every

command line interface uses multiple modalities, as sight and touch (and sometimes

sound) are vital to these systems. The same is true for graphical user interfaces, which

in addition use multiple channels of keyboard text entry, mouse pointing and clicking,

sound, images, etc.We use multiple modalities when we engage in face to face com-

munication leading to more effective communication. Most work on multimodal User

Interface as focused on computer input(for example using speech together with pen

10

based gestures).Multimodal output uses different modalities, like visual display, audio

and tactile feedback to engage human perceptual, cognitive and communication skills

in understanding what is being presented. In multimodal UI various modalities are

sometimes used independently or simultaneously or tightly coupled.Multimodal inter-

faces focus on integrating sensorrecognition based input technologies such as speech

recognition, pen gesture recognition, and computer vision, into the user interface.

Multimodal interface systems have used a number of non-traditional modes and

technologies. Some of the most common are the following:

• Speech recognition Speech is a very important and flexible communication

modality for humans, and is much more natural than typing or any other

way of expressing particular words, phrases, and longer utterances. Despitethe

decades of research in speech recognition and over a decade of commercially

available speech recognition products, the technology is still far from perfect,

due to the size, complexity, and subtlety of language, the limitations of micro-

phone technology, the plethora of disfluencies in natural speech, and problems

of noisy environments. Systems using speech recognition have to be able to

recover from the inevitable errors produced by the system.

• Language understanding Natural language processing attempts to model and

understand human language,whether spoken or written. In multimodal inter-

faces, language understanding may be hand-in-hand with speech recognition ,

or it may be separate, processing the users typed or handwritten input. Typi-

cally the more a system incorporates natural language, the more users will ex-

pect sophisticated semantic understanding from the system. Current systems

are unable to deal with completely unconstrained language, but can do quite

well with limited vocabularies and subject matter. Allowing for user feedback

to clarify and disambiguate language input can help language understanding

systems significantly.

• Pen-based gesture Pen-based gesture has been popular in part because of com-

11

puter form factors that include a pen or stylus as a primary input device. Pen

input is particularly useful for pointing gestures,defining lines, contours, and

areas, and specially-defined gesture commands e.g., minimizing a window by

drawing a large M on the screen. Pen-based systems are quite useful in mobile

computing,where a small computer can be carried.

• Haptic input and force feedback Haptic, or touch-based, input devices measure

pressure, velocity, locationessentially perceiving aspects of a users manipulative

and explorative manual actions. These can be integrated into existing devices

e.g., keyboards and mouse that know when they are being touched, and possi-

bly by whom. Or they can exist as standalone devices, such as the well-known

PHANTOM device by SensAble Technologies in Fig.1.4 or the DELTA device

by Force Dimension.4 These and most other haptic devices integrate force feed-

back and allow the user to experience the touch and feel of simulated artifacts

as if they were real. Through the mediator of a hand-held stylus or probe, hap-

tic exploration can now receive simulated feedback including rigid boundaries

of virtual objects, soft tissue, and surface texture properties. A tempting goal

is to simulate all haptic experiences and to be able to recreate objects with all

their physical properties in virtual worlds so they can be touched and handled

in a natural way.

• Computer vision Computer vision has many advantages as an input modality

for multimodal or perceptual interfaces. Visual information is clearly impor-

tant in human-human communication, as meaningful information is conveyed

through identity, facial expression,posture, gestures, and other visually ob-

servable cues. Sensing and perceiving these visual cues from video cameras

appropriately placed in the environment is the domain of computer vision.

12

Figure 2.6: Phantom input/output device

2.2.1.3 Multimedia Interfaces

Multimedia User Interface uses perceptual and cognitive skills to interpret

information presented to the user .Multimedia user interfaces combine several kinds

of media to help people use a computer. These media can include text, graphics, an-

imation, images, voice, music,and touch.Multimedia user interfaces are gaining pop-

ularity because they are very effective at keeping the interest of their users, improve

the amount of information users remember, and can be very cost-effective.

Successful multimedia designers build their products with primary emphasis on

the user. Multimedia designers determine which human sensory system is most effi-

cient at receiving specific information, then use the media that involves that human

sensory system. For example, to teach someone how to fixa part on a jet engine it is

probably most efficient for the student to watch someone else fixing the part rather

than hearing a lecture on how to fix the part. The human visual system is better

at receiving this complex information. So, the designer for this multimedia product

13

Figure 2.7: Different Medias

should probably usevideo as the medium of communication.

This heavy emphasis on the user’s senses, rather than the media, means that

we should probably call these product user interfaces multisensory rather than mul-

timedia. The human senses that designers use most frequently in their multimedia

products are sight, sound, and touch. Multimedia products often stimulate these

senses simultaneously. For example, a user can see and hear a video, and then touch

a graphical button to freeze the video image. Since so many media are available to the

multimedia user interface designer, it is very easy to overwhelm and confuse the users

of these products. The following guidelines are based on the way people think and

perceive information. Theseguidelines will help you build multimedia user interfaces

that are easy and comfortable for people to learn and use.

General Gudilines

some general guidelines of multimedia user interfaces are as follows:

• Keep the user interface simple This is the most important guideline to remem-

14

Figure 2.8: User cancels video by pressing Cancel push button

ber when you design multimedia user interfaces. Don’t show off what you can

do with multimedia technology. Instead, give the users only what they need to

do their tasks.

• Be consistent Use similar objects to perform similar functions throughout. That

way, what users learn in one place they can use in other places . This quality

helps users learn quickly and builds their confidence.

• Let the user control the interaction Let the user, rather than the computer,

control what happens next. Let the user decide where to go, what to see, and

when to leave. For example, when watching a video, let the user cancel the

video rather than watching the whole thing before being able to move on. That

way, if the user saw the video during an earlier interaction, he or she does not

have to see it all over again. This feature is shown in Figure 1.

• Give immediate, obvious feedback for every user action Let the user know that

15

the computer is working and that it received and is responding to the user

action. For example, when the user selects a push button, change the button

to inverse video, and have the computer emit a short beep and process the

action on the push button. This design characteristic will help keep the user

interested.

• Use familiar metaphors Take advantage of the user’s prior learning by using fa-

miliar objects and actions. This characteristic improves learning and increases

user comfort. For example, if the user needs to dial a telephone number, provide

an image of a telephone with a touch-sensitive key pad. The user recognizes the

telephone and knows how to make it work. Other familiar metaphors include

televisions,video cassette recorders, and calculators.

• Let the user safely explore your product The user should be able to navigate

through your product without worrying about breaking or deletingsomething.

If appropriate, don’t provide a delete function. If the user needs to delete

objects, design your user interface so the user can confirm or cancel a deletion.

Let the user undo actions that are not easy to reverse. Let the user go back to

the previous screen from the current screen. Make obvious what the user can

do next. These design characteristics help to make user interface inviting and

comfortable.

Medias of Multimedia Interfaces

Sight The medium of sight is helpful when you need to communicate detailed

information, such as instructions, that the user may need to refer to later. Here are

some guidelines that involve sight.

• Use pastel colors Except for videos and black and white text, it is generally a

good idea to use slightly washed out, desaturated, impure colors. This is most

important for the small objects that you put on screens. Scientists believe that

these kinds of colors let people focus on small objects better and are lesslikely

to cause objects to appear to float on the display. In your videos, stick with

16

the original colors in which the video was shot. Use high-contrast foreground-

background colors for text.

• Use fonts that have serifs Serifs are small lines that finish off the main strokes

of a letter and often appear at the top or bottom of a letter. The text you

are reading has a serif font. Use fonts with serifs because these fonts may help

readers differentiate letters and because readers seem to prefer fonts with serifs.

Sound The medium of sound is helpful when you need to communicate short,

simple information, such as confirmation beeps, that users don’t have to refer to

later. Here are some guidelines that involve sound.

• Limit your use of sound A product that is constantly buzzing, beeping, chat-

tering, and singing to its users quickly becomes annoying. The user is working

with your product to get a job done. Limit your use of sound to those instances

where sound informs, rather than simply entertains, the user.

• Use lower frequency sounds High-frequency sounds can be shrill and annoying

, especially when the user has to hear them repeatedly. Generally, try to use

lower frequency sounds, around 100 hertz to 1000 hertz.

• Let the user control sound volume quickly and easily The user may want to

make the sound louder or softer for personal preference or to avoid disturbing

people nearby.Design so the user can adjust the sound volume. Make sound

controls large and obvious and let the user turn the sound off. These features

are shown in Figure 2.

Touch The medium of touch is helpful when you need to ask the user to make

simple choices, such as navigation decisions, without using a keyboard. Here are some

guidelines that involve touch.

• Use large touch areas If your product has a touch-sensitive screen, make it easy

for the users to activate the touch areas by making those areas fairly large. One

17

Figure 2.9: User adjusts sound volume with slider and turns off sound with Sound off

push button

18

Figure 2.10: Suggested sizes for touch areas and visual targets

study (Hall, Cunningham, Roache, and Cox, 1988) suggests using rectangular

touch areas that are 40 mm wide and 36 mm high surrounding smaller visual

targets, such as push buttons, that are 27 mm wide and 22 mm high. This is

shown in Figure 3.

• Use consistent colors and shapes to designate touch areas To make it obvious

which objects can be selected with touch, use the same color (Christ and Corso,

1975; Teichner, Christ, and Corso, 1977) and shape for those objects that can

be selected with touch.For example, always use a blue square to designate

buttons that the user can select with touch.

• Use different touch styles where appropriate Depending on the touch technology

you use, there are three general ways to design the touch interaction. First,

you can process the touch command when the user’s finger touches down on

the touch area. This technique is useful when you have just a few large touch-

selectable objects, such as push buttons, on the screen and the user can easily

19

recover after selecting the wrong object. Second, you can process the touch

command when the user’s finger goes up from a touch area. This technique

allows the user to highlight touch areas, such as push buttons, by touching or

sliding a finger over them until the desired touch area is highlighted. The user

can select the highlighted touch area command by lifting the finger off it. This

technique reduces errors when your touch areasare close together. Third, you

can add a cursor that appears on the screen, say, half an inch above the user’s

finger location. This technique is useful when the user has to select text. Since

the user’s finger is not blocking the text, the user can read the text before

selecting it with the cursor.

2.3 Perceptual Intelligence

Perceptual Intelligence is the knowledge and understanding that every-

thing we experience (especially thoughts and feelings) are defined by our percep-

tion.Perceptual intelligence is paying attention to people and the surrounding situa-

tion in the same way another person would, thus allowing these new devices to learn

to adapt their behaviour to suit us, rather than adapting to them as we do today.In

the language of cognitive science, perceptual intelligence is the ability to deal with the

frame problem; it is the ability to classify the current situation,so that it is possible

to know what variables are important and thus can take appropriate action. Once a

computer has the perceptual ability to know who, what,when, where, and why, then

the probabilistic rules derived by statistical learning methods are normally sufficient

for the computer to determine a good course of action.

The key to perceptual intelligence is making machines aware of their environment,

and in particular, sensitive to the people who interact with them.They should know

who we are, see our expressions and gestures, and hear the tone and emphasis of

our voice. People often confuse perceptual intelligence with ubiquitous computing

or artificial intelligence,but in fact they are very different.The goal of the perceptual

20

intelligence approach is not to create computers with the logical powers envisioned

in most Artificial Intelligence research, or to have computers that are ubiquitous and

networked, because most of the tasks we want performed do not seem to require

complex reasoning or a gods-eye view of the situation.One can imagine, for instance,

a well-trained dog controlling most of the functions we envision for future smart

environments. So instead of logic or ubiquity, we strive to create systems with reliable

perceptual capabilities and the ability to learn simple responses.

2.4 Perceptual Intelligent Systems

We have developed computer systems that can follow peoples actions,recognizing

their faces, gestures, and expressions.

Some of the systems are:

• Gesture recognition system

• Speech recognition system

• Nouse perceptual vision interface

2.4.1 Gesture Recognition System

2.4.1.1 Gesture Recognition

Gesture Recognition deals with the goal of interpreting human gestures via

mathematical algorithms. Gestures can originate from any bodily motion or state

but commonly originate from the face or hand. Current focuses in the field include

emotion recognition from the face and hand gesture recognition. Many approaches

have been made using cameras and computer vision algorithms to interpret sign lan-

guage.Gesture Recognition can be seen as a way for computers to begin to understand

human body language, thus building a richer bridge between machines and humans

21

than primitive text user interfaces or even GUIs (Graphical User Interfaces), which

still limit the majority of input to keyboard and mouse.

Gesture Recognition enables humans to interface with the machine (HMI) and

interact naturally without any mechanical devices. Using the concept of Gesture

Recognition, it is possible to point a finger at the computer screen so that the cursor

will move accordingly. This could potentially make conventional input devices such

as mouse, keyboards and even touch-screens redundant.Gesture Recognition can be

conducted with techniques from computer vision and image processing.Often the term

gesture interaction is used to refer to inking ormouse gesture interaction, which is

computer interaction through the drawing of symbols with a pointing device cursor.

Strictly speaking the term mouse strokes should be used instead of mouse gesture

since this implies written communication,making a mark to represent a symbol.

2.4.1.2 Working of Gesture Recognition System

The system must be able to look at a user, and determine where they are

pointing. Such a system would be integral to an intelligent room, in which a user

interacts with a computer by making verbal statements accompanied by gestures to

inform the computer of, for example, the location of a newly filed,physical document.

The system was written in c++ using components in c.The operation of the

system proceeds in four basic steps:

• Image input

• Background subtraction

• Image processing and Data extraction

• Decision tree generation/parsing

Initial training of the system requires the generation of a decision tree,however sub-

sequent use of the system only requires the parsing of the decision tree to classify the

image.

22

Image input

To input image data into the system, an SGI IndyCam(Silicon Graphics Inc was

used, with an SGI image capture program used to take the picture. The camera was

used to take first a background imageand then to take subsequent images of a person

pointing in particular directions.

Background subtraction

Once images are taken, the system performs a background subtraction of the im-

age to isolate the person and create a mask. The background subtraction proceeds in

two steps.First, each pixel from the background image is channelwise subtracted from

the corresponding pixel from the foreground image. The resulting channel differences

are summed, and if they are above a threshold, then the corresponding image of the

mask is set white, otherwise it is set black.The resulting image is a mask that outlines

the body of the person (figure ).

Two important features of the image are the existence of a second right arm, which

is the result of a shadow of the right arm falling on the wall behind the person, and

the noise in the generated mask image. This phantom arm is the result of the poor

image quality of the input image, but could be corrected for by the conversion of the

color space of the images and the use of another method of background subtraction. If

the images were converted from RGB to HSB color space, than the subtracted values

of the pixels (before being set to white or black, could be inspected, and those pixels

that have a very low brightness could be discarded as well (set to black). Since a

shadow tends to be very dark when compared to the body of a person (in an image),

those pixels that have a low brightness can be inferred to be part of a shadow.The

noise in the mask image can be reduced significantly by running an averaging filter

over the mask data. The GRS runs two such filters over the mask data, one with a

radius of one pixel, and another of a radius of three pixels.

Image processing and Data extraction

Once a mask is generated, then that image can be processed for data to extract

23

Figure 2.11: Background subtracted mask

24

Figure 2.12: image illustrating body center and arm algorithim

into a decision tree.Two strategies for extracting data from the image were tried.

The first was to find the body and arms. Each column of pixels from the mask was

summed,and the column with the highest sum was assumedto be the an equalized

backg round subcenter of the body. This is a valid criteria for determining the tracted

image body center, based on the assumptions of the input image.In all of the samples

this technique was tried , it was successful in finding the center of the body, within a

few pixels.From the center of the body, horizontal rows are traced out to the left and

right until the edge of the mask is reached (pixels turn from white to black). The row

of pixels that extends the furthest is assumed to be the arm that is pointing. This

again is a valid decision based on the assumptions about the input image. Only one

arm is pointing at a time, and the arm is pointing in a direction. For this , algorithm

created was successful at finding the pointing arm to within a few pixel for all images

tested, including those with shadows.

// find the maximum distances left and right from the middle, and call them

arms for (y = 0; y ¡ mask-¿y(); y++)

if (mask-¿Red(xMiddle, y) != 0)

// left first

25

for (x = xMiddle; mask-¿Red(x, y) != 0; x);

if (x ¡ xLeftMax)

xLeftMax = x + 1;

yLeftMax = y;

// then right

for (x = xMiddle; mask-¿Red(x, y) != 0; x++);

if (x ¿ xRightMax)

xRightMax = x - 1;

yRightMax = y;

Decision tree generation/parsing

Once the data vectors are written to disk, is executed to create the decision tree.

This process is straightforward, involving only the manual classification of the test

vectors into six categories representing the images: right-up, right-middle, rightdown,

left-up, left-middle,left-down. The decision to use only these six vectors was a result of

the use of a decision tree, as well as the difficulty in classifying the and arm algorithim

images with more granularity. Since the system usesa decision tree, the method of

learning must be supervised. Besides being tedious, the manual classification of the

images is difficult to classify into more categories.Figure 6 - a mask converted to

vertical bar representation.The system as presented here is a combination of two

algorithms for image data extraction.To use the system, instead of generating the

tree,a single image is processed, and then run through the decision tree , resulting in

the classification of the image into one of the six mentioned categories.

2.4.1.3 Challenges of Gesture Recognition

There are many challenges associated with the accuracy and usefulness of

Gesture Recognition software. For image-based gesture recognition there are limita-

tions on the equipment used and image noise. Images or video may not be under

consistent lighting, or in the same location. Items in the background or distinct

26

features of the users may make recognition more difficult. The variety of implemen-

tations for image-based gesture recognition may also cause issue for viability of the

technology to general usage. For example, recognition using stereo cameras or depth-

detecting cameras are not currently commonplace. Video or web cameras can give

less accurate results based on their limited resolution.

2.4.2 Speech Recognition System

Speech recognition converts spoken words to machine-readable input (for

example, to the binary code for a string of character codes). The term voice recog-

nition may also be used to refer to speech recognition, but more precisely refers to

speaker recognition, which attempts to identify the person speaking, as opposed to

what is being said.

The following definitions are the basics needed for understanding speech recogni-

tion technology.

• Utterance

An utterance is the vocalization (speaking) of a word or words that represent a

single meaning to the computer. Utterances can be a single word, a few words,

a sentence, or even multiple sentences.

• Speaker Dependance

Speaker dependent systems are designed around a specific speaker. They gener-

ally are more accurate for the correct speaker, but much less accurate for other

speakers. They assume the speaker will speak in a consistent voice and tempo.

Speaker independent systems are designed for a variety of speakers. Adap-

tive systems usually start as speaker independent systems and utilize training

techniques to adapt to the speaker to increase their recognition accuracy.

• Vocabularies

Vocabularies (or dictionaries) are lists of words or utterances that can be rec-

27

ognized by the SR system. Generally, smaller vocabularies are easier for a

computer to recognize, while larger vocabularies are more difficult. Unlike nor-

mal dictionaries, each entry doesn’t have to be a single word. They can be

as long as a sentence or two. Smaller vocabularies can have as few as 1 or 2

recognized utterances (e.g.”Wake Up”), while very large vocabularies can have

a hundred thousand or more!

• Accuract

The ability of a recognizer can be examined by measuring its accuracy or how

well it recognizes utterances. This includes not only correctly identifying an

utterance but also identifying if the spoken utterance is not in its vocabulary.

Good ASR systems have an accuracy of 98acceptable accuracy of a system

really depends on the application.

• Training

Some speech recognizers have the ability to adapt to a speaker. When the

system has this ability, it may allow training to take place. An ASR system

is trained by having the speaker repeat standard or common phrases and ad-

justing its comparison algorithms to match that particular speaker. Training a

recognizer usually improves its accuracy. Training can also be used by speak-

ers that have difficulty speaking, or pronouncing certain words. As long as

the speaker can consistently repeat an utterance, ASR systems with training

should be able to adapt.

The speech recognition process is performed by a software component known as

the speech recognition engine. The primary function of the speech recognition engine

is to process spoken input and translate it into text that an application understands.

The application can then do one of two things:

• The application can interpret the result of the recognition as a command. In

this case, the application is a command and control application. An example

28

of a command and control application is one in which the caller says check

balance, and the application returns the current balance of the callers account.

• If an application handles the recognized text simply as text, then it is considered

a dictation application. In a dictation application, if you said check balance,

the application would not interpret the result, but simply return the text check

balance.

2.4.2.1 Types of Speech Recognition

Speech recognition systems can be separated in several different classes by

describing what types of utterances they have the ability to recognize. These classes

are based on the fact that one of the difficulties of ASR is the ability to determine

when a speaker starts and finishes an utterance. Most packages can fit into more

than one class, depending on which mode they’re using.

• Isolated Words

Isolated word recognizers usually require each utterance to have quiet (lack of

an audio signal) on BOTH sides of the sample window. It doesn’t mean that it

accepts single words, but does require a single utterance at a time. Often, these

systems have ”Listen/NotListen” states, where they require the speaker to

wait between utterances (usually doing processing during the pauses). Isolated

Utterance might be a better name for this class.

• Connected Words

Connect word systems (or more correctly ’connected utterances’) are similar

to Isolated words, but allow separate utterances to be ’runtogether’ with a

minimal pause between them.

• Continuous Speech

Continuous recognition is the next step. Recognizers with continuous speech

capabilities are some of the most difficult to create because they must utilize

29

special methods to determine utterance boundaries. Continuous speech recog-

nizers allow users to speak almost naturally, while the computer determines

the content. Basically, it’s computer dictation.

• Spontaneous Speech

There appears to be a variety of definitions for what spontaneous speech actu-

ally is. At a basic level, it can be thought of as speech that is natural sounding

and not rehearsed. An ASR system with spontaneous speech ability should

be able to handle a variety of natural speech features such as words being run

together, ”ums” and ”ahs”, and even slight stutters.

• Voice Verification/Identification

Some ASR systems have the ability to identify specific users. This document

doesn’t cover verification or security systems.

2.4.2.2 Working of Speech Recognition Systems

The speech recognition process is performed by a software component known

as the speech recognition engine. The primary function of the speech recognition

engine is to process spoken input and translate it into text that an application un-

derstands. The application can then do one of two things:

• The application can interpret the result of the recognition as a command. In

this case, the application is a command and control application. An example

of a command and control application is one in which the caller says check

balance, and the application returns the current balance of the callers account.

• If an application handles the recognized text simply as text, then it is considered

a dictation application. In a dictation application, if you said check balance,

the application would not interpret the result, but simply return the text check

balance.

30

Recognition systems can be broken down into two main types. Pattern Recog-

nition systems compare patterns to known/trained patterns to determine a match.

Acoustic Phonetic systems use knowledge of the human body (speech production, and

hearing) to compare speech features (phonetics such as vowel sounds). Most modern

systems focus on the pattern recognition approach because it combines nicely with

current computing techniques and tends to have higher accuracy. Most recognizers

can be broken down into the following steps:

1. Audio recording and Utterance detection

2. PreFiltering (preemphasis, normalization, banding, etc.)

3. Framing and Windowing (chopping the data into a usable format)

4. Filtering (further filtering of each window/frame/freq. band)

5. Comparison and Matching (recognizing the utterance)

6. Action (Perform function associated with the recognized pattern)

Although each step seems simple, each one can involve a multitude of different

(and sometimes completely opposite) techniques.

(1) Audio/Utterance Recording: can be accomplished in a number of ways. Start-

ing points can be found by

comparing ambient audio levels (acoustic energy in some cases) with the sample

just recorded. Endpoint detection is harder because speakers tend to leave ”artifacts”

including breathing/sighing,teeth chatters, and echoes.

(2) PreFiltering: is accomplished in a variety of ways, depending on other features

of the recognition

system. The most common methods are the ”BankofFilters” method which uti-

lizes a series of audio filters to prepare the sample, and the Linear Predictive Coding

method which uses a prediction function to calculate differences (errors). Different

forms of spectral analysis are also used.

(3) Framing/Windowing involves separating the sample data into specific sizes.

This is often rolled into step 2 or step 4. This step also involves preparing the sample

31

boundaries for analysis (removing edge clicks, etc.)

(4) Additional Filtering is not always present. It is the final preparation for each

window before comparison and matching. Often this consists of time alignment and

normalization.

(5), Comparison and Matching. Most involve comparing the current window

with known samples. There are methods that use Hidden Markov Models (HMM),

frequency analysis, differential analysis, linear algebra techniques/shortcuts, spectral

distortion, and time distortion methods. All these methods are used to generate a

probability and accuracy match.

(6) Actions can be just about anything the developer wants.

2.4.2.3 Performance of speech recognition systems

The performance of speech recognition systems is usually specified in terms of

accuracy and speed. Most speech recognition users would tend to agree that dictation

machines can achieve very high performance in controlled conditions. There is some

confusion, however, over the interchangeability of the terms ”speech recognition” and

”dictation”.

Commercially available speaker-dependent dictation systems usually require only

a short period of training (sometimes also called ‘enrollment’) and may successfully

capture continuous speech with a large vocabulary at normal pace with a very high

accuracy. Most of experiments claim that recognition software can achieve more than

75 percentage accuracy if operated under optimal conditions. Optimal conditions

usually assume that users:

• have speech characteristics which match the training data,

• can achieve proper speaker adaptation, and

• Work in a noise free environment (e.g. quiet office or laboratory space).

This explains why some users, especially those whose speech is heavily accented, might

32

achieve recognition rates much lower than expected. Speech recognition in video has

become a popular search technology used by several video search companies. Limited

vocabulary systems, requiring no training, can recognize a small number of words

(for instance, the ten digits) as spoken by most speakers. Such systems are popular

for routing incoming phone calls to their destinations in large organizations. Both

acoustic modeling and language modeling are important parts of modern statistically-

based speech recognition algorithms. Hidden Markov models (HMMs) are widely used

in many systems.

2.4.3 Nouse Perceptual Vision Interface

Evolved from the original Nouse ie Nose as Mouse. Nouse PVI has several

unique features that make it preferable to other hands-free vision-based computer

input alternatives. natives. Its original idea of using the nose tip as a single refer-

ence point to control a computer has been confirmed to be very convenient.The nose

literally becomes a new finger which can be used to write words, move a cursor on

screen, click or type. Being able to track the nose tip with subpixel precision within

a wide range of head motion, makes performing all control tasks possible.

2.4.3.1 Computer Control Actions

Nouse PVI is a perceptual vision interface program that offers a complete

solution to working with a computer in Microsoft Windows OS hands-free. Using a

camera connected to a computer, the program analyzes the facial motion of the user

to allow him/her to use it instead of a mouse and a keyboard. As such Nouse- PVI

allows a user, to perform the basic three computer-control actions:

• cursor control: includes

a) Cursor positioning

b) Cursor moving, and

33

c) Object drugging - which are normally performed using mouse motion

• clicking: includes

a) right-button click

b) left-button click

c) Double-click, and

d) Holding the button down - which are normally performed using the mouse

buttons

• key/letter entry: includes

a) typing of English letters

b) Switching from capital to small letters, and to functional keys

c) entering basic MS Windows functional keys as well as Nouse functional keys

- which would normally be performed using a keyboard

2.4.3.2 Tools Used In Nouse Perceptual Vision Interface

The program is equipped with such tools as:

1. Nousor (Nouse Cursor) - the video-feedback-providing cursor that is used to

point and to provide the feeling of ” touch” with a computer.

2. NouseClick - a nose-operated mechanism to simulate types of clicks

3. NouseCodes - configurable Nouse tool that allows entering computer commands

and operate the program using head motion codes.

4. NouseEditor - provides an easy way of typing and storing messages hands-free

using face motion. Typed messages are automatically stored in Clipboard (as with

CNTR+A, CNTR+C)

5. NouseBoard- a specially designed for face-motion-based typing on-screen key-

board that automatically maps to the user’s facial motion range.

34

6. NouseTyper - a configurable Nouse tool that allows typing letters by drawing

them inside the cursor (instead of using the NouseBoard)

7. NouseChalk - a configurable Nouse tool that allows writing letters as with a

chalk on a piece of paper. Written letters are automatically saved on hard- drive as

images that can be opened and emailed.

2.4.3.3 Working Of Nouse Perceptual Vision Interface

Once a camera is selected you will be in the inactive sleep state. You can

tell you are in this state because the nouse icon is white and turned to the left. Also

the buttons on the Nouse Cursor (in the top right corner of the screen) are red.

The program remains in this state until a user showing facial motion is detected by

camera.When user is detected, we switch to the inactive wake up state. You can

tell you are in this state because the nouse icon turns to face straight up. Also,

the buttons on the Nouse Cursor turn white. In this state the system verifies that

position of users face. If this position is not consistent over 10-20 secs, Nouse goes

back to sleep state.

Working in NouseCursor State

1. Sleep:Start nouse cursor. (Bring it to ”put face in centre mode”)

When you initially go to NouseCursor state, you are in the Nouse Sleep states all

Nousor buttons colours all red. To Make up control - produce significant face motion.

2. Wake-up: Putting nose (face) in center of nouse cursor to activate it.

The two middle buttons will remain red, while side buttons become bllue. This is

the indication that you must now put your face in the center of the cross-hair. Once

you have done so successfully, the buttons will be blue and you are ready to start

nousing.

3. Crude navigation:Move the cursor to a point on the screen as with a joystick.

When all of the buttons are blue, this means that you are in a general mouse

movement state. The nouse cursor works as a joystick. If your nose is in the center,

35

the nouse cursor will not move. Then as you get farther away from the center in

any direction, the nouse cursor will move proportionally faster is that direction. You

can use the mode to place the cursor approximately near where you’ll want to click.

There is no need to be exact. Once you are satisfied, move your nose back to the

center of the cross-hair and then wait for the buttons to change to yellow.

4. Fine navigation: Move the cursor as by pushing a mouse.

When the buttons are all yellow this means that we are in the precise movement

state. This state can be analogized to moving an actual mouse cursor when the mouse

pad is very small. In order to move to the right for example, you must nudge your

nose to the right and then move back to center and repeat this process. The mode is

used to place the cursor exactly where you would like to click. Once you are satisfied,

stay still and wait for the the cursor to switch to clicking mode.

It is important to note that when placing the cursor you should make note of

which corner of the nouse cursor will be used to make the click. This is indicated by

purple lines protruding from the clicking corner. If you wish to change the clicking

corner, enter the correct code as described later.

5. Prepare to Click:Now that the cursor has been placed, you move into the final

state; that of clicking.

This is indicated by all of the buttons being coloured green. A countdown starting

at 3 will count down and to click all you need to do is move. After you click, you will

go back to the putting face in center mode .

6. Performing different types of Clicking

a. Choose not to click.

b. Do a left click

To do a left click, move down and to the left.

c. Do a right click

To do a right click, move down and to the right.

d. Do a double click.

36

To do a double click, move up and to the left.

e. Drag an item.

If you select drag, the will emulate pressing down on the left mouse button and

holding it down. Once you select drag, a ’D’ will be drawn in the nouse cursor. When

you choose you next place to click, the action will be to release the mouse no matter

what direction you move.

7. Entering motion codes with nousor - NouseCode

Aside from clicking and moving the cursor, there are some actions that you may

want to perform while in the nouse cursor state. The way this can be done is using

the nousor motion coding scheme. Whenever you move your position to a corner of

the range, that corner’s corresponding number will be remembered and shown inside

Nousor as green dot. To enter a code, simply move to the correct corners in order.

So for example, if you want to enter the code 1-2-3-4, you need to move to the top

left, bottom left, bottom right and top right in that order. After any code is entered,

you need to confirm wether or not you actually want to enter that code. A letter

corresponding to that code will appear on the nouse cursor to let you know that you

are in confirmation mode. To confirm you must move in a vertical direction (if you

are up, move down and if you are down, move up). To cancel the code, move in the

horizontal direction.

a. Open Nouse Board

To open Nouse Board enter the code 1-2-3-4. Then to confirm go to 3.

b. Choose to not open nouse board

To cancel opening Nouse Board, go to 1.

c. Change clicking corner

If you don’t move, no click will be made and you will go back to the general

mouse movement state. To change the conner of the Nouse Cursor that will be used

for clicking (this is represented by purple lines protruding from the corder) enter code

1-2-1-2. Then to confirm go to 1.

37

d. Choose to not change clicking corner

To cancel changing the clicking corner, go to 3.

e. Glue/un-glue cursor to nousor

To glue the actual cursor to the nouse cursor, enter Glue code. The same code is

used to un-glue the cursor from the nousor.

f.Chose to not glue cursor to nouser

To cancel gluing the cursor, go to 2.

g. Press enter

To simulate hitting the enter key, enter code 4-1-4-3. Then to confirm go to 4.

h. Choose to not press enter

To cancel hitting the enter key, go to 2.

8. Reconnecting with nouse

Sometimes the nouse program will lose track of your nose. You will notice that

this is happening when you observe that the blue blob that normally appears on your

nose in the nouse cursor will stray. We have implemented many techniques to try to

regain the nose once it is lost. If none of these methods seem to work and the blob

keeps on straying from the nose, this is likely a case of bad callibration. The best

option in this case is to go through the express calibration once again. a. Try putting

nose back where blue is. This is the most intuitive way to get the nose back. If you

see the blue blob stray away from the nose, simply move your nose back to where the

blue blob is. Once you move your nose under it, the blob will likely start to follow

the nose once again. b. Move left right to go back to face in center mode. If you

quickly move your head all the way to the left of your range and then all the way to

the right, you will go back to the face in center mode. Once you put your face in the

center, the nouse program should put the blue blob back on nose nose. Working in

NouseBoard state

• Motion-based key entry using virtual (on-screen) NouseBoard and NousePad.

38

Figures/Chap3/nui2.png

Figure 2.13: Selecting Shift, selecting H

When you first open NouseBoard you will be in the state where you need

to put your face in the center of the nouse cursor. Once you do this, the

NouseBoard and NousePad will popup (see Figure) and youll be ready to type.

The first state of the NouseBoard is one where the whole board in coloured

white. This indicates that no key have been selected yet. Note that for each

cell of the NouseBoard, there are four characters (or commands), one at each

corner. The first thing you need to do is decide which corner the character you

would like to type resides in. When you move the NouseBoard selector to a

corner, all of the cells will have their respective characters in that corner stay

white while the rest of the NouseBoard turns gray. Now that a corner has been

selected, you can move the NouseBoard selector around to select the character

you would like to type. Once you have moved the selector to the requested cell,

stay still for a couple of seconds. This will cause the character or command to

turn green. Once a character has turned green you simply need to move the

selector to another cell and that character will be typed in the NousePad. The

NouseBoard will return to its initial all white state.

• Choosing to not type a letter.

If you have a character coloured green but you dont want to type it, simple

stay still in the same cell for a couple of seconds. This will cause the letter not

to be typed and the NouseBoard to return to its initial all white state.

• Getting back nouse after losing it.

All three methods described for the Nouse Cursor state will work equally here.

See section 6.6.

39

• Cancelling corner selection.

If you select a corner and then realize that the character you want to type is

not in that corner of its cell, there are two ways to cancel the corner selection.

The first is to highlight a letter and then not type it, as in section 7.b. The

other option is to move your face to left and to the right as described in section

6.6.b. This will cause the program to ask you to put your face in the center of

the NouseCursor. Once you do so, the keyboard will be reset to the all white

state.

• Exiting NousePad (copying to clipboard) Once your are satisfied with the text

in the NousePad, you can exit the NouseBoard state by selecting mouse on the

NouseBoard. This will cause the NouseBoard to close and the text in NousePad

to be copied to your systems clipboard. This text can then be accessed using

the paste option of what ever application you are using.

40

CHAPTER 3

Applications

There are some applications currently using Perceptual Intelligence technique for

future development.some of these applications are given below

3.1 SMART ROOMS

The idea of a smart room is little like having a butler; that is a passive

observer who usually stands quietly in the corner but who is constantly looking for

opportunities to help. Both smart rooms and smart clothes are instrumented with

sensors that allow the computer to see, hear, and interpret users actions (currently

mainly cameras, microphones, and electromagnetic field sensors, and also biosensors

like heart rate and muscle action.)People in a smart room can control programs,

browse multimedia information, and experience shared virtual environments without

keyboards, special sensors, or special googles.The key idea is smart room or clothing

can know something about what is going on, it can react intelligently.

Our first smart room was developed in 1989.Now there are smart rooms in Japan,

England, and throughout places in U.S.They can be linked together by ISDN tele-

phone lines to allow shared virtual environment and cooperative work experiments.

Some of the perceptual capabilities are available to smart rooms. To act intelli-

gently in a day to day environment, the first thing we need to know is: where are the

people? The human body is a complex dynamic system, whose visual features are

time varying, noisy signals.

Object tracking can be used to detect the location. Once the person is located,

and visual and auditory attention has been focused on them, the next question to

ask is: who is it? The question of identity is central to adaptive behaviour because

Figure 3.1:

who is giving a command is often as important as the command itself. Perhaps the

best way to answer the question is to recognize them by their facial appearance and

by their speech. For general perceptual interfaces, person recognition systems will

need to recognize people under much less constrained conditions. One method of

achieving greater generality is to employ multiple sensory inputs; audio and-video-

based recognition systems in particular have the critical advantage of using the same

modalities that humans use for recognition.

We have developed computer systems for speech-recognition, hand gesture recog-

nition face recognition etc. After a person is identified, next crucial factor is facial

expression. For instance a car should know if the driver is sleepy, and a teaching pro-

gram should know if the student looks bored.So, just as we can recognize a person once

42

we have accurately located their face, we can also analyse the persons facial motion

to determine their expression. The lips are of particular importance in interpreting

facial expression, so we are giving more importance to tracking and classification of

lip shape. First step of processing is to detect and characterize the shape of the lip

region. For this task, a system called LAFTER is developed. Online algorithms are

used to make maximum posteriori estimates of 2D head pose and lip shape. Using

lip shape features derived from LAFTER systems we can train Hidden Markov Mod-

els for various mouth configurations.HMMs are well developed statistical modelling

techniques for modelling time series data, and are used widely in speech recognition.

3.2 SMART CLOTHES

In the case of a smart room, cameras and microphones are watching people

from a third person perspective.However,when we build the computers,cameras,microphones

and other sensors into our clothes ,the computers view moves from a passive third

person to an active first person vantage point. That means smart clothes can be more

intimately and actively involved in the users activities. If these wearable devices have

sufficient understanding of the users situation-that is-enough perceptual intelligence-

then they should be able to act as an intelligent personal agent, proactively providing

the wearer with information relevant to the current situation.

Eg: suppose we are placing a GPS (Global Position Sensor) into our belt, then

navigation software can help us to find our way around by whispering directions in

your ear or showing a map on a display built into our glasses. Similarly body worn ac-

celerometers and tilt sensors can distinguish walking from standing from sitting, and

bio sensors such as galvanic skin response (GSR) are correlated with mental arousal,

allowing construction of wearable medical monitors. A simple but important applica-

tion for a medical wearable is to give people feedback about their alertness and stress

level. Centre for Future health at the University of Rochester has developed early

warning systems for people with high risk medical problems, and eldercare wearable

43

to help keep seniors out of nursing homes. These wearable devices are examples of

personalized perceptual intelligence, allowing proactive fetching and filtering of in-

formation for immediate use by the wearer. With the development of new self care

technology, tiny wearable health monitors may one day continuously collect signals

from the body and transmit data to a base station at home. Predictive software may

identify trends and make specific health predictions, so users can prevent crisis and

better manage daily routines and health interventions.

Consider that we have built wearables that continuously analyze background

sound to detect human speech. Using this information, the wearable is able to know

when you and another person are talking, so that they wont interrupt. Now researches

are going a step further, using microphones built into a jacket to allow wod-spotting

software to analyze your conversation and remind you of relevant facts. Cameras

make attractive candidates for a wearable, perceptual intelligent interface, because

a sense of environmental contexts may be obtained by pointing the camera in the

direction of users gaze. For instance, by building a camera into your eye glasses, face

recognition software can help you remember the name of the person you are looking

at.

44