ASL using dynamic gesture recognisation

7/29/2019 ASL using dynamic gesture recognisation

1/30

PRESENTED

BY:-


2/30

IIT-BHU.

Indian Institute of Technology (BHU)Varanasi [IIT(BHU)], formerly knownas Institute of Technology, BanarasHindu University (IT-BHU), is apublic engineering institution locatedin Varanasi, Uttar Pradesh, India.Founded in 1919 as part ofthe Banaras Hindu University, it wasdesignated anIndian Institute ofTechnology in 2012. IIT(BHU) has 13departments and three inter-

disciplinary schools.


3/30

Sixth Sense Technology..

Sixth Sense allows people to use thecomputer without a screen or a keyboard

It can turn any surface into a touch screen It is made with store-bought components The software program processes the video

stream data captured by the camera andtracks the locations of the colored markers[that you wear on your fingers] using simple

computer-vision techniques(http://www.pranavmistry.com/projects/sixthsense

/index.htm#ABOUT) According to a number of sources, graduate

students at MIT have been working on this forthe past four months day and night.
http://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.pranavmistry.com/projects/sixthsense/index.htm


4/30

AT A GLANCE.

The device recognizes items in stores, and thengathers and projects information about productsto the user. It also is able to provide quick signalsto let users know what the best products arebased on their needs.

Another amazing feature is the devices ability tolook at an airplane ticket and let the user knowwhether the flight is delayed or on time. It is alsoable to recognize books (based on an image orRFID) in a book store and then project readerreviews or author information from the Internetonto blank pages.

The device can also recognize articles innewspapers, retrieve the latest related stories orvideo from the Internet and then display them onpages for the user. Say goodbye to yourdictionary and encyclopedia


5/30

Educational Use

Engagement

Research

Collaboration

Cost-effectiveness


6/30

Engagement

.

If this technology does nothing else, the idea thatstudents can simply use their hands to draw images

on walls and take pictures with their hands will be (atleast initially) quite motivating


7/30

Research

Sixth Sense will give new meaning tolooking something up.

Since we are essentially talking about

the internet, students will be given theopportunity to portably research theworld.

They can all go on a virtual field trip

and project the images on their owndesks, for example


8/30

Collaboration

Students will be given an opportunitythrough this technology to feel moreconnected to others.

Although there does seem to be thesense that individuals could walk throughlife wired and lonely, I like to think theclassroom would serve as theenvironment in which students andteachers using Sixth Sense would wantto share.

Not only could the sharing take place inthe classroom, I suppose since the usersare connected online, sharing could takeplace around the world.


9/30

Cost-effectivenes

As mentioned earlier, this productcurrently costs only $350. From aneducators stand point, this is

minimum given the amount of moneyspent on textbooks, computer labsand LCD projectors.

For educators, Sixth Sense would be

a way to put technology (literally) intoeach students hands, because of itssize and its cost.


10/30

Sign Language

Used primarily by hearing-impairedpeople,

Uses a different medium: hands, face,

and eyes (rather than vocal tract orears).

NOT derived from spoken language

Study ofsign language providesunique insight into the nature oflanguage itself.

Brain function similarities indicate


11/30

American sign language


12/30

Structure of ASL

Five Basic Parameters Shape of the hand

Place of articulation (location)

Movement Palm Orientation Region of the hand contacting body

Orientation of the hand to body

Orientation of hands to each other Facial Expressions


13/30

INTRODUCTION TO IMAGE PROCESSING

1 pixel


14/30

Types Of Image

GREY SCALE IMAGE (IntensityImage


15/30

RGB IMAGE


16/30

Black and White Image(BinaryImage)


17/30

Conversion of Image


18/30

DEMO


19/30

Dynamic gesture and Computervision

Human-computer interaction (HCI) has become anincreasingly important part of our daily lives.Keyboards and mice are the most popular mode ofHCI.

Virtual Reality and Wearable Computing requirenovel interaction modalities.

in a way that humans communicate with each other.

Hand gesture is a natural and intuitive

communication mode.Other applications: Sign Language Recognition,video transmission, and so on.

C t Vi i i


20/30

Computer Vision usinggesture

Replace mouse and keyboardPointing gesturesNavigate in a virtual environmentPick up and manipulate virtual objectsInteract with a 3D worldNo physical contact with computer

Communicate at a distance


21/30

Gesture for Human-computerInteraction


22/30

Gesture

What is a gesture? Body motion used for communication.

There are different types of gestures.

Hand gestures (e.g., waving goodbye). Head gestures (e.g., nodding).

Body gestures (e.g., kicking).

Example applications: Human-computer interaction. Controlling robots, appliances, via gestures.

Sign language recognition.


23/30

Dynamic Gestures


24/30

Gesture Analysis

1 Gesture detection and featureextraction

- skin color clues based approaches

- motion clues based approaches- multiple clues based approaches

- features include gray image, binary

silhouette, moving region, edge, contour,and so on.


25/30

Decomposing GestureRecognition We need modules for:

Computing how the person moved. Person detection/tracking.

Hand detection/tracking.

Articulated tracking (tracking each body part). Handshape recognition.

Recognizing what the motion means.

Motion estimation and recognition are

quite different tasks. When we see someone signing in ASL, we

know how they move, but not what themotion means.


26/30

GESTURE SPOTTING

Obtaining the trajectory of dynamic gesture fromeach frame

Applying mean shift algorithm for each block.

Comparing the out with data base made bycontinues samples

Finally giving the out put in the voice form.

21 3

4

56 7 8

2

1

34

5

7 8 9

6


27/30

Project in a glance.

Hand Detection +Feature Extraction

Gesture Models+Mean shift algorithm

Spotting Rules +Trajectory comparing

Input trajectory Spotting

VideoStream Feature

Vector

Gesture id,sound output


28/30

About my professor

PhD

ProfessorDepartment of ElectricalEngineeringInstitute of Technology, BHU

Dr. S. K. Nagar

Areas of Interest:Control Systems

AI Applications


29/30

Career at IIT

Summer cum research intern with thestipend of 7000+HRA.

Project Associate for MHRD projects

accomplished by IIT with pay of10,000+HRA.

JRF for particular field with pay of

18000+HRA. SRF with pay of 26000+HRA.


30/30

ANY QUERies??

THANK YOU.

Documents

ASL using dynamic gesture recognisation