CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 1

Communication Assistance System for Speech Impaired

1 M.Sudhakar 2 B. Shruthi

1 Professor 2 PG Scholar

Dept of ECE, CMR College of Engineering & Technology, Hyderabad, Telangana-India.

Email: 1 vp@cmrcet.org

Email: 2 biradarshru54@gmail.com

Abstract-Sign Language is the only approach for communication of speech impaired persons. However normal

persons cannot perceive the linguistic communication, therefore it becomes difficult for speech impaired person

to communicate with others. In this paper an electronic speaking system has been developed to ease

communication of speech impaired persons with others. An electronic speaking system consists of 3 system

frameworks namely scene capturing, data processing and audio output. Camera as input device captures the

gestures and feed into raspberrypi3 processor. The processor identifies the gesture image from its data base

images and activates the corresponding recorded audio file from SD card to play to speaker. Simultaneously the

corresponding text command is displayed on the LCD. In this project ten most important gestures have been

captured and the data base for the same has been created. This device also helps in communicating between two

speech impaired persons using the texting feature.

Keywords- Communication of speech impaired, Raspberry Pi3, Open CV Python, Webcam, Speakers

I. INTRODUCTION Sign Language is the solely method for communication of speech impaired persons. However normal persons

cannot follow the sign communication therefore it becomes difficult for speech impaired to communicate with a

normal person. This necessitates development of a communication assistance system for the speech impaired.

Considering such necessity, an electronic speaking system has been developed in our paper to ease

communication of speech impaired persons. The electronic speaking system developed consists of 3 system

frame works namely scene capturing, data processing and audio output [1]. The camera captures the gestures of

speech impaired person and feed the same into raspberry pi3. The raspberry pi3 processes this information and

tries to match with the data in data base. A code is generated by the processor and accordingly the audio file is

picked and played. Simultaneously the processor displays the text on LCD, corresponding to the code. A data

base of ten commonly used gestures have been captured and placed in the data base. Accordingly the audio and

text files are created.

II. EXISTING METHOD

In existing framework a glove containing five flex sensors is worn by the speech impaired person. The gestures

of the hand are sensed by the flex sensors [2] [3] 4] [5] and the same are processed by a micro processor. The

disadvantage in this method is that the sensors are connected to the processor through wires [8].

III. SYSTEM PROPOSED

Our paper adopts a computerized talk me framework. There are no wearable sensors and connecting wires. An

electronic talking contraption ends up cutting edge which incorporates a 3 framework systems viz Scene

capturing, Data handling, Audio yield. The efficiency of communication between a normal person and speech

impaired individual is improved by texting the gesture made by speech impaired and then playing a

corresponding audio file. A camera and raspberry pi3 processor has been used to capture the gesture of the

speech impaired person. The image processing program has been written using OpenCV. The image is compared

with images available in the data based and a code is generated on matching. The generated code is used to

display corresponding text on the LCD. The same is also used to select the corresponding audio file and play it

[3].

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 2

IV. SYSTEM HARDWARE

Fig1: Block diagram

V. RASPBERRY Pi 3 PROCESSOR The Raspberry Pi 3 demonstrate B has particularly worked with the Broadcom BCM2837 System-On-

Chip(SoC) incorporates four superior ARM Cortex-A53 process centers running at 1.2 GHz with 32 kB Level

one and 512 kB Level a couple of reserve memory, a Video Core IV illustrations processor, and is associated

with a 1GB LPDDR2 memory module on the back of the board. It also alternatives 40-pins broadly useful info

yields (GPIO) and enhanced property with Bluetooth Low Energy (BLE) and BCM43143 Wi-Fi on board. It

likewise has an updated control administration wellspring of 5V USB control supply up to 2.5 Amps. As of

now, Raspberry Pi 3 Model B is best of Raspberry Pi PCs. The framework handling is colossal with 1.2 GHz

clock speed and 1GB RAM Raspberry Pi can play out every single propelled process. As per the association

shrewd, the board ought to be equipped for sending information to and from the board quickly. Another double

band Wi-Fi underpins for 2.4 GHz and 5 GHz 802.11b/g/n/air conditioning which is likewise guarantees

twofold all through the 802.11b/g/n/air conditioning Wi-Fi on the Raspberry Pi 3 Model B. With the expansion

of Gigabit Ethernet over USB 2.0, the wired Ethernet execution is additionally supported, with an extraordinary

throughput of around 300MB.

VI. WEB CAM Webcams are video capturing objects identified with PCs or pc frameworks, more often than not making

utilization of USB or, if they interface with frameworks, Ethernet or Wi-Fi. They are fabulous for low gathering

charges and versatile applications.

Webcams typically contain a central factor, a preview sensor, and a couple of assistance equipment. Exceptional

central components are available, basically the most extensively perceived being a plastic point of convergence

that might be settled and out to set the digicam's center interest. Settled mindfulness central angles, which have

no relationship for trade, are in addition open. Preview sensors can be CMOS or CCD, the past being

overpowering for insignificant exertion cameras, yet CCD cameras don't generally beat CMOS-based cameras

inside the straightforwardness esteem run. Client webcams are for the most part VGA choice with an aspect cost

of 30 plots for each second. Bigger goals, in uber pixels, are available and more prominent packaging charges

are building up to appear. The video get framework includes various making prepared advances [6]. To with the

simple video signal is digitized through an easy to-automated converter to make an unrefined, developed know-

how stream. As an outcome of composite video, the luminance and chrominance are then disengaged.

Consequent, the chrominance is demodulated to give shading qualification video know-how. Finally, the

aptitude is altered with the guide of a shading house converter to make know-how in conformance with any of a

couple of shading space standards, for instance, RGB and YCbCr. By and large, these methods constituted video

unraveling, on the grounds that they "translate" a basic video foundation, for instance, NTSC or PAL. Support

equipment is close by to scrutinize the photograph from the sensor and transmit it to the host pc. The advanced

camera imagined to 1 angle, for example, uses a Sonix SN9C101 to transmit its photo over USB. A few cameras

- , for instance, cell phone cameras - make utilization of a CMOS sensor with helping objects.

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 3

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 4

Features:

Smallest remote video and sound advanced camera.

Wi-Fi transmission and social affair

High affectability

Effortless foundation and movement

Effortless to vanity

Mild weight

Low control usage

Small measure

Particulars:

Output repeat: 900 MHz 1200 MHz

Output control: 50 mW -200 mW

Power convey: DC 6-12 V

Partition anchored: 10 m

VII. SPEAKERS Add some extra impact to your sound undertaking with these USB controlled enhancers. We tried a huge bit of a

tremendous measure of different models to find ones with a better than average repeat response in this way,

you'll get quality sound yield for music playback. There's even a volume control wheel on it so you can set it up

splendidly. Essentially, interface the standard 3.5 mm stereo associate with the Raspberry Pi, Wave shield, et

cetera. For control, interface the USB connector to anything that can give USB control. We can normally keep

the volume about halfway (which is still to a great degree riotous), where the current drawn is 200-400 mA. At

max volume, you can end up with up to 1Amp.

USB Powered Speakers Features:

Completely Compatible with the Raspberry Pi 100 Hz - 18 KHZ recurrence reaction

Power necessities: 5V DC 1A crest (at max volume yield) 4 Ohm impedance speakers, 3W each

Speaker Dimensions: (Height x Width x Depth) 80mm x 70mm x 70mm

Link Length: 1m

VIII. FLOWCHART

Compare sign?

Start

Yes

No

Display of Corresponding sign

Play voice of corresponding sign

Draw contours for image

Thresholding of image

Converting Blur image

Capture frames and Convert to grayscale

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 5

IX. SOFTWARE USED

This project is implemented using following software:

Open CV

Python

Raspbian OS

X. RESULTS

The images given below are various hand gestures made and corresponding American code that is generated.

Hand Gesture 1: “I am happy”

Hand Gesture 2: “Don‟t disturb me”

Hand Gesture 3: “I need your help”

Hand Gesture 4: as “please”

Hand gesture 5: “Hello” Hand Gesture 6: “Good morning”

Hand Gesture 7: “I am sorry”

Hand Gesture 8: “please can you help me with this

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 6

address”

Hand Gesture 9: “Shall we go”

Hand Gesture 10: “I need some water”

XII. CONCLUSION

In this paper a „Communication Assistance System‟ has been developed to ease communication of speech

impaired persons with normal people. An electronic speaking system has been developed which consists of 3

system framework was scene capturing, data processing, audio output. Camera as input device captures the

gestures and feed into raspberrypi3 processor. The processor identifies the gesture from its data base and

activates the corresponding recorded audio file from SD card to play to speaker .Simultaneously the

corresponding text command is displayed on the LCD. In this project ten most important gestures have been

captured and the data base for the same has been created. This device also helps in communicating between two

speech impaired persons using the texting feature.

XIII. FUTURE SCOPE

In this project the gesture of the speech impaired person is captured by the camera and the frames are

processed by the raspberry pi processor. This required faster processing speeds for enhancing the

response of the system. Therefore processors with higher speeds can be adopted.

Only ten commonly used gestures have considered for implementation in this project. The number of

gestures can be increased so as to expand the communication spectrum.

The system working is based on analyzing the captured frames where as if it extended to motion based

analysis; it can be become a more useful tool. The camera used in this project is of 5 MP, whereas the

resolution can be increased.

The audio system used in this project has low power output. The power output may be increased for

better audibility.

XV. REFERENCES

[1] A. Y. Satpute, A. D. Bhoi, and T. Designing, "Electronic Speaking System for Dumb, "VO! 6, no.3, pp.

1132-1139, 2013.

[2] S. F. Ahmed, S. Muhammad, B. Ali, S. Saqib, and M. Qureshi, "Electronic Speaking Glove for moronic

Patients a Tongue to," no. November, pp.56-60, 2010

[3] M. Wald, "Inscribing for Deaf and debilitating of Hearing people by redaction Automatic Speech

Recognition in Real Time", Proceedings of tenth International Conference on Computers serving to people with

Special wants ICCHP 2006, LNCS 4061, pp.683-690.

[4] R. R. Itkarkar and A. V. Nandi, "Hand motion to discourse change the use of Mat research facility," in 2013

Fourth International Conference on Computing, Communications and Networking Technologies (ICCCNT),

2013, pp.1-4

[5]. Jingdong Zhao, Li Jiang, Shicai Shi, Hegao Cai, Hong Liu, G. Hirzinger, "A Five-fingered underneath

spurred Prosthetic Hand System", Proceedings of the 2006 IEEE International Conference on Mechatronics and

Automation, June 2006, pp.1453-1458.

[6] S. U. N. Praveen Kumar S Havalagi, “THE magnificent DIGITAL GLOVES THAT lucid TO THE

VOICELESS," vol.6, no.1, pp.471-480, 2013.

[7] M. Sudhakar, Vandana Khare, D. Vijaya Krishna kanth, “Speech to text conversion & display using

CMR Journal of Engineering and Technology Vol.2 Issue.3 December, 2018

www.cmrjet.org 7

Raspberry Pi” in „IOSR Journal of Computer Engineering’ (IOSR-JCE) e-ISSN: 2278-0661,p-ISSN: 2278-

8727, Volume 19, Issue 4, Ver. III (Jul.-Aug. 2017), PP 14-18.

[8] M. Mahesh, M. Sudhakar, “eFACE Recognition System” in International Journal of Engineering Science &

Advanced Technology, Volume-3, Issue-4, 194-196, ISSN: 2250-3676

AUTHORS‟ BIOGRAPHY

Dr. M. Sudhakar: Graduated from JNTU College of Engineering, Hyderabad in 1979, with

specialization in ECE. He completed his M.Tech from Indian Institute of Technology

Madras in 1986 with the specialization in Instrumentation, Control & Guidance. Obtained

doctoral degree from Annamalai University. Successfully headed R&D Project assigned by

IAF on “Mathematical Modeling & Simulation of Aero Engine Control System‟ at

Aeronautical Development Establishment, Bangalore and Gas Turbine Research Establishment, Bangalore.

He is presently working as a Professor in the department of ECE and Dean (Planning & Development) at

CMR College of Engineering & Technology, Hyderabad.

B. Shruthi is a PG scholar with specialization in „Embedded System‟ in the department of

Electronics & Communications Engineering at CMR College of Engineering & Technology,

Hyderabad.