Upload
yogesh-madaan
View
132
Download
4
Embed Size (px)
Citation preview
GESTICULATION
Report submitted in partial fulfillment
Of the requirement for the degree of
B.Tech.
In
Information Technology Engineering
Under the Supervision of
Ms. Parul Yadav
By
Yogesh Madaan (0411503112)
Kanika Agarwal (05811503112)
To
Bharati Vidyapeeth’s College of Engineering
Paschim Vihar East, New Delhi
Of
Guru Gobind Singh Indraprastha University
Dwarka, New Delhi
Year 2015-2016
1
DECLARATION
This is to certify that Report entitled Gesticulation, which is submitted by us in
partial fulfillment of the requirement for the award of degree B.Tech. in Bharati
Vidyapeeth’s College of Engineering, Paschim Vihar East, New Delhi, of GGSIP
University, Dwarka, New Delhi, comprises only our original work and due
acknowledgement has been made in the text to all other material used.
DATE: NAME OF STUDENTS:
Yogesh Madaan (0411503112)
Kanika Agarwal (05811503112)
APPROVED BY Prof. (Dr.) Vanita Jain
(Head of Department, Information Technology
Bharati Vidyapeeth’s College of Engineering)
CERTIFICATE
2
This is to certify that Report entitled Gesticulation, which is submitted, by Yogesh
Madaan and Kanika Agarwal in partial fulfillment of the requirement for the award
of degree B.Tech. in Information Technology Engineering to BVCOE, Paschim Vihar
East, New Delhi, of GGSIP University, Dwarka, New Delhi is a record of the
candidates’ own work carried out by them under my supervision. The matter
embodied in this report is original and has not been submitted for the award of any
other degree.
DATE: SUPERVISOR:
Ms. Parul Yadav
3
ABSTRACT
The Internet of Things (IoT) is – at its most profound levels – about creating digital
representations of real-world objects. It is a phenomenon that draws on rapid
developments within IT to spark insights and to help companies create entirely new
types of services and business areas. Today Devices are becoming more capable and
more broadly utilized, and when they are connected to the internet, they are integrated
into vast numbers of different applications across sectors. Hence, IoT is evolving. [1]
In this project, an effort has been made to implement the above technology that is IoT,
using a hardware device called Myo Gesture Control Armband. This hardware device,
developed by Thalmic Labs, senses the motion of an arm of a human being and acts
accordingly. This project has three modules- The Windows System Module, The
Android OS Module and The Home Automation. These modules involve the
integration of the hardware device with an existing application on a windows system,
a special application development on an android OS supported mobile device, and the
operation of basic home appliances with the hardware device respectively.
4
TABLE OF CONTENTS
Title Page………………………………………………………………........................i
Declaration.....................................................................................................................ii
Certificate………………………… ……………………………………………..…...iii
Abstract……………………………………………………………………..………...iv
Table of Contents…………………………………………………..………………….v
List of Figures……………………………………………………..…………………vii
1. Introduction………………………....……………………………………..………81.1. Overview……………………………………………………………………...81.2. Existing System……. …………………………………………......................81.3. Scope of Proposed Project…………………………………………………....9
2. Pre-Requisite Technologies………………………………………………………102.1. Internet of Things……………………………………………………………102.2. Myo Gesture Control Armband……………………………………………..122.3. Bluetooth Low Energy………………………………………………………12
2.3.1. BLE Platform Support ……………………………………………….132.3.2. Technical Details……………………………………………………..14
2.4. Arduino……………………………………………………………………...152.4.1. Technical Details…………………………………………………….16
3. Myo Armband………………………………………………………………....…173.1. Design……………………………………………………………………….183.2. Setup and Calibration……………………………………………………….193.3. Performance and use………………………………………………………..203.4. Compatibility………………………………………………………………..213.5. Features……………………………………………………………………...213.6. Highs and Lows……………………………………………………………..22
4. Gesticulation…………………………………………………………………..…234.1. Idea………………………………………………………………………….234.2. Project Phases…………………………………………………………….…23
4.2.1. Phase 1……………………………………………………………….234.2.2. Phase 2……………………………………………………………….244.2.3. Phase 3……………………………………………………………….24
5. Selfie Camera and Gallery Viewer.……………………………………………..256. Smart Assistant……………………………………………………………….…277. Home Automation (Prototype)………………………………………………….28
5
8. Summary………………………………………………………………………..309. Conclusion……………………………………………………………………....3110. Future Scope…………………………………………………………………….3211. References………………………………………………………………………3312. Appendix………………………………………………………………………..34
6
LIST OF FIGURES
Fig.1 Type of Gestures of Myo…………………………..............…………………17
Fig.2 Hardware parts of Myo……………………………………………………….19
Fig.3 Flow diagram for windows’ camera application……………………………...26
Fig.4 Flow diagram for android utility application…………………………………27
Fig.5 Flow diagram for Home Automation………………………………………...29
INTRODUCTION
7
The objective of our project is to use technology for a social change. We have
tried to design a platform and set of utility tools that can be used by people
with visual impairment to help them in day to day life with a help of just
smartphones and reduce their effort. So with the vocabulary meaning of
gesticulation task performance can be accomplished with just a set of gestures.
With the initial idea of using a device to create interfaces for youngsters to
control their smart devices from a distance without physical contact with the
device, we are now building an aid to serve as a partner for blind and
physically handicapped persons.
1.1 Overview
Today, there are the inventors who work on various hardware devices with
some other hardware devices to make a new one. There are coders,
programmers, software developers, designers who are bringing new innovative
ideas and revolutionizing the world of web. There are brilliant hardware
devices, amazing software; the need of hour is to bring together the two and
kick-start a new era of IoT that is Internet of Things.
An effort has to be made to implement the above technology that is IoT. A hardware
device called MYO, developed by Thalmic Labs, can be used for the same. This
device senses the motion of an arm of a human being and acts accordingly. This
feature of MYO can be used in integration with various software platforms to develop
an innovation. These platforms may include windows, android OS and even other
hardware technology like Arduino.
1.2 Existing System
For the integration of MYO with the windows system, The Thalmic Labs has
provided the whole MYO market where the user can use any utility application
connector he/she wants to operate with the gesture of a hand. For example,
PowerPoint connector is used to switch between the slides of a presentation
with their hand gesture. The user has to download the connector from the MYO
market and add it in the application manager of for MYO, that is the MYO
Connect. The MYO connect is the windows application that acts as an interface
8
between the connector and the other windows application. The Thalmic Labs
also provide the facility for the developers. People can code their own
application connector and use for their application using the MYO Connect.
Just like for windows, The Thalmic Labs also provides application for MYO in
android for the mobile devices. Some of these applications are MYO music and
MYO dialer. It also provides developer options for android. The Myo gesture
control armband is never integrated with Arduino.
1.3 Scope of Proposed Project
In the integration of Myo Gesture control Armband with the windows system a
pre-existing application, camera is used. The Myo is integrated with the camera
so that the user do not have to touch the system to click his photograph, make a
video or view the gallery. All will be done by gesturing the arm in which Myo
is worn.
For android, a special general utility mobile application is developed in which
the user can use his selfie stick to click his photograph without setting any
timer. Also, it has a feature by which the user can feel all the notification of his
phone on his arm and can act accordingly.
An effort is made to generate a basic prototype for such integration through an
android mobile application. This has been done to solve the purpose of home
automation. Through the gesture of a hand having Myo, The electric appliances
like, a tube light, a fan, etc. of a room can be controlled.
PRE-REQUISITE TECHNOLOGIES
9
2.1 Internet of Things – IoT
The Internet of Things (IoT) is the network of physical objects or "things" embedded
with electronics, software, sensors, and network connectivity, which enables these
objects to collect and exchange data. The Internet of Things allows objects to be
sensed and controlled remotely across existing network infrastructure, creating
opportunities for more direct integration between the physical world and computer-
based systems, and resulting in improved efficiency, accuracy and economic benefit.
Each thing is uniquely identifiable through its embedded computing system but is able
to interoperate within the existing Internet infrastructure.
IoT offers advanced connectivity of devices, systems, and services that goes beyond
machine-to-machine communications (M2M) and covers a variety of protocols,
domains, and applications. The interconnection of these embedded devices (including
smart objects), is expected to usher in automation in nearly all fields, while also
enabling advanced applications like a Smart Grid, and expanding to the areas such as
smart cities.
"Things," in the IoT sense, can refer to a wide variety of devices such as heart
monitoring implants, biochip transponders on farm animals, electric clams in coastal
waters, automobiles with built-in sensors, or field operation devices that assist
firefighters in search and rescue operations. These devices collect useful data with the
help of various existing technologies and then autonomously flow the data between
other devices. Current market examples include smart thermostat systems and
washer/dryers that use Wi-Fi for remote monitoring.
Besides the plethora of new application areas for Internet connected automation to
expand into, IoT is also expected to generate large amounts of data from diverse
locations that is aggregated very quickly, thereby increasing the need to better index,
store and process such data.
Survey conducted by Mckinsey & Company in June 2015 To get a broader view of
the IoT’s potential benefits estimates that the IoT has a total potential economic
impact of $3.9 trillion to $11.1 trillion a year by 2025. At the top end, that level of
10
value—including the consumer surplus—would be equivalent to about 11 percent of
the world economy (exhibit).[1]
Achieving this kind of impact would require certain conditions to be in place, notably
overcoming the technical, organizational, and regulatory hurdles. In particular,
companies that use IoT technology will play a critical role in developing the right
systems and processes to maximize its value. Among our findings:
Interoperability between IoT systems is critical. Of the total potential
economic value the IoT enables, interoperability is required for 40 percent on
average and for nearly 60 percent in some settings.
Currently, most IoT data are not used. For example, on an oilrig that has
30,000 sensors, only 1 percent of the data are examined. That is because this
information is used mostly to detect and control anomalies—not for
optimization and prediction, which provide the greatest value.
Business-to-business applications will probably capture more value—nearly
70 percent of it—than consumer uses, although consumer applications, such as
fitness monitors and self-driving cars, attract the most attention and can create
significant value, too.
The IoT has a large potential in developing economies. Still, we estimate that
it will have a higher overall value impact in advanced economies because of
the higher value per use. However, developing economies could generate
nearly 40 percent of the IoT’s value, and nearly half in some settings.
Customers will capture most of the benefits. We estimate that IoT users
(businesses, other organizations, and consumers) could capture 90 percent of
the value that IoT applications generate. For example, in 2025 remote
monitoring could create as much as $1.1 trillion a year in value by improving
the health of chronic-disease patients.
11
A dynamic industry is evolving around IoT technology. As in other
technology waves, both incumbents and new players have opportunities.
Digitization blurs the lines between technology companies and other types of
businesses; makers of industrial machinery, for example, are creating new
business models by using IoT links and data to offer their products as a
service.[1]
2.2 Myo Gesture Control Armband
Myo Gesture Control Armband, launched in June 2013, is a wearable technology
developed by a Canada based start-up named Thalmic Labs. Using a technique known
as electromyography, this little armband is able to read electrical signals from the
muscles in the forearm, map them to gestures made with the hand, and control other
devices with those gestures. The Myo armband gives a touch-free control of
technology with hand gestures and motion. It aims to give a motion control without
the need for a Wii remote, a presentation controller, or anything else that has to be
held in hand. Instead, the armband just has to be slipped like a blood pressure sleeve
(or a Nintendo Power Glove) and things can be controlled with the movements of
fingers and wrist. Most gesture-control systems, like Microsoft’s Kinect, still detect
movements with cameras, which can be thrown off by poor lighting conditions,
distance, and simple obstructions. By drawing gesture information directly from the
arm muscles instead of a camera, Myo circumvents all these problems — and works
with devices that do not have a camera in the first place. It detects five distinct hand
gestures to wirelessly control tech, like music, games, and presentation slides, all with
the flick of a wrist.
Myo wraps it in eight different blocks, each of which contains a medical-grade EMG
sensor. The armband also uses a three-axis gyroscope, three-axis accelerometer, and
three-axis magnetometer to sense motion in any direction.
2.3 Bluetooth Low Energy
Bluetooth Low Energy (BLE), sometimes referred to as "Bluetooth Smart", is a light-
weight subset of classic Bluetooth and was introduced as part of the Bluetooth 4.0
12
core specification. While there is some overlap with classic Bluetooth, BLE actually
has a completely different lineage and was started by Nokia as an in-house project
called 'Wibree' before being adopted by the Bluetooth SIG.
There are plenty of wireless protocols out there for engineers and product designers,
but what makes BLE so interesting is that it's almost certainly the easiest way to
design something that can talk to any modern mobile platform out there (iOS,
Android, Windows phones, etc.), and particularly in the case of Apple devices it's the
only HW design option that doesn't require you to jump through endless hoops to be
able to legally market your product for iOS devices.
This guide will give you a quick overview of BLE, specifically how data is organized
in Bluetooth Low Energy, and how devices advertise their presence so that you can
connect to them and start passing data back and forth.
2.3.1 BLE Platform Support
Support for Bluetooth 4.0 and Bluetooth Low Energy (which is a subset of BT 4.0) is
available on most major platforms as of the versions listed below:
iOS5+ (iOS7+ preferred)
Android 4.3+ (numerous bug fixes in 4.4+)
Apple OS X 10.6+
Windows 8 (XP, Vista and 7 only support Bluetooth 2.1)
GNU/Linux Vanilla BlueZ 4.93+
Bluetooth Smart extends the use of Bluetooth wireless technology to devices that are
powered by small, coin-cell batteries such as watches and toys. Other devices such as
sports & fitness, health care, keyboards and mice, beacons, wearables and
entertainment devices are enhanced by this version of the technology. In many cases,
it makes it possible to operate these devices for more than a year without recharging.
As with previous versions of the specification, the range of the radio may be
optimized according to application. The majority of Bluetooth devices on the market
today include the basic 30 foot, or 10 meter, range of the Classic Bluetooth radio, but
there is no limit imposed by the Specification. With Bluetooth Smart, manufacturers
13
may choose to optimize range to 200 feet and beyond, particularly for in-home sensor
applications where longer range is a necessity.
Bluetooth Smart features provides:
Ultra-low peak, average and idle mode power consumption
Ability to run for years on standard coin-cell batteries
Lower implementation costs
Multi-vendor interoperability
Enhanced range
This enhancement to the Bluetooth Core Specification allows two types of
implementation, dual-mode and single-mode. In a dual-mode implementation,
Bluetooth low energy functionality is integrated into an existing Classic Bluetooth
controller. The resulting architecture shares much of Classic Bluetooth technology's
existing radio and functionality resulting in a minimal cost increase compared to
Classic Bluetooth technology. Additionally, manufacturers can use current Classic
Bluetooth technology (Bluetooth v2.1 + EDR or Bluetooth v3.0 + HS) chips with the
new low energy stack, enhancing the development of Classic Bluetooth enabled
devices with new capabilities.
Single-mode chips, which will enable highly integrated and compact devices, will
feature a lightweight Link Layer providing ultra-low power idle mode operation,
simple device discovery, and reliable point-to-multipoint data transfer with advanced
power-save and secure encrypted connections at the lowest possible cost. The Link
Layer in these controllers will enable Internet connected sensors to schedule
Bluetooth low energy traffic between Bluetooth transmissions.
2.3.2 Technical Details
Data Transfers – Bluetooth Smart (low energy) supports very short data
packets (8 octet minimum up to 27 octets maximum) that are transferred at 1
Mbps. All connections use advanced sniff-sub rating to achieve ultra low duty
cycles
Frequency Hopping – Bluetooth Smart (low energy) uses the adaptive
frequency hopping common to all versions of Bluetooth technology to
14
minimize interference from other technologies in the 2.4 GHz ISM Band.
Efficient multi-path benefits increase the link budgets and range
Host Control – Bluetooth Smart (low energy) places a significant amount of
intelligence in the controller, which allows the host to sleep for longer periods
of time and be woken up by the controller only when the host needs to
perform some action. This allows for the greatest current savings since the
host is assumed to consume more power than the controller
Latency - Bluetooth Smart (low energy) can support connection setup and data
transfer as low as 3ms, allowing an application to form a connection and then
transfer authenticated data in few milliseconds for a short communication
burst before quickly tearing down the connection
Range – Increased modulation index provides a possible range for Bluetooth
Smart (low energy) of over 100 meters
Robustness – Bluetooth Smart (low energy) uses a strong 24 bit CRC on all
packets ensuring the maximum robustness against interference
Strong Security – Full AES-128 encryption using CCM to provide strong
encryption and authentication of data packets
Topology – Bluetooth Smart (low energy) uses a 32 bit access address on
every packet for each slave, allowing billions of devices to be connected. The
technology is optimized for one-to-one connections while allowing one-to-
many connections using a star topology. [4]
2.4 Arduino
The Micro is a microcontroller board based on the ATmega32U4, developed in
conjunction with Adafruit. It has 20 digital input/output pins (of which 7 can be used
as PWM outputs and 12 as analog inputs), a 16 MHz crystal oscillator, a micro USB
connection, an ICSP header, and a reset button. It contains everything needed to
support the microcontroller; simply connect it to a computer with a micro USB cable
to get started. It has a form factor that enables it to be easily placed on a breadboard.
The Micro board is similar to the Arduino Leonardo in that the ATmega32U4 has
built-in USB communication, eliminating the need for a secondary processor. This
allows the Micro to appear to a connected computer as a mouse and keyboard, in
15
addition to a virtual (CDC) serial / COM port. It also has other implications for the
behavior of the board; these are detailed on the getting started page.
2.4.1 Technical specs
Microcontroller ATmega32U4
Operating Voltage 5V
Input Voltage (recommended) 7-12V
Input Voltage (limit) 6-20V
Digital I/O Pins 20
PWM Channels 7
Analog Input Channels 12
DC Current per I/O Pin 20 mA
DC Current for 3.3V Pin 50 mA
Flash Memory 32 KB (ATmega32U4)of which 4 KB used by bootloader
SRAM 2.5 KB (ATmega32U4)
EEPROM 1 KB (ATmega32U4)
Clock Speed 16 MHz
Length 48 mm
Width 18 mm
Weight 13 g
16
MYO ARMBAND
Myo Gesture Control Armband, launched in June 2013, is a wearable technology
developed by a Canada based start-up named Thalmic Labs. Using a technique known
as electromyography, this little armband is able to read electrical signals from the
muscles in the forearm, map them to gestures made with the hand, and control other
devices with those gestures. The Myo armband gives a touch-free control of
technology with hand gestures and motion. It aims to give a motion control without
the need for a Wii remote, a presentation controller, or anything else that has to be
held in hand. Instead, the armband just has to be slipped like a blood pressure sleeve
(or a Nintendo Power Glove) and things can be controlled with the movements of
fingers and wrist. Most gesture-control systems, like Microsoft’s Kinect, still detect
movements with cameras, which can be thrown off by poor lighting conditions,
distance, and simple obstructions. By drawing gesture information directly from the
arm muscles instead of a camera, Myo circumvents all these problems — and works
with devices that do not have a camera in the first place. It detects five distinct hand
gestures to wirelessly control tech, like music, games, and presentation slides, all with
the flick of a wrist.
Fig. 1
17
Myo wraps it in eight different blocks, each of which contains a medical-grade EMG
sensor. The armband also uses a three-axis gyroscope, three-axis accelerometer, and
three-axis magnetometer to sense motion in any direction.
Muscle activity and motion readings are handled by an onboard ARM Cortex M4
Processor, which communicates with your devices via Bluetooth. For devices that do
not have Bluetooth functionality built in, Myo also comes with a Bluetooth dongle
that plugs into any USB port.
Details on battery size are not readily available, but Thalmic claims the lithium-ion
cell inside is big enough to keep Myo going for a full day of continued use.
One can also map gestures to key strokes for customized control, or dig into the open
API and free, easy-to-use SDK to create scripts and applications that put technology
at the fingertips. The device already has a sizable library of apps and plugins to offer.
So far, the Armband has been used to serve the purpose PowerPoint
presentations and media playback for apps like Netflix, Windows Media Player,
and iTunes. [2]
3.1Design
Myo has undergone some major design changes since it was first conceived, but the
final version is about as polished as you can hope for. It is relatively slim and sleek
considering how many sensors are stuffed inside its hull, and despite being a
somewhat geeky piece of gear; it does have an undeniably badass look to it.
The band itself is a series of plastic, rectangular "pods," held together by a
perimeter of rubber lining. On the underside of each pod there are three
medical grade stainless steel EMG sensors that detect the electric impulses in
the arm's muscles. The design allows the Armband to fit a multitude of
differently sized forearms.
18
Fig. 2
One of the pods sports two indicator LEDs. One is simply for the Myo logo,
while the other, rectangular light turns green to indicate the band is charging,
and light blue when it is on and connected.
As for the cold, hard measurements, the Myo comes in at 7.5 inches for
circumference, expandable to 13 inches. It weighs only 3.3 ounces, and each
individual pod has a thickness of 0.45 inches. It is available in black or white.
[3]
3.2 Setup and calibration
Setting up Myo is a strange and unfamiliar process, but despite being unlike you’ve
ever used before, getting the armband up and running is pretty straightforward.
Once you’ve got it all charged up (via the included micro USB cable), all that’s left to
do is establish a Bluetooth connection with your computer and run through the
calibration process. Thalmic has put together a series of short videos to guide you
through the process, which make it a breeze to get going.
19
After you slip Myo onto your arm, you’ll need to wait for a couple minutes for the
sensors to warm up. The setup video explains that during this time, Myo is forming a
strong electrical connection with the muscles in your forearm. After this bond is
established, Myo vibrates to let you know it’s ready for sync.
To sync Myo and start using it, you need to perform Thalmic’s special “sync gesture,”
which involves holding your arm parallel to the ground, rotating your wrist outward,
and then swinging your forearm out 90 degrees like it’s on a hinge.
If this gesture works and Myo vibrates again, you’re off to the races.
3.3 Performance and use
Simply put, Myo’s gesture-recognition capabilities are everything they’re chalked up
to be — so long as you have the sensors calibrated properly. That’s the trick. It might
take a couple tries to get everything perfect, but once you do, Myo works like a
dream.
The first thing we tried out was Thalmic’s presentation demo, which allows you to
navigate through slideshows using gesture controls like tapping fingers together,
waving your hand right or left, and opening and closing your fist. At first we had
some trouble getting finger taps to register unless they were a bit exaggerated, but
after a quick recalibration, Myo recognized all our motions instantaneously, even if
the taps were light and quick.
After that, we tried it out on iTunes. It was awesome. Wave left and right to skip
tracks, play and pause by spreading your fingers, and roll your fist to crank up the
volume. Even when we were across the room, the gesture commands were recognized
on the first try every time, and their corresponding action was executed without any
sort of delay. So long as you’re within Bluetooth range of whatever you’re
controlling, Myo is extremely responsive.
If there’s a downside to all this, it’s probably the fact that, in its current state, Myo can
only recognize five different gestures and two types of motion. It’s not quite as
dynamic as we had initially thought, and can only read finger taps, left or right waves,
opening your hand, making a fist, and any up, down, left, right, and roll movements.
20
The sensors are likely capable of more –like, say, a “trigger pull” motion you make
with your index finger, or pointing with two fingers instead of one– but right now it’s
limited to the aforementioned gestures and motions. That’s not to say Myo feels
incomplete or incapable in any way, though. You can do a lot with the available
commands — we just expected more gestures to be available out of the box.
3.4 Compatibility
Flipping through your iTunes playlists and Keynote presentations is just the
beginning. Thalmic put Myo in the hands of developers long before the general
release, so now that it’s officially launched, the device already has a sizable library of
apps and plugins to offer.
In the Myo Marketplace, you can download integrations for all kinds of apps, games,
and utilities — Netflix, Spotify, YouTube, Minecraft, Civilization V, Kerbal Space
Program, and tons more. There’s even one that allows you to control a Parrot AR
drone like they showed in the pitch video two years ago.
Thalmic doesn’t really place any limits on how Myo can be used or what it can be
used for, so developers have basically run wild with it – which is awesome, but you’ll
definitely find that some of the integrations are much more practical than others.
One of the first applications I downloaded after setup was the Mac Desktop Utility,
which allows you to do things like switch between desktops, take a screenshot, or
show OS X’s Launchpad by using gestures. Cool, no doubt, but when you’ve already
got a keyboard and a trackpad at your fingertips, it’s pretty unnecessary to throw
gesture control into the mix.
For other things, however, Myo makes a whole lot of sense. Linked up to media
players like iTunes or Spotify, it functions as a sort of wearable hands-free controller.
The other night I wore it while I cooked dinner and didn’t have to whip out my phone
to skip tracks on Spotify – I could just wave my hand in the air over the stove and
continue sautéing my onions.
21
3.5 Features
Along with the Armband, the whole package is also provided with a standard
micro USB cable for charging and a USB Bluetooth adapter for pairing the
device with the computer. It also comes with ten sizing clips that can be used to
readjust the band if it doesn't fit the arm properly.
The Armband uses a three-axis accelerometer, a three-axis magnetometer, and a
three-axis gyroscope to sense motion. An onboard ARM Cortex M4 processor
talks with any connected devices via Bluetooth. The band is compatible with
PCs running Windows 7, Windows 8, and Mac OS X 10.8 and above. It also
works with Bluetooth 4.0-equipped mobile devices running iOS 7.0 and higher,
as well as Android 4.3 and above.
According to Thalmic Labs, a built-in rechargeable lithium-ion battery can
keep the Myo going for one full day on a single charge.
3.6 Highs and Lows of the Device
Highs
Easy setup
Expandable design
Broad compatibility
Super fast and responsive
Growing library of apps/integrations
Lows
Limited range of gestures
Perfect calibration takes a few tries[3]
22
GESTICULATION
4.1 Idea
Gesticulation is a project in which an effort has been made to implement IoT using a
hardware device called Myo Gesture Control Armband. This hardware device,
developed by Thalmic Labs, senses the motion of an arm of a human being and acts
accordingly. This project has three modules- The Windows System Module, The
Android OS Module and The Home Automation Module. These modules involve the
integration of the hardware device with an existing application on a windows system,
a special application development on an android OS supported mobile device, and the
operation of basic home appliances with the hardware device respectively.
With the initial idea of using a device to create interfaces for youngsters to control
their smart devices from a distance without physical contact with the device we are
now building an aid to serve as a partner for blind and physically handicapped
persons.
The idea is to use technology for a social change. We have tried to design a platform
and set of utility tools that can be used by people with visual impairment to help them
in day to day life with a help of their smartphones and this gesture control hardware
device, MYO and reduce their effort. So now various tasks of everyday life can be
accomplished with just a set of a few hand gestures.
4.2 Project Phases
4.2.1 Phase 1
The idea is very simple and unique with try to do ordinary task in a creative and smart
way using technology and gadgets. With the help of utility camera a gallery can be
smartly controlled from you hands from a distance apart removing the difficulty of
switching images/ slides during a slide show. The mobile version of the application is
targeting the youngster that are normally seen capturing a selfie using a stick which
can now only be completed by putting a phone at a distance and opening your fist.
23
4.2.2 Phase 2
The smart application is always running with you establishing a smart and battery
efficient Bluetooth connection with you smart device to assist you immediately. The
application gets triggered whenever some receiver is triggered.. The functionalities of
the application includes
• Handle various receivers of the mobile (Call Receiver, Message receiver)
• Accept and reject calls from a distance
• Set phone to speaker mode
• Read out incoming text messages loudly
• Handle notifications
4.2.3 Phase 3
Continuing with the most basic and commonly used IOT idea of making your home
smart we have designed a prototype to help unpaired people in controlling their home
appliances remotely with their virtual presence. The is just a prototype and we are
working on the research for this is order to create a stable product. The idea is to
create switch boards that have integrated wifi modules in them that will be handled by
a home server .The home server in return will be connected with various clients
(Android phone in our case) . The armband will be in continuous connection with the
app and will interact with the switch relay to control the appliances.
24
SELFIE CAMERA AND GALLERY VIEWER
The idea is very simple and unique with an effort made to be able to do
ordinary task in a creative and smart way using technology and gadgets. With
the help of this module, utility camera and gallery can be smartly controlled by
your hands from a distance apart removing the difficulty of switching images/
slides during a slide show. The mobile version of the application is targeting
the youngster that are normally seen capturing a selfie using a stick, which can
now be completed by putting a phone at a distance and opening your fist.
In the windows version of this module the effort is made to allow the user to
use the camera of the system without actually touching the system. All they
have to do is look into the camera, smile, twist the hand a little and the photo is
clicked. Through this tool the user can also make a video, can look into the
gallery of all his photos, play a video, increase and decrease the brightness of
the image before clicking and much more.
The module is providing support for two different type of platforms namely
Windows operating system and Android operating system. In windows system
default application “Camera.exe” and “Photos.exe” will be used and we have
provided MYO support by developing a connector which runs in background
under the surveillance of Myo Connect which handles the direct connection
with the MYO armband and manages its different connections states and
different pose actions. The connector gets notified about the change of states of
armband and acts accordingly. With mapping of various keyboard keys and
mouse controls the application is controlled with the hand gestures.
The working in android application is slightly different in case of mobile
application where direct connection is established between the mobile
application and MYO armband. Thalmic labs have provided the OS SDK to
25
interact directly with the armband. In order to have an efficient performance in
terms of battery BLE(Bluetooth low energy ) connection has been established
rather than the classical Bluetooth connection between the armband and the
application.
Fig. 3
26
SMART ASSISTANT
This module is based only on Android platform.
The smart application is always running with you establishing a smart and
battery efficient Bluetooth connection with your smart device to assist you
immediately. The application gets triggered whenever some receiver is
triggered. The functionalities of the application include
• Handle various receivers of the mobile (Call Receiver, Message
receiver)
• Accept and reject calls from a distance
• Set phone to speaker mode
• Read out incoming text messages loudly
• Handle notifications
Moving on the technical specification of the application it has support same as
BLE for android OS so it will be supported for Android OS version greater than
4.3 thus targeting the 76% of total android users.
Apart from call and message alerts the application will handle all the third
party notifications like WhatsApp messages and Hangout notifications. We
have designed the different broadcast receivers for different tasks. A
background service is designed which starts running automatically as soon as
phone boots up and tries to connect to the nearest armband.
27
Fig. 4
HOME AUTOMATION (PROTOTYPE)
Home automation is the use of one or more computers to control basic home functions
and features automatically and sometimes remotely. An automated home is
sometimes called a smart home .
Home automation can include the scheduling and automatic operation of water
sprinkling, heating and air conditioning, window coverings, security systems, lighting,
and food preparation appliances. Home automation may also allow vital home
functions to be controlled remotely from anywhere in the world using a computer
connected to the Internet. Besides the functions already mentioned, remote control
can be extended to telephones and answering machines, fax machines, amateur radios
and other communications equipment, and home robot s such as automatic vacuum
cleaners.
Continuing with the most basic and commonly used IOT idea of making your
home smart, we have designed a prototype to help impaired people in
controlling their home appliances remotely with their virtual presence. The is
just a prototype and we are working on the research for this in order to create a
stable product. The idea is to create switchboards that have integrated Wi-Fi
modules in them that will be handled by a home server. The home server in
return will be connected with various clients (Android phone in our case). The
28
armband will be in continuous connection with the app and will interact with
the switch relay to control the appliances.
Fig. 5
29
SUMMARY
Today, there are the inventors who work on various hardware devices with
some other hardware devices to make a new one. There are coders,
programmers, software developers, designers who are bringing new innovative
ideas and revolutionizing the world of web. There are brilliant hardware
devices, amazing software; the need of hour is to bring together the two and
kick-start a new era of IoT that is Internet of Things.
In this project, a hardware device called Myo Gesture control Armband is
integrated with a pre-existing application on the windows system, a special
application for android mobile devices, and with yet another technology called
Arduino.
In the integration of Myo Gesture control Armband with the windows system a
pre-existing application, camera is used. The Myo is integrated with the camera
so that the user do not have to touch the system to click his photograph, make a
video or view the gallery. All will be done by gesturing the arm in which Myo
is worn.
For android, a special general utility mobile application is developed in which
the user can use his selfie stick to click his photograph without setting any
timer. Also, it has a feature by which the user can feel all the notification of his
phone on his arm and can act accordingly.
In the final module, the Myo gesture control armband is integrated with
Arduino through an android mobile application. This has been done to solve the
purpose of home automation. Through the gesture of a hand having Myo, The
electric appliances like, a tube light, a fan, etc. of a room can be controlled.
30
CONCLUSION
Through this report it is concluded that the hardware device Myo Gesture
Control Armband has been studied properly. It has been integrated with a
windows system, an android mobile device and with Arduino in order to
achieve following targets
Control the predefined applications for Windows application with
keyboard mapping of MYO.
Handle calls and read phone notification with a gesture of MYO.
Control the various home appliances using the armband.
On an overall review MYO has turned to be a great device with so much
potential to exist in the market but still it has some shortcomings which is
needed to be overcome for better performance.
31
FUTURE SCOPE
• Home Automation
• Gaming
• Blind Teaching
• E-Learning
• Motion Detection
32
REFRENCES
[1] Dr. Ovidiu Vermesan and Dr. Peter Friess , Internet of Things: Converging
Technologies for Smart Environments and Integrated Ecosystems. River Publishers
2013, pp. 20-100
[2] Mithileysh Sathiyanarayanan, Tobias Mulling and Bushra Nazir, Controlling a
Robot Using a Wearable Device (MYO), School of Computing, Engineering and
Mathematics, UK, 2015, pp. 1-3
[3] Drew Prindle, Myo Gesture Control Armband Review, Digital Trends, 2015
[4] Josie Hughes, Jize Yan and Kenichi Soga, Development of Wireless Sensor
Network using Bluetooth Low Energy (BLE) for Construction Noise Monitoring,
June 15, pp.1394-1400
33
APPENDIX
9.1 Lua Script
scriptId = 'kanika.myoscript.camera'
scriptTitle = "Camera Connector"
scriptDetailsUrl = ""
temp=0
function onForegroundWindowChange(app, title)
myo.debug("onForegroundWindowChange: " .. app .. ", " .. title)
if(title=="Camera") then
temp =0
end
34
if(title=="Photos") then
temp=1
end
return platform == "Windows" and (app == "ApplicationFrameHost.exe" or
title == "Camera")
end
function activeAppName()
return "Camera"
end
function performAction()
myo.keyboard("space", "press")
end
function switchTab()
myo.keyboard("tab", "press")
end
function cancelFocus()
myo.keyboard("escape","press")
end
function switchOptions()
myo.keyboard("up_arrow","press")
end
35
function nextPhoto()
myo.keyboard("right_arrow","press")
end
function previousPhoto()
myo.keyboard("left_arrow","press")
end
function closePhotos()
myo.keyboard("alt+f4","press")
end
function conditionallySwapWave(pose)
if myo.getArm() == "left" then
if pose == "waveIn" then
pose = "waveOut"
elseif pose == "waveOut" then
pose = "waveIn"
end
end
return pose
end
36
function onPoseEdge(pose, edge)
myo.debug("onPoseEdge: " .. pose .. ", " .. edge)
if(temp==0) then
if (pose == "fist") then
if (edge == "on") then
performAction()
myo.setLockingPolicy("standard")
end
if edge == "off" then
myo.unlock("timed")
end
end
if (pose == "waveOut") then
if (edge == "on") then
switchTab()
myo.setLockingPolicy("none")
end
if edge == "off" then
myo.unlock("timed")
end
37
end
if (pose == "waveIn") then
if (edge == "on") then
switchOptions()
myo.setLockingPolicy("none")
end
if edge == "off" then
myo.unlock("timed")
end
end
if (pose == "fingersSpread") then
if (edge == "on") then
cancelFocus()
myo.setLockingPolicy("standard")
end
if edge == "off" then
myo.unlock("timed")
end
end
end
38
if(temp==1) then
if (pose == "fist") then
if (edge == "on") then
performAction()
myo.setLockingPolicy("standard")
end
if edge == "off" then
myo.unlock("timed")
end
end
if (pose == "waveOut") then
if (edge == "on") then
nextPhoto();
myo.setLockingPolicy("none")
end
if edge == "off" then
myo.unlock("timed")
end
end
39
if (pose == "waveIn") then
if (edge == "on") then
previousPhoto();
myo.setLockingPolicy("none")
end
if edge == "off" then
myo.unlock("timed")
end
end
if (pose == "fingersSpread") then
if (edge == "on") then
closePhotos()
myo.setLockingPolicy("standard")
end
if edge == "off" then
myo.unlock("timed")
end
end
end
end
40
9.2 Code snippet for background services that handle various gestures
private DeviceListener mListener = new AbstractDeviceListener() {
@Override
public void onConnect(Myo myo, long timestamp) {
Log.e("myo
connected",""+Hub.getInstance().getConnectedDevices().get(0).getName());
eventBus.post(new String("Connected"));
}
@Override
public void onDisconnect(Myo myo, long timestamp) {
eventBus.post(new String("Disconnected"));
}
// onPose() is called whenever the Myo detects that the person wearing it has
changed their pose, for example,
// making a fist, or not making a fist anymore.
@Override
public void onPose(Myo myo, long timestamp, Pose pose) {
// Show the name of the pose in a toast.
switch (pose) {
case UNKNOWN:
break;
case REST:
case DOUBLE_TAP:
switch (myo.getArm()) {
case LEFT:
break;
case RIGHT:
break;
41
}
break;
case FIST:
break;
case WAVE_IN:
break;
case WAVE_OUT:
break;
case FINGERS_SPREAD:
break;
}
if (pose != Pose.UNKNOWN && pose != Pose.REST) {
// Tell the Myo to stay unlocked until told otherwise. We do that here so you
can
// hold the poses without the Myo becoming locked.
myo.unlock(Myo.UnlockType.HOLD);
// Notify the Myo that the pose has resulted in an action, in this case changing
// the text on the screen. The Myo will vibrate.
myo.notifyUserAction();
} else {
// Tell the Myo to stay unlocked only for a short period. This allows the Myo
to
// stay unlocked while poses are being performed, but lock after inactivity.
myo.unlock(Myo.UnlockType.TIMED);
}
EventBus eventBus = EventBus.getDefault();
eventBus.post(pose);
Log.e("posting event",""+pose);
}
@Override
public void onArmSync(Myo myo, long timestamp, Arm arm, XDirection
42
xDirection) {
super.onArmSync(myo, timestamp, arm, xDirection);
eventBus.post(new String("Synced"));
}
@Override
public void onArmUnsync(Myo myo, long timestamp) {
super.onArmUnsync(myo, timestamp);
eventBus.post(new String("Unsynced"));
}
@Override
public void onUnlock(Myo myo, long timestamp) {
super.onUnlock(myo, timestamp);
eventBus.post(new String("Unlocked"));
}
@Override
public void onLock(Myo myo, long timestamp) {
super.onLock(myo, timestamp);
eventBus.post(new String("Locked"));
}
};
9.3 Code snippet for call receiver
public class IncomingCallReceiver extends BroadcastReceiver {
Context context;
43
private int mInterval = 4000; // 5 seconds by default, can be changed later
private Handler mHandler;
private AudioManager myAudioManager;
public int mod;
public static boolean isRinging = false;
public static Intent intent ;
@Override
public void onReceive(Context context, Intent intent){
try{
String state = intent.getStringExtra(TelephonyManager.EXTRA_STATE);
mHandler =new Handler();
myAudioManager =
(AudioManager)context.getSystemService(Context.AUDIO_SERVICE);
mod=myAudioManager.getRingerMode();
this.intent = intent;
if(state.equals(TelephonyManager.EXTRA_STATE_RINGING)){
Toast.makeText(context, "Phone Is Ringing",
Toast.LENGTH_LONG).show();
startRepeatingTask();
myAudioManager.setRingerMode(AudioManager.RINGER_MODE_SILENT);
isRinging = true;
Log.e("ringing","true");
}
if(state.equals(TelephonyManager.EXTRA_STATE_OFFHOOK)){
Toast.makeText(context, "Call Recieved", Toast.LENGTH_LONG).show();
stopRepeatingTask();
myAudioManager.setRingerMode(AudioManager.RINGER_MODE_VIBRATE);
isRinging = false;
Log.e("ringin","false");
}
44
if (state.equals(TelephonyManager.EXTRA_STATE_IDLE)){
Toast.makeText(context, "Phone Is Idle", Toast.LENGTH_LONG).show();
stopRepeatingTask();
myAudioManager.setRingerMode(AudioManager.RINGER_MODE_VIBRATE);
isRinging = false;
Log.e("ringin","false");
}
}
catch(Exception e){e.printStackTrace();}
}
Runnable mStatusChecker = new Runnable() {
@Override
public void run() {
if(Hub.getInstance().getConnectedDevices().size()>0)
{
if(isRinging){
Hub.getInstance().getConnectedDevices().get(0).vibrate(Myo.VibrationType.LONG);
Log.e("vibration","sent");}
}
mHandler.postDelayed(mStatusChecker, mInterval);
}
};
void startRepeatingTask() {
mStatusChecker.run();
}
void stopRepeatingTask() {
mHandler.removeCallbacks(mStatusChecker);
45
}
}
9.4 Code Snippet for usb serial connection
private SerialInputOutputManager mSerialIoManager;
private final SerialInputOutputManager.Listener mListener =
new SerialInputOutputManager.Listener() {
@Override
public void onRunError(Exception e) {
Log.d(TAG, "Runner stopped.");
}
@Override
public void onNewData(final byte[] data) {
SerialConsoleActivity.this.runOnUiThread(new Runnable() {
@Override
public void run() {
SerialConsoleActivity.this.updateReceivedData(data);
}
});
}
};
@Override
protected void onResume() {
super.onResume();
Log.d(TAG, "Resumed, port=" + sPort);
if (sPort == null) {
mTitleTextView.setText("No serial device.");
} else {
46
final UsbManager usbManager = (UsbManager)
getSystemService(Context.USB_SERVICE);
UsbDeviceConnection connection =
usbManager.openDevice(sPort.getDriver().getDevice());
if (connection == null) {
mTitleTextView.setText("Opening device failed");
return;
}
try {
sPort.open(connection);
sPort.setParameters(115200, 8, UsbSerialPort.STOPBITS_1,
UsbSerialPort.PARITY_NONE);
} catch (IOException e) {
Log.e(TAG, "Error setting up device: " + e.getMessage(), e);
mTitleTextView.setText("Error opening device: " + e.getMessage());
try {
sPort.close();
} catch (IOException e2) {
// Ignore.
}
sPort = null;
return;
}
mTitleTextView.setText("Serial device: " +
sPort.getClass().getSimpleName());
}
onDeviceStateChange();
}
47
48