BLUE EYES - A REVOLUTION

S. Salini Jeba Angel B.E.-ECE/Final year

Anand Institute of Higher Technology E-mail: salinijebaangel@gmail.com

ABSTRACT

Blue eyes technology aims at creating a computational machine having perceptual abilities and sensory abilities as humans through speech recognition and facial recognition system. It has the ability to recognize one’s mood, sense or perceive the user action. It can identify, feel the presence and start interacting with the user. It can even understand one’s emotions at the touch of the mouse. Human error can be eliminated with this technology in the field of security & controlling, where the contribution of human operator is required the whole time, as the continuous work may tire the operator. This blue eye records the user conscious brain involvement as well his physiological condition. As the operator works, the supervising system will look after his physiological condition too. In total, blue eyes technology is a revolution in the field of computers where the computers can recognize the physical, emotional and physiological state of the user. INTRODUCTION:

The blue eyes technology is a technical means of monitoring, recording the human operator’s state. The time is not far when one’s computer can talk, listen, interact or even display more human traits such as pacifying the temper of the user, realizing the urgency of the user and establishing a connection by dialing to a friend etc.,. This is exactly the research of a team of IBM at its Almaden Research center (ARC) in San Jose, California since 1997. Blue eyes are also being developed by the team of Poznan University of Technology& Microsoft. It makes use of the “blue tooth technology” developed by Ericsson. The key features of the system are:

• Visual attention monitoring (eye motility analysis)

• Physiological condition monitoring (pulse rate, blood oxygenation)

• Operator’s position detection (standing, lying)

• Wireless data acquisition using Bluetooth technology

• Real-time user-defined alarm triggering

• Physiological data, operator's voice and overall view of the control room recording

• Recorded data playback

What’s a blue eyes technology?

Blue Eyes uses sensing technology is to identify a user's actions and to extract key information. This information is then analyzed to determine the user's physical, emotional, which in turn can be used to help make the user more productive by performing expected actions or by providing, expected information. Human cognition depends primarily on the ability to perceive, interpret, and integrate audio-visuals information. A Blue Eyes-enabled television could become active when the user makes eye contact, at point the user could then tell the television to turn on. If one walks by the computer screen, for example, the camera would immediately sense the presence and automatically turn on room lights, the television, or radio while popping up your favorite Internet website on the display. The computers know to sense or perceive user action. They are also being programmed to know how users feel- depressed, ecstatic, bored, amused, or anxious and make a corresponding response. Computers, can on their own, play a funny flash animation feature to entertain the user if it notices a sad look on the face. Voice or sound capabilities can also be integrated, with the computer talking to his user about the task at hand or simply acknowledging a command with a respectful, "yes, sir."In these cases, the computer extracts key information, such as where the user is looking, what he or she is saying or gesturing or how

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201649

the subject’s emotions are with the grip on the pointing device.

Blue eyes system monitors the status of the operator’s visual attention through measurement of saccadic activity. Complex industrial environment can create a danger of exposing the operator to toxic substances, which can affect his cardiac, circulatory and pulmonary systems. Thus, on the grounds of plethysmographic signal taken from the forehead skin surface, the system computes heart beat rate and blood oxygenation. The system checks parameters like heart beat rate and blood oxygenation against abnormal and triggers user defined alarms. Thus it proves to be of additional benefit to the user since even the health of the user is monitored in addition to the physical and mental states of the user. Adding extraordinary perceptual abilities to computers would enable computers to work together with human beings and create a revolution. Etymology

The term ‘blue’ indicates the blue tooth technology used to provide efficient communication between the computer and the user. For an effective communication between the computer and the user, wiring is required. But this is a serious limitation of the user’s mobility and hence the use of wireless communication becomes essential. Since eyes are the most important parameter to read the expressions of the user the term ‘eyes’ are used, though many other parameters are taken into account.

OPERATION

The major parts in the Blue eye system are 1. Data Acquisition Unit (DAU)

The tasks of the mobile Data Acquisition Unit are to maintain Bluetooth connections, to get information from the sensor and sending it over the wireless connection, to deliver the alarm messages sent from the Central System Unit to the operator and handle personalized ID cards. 2. Central System Unit (CSU)

Central System Unit maintains the other side of the Bluetooth connection, buffers incoming sensor data, performs on-line data analysis, records the conclusions for further exploration and provides visualization interface.

Fig: Overall block diagram Data Acquisition Unit (DAU): Data Acquisition Unit is a mobile part of the Blue eyes system. Its main task is to fetch the physiological data from the sensor and to send it to the central system to be processed. To accomplish the task the device must manage wireless Bluetooth connections (connection establishment, authentication and termination). Personal ID cards and PIN codes provide operator's authorization. Communication with the operator is carried on using a simple 5-key keyboard, a small LCD display and a beeper. When an exceptional situation is detected the device uses them to notify the operator. Voice data is transferred using a small headset, interfaced to the DAU with standard mini jack plugs. Eye movement is measured by using direct infrared transducers. The eye movement is sampled at 1 kHz, the other parameters at 250 Hz. The sensor sends approximately 52kB of data per sec.

The Data Acquisition Unit comprises several hardware modules namely, 1. Atmel 89C52 microcontroller - system core. 2. Bluetooth module (based on ROK101008). 3. HD44780 - small LCD display. 4. 24C16 - I2C EEPROM (on a removable ID card). 5. MC145483 – 13bit PCM codec. 6. Jazz Multi sensor interface. 7. Beeper and LED indicators, 6 AA batteries and voltage level monitor.

Atmel 89C52 microcontroller is to be the core of the Data Acquisition Unit since it is a well-established industrial standard and provides necessary functionality (i.e. high speed serial port) at a low price. To provide the Data Acquisition Unit with necessary physiological data an off-shelf eye movement sensor i.e., a Jazz multisensory is used. It supplies raw digital data regarding eye position, the level of blood oxygenation. If the Bluetooth module supports synchronous voice data transmission (SCO link), the hardware PCM codec can be used to transmit operator’s voice and central system sound feedback. The codec reduces the microcontroller’s tasks and lessens the amount of data being sent over the UART.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201650

Additionally, the Bluetooth module performs voice data compression, which results in smaller bandwidth utilization and better sound quality. Communication between the Bluetooth module and the microcontroller is carried on using standard UART interface. The keyboard is used to react to incoming events (e.g. to silence the alarm sound) and to enter PIN code while performing authorization procedure. ID card interface helps connect the operator’s personal identification card to the DAU.

Fig: Data acquisition Unit

Fig: Jazz multisensor Microcontroller software specification

All the DAU software is written in 8051 assembler code, which assures the highest program efficiency and the lowest resource. At a low-level design, the software uses a state diagram. Such a diagram facilitates implementation, debugging and testing phases. CENTRAL SYSTEM UNIT (CSU)

Central System Unit hardware is the second peer of the wireless connection. It contains a Bluetooth module and a PCM codec for voice data transmission. The module is interfaced to a PC using a parallel, serial and USB cable. The audio data is accessible through standard mini jack sockets. To program operator's personal ID cards a simple programming device is used. The programmer is interfaced to a PC using serial and PS/2(power source) ports. Inside, there is an Atmel 89C2051 microcontroller, which handles UART transmission and I2C EEPROM (ID card) programming, data logger and Visualization.

Fig: Central Unit system Bluetooth connection

During the implementation of the DAU there was a need of a piece of software to establish and test Bluetooth connections. Therefore a tool had been created called Blue Dentist. The tool provides support for controlling the currently connected Bluetooth device. Its functions are local device management and connection management.

To test the possibilities and performance of the remaining parts of the Project Kit (computer, camera and database software) Blue Capture had been created. The tool supports capturing video data from various sources (USB web-cam, industrial camera) and storing the data in the MS SQL Server database. Additionally, the application performs sound recording. After filtering and removing insignificant fragments (i.e. silence) the audio data is stored in the database. Finally, the program plays the recorded audiovisual stream. The software was used to measure database system performance and to optimize some of the SQL queries. Since all the components of the application have been tested thoroughly they were reused in the final software, which additionally reduced testing time THE SOFTWARE

Blue Eyes software’s main task is to check the operator’s physiological condition to assure instant reaction to the operators' condition change. The software performs real time buffering of the incoming data, real-time physiological data analysis and alarm triggering. The Blue Eyes software comprises several functional modules. System core facilitates the transfers flow between other system modules (e.g. transfers raw data from the Connection

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201651

Manager to data analyzers, processed data from the data analyzers to GUI controls, other data analyzers, data logger etc.). The System Core fundamental are single-producer-multi-consumer thread safe queues. Any number of consumers can register to receive the data supplied by a producer. Every single consumer can register at any number of producers, receiving therefore different types of data. Naturally, every consumer may be a producer for other consumers. This approach enables high system scalability i.e., new data processing modules (i.e. filters, Data analyzers and loggers) can be easily added by simply registering as a costumer.

Fig: Software analysis diagram Connection Manager

It is responsible for managing the wireless Bluetooth communication between the mobile Data Acquisition Units and the central system. The Connection Manager handles: 1. Communication with the CSU hardware 2. Searching for new devices in the covered range 3. Establishing Bluetooth connections 4. Connection authentication 5. Incoming data buffering 6. Sending alerts Data analysis module

The module performs the analysis of the raw sensor data in order to obtain information about the operator’s physiological condition. The separately running Data Analysis Module supervises each of the working operators. The module consists of a number of smaller analyzers extracting different types of information. Each of the analyzers registers at the appropriate Operator Manager or another analyzer as a data consumer and, acting as a producer, provides the results of the analysis. An analyzer can be either a simple signal filter (e.g. Finite Input Response (FIR)

filter) or a generic data extractor (e.g. signal variance, saccade detector) or a custom detector module. The most important analyzers are:

1. Saccade detector- It monitors eye movements in order to determine the level of operator's visual attention. It uses the data to signal saccade occurrence. Since saccades are the fastest eye movements the algorithm calculates eye movement velocity and checks physiological constraints.

2. Pulse rate analyzer - uses blood oxygenation signal to compute operator's pulse rate. It registers the oxy hemoglobin and de-oxy hemoglobin level data streams. Since both signals contain a strong sinusoidal component related to heartbeat, the pulse rate can be calculated measuring the time delay between subsequent extremes of one of the signals

3. Custom analyzers- It recognize other behaviors than those which are built-in the system. The new modules are created using C4.5 decision tree induction algorithm. As the supervisor’s needs can’t be predicted, the custom modules are created by applying a supervised machine learning algorithm to a set of earlier recorded examples containing the characteristic features to be recognized. The computed features can be, for an e.g. the operator’s position (standing, walking and lying) or whether one’s eyes are closed or opened. As built-in analyzer modules we implemented a saccade detector, visual attention level, blood oxygenation and pulse rate analyzers.

Visualization module

The module provides user interface for the supervisors. It enables them to watch each of the working operator’s physiological condition along with a preview of selected video source and his related sound stream. All the incoming alarm messages are instantly signaled to the supervisor. Moreover, the visualization module can be set in the off-line mode, where all the data is fetched from the database. The physiological data is presented using a set of custom-built GUI controls:

1. A pie-chart used to present a percentage of time the operator was actively acquiring the visual information.

2. A VU-meter showing the present value of a parameter time series displaying a history of selected parameter’s value.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201652

Data Logger Module

The module provides support for storing the monitored data in order to enable the supervisor to reconstruct and analyze the course of the operator’s duty. The module registers as a consumer of the data to be stored in the database. Each working operator’s data is recorded by a separate instance of the Data Logger. TECHNOLOGIES USED a) Emotion Mouse b) Manual and Gaze Input Cascaded (MAGIC) c) Artificial Intelligent Speech Recognition d) Simple User Interest Tracker (SUITOR) e) The eye movement Sensor TYPES OF EMOTION SENSORS 1. For Hand:

• Emotion Mouse • Sentic Mouse

2. For eyes: • Expression Glasses • Magic Pointing • Eye Tracking

3. For Voice: • Artificial Intelligence Speech Recognition

a) EMOTION MOUSE Rosalind Picard (1997) describes the emotions are importance to the computing community. There are two aspects of affective computing: giving the computer the ability to detect emotions and giving the computer the ability to express emotions. Not only are emotions crucial for rational decision making as Picard describes, but emotion detection is an important step to an adaptive computer system. An adaptive, smart computer system has been driving our efforts to detect a person’s emotional state. An important element of incorporating emotion into computing is for productivity for a computer user.

Fig: Emotion mouse

Fig:Samples obtained from an emotional mouse People spend approximately 1/3 of their total computer time touching input device. Physiological data is obtained and emotional state is determined. A user model will be built that reflects the personality of user.

The non invasive method for gaining user information through touch is via a computer input device, the mouse. This then allows the user to relate the cardiac rhythm, the body temperature, electrical conductivity of the skin and other physiological attributes with the mood. This has led to the creation of the ‘Emotion Mouse’. The device can measure heart rate, temperature, galvanic skin response and minute bodily movements and matches them with six emotional states: happiness, surprise, anger, fear, sadness and disgust. The mouse includes a set of sensors, including infrared detectors and temperature-sensitive chips. These components, user’s stress, will also be crafted into other commonly used items such as the office chair, the steering wheel, the keyboard and the phone handle. Integrating the system into the steering wheel, for instance, could allow an alert to be sounded when a driver becomes drowsy.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201653

Information Obtained From Emotion Mouse 1) Behavior

• Mouse movements • Button click frequency • Finger pressure when a user presses his/her

button 2) Physiological information

• Heart rate (Electrocardiogram (ECG/EKG), • Photoplethysmogram (PPG)) • Skin temperature (Thermostatic device) • Skin electricity (Galvanic skin response,

GSR) • Electromyography activity (Electromyogram

,MG)

SENTIC MOUSE:

It is a modified computer mouse that includes a directional pressure sensor for aiding in recognition of emotional valence (liking/attraction vs. disliking/avoidance)

Fig: Sentic mouse EXPRESSION GLASSES

A wearable device which allows any viewer to visualize the confusion and interest levels of the wearer. Other recent developments in related technology are the attempt to learn the needs of the user just by following the interaction between the user and the computer in order to know what one is interested in at any given moment. For example, by remembering the type of websites that the user links to according to the mood and time of the day, the computer could search on related sites and suggest the results the user.

Fig: Expression Glasses

Manual And Gaze Input Cascaded (MAGIC)

Reduce the cursor movement needed for target selection by eliminating the need of clicking on the target than with a regular manual input device. The two magic pointing techniques are

• Liberal • Conservative

Fig: Manual And Gaze Input Cascaded (MAGIC) The liberal MAGIC pointing technique:

The cursor is placed in the vicinity of a target that the user fixates on. Actuate input device, observe the cursor position and decide in which direction to steer the cursor. The cost to this method is the increased manual movement amplitude. To initiate a pointing trial, there are two strategies available to the user. One is to follow ‘virtual inertia’: move from the cursor’s current position towards the new target the user is looking at. This is likely the strategy the user will employ, due to the way the user interacts with today’s interface. The alternative strategy, which may be more advantageous but takes time to learn, is to ignore the previous cursor position and make a motion which is most convenient and least effortful to the user for a given input device.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201654

Fig: Liberal magic pointing Technique

The conservative MAGIC pointing technique: The goal of the conservative MAGIC pointing method is the following. Once the user looks at a target and moves the input device, the cursor will appear in motion towards the target, on the side of the target opposite to the initial actuation vector. In comparison to the liberal approach, this conservative approach has both pros and cons. While with this technique the cursor would never be over-active and jump to a place the user does not intend to acquire, it may require more hand-eye coordination effort.

Fig: Conservative magic pointing Technique

ARTIFICIAL INTELLIGENT SPEECH RECOGNITION

Artificial intelligence (AI) involves two basic ideas. First, it involves studying the thought processes of human beings. Second, it deals with representing those processes via machines (like computers, robots, etc). AI is behavior of a machine, which, if performed by a human being, would be

called intelligent. It makes machines smarter and more useful, and is less expensive than natural intelligence. Natural language processing (NLP) refers to artificial intelligence methods of communicating with a computer in a natural language like English. The main objective of a NLP program is to understand input and initiate action. The input words are scanned and matched against internally stored known words. Identification of a key word causes in some action to be taken. In this way, one can communicate with the computer one’s language. No special commands or computer language are required. There is no need to enter programs in a special language for creating software. Application:

• To control weapons by voice commands. • Pilot give commands to computers by

speaking into microphones. • Can be connected to word processors and

instead of writing, simply dictate to them SIMPLE USER INTEREST TRACKER (SUITOR)

It helps by fetching more information at desktop and notices where the user’s eyes focus on the screen fills a scrolling ticker on a computer screen with information related to user’s task ex. If the user is reading the headline, the story pops up in the browser window. THE EYE MOVEMENT SENSOR

Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. An eye tracker is a device for measuring eye positions and eye movement. Eye trackers are used in research on the visual system in psychology, in cognitive linguistics and in product design. There are a number of methods for measuring eye movement. The most popular variant uses video images from which the eye position is extracted. Other methods use search coils or are based on the electro-occulogram.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201655

Fig: Saccade occurrence and visual attention levels

Fig: Eye movement sensor

BLUE EYES ENABLED DEVICES

• POD: It is a technology used in cars.

• PONG: A robot.

APPLICATIONS

• In retailing record and interpret customer movements

• In automobile industry. • In video games. • Prevention from dangerous incidents. • Human-operator’s physiological condition. • To create “Face Responsive Display” and

“Perceptive Environment” Generic control rooms.

• Power station. • Flight control centers. • Operating theatres. • Blue Eyes system can be applied in every

working environment requiring permanent operator's attention at power plant control rooms, at captain bridges, at flight control centers.

• Data security - This system implies data security which is requiring in the modern network system. Only registered mobile devices can connect to the system. Bluetooth connection authentication & encryption. Access rights restrictions are there and personal and physiological data encryption is present.

• It can be used in the field of security & controlling, where the contribution of human operator required in whole time.

• Another application would be in the automobile industry. By simply touching a computer input device such as a mouse, the computer system is designed to be able to determine a person's emotional state.

• The BLUE EYES technology ensures a convenient way of simplifying the life by

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201656

providing more delicate and user friendly facilities in computing devices

ADVANTAGES

• The unique feature of system relies on the possibility of monitoring the operator’s higher brain functions involved in the acquisition of the information from the visual environment.

• It covers such areas as industry, transportation (by air, by road and by sea), military command centers or operating theaters (anesthesiologists)

• It is intended that the system in its commercial release will help avoid potential threats resulting from human errors, such as weariness, oversight, tiredness.

• Manmade disasters are also avoided. CONCLUSION

Humans have tremendous expectations from the technologies. This tends to research new and helpful technologies which can make the life more comfortable and reliable. This technology is one of them that can make so. Artificial disasters due to consciousness of human brain can be overcome by this technology. The BLUE EYES technology ensures a convenient way of simplifying the life by providing more delicate and user friendly facilities in computing devices. Future applications of Blue Eyes technology are limitless -from designing cars and developing presentations, to interactive entertainment and advertising. In future it is possible to create a computer which can interact with us, as we interact with each other with the use of blue eye technology. It seems to be a fiction, but it is much possible in the near future. Ordinary household devices --such as televisions, refrigerators, and ovens may be able to do their jobs when we look at them and speak to them. Thus the Blue eyes technology proves to be revolution in the field of technology. REFERENCES 1. Binyamin.M, B.Tech. (CSE), Cochin University of science and technology, Cochin-682022, Seminar report on “Blue eyes technology”, July 2010. 2. Priya R. Baghe, UG Students Information Technology, VIT, Wadala (E), Mumbai, “Blue eyes sensing Intelligence Technology”.

3. M.Revanth, B.E(EEE) ,Third year, Valimammai Engineering college, “Blue eyes- Monitoring human operating system.

ISBN-13: 978-1537313573

www.iirdem.org

Proceedings of ICTPRE-2016

©IIRDEM 201657