98 C O M P U T E R P U B L I S H E D B Y T H E I E E E C O M P U T E R S O C I E T Y 0 0 1 8 - 9 1 6 2 / 1 6 / $ 3 3 . 0 0 © 2 0 1 6 I E E E

STUDENT DESIGN SHOWCASE

Millions of people worldwide have limb im-pairments or other disabilities that limit their dexterity, making it di� cult for them to perform tasks that others take for granted,

such as unlocking doors, turning on lights, or changing the TV channel.

By allowing remote control and programmability, smart-home devices can o� er people with disabilities greater independence and improved quality of life. How-ever, limited dexterity also constrains the types of con-trollers that a person can use. Although voice recogni-tion is one potential solution, a group of Colorado State University students were inspired to go a step further: they’re exploring the possibility of controlling devices with our thoughts.

Recently developed, inexpensive wireless electro-encephalography (EEG) devices enable real-time monitor-ing and measurement of brainwaves without specialized equipment. EEG patterns have been used to recognize emotions, providing a more engaging and enriched user experience in games and other environments. EEG sys-tems have also been trained to recognize speci� c thought patterns that can be translated into commands. The Brain- Controlled Smart Home (BCSH) project’s goal is to

apply this thought-recognition ap-proach to control household devices.

Figure 1 shows a high-level view of the proposed smart-home system. The EEG headset transmits wire-

lessly to a computer, which processes the brainwave sig-nals. Commands are then sent via Wi-Fi to enabled devices in the home.

Rather than equipping an actual smart home, the team constructed a virtual reality (VR) house with thought- activated virtual devices. The VR environment served as a testing ground for development and provided new users with a realistic training experience.

EEG HEADSET AND SOFTWAREThe enabling technology for this project is the Emotiv EPOC+ headset (http://emotiv.com). EPOC+ is a wireless, multichannel, high-resolution system that measures brain activity and transmits data to a computer. Its pro-prietary software—the Mental Commands detection suite—displays the signals or detects patterns. According to Emotiv, this software “reads and interprets a user’s con-scious thoughts and intents.”

The headset provides 14 signal channels and 2 ref-erence channels, sampling signals in the 0.2–45 Hz range at 128 samples per second. Contacts are provided through felt pads hydrated with standard, multipurpose contact lens saline solution. The headset’s structure

Home Sweet Mind-Controlled HomeGreg Byrd, North Carolina State University

Students at Colorado State University built a

virtual reality prototype for experimenting with

cog nitive control of connected household devices.

M AY 2 0 1 6 99

EDITOR GREG BYRD North Carolina State University; gbyrd@computer.org

ensures that the contacts are posi-tioned appropriately for signal detec-tion (see Figure 2). The headset uses a proprietary 2.4-GHz wireless connec-tion to transmit the data; the most re-cent version also supports Bluetooth 4.0. The system contains a recharge-able battery for which the charge typ-ically lasts 12 hours.

The Mental Commands detection suite identi� es brain patterns associ-ated with certain thoughts. It can learn up to 15 di� erent commands, although only 4 can be active at any given time. During training, the command can be custom labeled and linked to anima-tion that illustrates its intent.

The � rst step is to train the head-set’s software to recognize the user’s neutral, or background, mental state. This is done by recording a brief period of brain activity during which the user isn’t trying to activate any command. To train for a particular command, the user imagines the consequences of that command (for example, visu-alizes a light turning on and o� ) for 8 seconds. The system records the men-tal patterns and associates them with the command. The user can then prac-tice using the command, retraining if necessary. In normal use, the system monitors the user’s brain activity, dis-tinguishing between the neutral state and one of the active commands.

VR ENVIRONMENTTo test their solution, the students needed a realistic environment, but they couldn’t a� ord to build or equip a smart home. As a � rst step, they cre-ated a virtual home for development and demonstration purposes. A vir-tual environment is also a useful and safe place for the user to practice while training the Emotiv headset system.

The team developed a 3D model of the house using Maya modeling soft-ware (www.autodesk.com/products/maya/overview). The house included

several rooms with various smart de-vices: door locks, lights, and a televi-sion. Figure 3 shows a snapshot of such a model. Although Maya can create realistic models, it’s not interactive.

Thus, to make the devices respond to user commands, the team used the Unity 5 game engine (http://unity3d.com). With Unity, users can move around the virtual environment and

EPOC + headset

Smart devices

Computer

2.4 GHz signal

Wi-Fi

Figure 1. Brain-controlled smart-home system. A wireless electroencephalography (EEG) headset transmits brain data to software on the computer, which recognizes patterns and translates them into commands for Wi-Fi–connected devices.

Figure 2. Student wearing the Emotiv EPOC+ EEG headset and Oculus Rift virtual reality display.

100 C O M P U T E R W W W . C O M P U T E R . O R G / C O M P U T E R

STUDENT DESIGN SHOWCASE

perform actions, such as opening doors and turning on lights.

To make the user experience as realistic as possible, the system also employs an immersive VR headset. The team initially used Oculus Rift (www.oculus.com; see Figure 2), but later switched to the Samsung Gear VR (www.samsung.com/us/explore /gear-vr). The Gear VR uses a Sam-sung Galaxy phone for the display,

with the headset providing the gy-rometer, accelerometer, and proxim-ity sensors. The students found the Samsung solution to be lighter and more responsive.

By using a modeled interactive en-vironment, the students could focus on the use of the brain-controlled de-vices, rather than on the specifics of any particular smart device protocol or API.

SOFTWARE: CONNECTING THE PIECESThe team created the program Cogni-tive Control to link the EPOC+ headset and the virtual environment. Written in C++, Cognitive Control uses APIs provided by Emotiv’s Xavier SDK. Each time the Emotiv software detects a user command, an event is generated for Cognitive Control to process.

Using raw events generated many false positives: commands were de-tected even when the user wasn’t in-tentionally issuing a command. Even a few milliseconds of pattern detec-tion was enough to trigger an event. To reduce false positives, the program requires commands to be active for a minimum time. However, this active period can’t be too long, because that would demand extra mental effort from the user, detracting from ease of use. Because each user is differ-ent, this duration is a parameter that should be tuned during training.

Another limitation of the EPOC+ headset and software is that only four cognitive commands can be active at any given time. With several con-nected devices in a house, or even in a single room, it could be difficult for a user to issue different commands to different devices. The team has exper-imented with recognizing command pairs (one soon after another), which results in up to 16 different distin-guishable commands. Another solu-tion might be to add proximity as a distinguishing feature, either through a separate sensor or measurement of Wi-Fi signal strength.

The final software component, Cognitive Connect, will connect to the smart devices. A smart home might contain dozens of connected devices, and keeping track of them all will be challenging. The team envisions a net-work of controller nodes, one in each room. The main computer will con-nect to the nodes that then control the actual devices.

Cognitive Connect entails a server component and a node component, both written in the Go programming

Figure 3. 3D model of the virtual smart home.

SUBMIT YOUR PROJECT

We want to hear about interesting student-led design projects in computer

science and engineering. If you’d like to see your project featured in this

column, complete the submission form at www.computer.org/student-showcase.

PROJECT DETAILS

» Project: Brain-Controlled Smart Home

» School: Colorado State University, Fort Collins, Colorado

» Students: Colt Darien, Edward Okvath, Kyle Van Cleave

» Faculty mentor: Sudeep Pasricha

M AY 2 0 1 6 101

language, an open source project ini-tiated by Google (http://golang.com). The language is portable and has na-tive support for concurrency, which is important for handling connections from multiple nodes. Using this net-worked approach, the user can control devices from any room in the house. For the node hardware, the team plans to use a simple, low-cost platform such as Raspberry Pi.

The team has now completed the basic BCSH prototype system, and demonstrated the feasibility of using thoughts to control virtual appliances in their VR environment. They’ve also demonstrated the system’s success in controlling a few physical devices. The students’ next major step is to com-plete development of the Cognitive Connect network for the whole house.

A s a proof-of-concept system, the BCSH project has been a success. The next phase will

require testing with real users in real-istic settings; the team plans to work with occupational therapists to ac-complish this. The training and user interface will likely require adapta-tion to better meet the target popula-tion’s needs. If successful, the BCSH will grant persons with disabilities increased independence, decreased re-liance on assisted-living facilities, and improved quality of life.

GREG BYRD is associate head of

the Department of Electrical and

Computer Engineering at North

Carolina State University. Contact

him at gbyrd@computer.org.

Selected CS articles and columns are also available for free at http://ComputingNow .computer.org.

COMPUTER SCIENCE

ENGINEERING SC

IEN

CE

The Perfect BlendAt the intersection of science, engineering, and computer science, Computing in Science & Engineering (CiSE) magazine is where conversations start and innovations happen.

CiSE appears in IEEE Xplore and AIP library packages, representing more than 50 scientifi c and engineering societies.