Design and Implementation ofSmartLab Infrastructure

GRADUATE PROJECT REPORT

Submitted to the Faculty ofthe Department of Computing SciencesTexas A&M University-Corpus Christi

Corpus Christi, Texas

In Partial Fulfillment of the Requirements for the Degree ofMaster of Science in Computer Science

By

Vinay Datta PinnakaSummer 2017

Committee Members

Dr. Scott A. KingCommittee Chairperson

Dr.Ajay K. KatangurCommittee Member

ii

ABSTRACT

Human efforts to interact with the digital world are continuously increasing

ever since the invention of the first computer. Advancements in embedded devices,

sensor technologies and distributed computing have driven forward research of smart

environments. Recently, the usage of smart environments has become popular in or-

der to make everyday living more comfortable and to improve the quality of life. This

work is concerned with constructing low cost infrastructure for a smart environment

which is proposed for implementation in Pixel Island Lab over time. The SmartLab

system collects depth, RGB and infrared data from a Kinect sensor by using Single

Board Computers and stream reduced data over the network. A novel client-server

model based on TCP protocol suite is proposed for transferring 3D data from the

Single Board server to client. The complete software and hardware infrastructure

designed and implemented in this project will help the application designers build

intelligent services and applications on top of SmartLab more easily.

iii

TABLE OF CONTENTS

CHAPTER Page

ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

TABLE OF CONTENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

1 BACKGROUND AND RATIONALE . . . . . . . . . . . . . . . . . . . . 1

1.1 Single Board Computers(SBC) . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Microsoft Kinect 360 . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Computer Vision for SmartLab . . . . . . . . . . . . . . . . . . . . . 4

1.4 Network Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.5 Pixel Island lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.6 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 PROJECT OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1 Project objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 SYSTEM DESIGN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1 Hardware and Software requirements . . . . . . . . . . . . . . . . . . 11

3.2 Data collection phase . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 Network phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 Centralized client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 IMPLEMENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1 Server Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1.1 Kinect Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.1.2 TCP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1.3 Server Data structure . . . . . . . . . . . . . . . . . . . . . . . 21

4.1.4 Server Configuration . . . . . . . . . . . . . . . . . . . . . . . . 22

iv

CHAPTER Page

4.2 Client Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.2.1 TCP Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.2.2 Client Data structure . . . . . . . . . . . . . . . . . . . . . . . 25

4.3 Background Subtraction . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.4 Data Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5 RESULTS AND EVALUATIONS . . . . . . . . . . . . . . . . . . . . . . 27

5.1 Smartlab Hardware setup . . . . . . . . . . . . . . . . . . . . . . . . 27

5.2 Smartlab system usage . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.3 Smartlab software results . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.3.1 Depth data stream . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.3.2 Color data stream . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.3.3 Depth and Color data combination . . . . . . . . . . . . . . . . 36

5.3.4 Factors influencing SmartLab software performance . . . . . . . 38

6 CONCLUSION AND FUTURE WORK . . . . . . . . . . . . . . . . . . . 40

6.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

APPENDIX A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.1 Environment for SmartLab server (Linux Environment- ODROID XU4) 46

7.1.1 Ubuntu Image . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.1.2 Installing OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.1.3 Installing OPENNI . . . . . . . . . . . . . . . . . . . . . . . . 48

7.1.4 Installing Libfreenect . . . . . . . . . . . . . . . . . . . . . . . 49

7.1.5 Installing Boost . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7.2 Environment for Smartlab server (Windows Environment-Lattepanda) 50

7.3 Building SmartLab server in Linux . . . . . . . . . . . . . . . . . . . 50

7.4 Building SmartLab server in Windows . . . . . . . . . . . . . . . . . 51

7.5 Running SmartLab server in Linux . . . . . . . . . . . . . . . . . . . 52

7.6 Running SmartLab server in Windows . . . . . . . . . . . . . . . . . 52

7.7 SmartLab client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

7.8 Running SmartLab client . . . . . . . . . . . . . . . . . . . . . . . . . 53

v

LIST OF TABLES

TABLE Page

1 Benchmark analysis of three Kinect libraries . . . . . . . . . . . . . . 18

2 Data capture and receiving rate without compression on Odroid

and Lattepanda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3 Network throughput with out compression . . . . . . . . . . . . . . . 39

4 Data capture and receiving rate with compression on Odroid and

Lattepanda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5 Network throughput with compression . . . . . . . . . . . . . . . . . 39

vi

LIST OF FIGURES

FIGURE Page

1 Kinect V1 hardware configuration and images from RGB and

depth camera. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 SmartLab abstract architecture. . . . . . . . . . . . . . . . . . . . . . 10

3 Detailed architecture of data collection phase. . . . . . . . . . . . . . 13

4 Network interconnection for SmartLab. . . . . . . . . . . . . . . . . . 14

5 Centralized client module. . . . . . . . . . . . . . . . . . . . . . . . . 15

6 Class diagram of server module . . . . . . . . . . . . . . . . . . . . . 16

7 Class diagram of client module . . . . . . . . . . . . . . . . . . . . . 24

8 Arrangement of Microsoft Kinects in Pixel Island . . . . . . . . . . . 27

9 Microsoft Kinect connected to SBC. . . . . . . . . . . . . . . . . . . 28

10 Face detection application using SmartLab . . . . . . . . . . . . . . . 29

11 Depth stream output without data compresssion . . . . . . . . . . . 32

12 depth stream result with data compresssion . . . . . . . . . . . . . . 33

13 Color stream output without data compression . . . . . . . . . . . . 34

14 color stream output without data compression . . . . . . . . . . . . . 35

15 depth and color image streams output without compression . . . . . 36

16 network statistics for depth and color stream collected from two

odroids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

17 network statistics for depth and color stream collected from single

Lattepanda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

1

CHAPTER 1

BACKGROUND AND RATIONALE

Smart environment is defined as a physical world that is equipped with sen-

sors, actuators, embedded devices and interconnected seamlessly through a network.

Recently, smart environment has become prominent research topic since it plays

a key role in predicting health, lifestyle and wellbeing of humans. Due to its ap-

plication in various fields this research received several transformations in terms of

hardware utilization as well as software practices. The concept of smart environments

evolved from the term ubiquitous computing according to Mark Weiser. Ubiquitous

computing can occur on any device, this type of computing utilizes the supporting

technologies such as embedded hardware, internet protocols and sensors. The focus

of SmartLab is to develop smart environment with low cost infrastructure.

SmartLab is an interactive lab environment which can identify individuals and

monitor activities. SmartLab Infrastructure Project(SLIP) is focused on building a

low cost infrastructure to acquire a variety of data including RGB, depth and infrared

from motion sensing sensor like Kinect [1], placed on the ceiling and walls of Pixel

Island Lab at Texas A&M University-Corpus Christi using Single Board Comput-

ers(SBC). Availability of low-cost motion sensing sensors such as Microsoft Kinect

360 [1] reinforced the SmartLab research. Human-computer interaction became more

amicable with the advent of multi sensor devices like Kinect. However, interaction

with Kinect to obtain data is still in its naive stage. Particularly interacting with

multiple sensors simultaneously is a complex hardware limitation as mentioned by

Mario Mart and Miguel Pedraza [2]. There were few researchers who addressed this

problem and developed a system for it. One such system that we came across is

2

constructed as a distributed system that uses TCP protocol to stream the data from

the server to a client [2]. But the system still uses powerful desktop system which

are expensive and contradicts the idea of inexpensive infrastructure for smartlab.

In this work, the focus is on developing a distributed system that is economically

subtle and easy to setup. We have chosen cost effective yet powerful Single Board

Computers(SBC’s) for smartlab. Later sections talk more about the SBC’s and their

applications. In addition to these tiny computers, we have chosen a well-known con-

sumer camera, Microsoft Kinect xbox 360 [1]. These two in combination provide a

reasonable system for smartlab.

1.1 Single Board Computers(SBC)

A single-board computer (SBC) is a complete computer built on a single cir-

cuit board, with microprocessor(s), memory, input/output (I/O) and other features

required for a functional computer. Compared with desktop computers SBC’s are

compact and consume less power. Moreover, SBC’s are significantly cheaper than

PC’s and also powerful enough to perform solitary tasks, in turn they are good re-

placement to PC’s where multitasking is not a priority. For example, Odrioid-XU4

a $70 Single Board Computer has processor speed of 2GHz where as the first gen-

eration Pentium Quad Code desktop has 2.16GHz processor speed which is almost

comparable. Applications of SBC’s are typically home automation, robot control,

media players, and personal cloud storage servers etc. Among all the SBC’s available

in market Raspberry Pi is famous for its low cost and ease. In addition to Raspberry

Pi, other boards like Odroid, Banana Pi, Orange Pi, Cubieboard and recent windows

10 board Lattepanda are much more powerful and have diverse applications. The

choice of SBC for SmartLab is made by considering three features; cost, ethernet

3

specifications, and processing power. For this project the ability of different SBC’s

like Raspberry pi, Odroid-XU4 and Lattepanda were tested and found that all the

three boards perform satisfactorily, but for Raspberry pi, we have to compromise

with lesser video frame rate.

1.2 Microsoft Kinect 360

To acquire multiple data streams such as RGB, depth and infrared, there

are a number of devices available such as stereo cameras, time of flight cameras and

consumer motion sensing cameras such as Asus Xtion [3], Microsoft Kinect [1], Struc-

ture [4] sensors. Amongst all these motion sensing devices available in the market

Microsoft Kinect stands out for its reasonable cost and satisfactory frame rates and

resolution. Kinect 360 also called Kinect for Xbox 360 is a combination of Microsoft

built software and hardware. Kinect 360 hardware includes a range of chipset tech-

nology, which consists of an infrared projector, camera and a special microchip that

generates a grid from which the location of a nearby object in three dimensions can

be ascertained. Though the Kinect is released for games, the usability of the sensor

in the computer vision field makes it popular among the research groups. Presently,

there are several software tools available to interact with the Kinect, among them

the popular ones are Microsoft official SDK, OpenNI, and LibFreenect. Microsoft

SDK is windows only and the remaining two software’s are multi-platform. A de-

tailed comparison of these software’s are presented by Han, Jungong and Shao, Ling

[5]. These software provide access to the RGB and depth and infrared sensors of the

Kinect, Figure1 [5] shows the hardware configuration of Microsoft Kinect 360 and

sample images from RGB and depth sensors. OpenNI works along with a middleware

called NiTE to provide high-level features namely hand tracking and human skeletal

4

joints. The wide applications of Kinect in Computer vision and its ability to track

human skeleton are the primary reasons to choose it as data collection sensor.

Figure 1: Kinect V1 hardware configuration and images from RGB and depth

camera.

1.3 Computer Vision for SmartLab

Computer vision is an interdisciplinary field that deals with how computers

can be used to gain high-level understanding from digital images or videos. Computer

vision tasks include methods for acquiring, processing, analyzing and understanding

digital images, and in general, deal with the extraction of high-dimensional data

from the real world in order to produce numerical or symbolic information. When

streaming data from the static camera, we face the problem of sending the back-

ground data multiple times when is not changing. To avoid this problem and to

extract only sensible data, background subtraction is used. In this technique the

image’s foreground is extracted for further processing [6]. Background subtraction is

a widely used approach for detecting moving objects in videos from static cameras

[7]. The rationale for the approach is to detect moving objects from the difference

between the current frame and a reference frame, often called as ”background im-

age”, or ”background model”. Each server application employs this technique and

5

sends only essential data to the client. By employing this technique the network

bandwidth utilized by each server is reduced.

1.4 Network Interface

A novel TCP/IP based client-server paradigm is developed using low-level

sockets for devising a sophisticated network interface for the SmartLab. Sockets al-

low network software applications to communicate using standard mechanisms built

into network hardware and operating systems. TCP/IP (Transmission Control Pro-

tocol/Internet Protocol) is the basic Internet communication protocol. TCP/IP is a

four-layer protocol, the higher layer is Transmission Control Protocol that manages

assembling of a message into smaller packets that are transmitted over the Internet.

The lower layer is called as Internet Protocol that handles the address part of each

packet. In the SmartLab application, the data streamed over the network is crucial,

as the data is used for critical application such as face recognition, object detection

etc. on the client side. TCP is selected for its reliable, ordered, and error-checked

delivery of a stream of bytes. Moreover, network interface implemented via sockets

does not have the programming language barrier and provides low-level access to the

network. So, we can easily modify the application layer over it.

1.5 Pixel Island lab

Pixel Island Lab is the research facility available at Texas A&M University

- Corpus Christi, the mission of Pixel Island is to conduct research in computer

graphics and visualization, to be a source of expertise in these areas for the campus,

and to educate students in these areas. Pixel Island currently has two major focus

areas: face-to-face human-computer interaction and visualizing scientific data.

6

1.6 Related work

Implementation and selection of sensors differentiate many smart environ-

ments from one another. Study for smart environments recommended three foremost

research works. Firstly, Aware Home Research Initiative (AHRI) [8] at Georgia In-

stitute of Technology is a interdisciplinary research project for analyzing health and

well-being, providing entertainment and child study. Secondly, MIT Media Lab has

a research project named House n [9], this project provides the environment for sci-

entific study. Finally, MavHome [10], the smart home research project developed by

University of Texas at Arlington, this project involved building a intelligent agent

that maximizes the comfort of inhabitants. Alongside the research work done by

Maurizio Caon and Yong Yue [11] talks about context aware gesture interactions

based on multiple Kinects. Moreover, all these systems acts as an inspiration to

implement a personalized smart environment in Pixel island lab.

7

CHAPTER 2

PROJECT OVERVIEW

2.1 Project objective

The goal of SmartLab Infrastructure Project is to build low cost infrastructure

for extracting 3D data from multiple Microsoft Kinects. This system also focuses

on transferring the data over the network effectively. The use of this extensive

data collection setup helps researchers easily set up monitoring system that can be

employed in useful applications, such as those related to security and surveillance,

human behavior analysis and patient health care.

In this project we setup multiple Kinects on the walls and ceiling of our pixel

island lab. In general, a personal computer is required to interact with each Kinect,

but in our system, in opposite to this general approach, we utilized Single Board

Computers that are hooked with Kinect. Server module running on SBC’s obtain

RGB, depth and human skeleton information from Microsoft Kinect. The server

application performs Gaussian background subtraction [6] on RGB and creates a

mask to obtain only sensible data that changes from frame to frame. Apart from

data collection and understanding, the server module also streams the data over

the network with a novel protocol built on top of TCP/IP. On the other hand, the

client module is responsible for interacting with multiple servers simultaneously and

collecting the data obtained over the network. Client module process the data and

reconstructs it to obtain the original RGB, depth and infrared images. Moreover, the

huge data obtained on the client side is utilized for human monitoring applications.

8

2.2 Requirements

The main focus of this project as stated in the earlier section, is building the

infrastructure, to interact with multiple Kinects. More specifically, this project will

enable researchers to develop futuristic application for human monitoring. Addition-

ally, the following are the required characteristics of the SmartLab Project.

• Inexpensive: Building a multiple data collection system is never inexpen-

sive but, in order to achieve this goal, the low cost motion sensor Microsoft

Kinect 360 is selected. Dealing with multi-array data sensors require power-

ful computers which is not an economic solution. Research on various Single

Board Computers showed that SBC’s can work with Microsoft Kinect effec-

tively. SBC’s make a viable solution to cut off the costs. Overall the hardware

selected for building SmartLab is economical and also unprecedented.

• Portability: While selecting SBC’s we came across several effective boards

which have varying operating software. Specifically, to mention Raspberry Pi

and Odroid are compatible with running Ubuntu mate 16.04 and Lattepanda

is able to run only windows 10. SmartLab application is intended to run on

any kind of SBC, hence the development of server application should be done

in such a way that the application is highly portable with minor changes.

• Incorporate: SmartLab Infrastructure Project can be incorporated into many

applications such as Surveillance system, clinical applications, gait analysis,

and fall detection. To make this application generalized to all other fields

the design and development are planned accordingly. Data obtained from the

SmartLab, RGB and depth and infrared are collected as simple data format.

This allows researchers to comfortably combine SmartLab to any functional

9

module.

10

CHAPTER 3

SYSTEM DESIGN

This chapter illustrates the functional phases of SmartLab system. Each block

contains different software components that are executed sequentially. Figure 2 show

the abstract flow diagram of the SmartLab system. Section 3.1 lists the hardware

and software requirements for SmartLab system.

Figure 2: SmartLab abstract architecture.

In the fallowing sections, details about the three phases of the SmartLab

are discussed. The first phase deals with data collection from Microsoft Kinect,

this phase is responsible for collecting RGB, depth, infrared and human skeleton

joints(only with Lattepanda) at 30 frames per second. The second phase is network

11

phase where data is transmitted to the client. The last phase is a centralized client,

data collected over the network is processed and displayed to the user.

3.1 Hardware and Software requirements

• Microsoft Kinect 360

– RGB video camera

– Depth sensor

– Infrared camera

• Single Board Computers(SBC’s)

– Lattepanda [12]

– Ordroid XU4 [13]

– Raspberry Pi

• OpenNI 2 and NiTe 2

– OpenNI 2 - Open Natural Interaction [14]

– NiTE 2 - Natural Interaction Technology for End-user [15]

3.2 Data collection phase

In this phase, Single Board Computers interact with Microsoft Kinect and

obtain RGB, Infrared, depth and human skeletal joints information. Figure 3 shows

each component of data collection phase. In order to communicate with the Kinect, a

range of software tools is available namely, Libfreenect, OpenNI and official Microsoft

Kinect SDK[18]. OpenNI is multiplatform and open source tool which made it a

12

preferable Kinect software tool. OpenNI works along with compliment middleware

NiTE and provides human skeletal data along with RGB and depth streams. Kinect

provides 640x480 pixels 8-bit RGB image and also 11-bit monochrome depth image

with the same resolution at a default frame rate of 30Hz. Typically, NiTE also

processes the human joints at the same rate. The size of the data generated by

Kinect per second by default is provided in the equation 3.1

Total size of a data generated per second = [ (640 * 480) * 2 bytes depth+

(640 * 480) * 3 bytes RGB+ 4 * 3 * 15 Joints] * 30 fps = 43.95 Mbytes per second

(3.1)

Theoretically, the size of the data generated by Kinect is 43.95 MB per second. This

summarizes that Kinect generates a lot of data. It is not a good idea to send this

huge data over the network because most of the collected data is redundant. Firstly,

after data collection the important step is to apply data reduction techniques based

on background subtraction. MOG2 is a sophisticated background and foreground

separator algorithm proposed by Z.Zivkovic [6]. MOG2 extracts a foreground mask

by learning from the video stream. MOG2 background subtractor works indefinitely

for the RGB image. Human skeleton joints are insignificant in size compared to

depth and RGB images and do not need any reduction mechanisms.

13

Figure 3: Detailed architecture of data collection phase.

Secondly, serialization of data into character array is a major task. Processed

RGB, depth, and skeleton joints are serialized into a string object, in order to send

over the network. The data serialization involves a specific encoding format with

RGB+depth+joints, selection of this flow makes de-serialization simple on the client.

Thirdly, compression using Zlib [16] is an essential part of data collection

phase. Compression further minimizes the redundant data in the serialized string.

Zlib is free, general-purpose, lossless compression library for both Windows and Linux

operating systems.The estimated reduction of actual data(43.95 MBps) is by 50%,

approximately equivalent to 20 MBps. This estimate is based on the analysis per-

formed on a single RGB image of size 900KB. Utilizing both background subtraction

and Zlib compression, this RGB image size is reduced to 141KB, which is 82% re-

duction. But, taking the worst case scenarios into consideration and also scaling to

14

1.46 MB(each frame size contains RGB, depth, infrared and human skeleton joints)

will justify the estimated 50% reduction. The compressed data is forwarded to the

network phase where the server streams over the network.

3.3 Network phase

Network phase completely deals with client-server communication. Intercon-

nection and setup of this phase are illustrated in figure 4. In order to transmit

data over the network TCP/IP based client-server protocol is implemented Multiple

servers running simultaneously provides large network traffic. To simulate the actual

inter networking of SmartLab environment a local area network is setup with a router

and a network switch. A centralized client collects RGB, depth and infrared data

from individual server. Data obtained by the client is further processed for obtaining

original 3D data.

Figure 4: Network interconnection for SmartLab.

3.4 Centralized client

Figure 5 show the modules in central client, also the implementation details

and hardware specifications of the client system are mentioned. The centralized

15

client module collects the data from the various servers and display it to the user

or provides a way to utilize the data in some other applications such as human

monitoring.

Figure 5: Centralized client module.

The centralized client is implemented on Intel x86 i7 computer with 12 GB

of RAM. The multi-threaded program runs on the client with each thread dedicated

to the single server. Once the data is collected by the client a series of operations

occur. Firstly, decompression and then de-serialization at last image reconstruction.

RGB, depth and skeleton data can be obtained by any application using a function

call.

16

CHAPTER 4

IMPLEMENTATION

To build an effective and reliable infrastructure for Smartlab, the system is

modularized to have good scope for re-usability. Figure 6 and 7 shows the modules

of Smartlab architecture, on the higher level the entire Smartlab architecture is

divided into two modules the server module and client module and these modules

are further divided into sub-components. Appendix A explains in detail about, how

to set up Smartlab environment on both Single Board Computers and desktop client.

The functioning of each module and interaction between the sub-components will be

discussed in the following sections.

Figure 6: Class diagram of server module

4.1 Server Module

As discussed in section 3.1, Single Board Computers interact with Microsoft

Kinect and obtain RGB, Infrared, depth and human skeletal joints information. It

17

is the server module which is set to run on the SBC to obtain data from Microsoft

Kinect sensor. In order to implement this module, the main challenge is finding

the most suitable library that helps to interact with Kinect. After intense research,

we noticed that the top three libraries on the list are OpenNI 2.2 [14], Libfreenect

[17] and Microsoft Kinect SDK 1.8 [18]. The next task is to perform a benchmark

analysis on all these three libraries and select the best amongst them. The factors

which are considered to preform benchmark analysis are as follows: using simple API

calls, multi-platform support, easy integration with other modules, and open source

support. Table 1 provides a comparison overview of all the three Kinect interface

libraries. In addition to the benchmarking features, libraries have been tested by

writing sample programs that actually simulates the server module. Finally, OpenNI

is selected for implementing interaction with Kinect. This library is selected not only

to achieve all our benchmarks but to support wide verity of sensors such as Asus

Xtion [3], structure sensor [4]. We are positive that in the near future with the help

of OpenNI it is possible to replace Microsoft Kinect with any of the aforementioned

sensors. OpenNi 2.2 library provides access to three video streams of the Kinect

namely, depth, RGB, and infrared. However, to obtain the skeleton joints of a

person in the video stream we need to use a third-party middleware library NiTE

[15]. NiTE is developed by Primesense, the same company that developed OpenNI

as well. For network interface Boost.asio [19] library is selected, boost is a header

only library for c++ and also users have the option to select only a few components

of the library for linking the program, this makes it more viable when developing on

low power devices such as Single Board Computers.

18

Table 1: Benchmark analysis of three Kinect libraries

OpenNI Libfreenect Microsoft KinectSDK

API calls simple to use not that simple Complex

Multi-platform sup-port

Linux, windows,Mac

Linux, Windows,Mac

Only windows

Integration withother modules

Easily integrates Low level driver,complex to inte-grate

Compatible withwindows only soft-ware

Open source support Available Available Support availablefrom Microsoft blog

Overall, the server module is subdivided into different components such as

Kinect data, TCP server, Data structure, and configuration. A multi-threaded ap-

proach is followed to implement server module, Figure 6 shows the overview of the

server module class design and thread execution. The following sections give more in-

formation about each component and how they are coordinated to achieve a reliable

system that runs on low-powered devices.

4.1.1 Kinect Data

Kinect data is a component of the server module that utilizes OpenNI [14]

for obtaining depth and color streams, kinectdata object is instantiated in the main

function and all the operations of the kinectdata class runs on the main thread. The

server module continuously collects data from Kinect on a dedicated program thread.

• kinectdata::run(): run is the first function called from the main method,

this method is responsible for updating the Kinect data between several client

interactions. Initializing Kinect, starting video capture and reading data from

different streams is performed by calling the corresponding function in run()

method. After completion of the each client session, all the previously selected

19

Kinect streams are suspended thereby making the server module ready for the

next session.

• kinectdata::init kinect(): This function opens the Kinect device instance

and returns STATUS OK signal. If any error occurs in accessing the Kinect

device, an exception is raised and a STATUS ERROR signal is returned. All

other Kinect related functionality such as starting a video stream or reading a

frame happens only after this function is called.

• kinectdata::init videocapture(): init videocapture is responsible for creat-

ing the selected stream among the available three video streams depth, RGB

and infrared. For creating a particular stream the required parameters are to be

initialized openni::device, selected openni::VideoStream, and its corresponding

SensorType. If the selected stream is already created it returns STATUS OK

or otherwise it opens a new stream with default video mode.

• kinectdata::read frames(): Once the client session is initiated the corre-

sponding streams selected by the client are created by kinectdata::init videocapture().

After stream creation, kinectdata::read frames() functions continioulsy looks

for any available stream using openni::OpenNI::waitForAnyStream function for

a predefined timeout, if any stream is available it grabs the frame and writes

to the corresponding data buffer.

4.1.2 TCP Server

TCP server component is responsible for network operations in server mod-

ule, Boost.Asio library [19] is used to implement tcp server. Boost.Asio [19] is a

cross-platform C++ library for network and low-level I/O programming that pro-

20

vides developers with a consistent synchronous and asynchronous model. tcp server

is basically a synchronous program that creates a TCP socket and continuously ac-

cepts clients on a user defined port. Once the client is connected, the configuration

file is read from the client and the configuration parameters are updated locally so

that the Kinect starts reading the selected frames. Whenever the data is available

from the Kinect, the tcp server sends data to the client over the TCP socket. The

main() method creates a tcp server thread for network operations. The members of

tcp server class are as follows:

• tcp server::run(): The main() method creates a network boost::thread, so

that all the client sessions that only in the network thread. run() method is the

starting point of the tcp server class which will internally call the accept client

method.

• tcp server::accept client(): accept client() method continuously accepts

client connections over the socket and once the client gets connected tcp session

is initiated. If tcp session is completed tcp server again waits on acceptor.accept(socket)

until the next connection

• tcp server::tcp session(): A single tcp server::tcp session() consists of one

configuration read and several data writes Initiallly tcp server reads the con-

figuation sent by the client and updates it to local config file. The data

in data buffer is updated by the thread and set the ready flag. Whenever,

tcp server::tcp session() reads the data ready flag it will initiate the write ses-

sion.

21

4.1.3 Server Data structure

In the server module, data is obtained from a Kinect and transferred over

the network. However, Kinect data obtained in different formats such as RGB,

depth , infrared and person skeleton joints. In order to format the data obtained

from the Kinect, a data structure is necessary. Server module data structure have

two buffers, one for reading configuration data from the client and second buffer to

collect data from the Kinect. In addition to these buffers, a lock-free queue is also

implemented to share data between the Kinect thread and the network thread. A

static data structure object is shared across different threads to maintain consistency.

The read buffer only holds configuration data received from the client, read buffer

is updated only when a new client session occurs. The write buffer is a complex

structured buffer formed by concatenation of selected data frames. In other words,

if client requests depth, color and body joints, the write buffer is constructed by

appending depth followed by color and body joints. This complex design is chosen

for two main reasons, one is network aspect and other is coordination between the

client and server.

The data structure also houses a synchronous communication mechanism be-

tween Kinect thread and TCP server thread. A lock-free communication system

is implemented in server data structure component, the main intention to select

lock-free mechanism is to avoid thread waiting and improve performance without

mutex locks. Also, server module runs on low power devices, consequently choos-

ing a wait-free mechanism significantly improves performance when compared to

locks. boost::lockfree:spsc queue [19] provided by boost library is thread-safe lockfree-

container. Multiple threads can access lockfree containers without having any syn-

chronization, the abbreviation spsc stands for single producer/single consumer. The

22

lockfree queue is technically implemented as a ring buffer with ordering mechanism,

unlike mutex locks, lock-free buffers does not have busy waiting, so they are fast and

have less overhead on the operating system. The Kinect thread is responsible for

serializing different data streams into write buffer and then sends it to spsc queue,

where as TCP server thread de-queues the data and sends it to the client.

4.1.4 Server Configuration

The principal design of smartlab is modular in other words each phase of

smartlab is configurable. Users can select the type of data they wants from the

server module, to enable this the server module is equipped with configuration com-

ponent. The config component obtains data from the client and parses it to select

the necessary data stream from Kinect. Configuration buffer is in JSON format as

shown below with parameters either set to true or false.

"VIDEO_CONFIG":{

"IMAGE_CONFIG":{

"RGB_IMAGE":true,

"IR_IMAGE":false

},

"DEPTH_CONFIG":{

"DEPTH_MAP":true,

},

"BODYJOINTS_CONFIG":{

"SKELETON":true

},

"BACKGROUND_SUBTRACTION":{

23

"RGB_SUBTRACTION":false,

},

"COMPRESSION":{

"SELECTED":true

}

}

For example, in the above configuration RGB, depth images and body joints

are selected. However, background subtraction is set to false and data compression

is enabled. Server program by parsing the config buffer will enable the respective

module and prepare the data before sending to the client.

4.2 Client Module

As discussed in section 3.3 the client module is responsible for collecting data

from multiple servers and provide a way for the user to utilize this data for further

processing. The implementation of client module is different from the server, it

consists a network componet, user interaction component and display component.

However, the data structure and configuration components are almost similar to that

of the server. The network component houses a TCP client module to communicate

with the TCP server. The client module dedicates a thread to each server, data

collected by each thread is added to a one dimensional array of lockfree queue.

Figure 7 shows the implementation of multi-threaded client module. The following

sections describe more about sub components of client module.

24

Figure 7: Class diagram of client module

4.2.1 TCP Client

The TCP client component is responsible for interacting with a server, to send

user configuration and retrieve data accordingly. The implementation of TCP client

is similar to that of the server. Boost.Asio [19] the library used for implementing

server is also used for the client. The client module creates an individual thread

for each server and assigns them the task to continuously interact with the server.

Each TCP client thread connects to its corresponding server using IP address and

port number read from the configuration file. In addition, to connection details, each

thread is assigned an id number so that they can write the retrieved data into a buffer

with the same id. The static member function run() of tcp client class is passed to

boost::thread with connection details and server id number. The run() function

establishes the connection to the server and initiates a TCP session. The following

tcp client class members give more details about the TCP Client component.

• tcp client::run(): run() is the static member function that is first called

when initializing the client thread. The parameters that are required to be

passed to this function are server details such as port number, IP address

25

and Id number. After initializing all the parameter in tcp client object, it is

connected to the server.

• tcp client::connect server(): The client thread connects to the server using

IP address and port number read from the configuration file by the main thread.

• tcp client::tcp session(): After the connection is established, client thread

writes the video configurations read from config.json file to the server. The

server then writes the serialized data based on the configuration parameters

and is followed by writing data to the lock-free buffer.

4.2.2 Client Data structure

The data structure of client module is very similar to that of server except

for one-dimensional lock-free buffer. This data structure also has a config buffer and

a read buffer, the config.json file is read into config buffer which is later sent to the

server. The serialized data received from the server is read into the read buffer. In

addition, to these two buffers one-dimensional lock-free queue is also initialized and

the client thread queues the read buffer to the respective index of the lock-free buffer

queue.

4.3 Background Subtraction

Background subtraction is a major pre-processing step in many vision based

applications. If we have a static camera we can always subtract the background

from the current frame and obtain only the changes, but this is not easy when the

room is illuminated with multiple lights. The shadows of the objects confuse vi-

sion algorithms. In order to avoid this MOG2 [6] background subtractor is utilized.

26

This subtraction algorithm is Gaussian Mixture-based Background/Foreground Seg-

mentation Algorithm. One important feature of this algorithm is that it selects the

appropriate number of Gaussian distribution for each pixel. It provides better adapt-

ability to varying scenes due to illumination changes etc. Including this background,

subtraction not only reduces the data to be transferred but also eliminates initial

preprocessing of data for many vision applications.

4.4 Data Compression

Data compression step in server module is performed before the network trans-

mission. The use of data compression excludes the redundant data and decreases

the load on the network. On the client side, the received data is decompressed and

deserialized in further steps. Zlib [16] is the lossless compression library that is used

in the smartlab application.

27

CHAPTER 5

RESULTS AND EVALUATIONS

The foremost objective of SmartLab Infrastructure Project is to set up a

reliable infrastructure to collect 3D data from depth sensors. In order to achieve this

goal we have arranged hardware in the Pixel Island Lab and developed software for

it to attain a reliable data acquisition rate. This chapter illustrates how hardware

components such as Microsoft Kinect and Single Board computers are coupled with

each other. Furthermore, an analysis of data acquisition rates on different SBC’s is

presented.

5.1 Smartlab Hardware setup

Smartlab infrastructure involves two components, motion sensing sensor Mi-

crosoft Kinect and Single Board Computer. Firstly, the Kinects are mounted on the

ceiling of the Pixel Island Lab.

Figure 8: Arrangement of Microsoft Kinects in Pixel Island

28

The Kinects are arranged to cover most of the lab, figure 8 shows a part of the

lab where Microsoft Kinects are arranged. Each Kinect is connected with a Single

Board Computer(SBC) and these SBC’s are connected to the network through a

switch. Figure 9 shows an individual Kinect and SBC pair.

(a) Kinect mounted on ceiling(b) Single Board Computers

Figure 9: Microsoft Kinect connected to SBC.

5.2 Smartlab system usage

Infrastructure for SmartLab is set up with a focus to provide reliable and

effective data collection mechanism for research. The SmartLab system can be ex-

tended to implement sophisticated applications such as Facial recolonization, Object

detection and health care analysis. As a naive step to implement such complex

application, a face detection program is developed. Figure 10 shows results for face

detection application. Similar to this application researchers can make great progress

and develop many other sophisticated applications.

29

Figure 10: Face detection application using SmartLab

5.3 Smartlab software results

Smartlab Infrastructure project comprises of two software modules server

module and the client module. The server module collects different data streams of

Microsoft Kinect 360 [1] such as depth, color(RGB), IR and human skeleton joints.

Client module can select individual streams or combination of the streams and re-

ceive data accordingly. Along with data streams, data compression and background

subtraction are additional features that can be selected in server module. The per-

formance of the smartlab system is analyzed on two factors. Firstly, the frame rate

at which the server module is capturing data from Kinect. Typically, the default

output video frame rate of Kinect 360 is 30fps. But, the computational capability of

the host machine effects the capture frame rate. Secondly, the data retrieval rate at

30

client side is also taken into consideration for analyzing network bandwidth utiliza-

tion. Analysis is done on selecting each data stream and the results are presented in

the following sections

The hardware components utilized in the analysis are two Single Board Com-

puters(SBC) Odroid-XU4 [13], Lattepanda [12] and depth sensor Microsoft Kinect

360 [1]. Odroid-XU4 has cortex A15 2GHz processor with 2GB of RAM and Lat-

tepanda has intel quad core 1.8GHz processor and 2GB of RAM. Odroid runs on

linux operating system where as Lattepanda is windows SBC. The server module is

tested on the above mentioned SBC’s and the client module is tested on general PC

with an i7 processor, 12 GB of RAM and Gigabit ethernet port. Odroid-XU4 also

has a gigabit ethernet port and Lattepanda has 100Mbits Ethernet capability. The

aforementioned properties of the hardware impacts data capture and transfer rate.

The following sections discuss more on how a trade off occurs between computational

power and network bandwidth usage.

5.3.1 Depth data stream

A depth frame collected from the Kinect sensor is 640x480 pixels with 2 bytes

(by default 11 bits per pixel but computers store each pixel in 16 bits) for each pixel.

This sums up to 614,400 bytes collected for each frame. The default frame rate of

depth stream is 30 fps(frames per second). So, the total data collected from Kinect

by SBC per second is 17.578MBytes(140.625Mbits). Figure 12a is depth stream out-

put on client side. Odroid and Lattepanda are efficient to capture the data at 30

frames per second. However, Odroid is able to stream data to the client at 30 fps

as evident from the Figure 12b, but Lattepanda suffers from bandwidth limitations

while transferring data over the network, considering it’s 100Mbps LAN port. Fig-

31

ure 12c shows data receiving rate on the client side when depth data is streamed

from Lattepanda and the frame rate is approximately equivalent to 20fps. However,

data compression reduces the load on the network but computation performed for

compression reduces the data capture rate in SBC. Subsequently, the frame rate on

the client is reduced to 16fps. Figure 12b and 12c shows network throughput of

compressed depth data. This is incident both in Odroid and Lattepanda.

32

(a) depth stream output on client module (b) network throughput for depth streamfrom odroid

(c) network throughput for depth stream from Lattepanda

Figure 11: Depth stream output without data compresssion

33

(a) depth stream frame rate with compression from Odroid

(b) network throughput for depth streamwith compression from Odroid

(c) Network throughput for depth streamwith compression from Lattepanda

Figure 12: depth stream result with data compresssion

5.3.2 Color data stream

The default color frame uses 8-bit video resolution (640x480 pixels and each

pixel have three 8-bit channels each for storing R, G and B values) as shown in

figure 13a . At default frame rate of 30fps, the data captured by the SBC is

26.367MBytes(210.937Mbits). Odroid XU4 is successful in streaming the data at

30fps but Lattepanda is not that efficient and it is able to attain an average frame

rate of 10fps. Figure 13b and Figure 13c shows the network statistics of color frame

34

streaming done by Odroid and Lattepanda.

(a) color stream(RGB) output on client mod-ule

(b) network throughput for color stream fromodroid

(c) network throughput for color stream from lattepanda

Figure 13: Color stream output without data compression

Data compression of color streams also has an impact on data capture rate

of the SBC. The data capture frame rate is reduced to 9 fps in odroid and 4 fps

in Lattepanda(originally 30 fps from Kinect). Utilization of background subtraction

further reduces the data capture rate and but provides a better compression result.

35

Figure 14 shows the results obtained by utilizing data compression and background

subtraction.

(a) network throughput with data compress-sion

(b) frame rate of color stream with data com-presssion

(c) color stream output with backgroundsubtraction and data compresssion

(d) network statistics for color stream outputwith background subtraction and data com-presssion

Figure 14: color stream output without data compression

36

5.3.3 Depth and Color data combination

Figure 15: depth and color image streams output without compression

Depth data and color(RGB image) data together sum up to 43.95Mbytes at a

frame rate of 30fps. When the same data is collected from two SBC’s then the size is

doubled equivalent to 2x43.95 Mbytes(703.125 Mbits). Two Odroids simultaneously

sending data to client are comfortably able to achieve 100% frame rate(30 fps),

where as single Lattepand is able to achieve only 8 fps. Figure 16 shows the network

throughput obtained from two odroids and Figure 17 shows throughput from single

Lattepanda.

37

Figure 16: network statistics for depth and color stream collected from two odroids

Figure 17: network statistics for depth and color stream collected from single Lat-

tepanda

38

5.3.4 Factors influencing SmartLab software performance

In order to analyze the SmartLab system, different performance metrics have

been tested by setting up a wired connection between the SmartLab server and client.

Firstly, network bandwidth of the system including the Ethernet specifications of

SBC and desktop client. Table 2 and Table 3 shows that Odroid is able to send

both color(RGB) and depth data to the client at 30 frames per second. On the

other hand, Lattepanda is able to achieve only 8 fps. The major difference between

these two boards is Ethernet port, Odroid has a Gigabit port where as Lattepanda

has 100Mbps port. Therefore, network bandwidth seriously effects the data transfer

rate and it is similar case with wireless network. Secondly, the data compression

rate on the server side also effects the SmartLab performance. Table 4 and Table 5

shows data compression results. From the results it is evident that data compression

effects the data collection rate of the server, because most of the computational power

is invested on compression. However, compression reduces the load on network as

shown in Table 5. In addition, to the aforementioned two factors processing power

and type of compression technique also effects the performance of SmartLab.

Note: All the experiments were conducted in wired network.

Table 2: Data capture and receiving rate without compression on Odroid and Lat-

tepanda

Odroid Lattepanda

Analysis Factor Capture rate(in fps)

Receiving rate(in fps)

Capture rate(in fps)

Receiving rate(in fps)

Depth 30 30 30 20

Color 30 30 30 13

Depth & Color 30, 30 30, 30 29, 27 8

Infrared 30 30 30 30

39

Table 3: Network throughput with out compression

Odroid Lattepanda

Analysis Factor data collected(In Mbps)

networkthroughput(In Mbps)

data collected(In Mbps)

networkthroughput(In Mbps)

Depth 140 153 140 98.7

Color 210 225 210 67

Depth & Color 351 370 328 85

Table 4: Data capture and receiving rate with compression on Odroid and Lat-

tepanda

Odroid Lattepanda

Analysis Factor Capturerate(In fps)

Receivingrate(In fps)

Capturerate(In fps)

Receivingrate(In fps)

Depth 17 17 4 4

Color 9 9 3 3

Depth & Color 5, 5 5, 5 2, 2 2,2

Color backgroundsubtracted

11 11 5 5

Table 5: Network throughput with compression

Odroid Lattepanda

Analysis Factor data collected(In Mbps)

networkthroughput(In Mbps)

data collected(In Mbps)

networkthroughput(In Mbps)

Depth 79 9.3 18.73 1.3

Color 63.21 55.5 21.09 7

Depth & Color 59 48 23.43 6.9

40

CHAPTER 6

CONCLUSION AND FUTURE WORK

The impact of Smart environments on human life is continuously increasing,

with the advent of low priced hardware. The research in the development of smart

environments is revamping. In order to realize our dream to have a smart environ-

ment in Pixel Island lab at Texas A&M University-Corpus Christi, the SmartLab

project has been started. As an initial step of this extensive project, we proposed

a novel and promising system for SmartLab infrastructure using Single Board Com-

puters and depth sensor like Microsoft Kinect. The main motivation to build this

system is to provide an infrastructure to the researchers in order to efficiently set up

data collection mechanism that can be employed in SmartLab applications such as

security surveillance, human behavior analysis, and smart interaction etc.

The SmartLab infrastructure is started with the focus on having a compact

and cost effective system for data collection. So, the experiments are conducted with

the pocket sized Single Board Computers to collect data from depth sensors like Mi-

crosoft Kinect. Another reason for using SBC’s is their cost effectiveness (typically

below $100). Although these SBC’s have less computation power as compared to

general purpose PC’s, our research demonstrated that they are a good enough for

developing smart environments. We experimented with a variety of Single Board

Computers such as Odroid XU4 and Lattepanda. The experimental results revealed

that Odroid outperforms both in wired and wireless connection. On the other hand,

Lattepanda provides acceptable frame rate for image processing or any other appli-

cations that use 3D data on a wired network.

We utilized multi threading principles and also avoided busy waiting using

41

lock-free buffers to overcome computational limitations of Single board computer.

This approach boosted the data capture rate from Kinect on the server side and also

increased the data retrieval rate on the client side. Moreover, Smartlab software is

also equipped with background subtraction as well as data compression mechanism

and the experimental results reveal that the aforementioned system reduces band-

width usage of each server. Additionally, in the SmartLab system, we also provide

an easy access to specific data that can be one of the fallowing data streams: depth,

color, infrared and body joints unlike other systems where we get everything together

and segregation has to be performed separately to get specific data stream. Overall,

the SmartLab infrastructure project created a reliable and scalable infrastructure for

data collection and processing.

6.1 Future work

The proposed design for creating infrastructure for smartlab can be improved

in multiple ways. In this section, the future work is discussed by highlighting the

addtional components that can be integrated to server module and also the improve-

ments that can be made to client module.

• Server Module: As a part of implementing server on a Single Board Com-

puter, different SBC’s are utilized specifically Odroid-XU4, Lattepanda and

Raspberry PI. Though all these Single Board Computers are efficiently used

for data collection and transferring, their capability is not completely exploited.

To utilize the complete computational power of SBC, all the server components

should run in parallel. At present, the server implementation has data acqui-

sition thread and network thread but it can be extended to have individual

thread for data compression as well as background subtraction. Having mul-

42

tiple threads also reduces the throughput as synchronization between threads

is expensive. However, it would be ideal to implement all the components in

parallel and compare the results with the existing system.

The data collected from the Kinect contains RGB, depth, infrared and

body joints. However, Microsoft Kinect has multi-array microphone, this en-

ables the server module to collect voice commands as sound signals. Implemen-

tation of sound component in server module is part of future research. The

Zlib compression library utilized for data compression on server side is compu-

tationally expensive, implementing a state of the art compression technique is

a necessary step in future.

• Client Module: In the client module, the interest is more about easy extensi-

bility of the project by fellow researchers. The client module is coded in C++

but it is possible to integrate Python code as well using Boost.Python [19].

Since Python is the widely used language for data analysis and machine learn-

ing, it would be more convenient to code when Boost.Python [19] is integrated

in to the client module.

43

REFERENCES

[1] Microsoft Kinect developer page. https://developer.microsoft.com/en-us/

windows/kinect. Accessed: 07-30-2017.

[2] Mario Martınez-Zarzuela, Miguel Pedraza-Hueso, Francisco Javier Dıaz Pernas,

David Gonzalez Ortega, and Miriam Anton-Rodrıguez. Indoor 3d video moni-

toring using multiple kinect depth-cameras. CoRR, abs/1403.2895, 2014.

[3] Asus Xtion specifications. https://www.asus.com/3D-Sensor/Xtion_PRO/.

Accessed: 07-30-2017.

[4] Structure sensor developer page. https://structure.io/. Accessed: 07-30-

2017.

[5] Jungong Han, Ling Shao, Dong Xu, and Jamie Shotton. Enhanced computer

vision with microsoft kinect sensor: A review. IEEE transactions on cybernetics,

43(5):1318–1334, 2013.

[6] Zoran Zivkovic. Improved adaptive gaussian mixture model for background

subtraction. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th

International Conference on, volume 2, pages 28–31. IEEE, 2004.

[7] Roberto Arroyo, J. Javier Yebes, Luis M. Bergasa, Ivan G. Daza, and Javier

Almazan. Expert video-surveillance system for real-time detection of suspicious

behaviors in shopping malls. Expert Systems with Applications, 42(21):7991 –

8005, 2015.

44

[8] Julie A Kientz, Shwetak N Patel, Brian Jones, ED Price, Elizabeth D Mynatt,

and Gregory D Abowd. The georgia tech aware home. In CHI’08 extended

abstracts on Human factors in computing systems, pages 3675–3680. ACM, 2008.

[9] Vincent Ricquebourg, David Menga, David Durand, Bruno Marhic, Laurent

Delahoche, and Christophe Loge. The smart home concept: our immediate

future. In E-Learning in Industrial Electronics, 2006 1ST IEEE International

Conference on, pages 23–28. IEEE, 2006.

[10] G Michael Youngblood, Diane J Cook, and Lawrence B Holder. The mavhome

architecture. Department of Computer Science and Engineering University of

Texas at Arlington, Techinal Report, 33, 2004.

[11] Maurizio Caon, Yong Yue, Julien Tscherrig, Elena Mugellini, and O Abou

Khaled. Context-aware 3d gesture interaction based on multiple kinects. In

Proceedings of the first international conference on ambient computing, applica-

tions, services and technologies, AMBIENT, pages 7–12. Citeseer, 2011.

[12] Lattepanda. [Online; accessed 21-July-2017].

[13] Odroid xu4 - hardkernel. [Online; accessed 21-July-2017].

[14] Openni 2.2. [Online; accessed 21-July-2017].

[15] Nite software. http://openni.ru/files/nite/index.html. Accessed 21-

July-2017.

[16] Zlib manual. https://www.zlib.net/manual.html. Accessed: 07-30-2017.

[17] Libfreenect. Accessed 21-July-2017.

45

[18] Microsoft kinect sdk. Accessed 21-July-2017.

[19] Boost library. [Online; accessed 21-July-2017].

46

APPENDIX A

HOW TO SET UP SMARTLAB ENVIRONMENT

Setting up the smartlab environment is a two stage process. Firstly, we need to setup

the server environment and subsequently the client. From the architecture diagram

2, it is noticeable that smartlab server runs on Single-Board Computer whereas the

client is meant to be run on a general-purpose PC. Smartlab server is designed in

such a way that it can run on both Linux and Windows operating systems. In this

section, we will look at setting up the environment for the smartlab server in both

Linux and Windows.

7.1 Environment for SmartLab server (Linux Environment- ODROID XU4)

This section walks through the steps involved in preparing the environment

for a smartlab server on a Linux Single-Board Computer. ODROID XU4 is a Sin-

gle Board Computers with ARM architecture, It has an arm-cortex-a7 processor

with 2.0GHz clock speed. Odroid supports Debian distributed operating systems

(Ubuntu) as well as Embedded operating-system (Android). For developing smart-

lab infrastructure we have selected Ubuntu 15.04 with Mate desktop as our host op-

erating system, all the additional dependency libraries required for running smartlab

server are listed below:

• OpenCV 3.2

• OpenNI 2.2 (Libfreenect or Primesenese driver required)

• Libfreenect(for Kinect Xbox 360)

• Boost 1 63 0 (For Boost.asio network library)

47

7.1.1 Ubuntu Image

Firstly download the Ubuntu image from this link. Burn the image into an

SD card with Win32DiskImager(or any other similar software). Insert the SD card

into ODROID and power it on, if it prompts for password type ’odroid’. Connect

the odroid to internet. The very first step is to install the necessary software with

the following commands.

#remove the unwanted so f tware which comes with the Ubuntu imagesudo apt−get updatesudo apt−get remove −−purge l i b r e o f f i c e ∗ plank simple−scan

sho twe l l imagemagick∗ pidg in hexchat thunderbird brase ro kodirhythmbox xzoom gnome−orca onboard a t r i l mate−u t i l s s eahor s et i l d a

sudo apt−get purge f i r e f o xsudo rm −r f ˜/ . moz i l l a / f i r e f o x ˜/ . macromedia ˜/ . adobe / e t c /

f i r e f o x / usr / l i b / f i r e f o x / usr / l i b / f i r e f o x−addonssudo apt−get c l eansudo apt−get autoremove

#Now i n s t a l l ext ra dependenc ies which are e s s e n s t i a lsudo apt−get updatesudo apt−get d i s t−upgradesudo apt−get i n s t a l l bui ld−e s s e n t i a l c h e c k i n s t a l l cmake cmake−

curses−gui pkg−c on f i g gparted guvcview lightdm−gtk−gree t e r−s e t t i n g s

7.1.2 Installing OpenCV

On the smartlab server OpenCV is required for image processing opera-

tions(eg: background subtraction) and in future, we can also select a region of interest

from the camera feed. Follow the steps to build OpenCV from source.

#I n s t a l l OpenCV dependenc iessudo apt−get i n s t a l l bui ld−e s s e n t i a lsudo apt−get i n s t a l l cmake g i t l i b g t k 2 .0−dev pkg−c on f i g

l ibavcodec−dev l ibavformat−dev l i b sws ca l e−devsudo apt−get i n s t a l l python−dev python−numpy l ib tbb2 l ibtbb−dev

48

l i b j p eg−dev l ibpng−dev l i b t i f f −dev l i b j a s p e r−dev l ibdc1394−22−dev

#Get Opencv from source ” https : // github . com/opencv/opencv/ arch ive/ 3 . 2 . 0 . z ip ” OpenCV 3 .2

cd ˜g i t c l one https : // github . com/opencv/opencv . g i tcd ˜/opencv#Build and i n s t a l l OpenCVmkdir bu i ldcd bu i ldcmake −DCMAKE BUILD TYPE=RELEASE −DCMAKE INSTALL PREFIX=/usr /

local −DWITHOPENGL=ON −DWITH V4L=ON −DWITH TBB=ON −DBUILD TBB=ON −DENABLE VFPV3=ON −DENABLENEON=ON . .

make −j 4sudo make i n s t a l l

7.1.3 Installing OPENNI

OpenNI provides access to PrimeSense compatible depth sensors such as Mi-

crosoft Kinect, Asus Xtion, etc. It allows an application to initialize the sensor and

obtain RGB, depth, and IR video streams from the device. OpenNI in coordination

with NiTE middleware provides Skeleton tracking. But NiTE libraries are not avail-

able for ARM architecture, so we can’t install NiTe on ODROID. The subsequent

steps help you to build OpenNI libs from source.

#I n s t a l l addt iona l dependenc ies f o r OpenNIcd ˜sudo apt−get i n s t a l l −y g++ python l ibusb −1.0−0−dev l ibudev−dev

openjdk−6−jdk f r e e g l u t 3−dev doxygen graphviz#Get OpenNI from source https : // github . com/ o c c i p i t a l /OpenNI2

OPENNI 2.2</a>g i t c l one https : // github . com/ o c c i p i t a l /OpenNI2cd OpenNI2#Build OpenNI l i b r a r i e s f o r ARM ar ch i t e c t u r ePLATFORM=Arm makecd Packaging && python Re leaseVers ion . py Armmv Fina l /OpenNI−Linux−Arm−2.2 . ta r . bz2 ˜cd ˜

49

ta r −xvf OpenNI−Linux−Arm−2.2 . ta r . bz2rm −r f OpenNI2rm OpenNI−Linux−Arm−2.2 . ta r . bz2#I n s t a l l OpenNIcd OpenNI−Linux−Arm−2.2sudo sh i n s t a l l . sh

7.1.4 Installing Libfreenect

Libfreenect is a userspace driver for the Microsoft Kinect. It runs on Linux,

OSX, and Windows and supports RGB and depth Images, Motors, Accelerome-

ter and LED. Libfreenect is open source driver and interacts with sensor raw data

streams, this driver acts as an underlying layer between Kinect hardware and high-

level layer OpenNI. Here are the instructions to build Libfreenect in Linux environ-

ment

#I n s t a l l dependenc iescd ˜sudo apt−get i n s t a l l libxmu−dev l i b x i−dev l ibusb−dev#Fetch the codeg i t c l one https : // github . com/OpenKinect/ l i b f r e e n e c tcd l i b f r e e n e c t#Build l i b r a r i e scd l i b f r e e n e c tmkdir bu i ldcd bu i ldcmake . . −DBUILD OPENNI2 DRIVER=ONmake −j 4

#Copy L ib f r e en e c t d r i v e r to OpenNI d i r e c t o r yRepos i tory=˜/OpenNI−Linux−Arm−2.2/ Redist /OpenNI2/Dr iver s /cp −L l i b /OpenNI2−FreenectDr iver / l i bFr e ene c tDr i v e r . so ${

Repos i tory }

#Copy Kinect r u l e s to user devsudo cp ˜/ l i b f r e e n e c t / plat form/ l inux /udev/51−k ine c t . r u l e s / e t c /

udev/ r u l e s . d

50

7.1.5 Installing Boost

Boost is a set of libraries for the C++ programming language that provides

support for multi-threading, network communication, regular expressions, and com-

pression. It contains over eighty individual libraries. Install boost by following this

link

7.2 Environment for Smartlab server (Windows Environment-Lattepanda)

Lattepanda is x86 Single board computer and it is able to run windows 10

operating system. This SBC provides the advantage of utilizing Microsoft driver for

accessing Kinect. It has 1.8GHz clock speed and 2GB of DDR3 RAM and 100Mbps

Ethernet port. Windows board is specifically chosen to analyze the advantage of

using Microsoft SDK. The fallowing are the set of libraries that are to be installed

on Lattepanda. Another advantage for choosing x86 SBC is to obtain Skeleton joints

using NiTE middleware.

• OpenCV 3.2

• OpenNI 2.2

• NiTE 2.0

• Microsoft Kinect SDK 1.8(driver for Kinect Xbox 360)

• Boost 1 63 0 (For Boost.asio network library)

7.3 Building SmartLab server in Linux

SmartLab server is build in linux using cmake, and executable is generated.

Before creating the executable make sure that all the dependencies are configured

51

and appropriate paths are set. There after compile the code link the binary with all

dependencies. The following CMakeLists.txt file illustrates the flow of command for

building SmartLab server.

p ro j e c t ( smar t l ab s e rv e r )cmake minimum required (VERSION 2 . 8 )# CONFIGURE OPENCVf ind package (OpenCV REQUIRED)# CONFIGURE BOOSTf ind package ( Boost REQUIRED COMPONENTS system thread io s t r eams )# CONFIGURE OPENNI2f i n d l i b r a r y (OPENNI2 LIBRARY

NAMES OpenNI2PATHS ”˜/OpenNI−Linux−Arm−2.2/ Redist ”

)f i nd path (OPENNI2 INCLUDE DIR OpenNI . h

PATHS ”˜/OpenNI−Linux−Arm−2.2/ Inc lude ”)# CREATE EXECUTABLEl i n k d i r e c t o r i e s ( ${OPENNI2 LIBRARY})i n c l u d e d i r e c t o r i e s ( ${OPENNI2 INCLUDE DIR} ${Boost INCLUDE DIRS})f i l e (GLOB RECURSE SRC FILES s r c /∗ . cpp )add executab le ( ${PROJECTNAME} ${SRC FILES})t a r g e t l i n k l i b r a r i e s ( ${PROJECTNAME} ${OPENNI2 LIBRARY} ${

OpenCV LIBS} ${Boost LIBRARIES})

7.4 Building SmartLab server in Windows

In windows platform SmartLab is build using Visual studio and all the includes

and library dependencies are configured in in Project solution. The executable can

be moved into another system and it can be executed perfectly fine. But, OpenNI

headers and library are required along with the executable

52

7.5 Running SmartLab server in Linux

Initializing the server script on SBC startup is easy way to run smartlab server.

To make a startup script, create a shell script mystartuo.sh with following contents

under /etc/init.d/ directory and provide executable permissions. Also, create an-

other script named startupscript.sh under home directory and provide executable

permissions. Start the mystartup service with the instructions shown in the listing.

########## / etc / i n i t . d/mystartup . sh #############!/ bin /bashecho ” S ta r t i ng smartlab s e r v e r ”# smar t l ab s e rv e r s c r i p t path/home/ odro id / s t a r t up s c r i p t . sh

######## s t a r t up s c r i p t . sh ###########!/ bin /bashecho ”Smartlab s e r v e r s e s s i o n s t a r t ed ” > ˜/ s t a r tup l og . txtdate >> ˜/ s t a r tup l og . txt#Running the smartlab s e r v e r s c r i p t˜/ servermodule−l inux−arm/ smar t l ab s e rv e r >> ˜/ s t a r tup l og . txt 2>&1

#s t a r t your s e r v i c es e r v i c e mystartup . sh s t a r t

#I n s t a l l s e r v i c e to be run at boot−timeupdate−rc . d mystartup . sh d e f a u l t s

7.6 Running SmartLab server in Windows

In windows initiating the SmartLab server is a easy task, just copy the server

module executable to startup folder. Everytime when windoes SBC boots smart-

lab server is initiated.

53

7.7 SmartLab client

SmartLab is designed for desktop PC, and it is platform independent. On

windows platform SmartLab client application is compiled using Visual studio, where

as on linux platforms cmake is used. Building the executable in both platforms is

similar to that of server.

7.8 Running SmartLab client

For running SmartLab client module necessary details required are server IP

address and port number. Place these details in config.json file and also set the

selected data stream to true. The config.json file is shown below.

{”NETWORKCONFIG” :{

”SERVER1” :{”IP ADDRESS” : ” 1 2 7 . 0 . 0 . 1 ” ,”PORTNO” :2000

}} ,”VIDEO CONFIG” :{

”SOURCE CONFIG” :{”KINECT” : true ,”CAMERA” : fa l se

} ,”IMAGE CONFIG” :{

”BGR IMAGE” : true ,”IR IMAGE” : fa l se

} ,”DEPTH CONFIG” :{

”DEPTHMAP” : true ,”POINT CLOUDMAP” : fa l se

} ,”BODYJOINTS CONFIG” :{

”SKELETON” : true} ,”BACKGROUNDSUBTRACTION” :{

”RGB SUBTRACTION” : false ,”DEPTH SUBTRACTION” : fa l se

54

} ,”COMPRESSION” :{

”SELECTED” : fa l se}

}}

Save the config.json file in the same folder where executable is located. Run the

client executable to view the selected data stream.