Experience certainty.

Copyright © 2011 Tata Consultancy Services Limited

Signal Processing, Communication and Computing Aspects of Internet-of-Things - Research Challenges

Arpan PalHead of ResearchInnovation Lab, Kolkata

Research Theme

3 Experience certainty.

Human-in-Loop Cyber-physical Systems

Humans

Physical Objects

and Infrastruct

ure

Computing Infrastruct

ure

Perso

nal

Conte

xt

Disco

very

PhysicalContext Discovery

4 Experience certainty.

Signal

Processing

Internet-of-Things - towards Intelligent Infrastructure

Sense

Extract

Analyze

Respond

Learn

Monitor

IntelligentInfra

@Home

@Building

@Vehicle@Utility

@Mobile

@Store

@Road

“Intelligent” (Cyber) “Infrastructure” (Physical)

APPLICATION SERVICES

BACK-END PLATFORM

INTERNET

GATEWAY

Internet-of-Things (IoT) Framework

Sense

Extract

Analyze

Respond

Communication

Computing

5

Integrated Platform for Intelligent Infrastructure

People Feedback & Emotions

Social Media

Integrated Services

Sensors & IoTPlatform

Traditional Monitoring & Control Systems Citizen Data

Smart Integration Platform

Transportation Healthcare Electricity

WaterPublic Safety Tourism

Smart Integrated Services

Sense

Analyze

Extract

Respond

Intelligence

Smart Domain Services

Community

etc.

Sense: People Activity, Appliances, Vehicles , Road, Home/Bldg, Utility Infrastructure

Detect gas leakage/water contamination : mobilize rescue team, suggest optimum route

Divert Road Traffic in case of Water Pipeline Burst

Correlate Electricity/Water /Gas consumption patterns

Intelligent Integration Platform

Integrated Intelligent Services

RIPSAC – Real-time Integrated Platform Services & Analytics for Cyber-physical Systems

6

Context Discovery

Physical Context Discovery(What is happening where and when)

• Localization and Spatiotemporal data fusion

• Sensor Informatics (Noise Cleaning, Disaggregation, Feature Extraction, Machine Learning)

• Semantic Interop. and Analytics of Sensor Data – Semantic Sensor Web

• Model driven Analytics – CPS modeling

• New modes of Sensing (5 senses computing)

• New Platforms for Sensing (UAV, Mobile Robots, Mobile Phones – Participatory Sensing)

7

Context Discovery

Human Context Discovery(Who is doing what, where and when, who is thinking what)

- Identity, Location, Activity, Physiology, Psychology

• Unobtrusive Sensing (Mobile phones and Surveillance cameras – Kinect)

• Non-invasive Physiological Sensing

• Psychology sensing via EEG

• Model driven Analytics - Human behavior modeling

• Sensing from Social Networks

8

Challenges

Signal Processing• Preserving the Battery power of edge devices• Extracting information from ambient-noise-corrupted sensor data

Low-power computing algorithms Adaptive Signal Processing for Ambient Noise Removal

Communication• Reducing the cost of Communication while preserving reliability

and security Secure, Lightweight yet Reliable communication protocols over

Internet• Preserving the Privacy

Quantifying Privacy vs. Utility measures

Computing• How to do big data analytics in Cloud• Distributed Computing - Bringing Edge into the Grid• Semantic Model of Physical World and Human World – Human-in-

loop CPS

Application Use Case – Unobtrusive Personal Context Discovery

10

Ubiquitous Healthcare – Elderly / Chronic Patient Monitoring

ECG

Blood PressureMonitor

Pulse OxyMeter

Healthcar

e Portal

Mobile phone as medical

gateway

Web Request

PatientRecords

Health Center / Home

Expert Doctor

Social Network

“Sensor Observation Service based Medical Instrument Integration”, SMART 2012

Requirement from NUH (Peritonial Dialysis) and SingHealth (Elderly Care)

11

Ubiquitous Healthcare – Research on Frugality and Usability

Healthcar

e Portal

Mobile phone as medical Sensor and

Gateway

Web Request

PatientRecords

Health Center / Home

Expert Doctor• Replace Medical Sensors with Mobile Phone

Sensing• Activity and Localization using Mobile Phones• Activity and Identification using Kinect Camera• Social Media as a Soft sensor• Multimodal Fusion

Social NetworkIdentification Localization

Activity DetectionPhysiological Sensing

“Mobile Healthcare Infrastructure for Home & Small Clinic”, Mobilehealth @ Mobihoc 2102

12

PPG based Pulse Measurement using Phone Camera

Subject1 Subject2 Subject3

Actual Detected Actual Detected Actual Detected

68 66 66 63 85 84

2.9% 4.5% 1.1%

Sources of noise:i. improper finger

placementii. imparting

excessive pressureiii. finger movement

Challenge: too much noise

13

PPG based Pulse – Proposed Robust Algorithm

Rejection FSM– Detect onset of good signal– Continuous consistency check– Reject unusable input data

o Feedback: notify user– Once enough signal received,

perform FFT

“A Robust Heart Rate Detection using Smart-phone Video”. MobileHealth @ Mobihoc 2013

14

Blood Pressure and ECG Monitoring from PPG

Extract PPG Features

BP Ground Truth Create BP Model

Extract PPG Features

Predict BP levels

(SP, DP, PP)

Extract ECG Features

Create ECG Parameters’ Model

Predict ECG Parameters

ECG Parameters

BP levels(SP, DP, PP)

Training Phase

Testing Phase

Data set Pd Ps PP-diff < 15

Standard dataset (14 features) 92.9% 74.7% 77.9%

TCS dataset - add height, weight, age

99.3% 82.7% 85.5%

BP Level

Pd Ps

Very Low < 50 < 70

Low 50-65 70-100

Normal 65-90 100-135

High 90-100

135-160

Very High

> 100 >160

“Estimation of Blood Pressure Levels from Reflective Photoplethysmograph using Smart Phones” – IEEE BIBE 2013“Estimation of ECG Parameters using Photoplethysmography” – IEEE BIBE 2013“HeartSense – Estimating Blood Pressure and ECG from Photoplethysmograph using Smart Phones” – Demo @ Sensys2013

15

Mobile Phone based Activity Detection for Wellness

Activity Detection– Uses Accelerometer Data– Gyroscope and Magnetometer for orientation

correction– Step Count, Stride Length Estimation– Walking, Brisk Walking, Running

Classification

Peak Detection and Step Validation using IPA;Calculating Step cycle lengths for all valid steps in the window

Classification of window activity using step frequencies derived from step cycle lengths

16

Mobile Phone based Activity Detection for Wellness

Noise Cancellation and pre-processingCalorie Count from Step Count and Type of Activity

UbiHeld - Ubiquitous Healthcare Monitoring System for Elderly and Chronic Patient”, Recognize2Interact @ UbiComp 2013

17

Mobile Phone based Indoor Localization – Geo Fencing and Proximity

Indoor Localization– Initial Referencing through GPS and Magnetometer– Inertial Navigation through accelerometer, gyro,

magnetometer – Improved accuracy through Stride Length, Kalman Filter /

Particle Filter– Tracking of non-smartphones via Bluetooth– Augmentation with Wi-Fi Triangulation

Location ID LOC I LOC II LOC III

Actual (ft) 2 4 6

Estimated (ft) 1.95 4.16 6.27

% Error 2.5 4.12 4.4

“BlueEye A System for Proximity Detection Using Bluetooth on Mobile Phones”, PUCAA @ �UbiComp 2013

Geo-fencing through Magnetometer

Proximity Sensing via Bluetooth

Based on RSSI Done by capturing RSSI in android in respect to

another phone Able to model the constants in equation as a

function of distance in given environment

18

Inertial Navigation– Step Count + Stride Length (personalized model)– Gyroscope and Magnetometer-corrected Inertial Navigation

Wi-Fi based Triangulation– Based on RSSI of known location of 3 or more access points– Attenuation modeling of the building

Fusion, Tracking and Correction– Kalman Filter based Tracking– Particle Filter based Correction

Mobile Phone based Indoor Localization – Inertial and Wi-Fi

Other Requirements• Colleague Finder in Large Offices• Shopper Localization in Retail Stores• Emergency Evacuation in Large Buildings

19

Kinect Based Human Identification

Human Identification– Skeleton Model Based / Depth

based– 20 joints of skeleton data for a

person captured at 30 frames per sec in side way walking pattern

• 2D Camera with IR depth sensor

• Excitation by IR light pattern

• Directional Mic.

• Human Identification • Gait cycle detection• Feature extraction from

skeleton joints• Training• Recognition

“Pose Based Person Identification Using Kinect”, IEEE SMC 2013“Stabilization of Cluster Centers over Fuzziness Control Parameter in Component-wise Fuzzy C-means Clustering”, Fuzz IEEE 2013“Feature Selection by Differential Evolution Algorithm - A Case Study in Personnel Identification”, CEC 2013

20

Kinect Based Activity Detection

H H NH NH

Human and non-Human Classification

“Human Localization from Kinect Captured Data for Activity Recognition at Home”, HomeSys @ Ubicomp 2013

21

Social Media as a Soft-sensor for Healthcare

Support Community Discovery– Use hidden community detection by applying

NLP on posts to create the social graph to identify the undeclared community for a given disease

Disease onset discovery like dementia or Alzheimer's disease or psychological disorders from social network posts– Search for patterns in posts to detect possible

symptoms to diseases– E.g. - sentiment analysis on posts will give

whether the given post’s emotion is positive or negative. If the emotions are cycling between positive and negative extremes with some periodicity, probably the person has bipolar disorder

“Using Social Network Graphs for Search Space Reduction in Internet of Things”, Ubicomp 2013

Prototyping in TCS own internal social network platform - Knome

Application Use Case – Human Cognitive Load Detection

23

Cognitive Load on Human Brain

Add in your mind:

23+45=?

1846890129 + 2374609823=?

How to Measure Cognitive Load

User Study Can be biased and it is Indirect

measurement

Biological Response Unbiased & more reliable method

ECG Pupil diameter Skin conductivity, other EEG (this is a more direct way as it

measures brain activity directly) Cheaper that fMRI, PET and other brain

activity measurement means

Application Personalize education based on

real-time measurement of one’s cognition state through EEG signals

Possibly a better measure for testing understanding of a subject during a course during taking a test

Getting unbiased feedback from subject on user interface design

Stress during critical operation like ATC.

24

Two class Cognitive Load – User Experience Testing

• People are given two different types of Onscreen Keyboards to use (one with easy and other is more complex to use for text entry)

• Corresponding EEG is recorded ((14 channel Emotiv).

Results shows clear classification of these two cases based on measuring cognition while subject is using a particular keyboard

Constraints: • Subject training required• Only two level classification is addressed

“Evaluation of Different Onscreen Keyboard Layouts using EEG signals”, SMC 2013“Unsupervised Approach for Measurement of Cognitive Load using EEG Signals”, BIBE 2013

25

Multi class Cognitive Load – Task Difficulty Testing

• People are given to analyze pseudo codes of three different difficulty levels (Low, Medium and High).

• EEG response is recorded (14 channel emotiv device).

Results shows clear formation of clusters each for corresponding task difficulty level with some overlaps.

Constrains: Multi class approach ( no continuous score ) Subjects are heavily constrained while they work out

Easy Task Difficult Task

“EEG-Based Fuzzy Cognitive Load Classification”, Fuzz IEEE 2013

Application Use Case – 3D reconstruction from 2D images from mobiles

27 Experience certainty.

5 senses Computing - 3D Reconstruction with 2D images from mobiles

• Low cost solution for 3D reconstruction from multiple 2D images captured from mobile device.

• Derive the motion information from the inbuilt sensors of the mobile phone and then aid in increasing the accuracy of the 3D reconstruction.

• Support heterogeneous and homogeneous objects • Future research focus on the multi-modal fusion of the 3D information

and the other sensors’ data for a given object or environment

Applications• Agro-advisory Service• Remote Diagnostics of Machines• Remote Healthcare

IoT Platform Enhancements – Communication and Computing

29

Communication - Constrained Application Protocol (CoAP)

– Provides RESTful web interface suitable for constrained devices

– Ideally to run on unreliable transport (e.g., UDP)– Confirmable (CON) mode for optional reliability with an

ACK feed backo Retransmissions ensure best-effort delivery

– Non-confirmable (NON) mode for unreliable deliveryo No ACK – no retransmission

– Supports HTTP like request/response– Supports resource-observe

o A variant of publish/subscribeo Useful for real-time updates

– Supports both unicast and multicast

30 Experience certainty.

CoAP vs. HTTP

Constrained Object Access Protocol (CoAP) Improved version of CoAP using dynamic network condition sensing Lightweight security protocols sensor authentication and data delivery

Use suitable lightweight application protocol between edge devices and core network

• http://people.inf.ethz.ch/mkovatsc/californium.php• Ralf Koetter, Muriel Medard, 2003 IEEE/ACM transaction http://web.mit.edu/medard/www/NWCFINAL.pdf• Bandyopadhyay, S. and Bhattacharyya, A. Lightweight Internet protocols for web enablement of sensors using constrained gateway devices. In Proc.

International Conference on Computing, Networking and Communications (ICNC), 2013, San Diego, CA, IEEE(2013), 334 – 340

31

Fog Computing – the Grid with an Edge

• Flavio Bonomi et.al. MCC2012, Helsinki, Finland

Intelligent Systems - Intelligence comes from Analytics Need for crunching huge amount of sensor data and respond in real-

time Needs large computing infrastructure in cloud Another option is to distribute computing load to the edge devices

Edge Devices computing power remain unused most of the timeFree Computing resource for the gridPotentially millions of ~1GHz Processors on the grid depending upon use case

Energy cost at edge is typically at consumer rates << Energy cost at cloud which is at Enterprise ratesEnergy cost account for 50% of Data Center Opex

32

Fog Computing - Solution Approach

• Agent-based grid Computing using CONDOR• Need for agents in diverse types of edge devices via a common

framework

• Min-Jen Tsai, ,Yuan-Fu Luo , Expert Systems with Applications, Volume 36, Issue 7, Sept. 2009, Elsevier

33

Privacy – Smart Meter Data

Activity monitoring Advantage: Personalized services and recommendation like theft

detection, elderly monitoring (University of Virginia’s ALARMNET, Harvard’s CodeBlue)

Privacy issue: Leads to private data (smart meter data) leakage

[1] www.winlab.rutgers.edu/~gruteser/papers/fp023-roufPS.pdf

[1]

34 Experience certainty.

Privacy Preservation

• Information-theoretic approach for sensitivity analysis of sensor data• Requirement based application utility measurement• Balancing of Privacy vs. Utility

Privacy

Utility

Privacy Preservati

on Tool

35 Experience certainty.

Results: Sensitivity Analysis and Detection

[5] J. Zico Kolter and Matthew J. Johnson, "REDD: A public data set for energy disaggregation research," SustKDD, 2011.

[5]

36 Experience certainty.

Semantic Query and Analysis on Spatiotemporal Sensor Data

Analytics Engine

Time Series

Database

RIPSAC

Sensor Manufacturer

Sensor KnowledgeDatabase

(Domain and Resource Ontology)

Provision Sensors & Actuators

Algorithm Catalog

Algorithm(s) Selection &Execution

Raw Sensor Data

App Developer

Query on Sensor Property & Capability

User

Algorithm DiscoveryRegister Sensor

Specification Metadata

Sensor Installer

Sensors & Actuators

Algorithm provider

Concept Developer

Algorithm Registration

Algorithm Instantiation

Innovation @ TCS

38

Innovation@TCS - Innovation Labs

Bangalore, India1

TCS Innovation Labs - Bangalore

Chennai, India2

TCS Innovation Labs - ChennaiTCS Innovation Labs - RetailTCS Innovation Labs - Travel & HospitalityTCS Innovation Labs - InsuranceTCS Innovation Labs - Web 2.0TCS Innovation Labs - Telecom

Cincinnati, USA3

TCS Innovation Labs - Cincinnati

Delhi, India4

TCS Innovation Labs - Delhi

Hyderabad, India5

TCS Innovation Labs - HyderabadTCS Innovation Labs - CMC

Kolkata, India6

TCS Innovation Labs - Kolkata

Mumbai, India7

TCS Innovation Labs - MumbaiTCS Innovation Labs - Performance Engineering

Peterborough, UK8

TCS Innovation Labs - Peterborough

Pune, India9

TCS Innovation Labs - TRDDC - Process EngineeringTCS Innovation Labs - TRDDC - Software EngineeringTCS Innovation Labs - TRDDC - Systems ResearchTCS Innovation Labs - Engineering & Industrial Services

1 2

3

4

597

6

8

2000+

Associates in Research, Development and Asset Creation

19 Innovation Labs

39

Academic Co-Innovation Network (COIN )

Fostering joint research and innovation through a mutually beneficial alliance between TCS and academia

Academic context

Thoughts and research towards disruptive InnovationKnowledge exchange and people development

Industry-oriented Business context

innovation scalability of academia context of real-world problems

Collaborativeresearch

environment

Collaboration Mechanisms•MoU based Alliances•Sabbaticals – Academia to TCS Innovation Lab and TCS Innovation Lab to Academia•TCS Research Scholar Program•Masters and PhD Internships

Joint publications and IPRs

40

Innovation Lab Kolkata, at-a-Glance

Research Areas• Sensor Signal Processing• 2D/ 3D Image / Video

Processing • Protocols, Security and

Privacy• Parallel and Distributed

Computing• Stream Processing and

Reasoning• System Modeling and

Identification• Sematic Sensor Web• Social Media Analytics

Academic Collaborations• Singapore Management University (iCity

Platform)• Indian Statistical Institute

(Protocol/Privacy/Security, Image / Video Processing)

• IIT Kharagpur (Analytics, Semantic Processing)

• IIT Bombay (Energy and Utilities)• Jadavpur University (Signal Processing)

Higher Studies

• PhD - 4• Masters - 4

Total Researchers - 37

Thank You

arpan.pal@tcs.com