5º MeetUP ARQconf 2016 - IoT: What is it really and how does it work?

©2014 GlobalLogic Inc. All Rights Reserved.

Juan Caracoche

[email protected]@jcaracoche

2 ©2015 GlobalLogic Inc. All Rights Reserved.

IoT: What is it really and how does it work?


Similar quote attributed in various forms to Nobel prize-winning physicists Albert Einstein and Richard Feynman

“If you can’t explain something to an intelligent twelve year old, you probably don’t understand it yourself.”


Lego Mindstorm Robot courtesy of TM & © 2015 Marvel & Subs.


+ “Brain”

+ Sensors

+ Actuators


“Brain”

Cloud


+ Sensors “Brain”

Cloud


+ Sensors “Brain”

Cloud

+ Actuators

9

IoT Architecture Paradigm


Immediate Actions (Alerts, Actuators,

Notifications)

End User and Management Clients

Complex Actions (Workflow)

Resources Platform Applications

External Federated Data

Context

Historical Data

Deep Insights

Fast Decision Making

Observations and Requests

Connected Devices and Systems

Action Triggers

ActuatorsSensors

Sensors

Brain Actuators

10

IoT Architecture Paradigm



Notifications)


Observations and Requests

Sensors


Connected Devices and Systems

Action Triggers


Actuators

Resources Platform Applications

Context

Historical Data

Deep Insights


11

The Internet of Things


A set of technologies that connects data from a variety of sensors with the ability to make contextually aware decisions and act on them in near real-time

12

IoT Architecture Paradigm: Sensor Technologies


ResourcesSensors

Sensors

• Alljoyn• MQTT • HomeNET (Apple) • Thread (Google) • EMIT (Panasonic) • Electric Imp • CCNx (PARC) • OIC / IIC (Intel)

Sensors

13

Speed-layer Technologies: • Spark Streaming • Storm • Samza• IBM Streams• Esper• Sidhi CEP • AWS Kinesis (Cloud based)

IoT Architecture Paradigm: Analytics Technologies



Context

Historical Data

Deep Insights


Brain

Batch-layer Technologies: • Hadoop Map-Reduce (Cloudera, Hortonworks,

MapR ) • HPCC• Spark• Hive• Spark SQL• Pig• Analytic Databases (MarkLogic, Vertica, Infobright)

Fast Decision Making Deep Insights

14

• Impala• VoltDB• ElephantDB• SploutSQL• Hive on Spark• Hbase+Phoenix• Couchbase• MongoDB

IoT Architecture Paradigm: Analytics Technologies



Context

Historical Data

Deep Insights


Brain

• Oracle Data Federation• Redhat Data Virtualization• Mule ESB + Custom code• EspressoLogic

View Layer

Data Federation

15

IoT Architecture Paradigm: Applications Technologies



Notifications)



ApplicationsConnected Devices

and SystemsActuators

Actuators

Technologies: • MBaaS (Mobile Backend as a service• mPaaS / aPaaS systems• Workflow and Rules systems• Pub / Sub• Mobile app frameworks• JS / Browser apps• Sensors / “M2M” Technologies listed

Actuators and Applications

16 InternetofThings

Architectural Challenges And Design Considerations

17 CONFIDENTIAL

Architectural Challenges

InternetofThings

• Scalability

- Billions of devices and ZBs of data

• Interoperability

- Heterogeneous systems and distributed resources

• Extensibility

- New things and new opportunities

• Resiliency

- Network partition and system errors

• Performance

- Near real-time processing and high processing

• Security

- Privacy and Deceit

• Maintenance

- Resource and Platform

18 CONFIDENTIAL

Design Considerations

InternetofThings

• Identity Management

- Identity of things and identity of people

• Context awareness

- Location and identity

• Data generation mode

- Event driven and time driven

• Resource capability

- Processing power and memory size

• Forward and backward capability

19 CONFIDENTIAL

Reference Architecture and Platform

InternetofThings


z

z z

M2M, Devices, and Intelligent Gateways


Historical Information

Notifications

Deep Insights Complex Actions

Platform Functions



©2015 GlobalLogic Inc. CONFIDENTIAL

Embedded systems in IoT

CONFIDENTIAL21 Internet of Things

• Secure communication capabilities

• Interoperability means talking to each other

• Sensing/interacting embedded technology

IoT = network of physical objects

22 InternetofThings

Real world interaction is provided by sensors and actuators

• Light, pressure, gyro, movement, temperature, etc. sensors

• MEMS, motors, valves, relays, etc.

• Machine vision, human interaction and other systems

Communication capabilities are provided by protocols

CoAP

Well-defined and secure interoperability means:

• AAA, provisioning and network management

• Inter-system capabilities negotiations and configuration

• Cloud/edge/fog optimal work distribution

• Security, security and also security

23 InternetofThings

• SSL/TLS based security (certificates)

• Flexible transport plugins –AllJoyn or MQTT or REST or …

• Transport-independent initial provisioning mechanism

• Capabilities negotiation between all endpoints based on GUIDs

• “Edge” and “Fog” computing concepts support

Security Transport

AllJoyn MQTT REST …

Pro

visi

onin

g

Capabilities negotiation

IFTTT rules & computing

Data flow

Vision of embedded systems in IoT

24 CONFIDENTIAL

Definitions

InternetofThings

• Sensor/Actuator- Pressure sensors, camera, stepper motor, etc.

• Controller-Collect, process and possibly store data,

implement some custom IFTTT or other logic-Can have a set of Sensors or Actuators

connected to it-Can talk to other Controllers or M2M Gateway

• M2M Gateway- Provides Internet access to one or some

set of Controllers- Can be smart enough to implement some

custom IFTTT or other logic

25

Device Gateway PlatformController M2M Gateway

Requestcertificate signing

M2M transport connection

Sign GW certificate

with Private Key

Create private keyby serial

Createplatform certificate

Returnsigned certificateCreate private key

by serial

Request certificate signingSign CTL certificate

with Private Key

Return signed certificate

Controller-Gateway certificate exchange(Transport authentication mechanism)

Device registration process

Gateway and Controller are authorized by the Platform, secured connection is established

Prov

isio

ning

26

Device Gateway PlatformController M2M Gateway

Authentication

Check certificate

authorize

data processing actions

Connection → announce DeviceInfo + certificate

Device Authorized, set ResourceID

getData events

setData eventsMessage loop

Announce capabilitiesUpdate Device Info

AA

A


Platform Core

28 CONFIDENTIAL


InternetofThings


z

z z




Notifications


Platform Functions



29 CONFIDENTIAL

Platform Core | Cloud

InternetofThings

Heterogeneous Data

Large Volume

Near Real-Time Processing Security Notifications

30

Heterogeneous Data Solutions

Internet of Things

• Transform any format of data to ONE format

• Apply basic rules

• Cache & Aggregate data

• Platform Gateways

31

Large Volume Solutions

Internet of Things

• Fast Hundreds of MBs of RW per second, serialize/deserialize

• Scalable Elastically and transparently expanded without downtime

• Durable persisted and replicated to prevent data loss

• Distributed fault-tolerance guarantees

32

Near Real-Time Processing

Internet of Things

• Low Latency persisted and replicated to prevent data loss

• Scale High throughput and scale-out by adding instances

• Dynamic Change the rules dynamically using API interface

• Complex detecting relationships between events

33

Security Solutions

Internet of Things

• Privacy Multi tenancy and data sharing

• Security Policies apply security policies on the data before storing

• In Memory Process non-persistable data in memory

• Transport Secure data transfer to the DMZ


IoT Analytics

35 CONFIDENTIAL


InternetofThings


z

z z




Notifications


Platform Functions



36 InternetofThings

The Internet of Things Analytics(IoTA) , the measurement and transformation into business intelligence of the Internet of Things

Data Paradigms

Volume application and transactional data

Velocity Machine VarietySocial Data

37 CONFIDENTIAL

IoT Analytics Challenges

InternetofThings

• Data Integration– Aggregation and integration of the collected data streams in a manner that makes them suitable for

analysis.

– Key challenge in dealing with more diverse and disparate data sources before meaningful analysis is viable.

• Data Storage– Implementation and management of the data store for the analytics process.

– Key issue to address in this context is how to store time-series sensor data, which can increase dramatically in volume compared to e.g., transactional data readings

• Different components of IOT analytics

38 CONFIDENTIAL

IoT Analytics Challenges (Cont’d)

InternetofThings

• Core Analytics– Processing of the data by an analytics engine and the subsequent delivery of insights.

• Data Presentation – Further presentation of the delivered analytical insights to the end-user.

– In IoT analytics, the geography of data is a particularly important presentation element, given that the location of physical things matters more to analysis than, say, the location of an ecommerce transaction.

• Different components of IOT analytics

39 CONFIDENTIAL

Enabling IOT Analytics using Lambda Architecture

InternetofThings

Components of Lambda Architecture

new data

batch layer serving layer

speed layer

Query

Query

master data set

batch view

batch view

real time viewreal time view

40

Technology Stack

InternetofThings

Component Technology Choices

Rationale

Batch View Generator (Batch Layer)

Hive on Spark , Impala

•Queries, especially those involving multiple reducer stages, runs faster.• Can anytime move to using only Hive by just a JDBC connection parameters change.• Easy of use than Map Reduce and other processing framework.• Support SQL by default.

Real Time View Generator (Speed Layer)

Volt DB , Spark Streaming ,Storm

•Processes data fast by keeping it in memory and runs on commodity hardware. •Data can be ingested directly from Kafka queues so would be on low latency as there is no intermediate layer between data ingestion.

Consolidate View Generator (Serving Layer)

Hbase + Phoenix , Couchbase

•Is easy to use and provides real time read and write.•Good for time series based evaluation.• Phoenix is used to provide SQL support over hbase

Analytics Engine(Statistical programming language)

R , Mahout •Gives wide variety of statistical and graphical algorithm.•It has linear and non linear modeling, time series analysis, classification, clustering etc.

Real time Machine Learning (Machine Learning Library)

SparkML •Distributed machine learning library.• Has support for common algorithms.

Data Ingestion Layer (Messaging System)

Kafka , RabbitMQ •High Throughput , data can be pulled at consumer's own pace.•Can replay any messages or set of messages given the necessary selection criteria.

41

IOT Analytics Industry Trends

InternetofThings

Informatica Vibe Data Stream for Machine Data

ParStream IOT Analytic Framework

Source- http://www.informatica.com/in/products/big-data/vibe-data-stream/, https://www.parstream.com/

42

IOT Analytics Industry Trends

InternetofThings

Source- http://voltdb.com/


Applications in IoT

44 CONFIDENTIAL

GlobalLogic Reference Architecture and Platform

InternetofThings


z

z z




Notifications


Platform Functions



45 CONFIDENTIAL

“There’s an app for that”– Apple’s advertising tagline, circa 2011

With the brain layer disappearing and becoming part of the background, how do we build apps for IoT?

46 CONFIDENTIAL

What are the Challenges?

InternetofThings

The Past The Present The Near Future The Future

ConceptfromJefferyS.Engelhardt CourtesySony CourtesyNokia/Issam TrabelsiCourtesybushlemon.deviantart.com

A rapidly changing ecosystem

47 CONFIDENTIAL


InternetofThings

Very high fluctuation in usage

48 CONFIDENTIAL


InternetofThings

Ability to handle all types of use cases

Control home appliances(Connected Home)

Gather meteorological data from sensors for weather forecasting

Communicate to an aero engine manufacturer that

this engine needs servicing

49 CONFIDENTIAL


InternetofThings

A completely new UX paradigm

How do we do UX for agriculture?

Will apps on TVs require a new UX?

How will appliances work in a vehicle?

50

What are the Challenges? The Future is an Unknown

Internet of Things

2020

2010

2000

1990

1980

1970

1960

1950

1944

1 10 1000s 100s of 1000s 1 Million 100s of Millions 1 Billion 8 Billion 50 Billion

One

Ten

Thousands

Hundreds of Thousands

One Million

Hundreds of Millions

Billion

8 Billion

50 Billion

51 CONFIDENTIAL


InternetofThings

Being comfortable with legacy

52 CONFIDENTIAL


InternetofThings

Lack of a default standard

What will become the default standard is just a guess for now

53 CONFIDENTIAL

Summary

InternetofThings

With challenges of this kind, it is best to build IoT applications on a platform

So: how do we do it, and what would an application platform look like?

54 CONFIDENTIAL

Plenty of Platforms Already Exist

Internet of Things

Xively There are plenty more being developed . . .And all of them do ingestion and analyticsThingWorx

MnuboBug Labs

55

App Platform Feature Set

Internet of Things

Challenges Solutions

• A rapidly changing ecosystem

• Extreme difference in consumption based on success

• Ability to handle all types of use cases

• A completely new UX paradigm

• The future is an unknown

• Being comfortable with legacy

• A lack of standardization

• Use of Auto-update like features (similar to Google’s Chrome

• Have a plug n play approach to building applications

• Use of cloud and cloud patterns to handle varying levels of consumption and to make this scalable and fault tolerant and highly available

• Have the ability to Integrate with all existing systems via an ESB

• Ability to incorporate new devices \ displays in a seamless manner by separating out the rendering out of the view from the data and the view

• Build once, Reuse everywhere, Continuously update

• Follow the Rapid Application Delivery (RAD model)

With the right Platform as your partner and with the right vendor, go to market would be reduced by months

56 CONFIDENTIAL

Proposed Architecture of the Application Platform

Internet of Things

Views

App

Engi

ne S

DK

App

Ren

dere

r

Web

App

s (M

V*)

Nat

ive

App

Cus

tom

Ap

pDesktops /

Laptops / Tablets

Smart Phones

Devices & Gateways

Business Logic

App

Pla

tfor

m A

PI Worker VM

(Task A)

Worker VM

(Task B)

Worker VM

(Task N)

OrchestrationManager

Integration ESB

Integration layer(3rd Party Adapters)

Auto ScaleManager

Plat

form

Acc

ess

Laye

r

App DB

Platform Tier

DataServices

Streaming Service

Platform ConfigServices

AnalyticsServices

Platform Management

…

57 CONFIDENTIAL

Proposed Architecture of the Application Platform

Internet of Things

Views

App

Engi

ne S

DK

App

Ren

dere

r

Web

App

s (M

V*)

Nat

ive

App

Cus

tom

Ap

pDesktops /

Laptops / Tablets

Smart Phones

Devices & Gateways

Business Logic

App

Pla

tfor

m A

PI Worker VM

(Task A)

Worker VM

(Task B)

Worker VM

(Task N)

OrchestrationManager

Integration ESB

Integration layer(3rd Party Adapters)

Auto ScaleManager

Plat

form

Acc

ess

Laye

r

App DB

Platform Tier

DataServices

Streaming Service

Platform ConfigServices

AnalyticsServices

Platform Management

…

Custombuilt onceforeachstack

Custombuilt onceforeachstack

Vert.x /Node.js/WebAPI

PolyglotDB

PolyglotProgramming

Custom/Drools/WWF

Mule/Neuron /Fuse/etc.

Vert.x /Node.js/WebAPI

58 CONFIDENTIAL


InternetofThings


z

z z




Notifications


Platform Functions




Software

5º MeetUP ARQconf 2016 - IoT: What is it really and how does it work?