Internet of Things and Big Data

BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH

#SDF16

Internet of Things (IoT) and Big Data

Guido Schmutz

#SDF16

#SDF16

Guido Schmutz

Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOAConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

#SDF16

Agenda

1. Introduction

2. Towards an IoT Architecture3. Summary

#SDF16

Introduction

#SDF16

Internet of Things (IoT) Wave

Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans

Term was first proposed by Kevin Ashton in 1999

Source:TheEconomistSource:Ericsson,June2016

#SDF16

Reasons why IoT opportunity is occurring now ?

Affordable hardware

• Costs of actuators & sensors have been cut in half over last 10 years

Smaller, more powerful hardware

• Form factors of hardware have shrunk to millimeter or even nanometer levels

Ubiquitous & cheap mobility

• Cost for mobile devices, bandwidth and data processing has declined over last 10 years

Availability of supporting tools

• Big data tools & cloud based infrastructure have become widely available

Mass market awareness

• IoT has surpassed a critical tipping point• Vision of a connected world has reached

such a followership that companies have initiated IoT developments

• Commitment is irreversible

#SDF16

The Sensing-as-a-Service Model

#SDF16

Towards an IoT Architecture

#SDF16

Key Challenges for building an IoT application

1. Connect: How to collect data from intelligent devices?• Abstract complexity associated with device connectivity• Standardize integration of devices with enterprise

2. Analyze: How to analyze IoT data?• Reduce noise and detect business event at real-time• Enable historical big-data analysis

3. Integrate: How to integrate IoT data & events with enterprise infrastructure?• Make enterprise processes IoT friendly• Allow enterprise & mobile applications to control devices

#SDF16

Today) Existing Service-/API Architecture as a base

10

MobileApps

DB

Rich(Web)ClientApps

APIG

atew

ay

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS

ExternalCloudServiceProviders

BPMandSOAPlatform

Event

BusinessLogic/Rules

BusinessIntelligence

Services

WS

Event

Processes

Visualization

Analytics

DB

REST/SOAP

REST/SOAP

REST/SOAP

SOAP

Various

SQL

SOAPREST

ServiceBu

sOracleDa

taIntegrator

APIG

atew

ay

SOASuite

BPMSuite

BusinessActivityMonitoring

=oneway =request/response

DB

#SDF16

REST/SOAP

REST/SOAP

IoT 1a) Reuse exiting Service-/API-based Architecture

IoT SmartDevices

11

MobileApps

DB

Rich(Web)ClientApps

DB

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

REST/SOAP

REST

REST

JMS/REST

SOAP

Various

SQL

SOAP REST

WebSocketJMS

ServiceBu

sOracleDa

taIntegrator

APIG

atew

ayAP

IGatew

ay

JMS

JMS

WeblogicJM

S

SOASuite

BPMSuite



#SDF16

IoT 1a) Challenges

• Are IoT devices powerful enough to communicate over the internet (HTTP protocol) ?

• Device only collects data (sense) or is there also the way back necessary (actuator) ?

• Can JMS be used from external devices (firewalls allow traffic over JMS) ?

• How many IoT devices are planned short and long term ? • How frequently do they produce data ? • Is JMS server and ESB capable for dealing with resulting message volume ?

• What operations are needed on IoT data (messages / events) ‘

• only simple transformations, filter and routing operations (VETRO pattern) ?

#SDF16

REST/SOAP

REST/SOAP

IoT 1b) Reuse existing Service-/API-based Architecture

IoT SmartDevices

13

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

RESTHTTP

REST

REST HTTP

JMSAP

IGatew

ay

ServiceBu

sOracleDa

taIntegrator

SOAP

Various

SQL

SOAP REST

WebSocketJMS

JMS

JMS

REST/SOAP

WeblogicJM

S

SOASuite

BPMSuite



#SDF16

REST/SOAP

REST/SOAP

IoT 2) Adding Event Hub and optional IoT Gateway

14

MobileApps

DB

Rich(Web)ClientApps

DB

ESB/D

ataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

REST

Kafka/MQTT/REST

REST

Kafka/MQTT/REST

MQTT

Kura

Even

tHub

Kafka

ServiceBu

s

OracleDa

taIntegrator

REST

REST

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMS

APIG

atew

ayAP

IGatew

ay

REST

Kafka

REST/SOAP

REST

REST

SOASuite

BPMSuite


MQTT

MQTT


#SDF16

How to implement an Event Hub? Apache Kafka to the rescue

• Distributed publish-subscribe messaging system

• Designed for processing of high-volume, real time activity stream data

• Stateless (passive) architecture, offset-based consumption

• Provides Topics, but does not implement JMS standard

• Initially developed at LinkedIn, now part of Apache• Peak Load on single cluster: 2 million messages/sec, 4.7

Gigabits/sec inbound, 15 Gigabits/sec outbound

Oracle’s Service Bus as a consumer of Kafka

ServiceBus12c

CloudApps

BusinessService Cl

oud

ProxyServiceKa

fka

Clou

dAP

I

MobileApps Pipeline

Routing

KafkaSensor/IoT

WebApps

BusinessService RE

ST

BusinessService W

SDL

BackendAppsRE

ST

BackendAppsW

SDLProxy

ServiceKafka Pipeline

Routing

Database

DB CDC

StreamProcessing

#SDF16

IoT 2) Solutions & Challenges

Solutions

• Event Hub solves the potential scalability issue of JMS

• Event Hub can interoperate / integrate with existing SOA platform (ESB)

• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data

Challenges

• Where to do complex analytics on the events? Is it scalable?

• Can we really send all data down to backend? Network bandwidth?

#SDF16

REST/SOAP

REST/SOAP

IoT 3) Adding Stream Processing / Analytics in Backend

18

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

MQTT Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST

MQTT

REST

REST

Kafka

Kafka/MQTT/REST

Kafka/MQTT/REST

REST

Kafka

REST/SOAP

REST

REST

Kura

SOASuite

BPMSuite



Kafka

MQTT

#SDF16


Solutions

• Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems

Challenges

• Can we really send all data down to backend? Network bandwidth?

#SDF16

Oracle’s Stream Analytics as consumer of Kafka/MQTT

OracleStreamAnalytics

StreamAnalytics

Kafka

Kafka

MobileAppsKafka

Sensor/IoT

WebApps

MachineData

DB CDC

Kafka

MQTT

REST/SOAP

REST/SOAP

IoT 4) Adding Industry 4.0 Data Sources (machine data)

21

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

MQTT

Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST

MQTT

REST

REST

Kafka

Kafka/MQTT/REST

Kafka/MQTT/REST

REST

Kafka

REST/SOAP

REST

REST

Kura

SOASuite

BPMSuite



I4.0Machine

DB CDC GoldenGate

Kafka/MQTT/REST

Kafka

MQTT

GoldenGate Gateway

Oracle’s GoldenGate for Change Data Capture ofexisting database

MachineData

DB

Kafka

OracleGoldenGate

Delivery

Capture

Pump

JMS

MachineData

DB OracleGoldenGate

Delivery

Capture

Pump HBase

HDFS

REST/SOAP

REST/SOAP

IoT 5) Adding Stream Processing / Analytics at Edge

23

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

Kafka/MQTT/REST

REST REST

Kafka

Kafka

REST

RESTREST

REST/SOAP

REST

MQTT

SOASuite

BPMSuite


ESP/CEP

EdgeAnalytics

MQTT

Kafka/MQTT/REST

MQTT


I4.0Machine

DB CDC GoldenGate

Kafka/MQTT/REST

Kafka

#SDF16


Solutions

• Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary

Challenges

• How to handle integration between the components on the IoT side (fan in)

#SDF16

REST/SOAP

REST/SOAP

IoT 6) Adding Data Flow Engines

25

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

Kafka/MQTT/REST

REST REST

Kafka

Kafka

REST

RESTREST

REST/SOAP

REST

MQTT

SOASuite

BPMSuite


ESP/CEP

EdgeAnalytics

MQTT

Kafka/MQTT/REST

MQTT


I4.0Machine

DB CDC GoldenGate

Kafka/MQTT/REST

Kafka

NiFiNiFi

Continuous Ingestion / Fan-In from the Edge

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

26

DataflowGW

Topic

Topic

Queue

MessageGW

Topic

DataflowGW

Dataflow

TopicRE

ST26FileSource

Log

Log

Log

Social

Native

Topic

Topic

#SDF16

Challenges for Ingesting Sensor Data

• Multitude of sensors

• Multiple Firmware versions

• Bad Data from damaged sensors

• Data Quality

#SDF16


Solutions

• Data Flow engines allow for passing information onwards, potentially over multiple hops (fan-in)

Challenges

• What if we want to keep the raw data? Where to persist it?

REST/SOAP

REST/SOAP

IoT 7) Adding Raw Data Storage and Batch Analytics

29

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

ServiceBu

s

OracleDa

taIntegrator

Kafka

KafkaKafka

SOAP

Various

SQL

SOAP REST

WebSocket

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST REST

Kafka

Kafka

SQL

RESTREST

REST/SOAP

StreamProcessing

ESP/CEP

DB

DB

BigDataProcessingHDFS

BatchProcessing

DB

Kafka

Kafka

HDFS

ESP/CEP

EdgeAnalytics

MQTTMQTT

StreamAnalytics

Hadoop/Spark

OracleBigDataAppliance

SOASuite

BPMSuite


Kafka/MQTT/REST

Kafka/MQTT/REST


I4.0Machine

DB CDC GoldenGate

MQTT

Kafka/MQTT/REST

Kafka

NiFiNiFi

#SDF16


Solutions

• Adding Big Data platform allows to store all raw data in a distributed file system in a scalable and reliable manner

Challenges

• How can we leverage the Big Data platform for more than just storing raw data?

• How does it integrate with the stream processing?

REST/SOAP

REST/SOAP

IoT 7a) Adding Data Mining / Machine Learning and Model execution

31

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

ServiceBu

s

OracleDa

taIntegrator

Kafka

KafkaKafka

SOAP

Various

SQL

SOAP REST

WebSocket

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST REST

Kafka

Kafka

SQL

RESTREST

REST/SOAP

StreamProcessing

ESP/CEP

DB

DB


BatchProcessing

DB

Kafka

Kafka

HDFS

ESP/CEP

EdgeAnalytics

MQTTMQTT

StreamAnalytics

Hadoop/Spark

OracleBigDataAppliance

SOASuite

BPMSuite


Kafka/MQTT/REST

Kafka/MQTT/REST


I4.0Machine

DB CDC GoldenGate

MQTT

Kafka/MQTT/REST

Kafka

NiFiNiFi

REST/SOAP

REST/SOAP

Oracle Cloud Services and Internet of Things / Big Data

32

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

Gateway

IoT SmartDevices

Even

tHub

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

=oneway=request/response

KafkaKafka

SOAP

Various

SQL

SOAP REST

WebSocket

JMS

JMSAP

IGatew

ay

Kafka/REST

REST REST

Kafka

Kafka

SQL

RESTREST

REST/SOAP

StreamProcessing

ESP/CEP

DB

DB


BatchProcessin

g

DB

Kafka

Kafka

HDFS

MQTT

Hadoop/Spark

OracleIoT CloudService

OracleBigDataCloudService

OracleBICloudService

OracleIntegrationClou

dService

Oracle

MessagingCloud

Service

OracleProcessCloudService

OracleSOACloudService

OracleMobileCloud

Service

ESP/CEPEdgeAnalytics

OracleIoT CSGateway

OracleIoT CSClientLibrary


#SDF16

Summary

#SDF16

Summary

Treat events as events! Infrastructures for handling lots of events are available!

IoT tends to make Big Data / Fast Data infrastructures necessary

Know your use case/requirements to choose the right architecture!• Can my existing backend landscape handle the new IoT load?

• Do I have to handle huge amount of events in “real-time”?

• Do I need to filter/aggregate data before invoking existing backend systems?

• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?

• What is the network bandwidth between device/gateway and cloud/backend?

• Centralized or Decentralized IoT solution?

#SDF16

J

#SDF16

Guido SchmutzTechnology Manager

[email protected]

#SDF16

#SDF16

IoT Services

IoT Logical Reference Architecture

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics

EnterpriseApplications

BPMandSOAPlatformStreaming

Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA

MQTTCoAPXMPPDDSAMQPKAFKAWIFI

BLEZigBeeWIFIWired

Internet of Things (IoT) and Big Data38

#SDF16

IoT Services

IoT Logical Reference Architecture – Oracleon premises

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

EdgeAnalytics


SOASuite

BPMSuiteServiceBus

OracleDataIntegrator

StreamAnalytics

BigDataAppliance

StreamAnalytics

ServiceBus

APIG

atew

ayInternet of Things (IoT) and Big Data39

OracleIoT CSGateway


#SDF16

IoT Services

IoT Logical Reference Architecture – OracleCloud Services

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

EdgeAnalytics

OracleBICS

OracleBigDataCS

OracleSOACS

OracleIntegrationCS

OracleIoT CS

OracleStreamingAnalyticsCS

OracleMessagingCS

OracleBigDataDiscoveryCS

OracleMobileCS


OracleIoT CSGateway


OracleProcessCS

OracleDataFlowMLCS

BigDataPreparationCS

ApplicationContainerCS

ContainerCS

#SDF16

IoT Services

IoT Logical Reference Architecture – MicrosoftAzure

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

AzureIoTHu

b

AzureStreamingAnalytics

AzureServiceBu

s

AzurePowerBI

HDInsight

AzureStorage

AzureIoT GatewaySDK Az

ureEventH

ub

AzureStorage

AzureMachineLearning


#SDF16

IoT ServicesIoT

Device

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

IoT Logical Architecture – Open Source


Data & Analytics

Internet of Things and Big Data