Upload
lyminh
View
228
Download
7
Embed Size (px)
Citation preview
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Internet of Things (IoT) and Big Data30.9.2016 – DOAG 2016 Big Data Days
Guido Schmutz
Guido Schmutz
Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis
More than 25 years of software development experience
Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz
Internet of Things (IoT) and Big Data2
Agenda
1. Introduction
2. Towards an IoT Architecture3. IoT Refererence Architecture
4. Summary
Internet of Things (IoT) and Big Data3
Introduction
Internet of Things (IoT) and Big Data4
Internet of Things (IoT) Wave
Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans
Term was first proposed by Kevin Ashton in 1999
Source:TheEconomistSource:Ericsson,June2016
Internet of Things (IoT) and Big Data5
Reasons why IoT opportunity is occurring now ?
Affordable hardware
• Costs of actuators & sensors have been cut in half over last 10 years
Smaller, more powerful hardware
• Form factors of hardware have shrunk to millimeter or even nanometer levels
Ubiquitous & cheap mobility
• Cost for mobile devices, bandwidth and data processing has declined over last 10 years
Availability of supporting tools
• Big data tools & cloud based infrastructure have become widely available
Mass market awareness
• IoT has surpassed a critical tipping point• Vision of a connected world has reached
such a followership that companies have initiated IoT developments
• Commitment is irreversible
Internet of Things (IoT) and Big Data6
The Sensing-as-a-Service Model
Internet of Things (IoT) and Big Data7
Towards an IoT Architecture
Internet of Things (IoT) and Big Data15
Key Challenges for building an IoT application
1. Connect: How to collect data from intelligent devices?
• Abstract complexity associated with device connectivity• Standardize integration of devices with enterprise
2. Analyze: How to analyze IoT data?
• Reduce noise and detect business event at real-time• Enable historical big-data analysis
3. Integrate: How to integrate IoT data & events with enterprise infrastructure?
• Make enterprise processes IoT friendly• Allow enterprise & mobile applications to control devices
Internet of Things (IoT) and Big Data18
Today) Existing Service-/API Architecture as a base
19
MobileApps
DB
Rich(Web)ClientApps
DB
APIG
atew
ay
EnterpriseServiceBu
s(ESB)/Da
taIntegration
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST/SOAP
REST/SOAP
REST/SOAP
SOAP
Various
SQL
SOAPREST
ServiceBu
sOracleDa
taIntegrator
APIG
atew
ay
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data19 =oneway =request/response
REST/SOAP
REST/SOAP
IoT 1a) Reuse exiting Service-/API-based Architecture
IoT SmartDevices
20
MobileApps
DB
Rich(Web)ClientApps
DB
EnterpriseServiceBu
s(ESB)/Da
taIntegration
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST/SOAP
REST
REST
JMS/REST
SOAP
Various
SQL
SOAP REST
WebSocketJMS
ServiceBu
sOracleDa
taIntegrator
APIG
atew
ayAP
IGatew
ay
JMS
JMS
WeblogicJM
S
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data20 =oneway =request/response
IoT 1a) Challenges
Internet of Things (IoT) and Big Data21
• Do IoT devices contain enough resources to communicate directly over the internet (HTTP or JMS) ?
• Should the device only collect data (sense) or is there also the way back necessary (actuator) ?
• Can JMS be used from external devices (firewalls allow traffic over JMS) ?
• How many IoT devices are planned short and long term? How often do they send to the backend? Are the JMS server as well as the ESB capable to deal with the resulting message volume ?
• What are the operations on IoT messages / events, only simple transformations, filter and routing operations?
REST/SOAP
REST/SOAP
IoT 1b) Reuse existing Service-/API-based Architecture
IoT SmartDevices
22
MobileApps
DB
Rich(Web)ClientApps
DB
APIG
atew
ay
EnterpriseServiceBu
s(ESB)/Da
taIntegration
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
RESTHTTP
REST
REST HTTP
JMSAP
IGatew
ay
ServiceBu
sOracleDa
taIntegrator
SOAP
Various
SQL
SOAP REST
WebSocketJMS
JMS
JMS
REST/SOAP
WeblogicJM
S
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data22 =oneway =request/response
REST/SOAP
REST/SOAP
IoT 2) Adding Event Hub and optional IoT Gateway
23
MobileApps
DB
Rich(Web)ClientApps
DB
ESB/D
ataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST
Kafka/MQTT/REST
REST
Kafka/MQTT/REST
MQTT
Kura
Even
tHub
Kafka
ServiceBu
s
OracleDa
taIntegrator
REST
REST
SOAP
Various
SQL
SOAP REST
WebSocketKafka
JMS
JMS
APIG
atew
ayAP
IGatew
ay
REST
Kafka
REST/SOAP
REST
REST
SOASuite
BPMSuite
BusinessActivityMonitoring
MQTT
Internet of Things (IoT) and Big Data23
MQTT
=oneway =request/response
How to implement an Event Hub? Apache Kafka to the rescue
Distributed publish-subscribe messaging system
Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …)
Topic Semantic
does not implement JMS standard
Initially developed at LinkedIn, now part of Apache
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Internet of Things (IoT) and Big Data24
Oracle’s Service Bus as a consumer of Kafka
ServiceBus12c
CloudApps
BusinessService Cl
oud
ProxyServiceKa
fka
Clou
dAP
I
MobileApps Pipeline
Routing
KafkaSensor/IoT
WebApps
BusinessService RE
ST
BusinessService W
SDL
BackendAppsRE
ST
BackendAppsW
SDLProxy
ServiceKafka Pipeline
Routing
Database
DB CDC
StreamProcessing
Internet of Things (IoT) and Big Data25
IoT 2) Solutions & Challenges
Internet of Things (IoT) and Big Data26
Solutions
• Event Hub solves the potential scalability issue of JMS
• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data
Challenges
• Where to do complex analytics on the events? Is it scalable?
• Can we really send all data down to backend? Network bandwidth?
REST/SOAP
REST/SOAP
IoT 3) Adding Stream Processing / Analytics in Backend
27
MobileApps
DB
Rich(Web)ClientApps
DB
(ESB
)/DataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
Even
tHub
StreamProcessing
ESP/CEP
DB
DB
Even
tHub
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
StreamAnalytics
ServiceBu
s
OracleDa
taIntegrator
Kafka
MQTT Kafka
Kafka
SOAP
Various
SQL
SOAP REST
WebSocketKafka
JMS
JMSAP
IGatew
ayAP
IGatew
ay
REST
MQTT
REST
REST
Kafka
Kafka/MQTT/REST
Kafka/MQTT/REST
REST
Kafka
REST/SOAP
REST
REST
Kura
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data27 =oneway =request/response
Kafka
MQTT
IoT 3) Solutions & Challenges
Internet of Things (IoT) and Big Data28
Solutions
• Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems
Challenges
• Can we really send all data down to backend? Network bandwidth?
Oracle’s Stream Analytics as consumer of Kafka/MQTT
OracleStreamAnalytics
StreamAnalytics
Kafka
Kafka
MobileAppsKafka
Sensor/IoT
WebApps
MachineData
DB CDC
Kafka
MQTT
Internet of Things (IoT) and Big Data29
REST/SOAP
REST/SOAP
IoT 4) Adding Industry 4.0 Data Sources (machine data)
30
MobileApps
DB
Rich(Web)ClientApps
DB
(ESB
)/DataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
Even
tHub
StreamProcessing
ESP/CEP
DB
DB
Even
tHub
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
StreamAnalytics
ServiceBu
s
OracleDa
taIntegrator
Kafka
MQTT
Kafka
Kafka
SOAP
Various
SQL
SOAP REST
WebSocketKafka
JMS
JMSAP
IGatew
ayAP
IGatew
ay
REST
MQTT
REST
REST
Kafka
Kafka/MQTT/REST
Kafka/MQTT/REST
REST
Kafka
REST/SOAP
REST
REST
Kura
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data30 =oneway =request/response
I4.0Machine
DB CDC GoldenGate
Kafka/MQTT/REST
Kafka
MQTT
GoldenGate Gateway
Oracle’s GoldenGate for Change Data Capture ofexisting database
Internet of Things (IoT) and Big Data31
MachineData
DB
Kafka
OracleGoldenGate
Delivery
Capture
Pump
JMS
MachineData
DB OracleGoldenGate
Delivery
Capture
Pump HBase
HDFS
REST/SOAP
REST/SOAP
IoT 5) Adding Stream Processing / Analytics at Edge
32
MobileApps
DB
Rich(Web)ClientApps
DB
(ESB
)/DataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
Even
tHub
StreamProcessing
ESP/CEP
DB
DB
Even
tHub
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
StreamAnalytics
ServiceBu
s
OracleDa
taIntegrator
Kafka
Kafka
Kafka
SOAP
Various
SQL
SOAP REST
WebSocketKafka
JMS
JMSAP
IGatew
ayAP
IGatew
ay
Kafka/MQTT/REST
REST REST
Kafka
Kafka
REST
RESTREST
REST/SOAP
REST
MQTT
SOASuite
BPMSuite
BusinessActivityMonitoring
ESP/CEP
EdgeAnalytics
MQTT
Internet of Things (IoT) and Big Data32
Kafka/MQTT/REST
MQTT
=oneway =request/response
I4.0Machine
DB CDC GoldenGate
Kafka/MQTT/REST
Kafka
IoT 5) Solutions & Challenges
Internet of Things (IoT) and Big Data33
Solutions
• Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary
Challenges
• What if we need the raw events persisted? Where do to that?
REST/SOAP
REST/SOAP
IoT 6) Adding Raw Data Storage and Batch Analytics
34
MobileApps
DB
Rich(Web)ClientApps
DB
(ESB
)/DataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
Even
tHub
Even
tHub
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
ServiceBu
s
OracleDa
taIntegrator
Kafka
KafkaKafka
SOAP
Various
SQL
SOAP REST
WebSocket
JMS
JMSAP
IGatew
ayAP
IGatew
ay
REST REST
Kafka
Kafka
SQL
RESTREST
REST/SOAP
StreamProcessing
ESP/CEP
DB
DB
BigDataProcessingHDFS
BatchProcessing
DB
Kafka
Kafka
HDFS
ESP/CEP
EdgeAnalytics
MQTTMQTT
StreamAnalytics
Hadoop/Spark
OracleBigDataAppliance
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data34
Kafka/MQTT/REST
Kafka/MQTT/REST
=oneway =request/response
I4.0Machine
DB CDC GoldenGate
MQTT
Kafka/MQTT/REST
Kafka
IoT 6) Solutions & Challenges
Internet of Things (IoT) and Big Data35
Solutions
• Adding Big Data platform allows to store all raw data in the distributed file system in a scalable and reliable manner
Challenges
• How can we leverage the Big Data platform for more than just storing raw data? How does it combine with the stream processing?
Continuous Ingestion / Fan-In from the Edge
DBSource
BigData
Log
StreamProcessing
IoT Sensor
EventHub
Topic
Topic
REST
Topic
IoT GW
CDCGW
Conn
ectCDC
DBSource
Log CDC Native
IoT Sensor
IoT Sensor
37
DataflowGW
Topic
Topic
Queue
MQTTGW
Topic
DataflowGW
Dataflow
TopicRE
ST37FileSource
Log
Log
Log
Social
Native
Internet of Things (IoT) and Big Data37
Topic
Topic
Challenges for Ingesting Sensor Data
Internet of Things (IoT) and Big Data
• Multitude of sensors
• Multiple Firmware versions
• Bad Data from damaged sensors
• Data Quality
38
REST/SOAP
REST/SOAP
IoT 6a) Adding Data Mining / Machine Learning and Model execution
40
MobileApps
DB
Rich(Web)ClientApps
DB
(ESB
)/DataIntegration
IoTDevices
IoTGateways
IoT SmartDevices
Even
tHub
Even
tHub
EnterpriseApps
WS
ExternalCloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
BusinessIntelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
ServiceBu
s
OracleDa
taIntegrator
Kafka
KafkaKafka
SOAP
Various
SQL
SOAP REST
WebSocket
JMS
JMSAP
IGatew
ayAP
IGatew
ay
REST REST
Kafka
Kafka
SQL
RESTREST
REST/SOAP
StreamProcessing
ESP/CEP
DB
DB
BigDataProcessingHDFS
BatchProcessing
DB
Kafka
Kafka
HDFS
ESP/CEP
EdgeAnalytics
MQTTMQTT
StreamAnalytics
Hadoop/Spark
OracleBigDataAppliance
SOASuite
BPMSuite
BusinessActivityMonitoring
Internet of Things (IoT) and Big Data40
Kafka/MQTT/REST
Kafka/MQTT/REST
=oneway =request/response
I4.0Machine
DB CDC GoldenGate
MQTT
Kafka/MQTT/REST
Kafka
IoT Reference Architecture
Internet of Things (IoT) and Big Data42
IoT Services
IoT Logical Reference Architecture
IoTDevice
Sensor
Actuator
IoT Gateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
Internet of Things (IoT) and Big Data43
IoT Services
IoT Logical Reference Architecture – Oracleon premises
IoTDevice
Sensor
Actuator
IoT Gateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
EdgeAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuiteServiceBus
OracleDataIntegrator
StreamAnalytics
BigDataAppliance
StreamAnalytics
ServiceBus
APIG
atew
ayInternet of Things (IoT) and Big Data44
OracleIoT CSGateway
OracleIoT CSClientLibrary
IoT Services
IoT Logical Reference Architecture – OracleCloud Services
IoTDevice
Sensor
Actuator
IoT Gateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
EdgeAnalytics
OracleBICS
OracleBigDataCS
OracleSOACS
OracleIntegrationCS
OracleIoT CS
OracleStreamingAnalyticsCS
OracleMessagingCS
OracleBigDataDiscoveryCS
OracleMobileCS
Internet of Things (IoT) and Big Data45
OracleIoT CSGateway
OracleIoT CSClientLibrary
OracleProcessCS
OracleDataFlowMLCS
BigDataPreparationCS
ApplicationContainerCS
ContainerCS
IoT Services
IoT Logical Reference Architecture – MicrosoftAzure
IoTDevice
Sensor
Actuator
IoT Gateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
AzureIoTHu
b
AzureStreamingAnalytics
AzureServiceBu
s
AzurePowerBI
HDInsight
AzureStorage
AzureIoT GatewaySDK Az
ureEventH
ub
AzureStorage
AzureMachineLearning
Internet of Things (IoT) and Big Data46
IoT ServicesIoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
IoT Logical Architecture – Open Source
Internet of Things (IoT) and Big Data47
Summary
Internet of Things (IoT) and Big Data48
Summary
Treat events as events! Infrastructures for handling lots of events are available!
IoT tends to make Big Data / Fast Data infrastructures necessary
Know your use case/requirements to choose the right architecture!• Can my existing backend landscape handle the new IoT load?
• Do I have to handle huge amount of events in “real-time”?
• Do I need to filter/aggregate data before invoking existing backend systems?
• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?
• What is the network bandwidth between device/gateway and cloud/backend?
• Centralized or Decentralized IoT solution?
Internet of Things (IoT) and Big Data49
J
Internet of Things (IoT) and Big Data50