IoT Architecture - are traditional architectures good enough?

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Internet of Things (IoT) Are traditional architectures good enough?

Guido Schmutz

Guido Schmutz

Working for Trivadis for more than 18 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Our company.

Trivadis is a market leader in IT consulting, system integration, solution engineeringand the provision of IT services focusing on and and Open Source technologiesin Switzerland, Germany, Austria and Denmark. We offer our services in the followingstrategic business fields:

Trivadis Services takes over the interacting operation of your IT systems.

O P E R A T I O N

COPENHAGEN

MUNICH

LAUSANNEBERN

ZURICHBRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

14 Trivadis branches and more than600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:CHF 5.0 million

Financially self-supporting andsustainably profitable

Experience from more than 1,900 projects per year at over 800customers

Agenda

1. Introduction

2. Are Traditional Solution architectures good enough?3. IoT Architecture & Components

4. Demo Use Case – Truck Movements

5. Oracle IoT Cloud Service6. Summary

Introduction

Internet of Things Wave

Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans

Term was first proposed by Kevin Ashton in 1999

The concept of IoT first became popular at the Auto-ID center and MIT

IoT can also be named M2M

Internet Of Things – Sensorsare/will be everywhereThere are more devices tapping into the internet than people on earth

How do we prepare our systems/architecture for the future?

Source:CiscoSource:TheEconomist

Reasons why IoT opportunity is occurring now ?

Affordable hardware

• Costs of actuators & sensors have been cut in half over last 10 years

Smaller, but more powerful hardware

• Form factors of hardware have shrunk to millimeter or even nanometer levels

Ubiquitous & cheap mobility

• Cost for mobile devices, bandwidth and data processing has declined as much as 97% over last 10 years

Availability of supporting tools

• Big data tools & cloud based infrastructure have become widely available and fairly sophisticated

Mass market awareness

• IoT has surpassed a critical tipping point• Vision of a connected world has reached such a

followership that companies have initiated IoTdevelopments

• Commitment is irreversible

The Sensing-as-a-Service Model

Are Traditional Solution architectures good enough?

A little story of a “real-life” customer situation

Traditional system interact with its clients and does its workImplemented using legacy technologies (i.e. PL/SQL)

New requirement:• Offer notification service to notify

customer when goods are shipped• Subscription and inform over different

channels• Existing technology doesn’t fit

delivery

LogisticSystem

Oracle

MobileApps

Sensor ship

sort

12

Rich (Web)Client Apps

DB

schedule

Logic(PL/SQL)

delivery


Events are “owned” by traditional application (as well as the channels they are transported over)

Implement notification as a new Java-based application/system

But we need the events ! => so let’s integrate

delivery

LogisticSystem

Oracle

MobileApps

Sensor ship

sort

13


DB

schedule

Notification

Logic(PL/SQL)

Logic(Java)delivery

SMS

Email

…


integrate in order to get the information! Oracle Service Bus was already there

Rule Engine implemented in Java and invoked from OSB message flowNotification system informed via queueHigher Latency introduced (good enough in this case)

delivery

LogisticSystem

Oracle

OracleServiceBus

MobileApps

Sensor AQship

sort

14


DB

schedule

Filter

Notification

Logic(PL/SQL)

JMS

RuleEngine(Java)

Logic(Java)delivery

shipdelivery

delivery true SMS

Email

…


Treat events as first-class citizens

Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !!

Event (stream) processing can be introduced and by that latency reduced!

delivery

LogisticSystem

Oracle

OracleServiceBus

MobileApps

Sensor AQship

sort

15


DB

schedule

Filter

Notification

Logic(PL/SQL)

JMS

RuleEngine(Java)

Logic(Java)delivery

shipdelivery

delivery true SMS

Email

…

Treat Events as Events and make them globally available

delivery

LogisticSystem

Oracle

OracleServiceBus

MobileApps

Sensorship

sort

16


DB

schedule

Filter

Notification

Logic(PL/SQL)

JMS

RuleEngine(Java)

Logic(Java)

delivery

ship

delivery true SMS

Email

…

EventBus/Hub

Stream/EventProcessing

Traditional Data Processing - Challenges

• Introduces too much “decision latency”

• Responses are delivered “after the fact”

• Maximum value of the identified situation is lost

• Decision are made on old and stale data

• “Data a Rest”

The New Era: Stream Data Processing / Fast Data

• Events are analyzed and processed in real-time as the arrive

• Decisions are timely, contextual and based on fresh data

• Decision latency is eliminated

• “Data in motion”

IoT Architecture

Key Challenges for building an IoT application

1. Connect: How to collect data from intelligent devices?

• Abstract complexity associated with device connectivity• Standardize integration of devices with enterprise

2. Analyze: How to analyze IoT data?

• Reduce noise and detect business event at real-time• Enable historical big-data analysis

3. Integrate: How to integrate IoT data & events with enterprise infrastructure?

• Make enterprise processes IoT friendly• Allow enterprise & mobile applications to control devices

Today) Existing Service-/API-based Architecture

21

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseS

ervic

eBus(ESB)/DataIntegratio

n

EnterpriseApplications

WS

External CloudServiceProviders

BPMandSOAPlatform

Event

BusinessLogic/Rules

Business Intelligence

Services

WS

Event

Processes

Visualization

Analytics

DB

=oneway=request/response

REST SOAP

REST SOAP

SOAP

SOAP

any

SQL

SOAPREST

APIG

atew

ay

ServiceBu

sOracleDa

taIntegrator

SOASuite

BPMSuite

BusinessActivityMonitoring

IoT 1a) Reuse Service-/API-based Architecture

IoT Device

22

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseS

ervic


n


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB


SOAP

REST

REST HTTP

JMS

SOAP

any

SQL

SOAP REST

WebSocket

REST SOAP

REST SOAP

APIG

atew

ay

ServiceBu

sOracleDa

taIntegrator

SOASuite

BPMSuite


JMS

HTTP

IoT 1b) Reuse existing Service-/API-based Architecture

IoT Device

23

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseS

ervic


n


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

WebSocket

REST SOAP

REST SOAP REST SOAP

RESTHTTP

REST

APIG

atew

ay

ServiceBu

sOracleDa

taIntegrator

SOASuite

BPMSuite

JMS


REST HTTP

JMS


IoT 1c) Reuse existing Service-/API-based Architecture

24

MobileApps

DB

Rich(Web)ClientApps

DB

EnterpriseS

ervic


n

APIG

atew

ay

IoT Device Gateway

IoT SmartDevice


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

REST SOAP

REST SOAP

SOASuite

BPMSuite

APIG

atew

ay

ServiceBu

sOracleDa

taIntegrator


REST HTTP

JMS MQTT Kafka

REST HTTP

JMS MQTT Kafka


Event Hub - Apache Kafka

Distributed publish-subscribe messaging system

Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

Does not use JMS

Initially developed at LinkedIn, now part of Apache

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

Service Bus as a consumer of Kafka

ServiceBus12c

CloudApps

BusinessService Cl

oud

ProxyServiceKa

fka

Cloud

API

MobileApps Pipeline

Routing

KafkaSensor/IoT

WebApps

BusinessService RE

ST

BusinessService W

SDL

BackendAppsRE

ST

BackendAppsW

SDLProxy

ServiceKafka Pipeline

Routing

Database

DB CDC

StreamProcessing

IoT 2) Stream Processing / Analytics in Backend

27

MobileApps

DB


DB

(ESB)/DataIntegratio

n

APIG

atew

ay

IoT Device Gateway

IoT SmartDevice

EventH

ub

StreamProcessingInfrastructure

ESP/CEP

DB

DB

EventH

ub


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB


REST SOAP

REST SOAP


SOASuite

BPMSuite

APIG

atew

ay

ServiceBu

sOracleDa

taIntegrator

StreamAnalytics

Oracle Stream Analytics as consumer of Kafka/MQTT

OracleStreamAnalytics

StreamAnalytics

Kafka

Kafka

MobileAppsKafka

Sensor/IoT

WebApps

Machine Data

DB CDC

Kafka

MQTT

Distribute runtime portion of IoT Analytics to the edge

Some IoT Analytics applications need to be distributed, so that processing can take place in devices, gateways, smart routers, servers at the site where sensor data is generated

Three main reasons:

• The factory, vehicle, home or other edge location needs to stay in operation, even when the corporate data center or cloud is down or not reachable

• Wide-are communication is generally to slow

• Transmitting all sensor data to corporate or cloud data center may be impractical or impossible if the volume of data is high or if reliable, high-bandwidth networks are unavailable

IoT 3) Stream Processing / Analytics at Edge


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

30

MobileApps

DB


DB


n

APIG

atew

ay

IoT DeviceGateway

IoT SmartDevice

EventH

ub


ESP/CEP

DB

DB

EventH

ubESP/CEP


DB

REST SOAP

REST SOAP

StreamAnalyticsEdgeAnalytics


SOASuite

BPMSuite

ServiceBu

sOracleDa

taIntegrator

APIG

atew

ay

IoT 4) Raw Data Storage and Batch Analytics


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

31

MobileApps

DB


DB


n

APIG

atew

ay


IoT Device

IoT SmartDevice

EventH

ub


ESP/CEP

DB

DB

EventH

ub

BigDataInfrastructureHDFS

BatchProcessing

DB

REST SOAP

REST SOAP

Gateway

ESP/CEP

DB

EdgeAnalytics

StreamAnalytics


SOASuite

BPMSuite

ServiceBu

sOracleDa

taIntegrator

BigDataAppliance

APIG

atew

ay

IoT 4a) Raw Event Replay Feature


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

32

MobileApps

DB


DB


n

APIG

atew

ay


IoT Device

IoT SmartDevice

EventH

ub


ESP/CEP

DB

DB

EventH

ub


Replay

REST SOAP

REST SOAP

Gateway

ESP/CEP

DB

EdgeAnalytics


SOASuite

BPMSuite

ServiceBu

sOracleDa

taIntegrator

StreamAnalytics

BigDataAppliance

APIG

atew

ay

IoT 5) Data Mining and Model evaluation


WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

33

MobileApps

DB


DB


n

APIG

atew

ay


IoT Device

IoT SmartDevice

EventH

ub


ESP/CEP

DB

DB

EventH

ub


BatchProcessing

DB

REST SOAP

REST SOAP

Gateway

ESP/CEP

DB

EdgeAnalytics


SOASuite

BPMSuite

ServiceBu

sOracleDa

taIntegrator

StreamAnalytics

BigDataAppliance

APIG

atew

ay

IoT Services

IoT Logical Reference Architecture

IoTDevice

Sensor

Actuator

IoTGateway

Storage

UIApp

StreamingAnalytics


BPMandSOAPlatformStreaming

Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA

MQTTCoAPXMPPDDSAMQPKAFKAWIFI

BLEZigBeeWIFIWired

IoT Services

IoT Logical Reference Architecture – Oracleon premises

IoTDevice

Sensor

Actuator

IoTGateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

EdgeAnalytics


SOASuite

BPMSuiteServiceBus

OracleDataIntegrator

StreamAnalytics

BigDataAppliance

StreamAnalytics

ServiceBus

APIG

atew

ay

IoT ServicesIoT

Device

Sensor

Actuator

IoTGateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

IoT Logical Architecture – Open Source

Demo Use Case – Truck Movements

Demo Use Case – Truck Movements

Truck DataIngestion Geo-Fencing

2016-06-0214:39:56.605|98|27|MarkLochbihler|803014426|Wichita toLittle RockRoute 2|Normal|38.65|-90.21|5187297736652502631

{"timestamp": "2016-06-0214:39:56.991", "truckId":99,"driverId":31,"driverName":"Rommel Garcia", "routeId":1565885487, "routeName":"Springfield toKCViaHanibal","eventType":"Normal", "latitude":37.16,"longitude": "-94.46","correlationId":5187297736652502631}

RecklessDrivingDetector

NEAR

ENTER

TruckDriver

DashboardMovement MovementJSON

RecklessDriver

Oracle IoT Cloud Service

Oracle IoT Cloud Service within Oracle Cloud Platform

DocumentsCloud

ProcessCloud

SitesCloud

SocialCloud

CONTENTANDCOLLABORATION

BusinessIntelligence Cloud

BigDataDiscoveryCloud

BigDataPreparation

Cloud

InternetofThingsCloud

BUSINESSANALYTICS

NoSQLDatabaseCloud

BigDataSQLCloud

ExadataCloud

BigDataCloud

DatabaseBackupCloud

DatabaseCloud

BigDataPreparation

Cloud

DATAMANAGEMENT

NodeCloud

DeveloperCloud

MobileCloud

JavaCloud

ApplicationBuilderCloud

APPLICATIONDEVELOPMENT

Integration Cloud

SOACloud

GoldenGateCloud

InternetofThingsCloud

INTEGRATION

ApplicationPerformance Monitoring

Cloud

ITAnalyticsCloud

LogAnalyticsCloud

ENTERPRISEMANAGEMENT

MobileCloud

MOBILEIntegration Cloud

Internet ofThings Cloud

SitesCloud

DocumentsCloud

Oracle IoT Cloud Service

DeviceVirtualization

High SpeedMessaging

StreamProcessing

EndpointManagement

EventStore

IoTCloudService

EnterpriseConnectivity

Integration CloudService

BI&Big DataCloudService

OracleCloudServices

MobileCloudService

3rd partyapps

IndustryVertical Apps

Enterprise AppsCloudorOnPremise

Manufacturing

Transportation

ServiceMgmt

AssetMgmt

Firewall

OracleIoTCSGateways/w

3rd partygateways/wwith OracleIoT

Client Library

IoTCloudServiceClientLibraries &Gateway

IndirectlyconnectedDevices

Directlyconnecteddevice

Anatomy of an IoT Application in Oracle IoT CS

IoTApplication

Devices Messages Analytics Integrations Configurations

Real-time Analytics Applications Workflow

Devices Data/Alerts

Devices Data/Alerts

Devices Data/Alerts

Stream

Stream

......

Exploration

Patterns

Stream Processing Engine

Publish

Stream

StreamExplorerEditor StreamProcessingEngine

Stream Processing Applications

AnalyzedDataIoT CloudService

Summary

Summary

Treat events as events! Infrastructures for handling lots of events are available!

IoT tends to make Big Data / Fast Data infrastructures necessary

Know your use case/requirements to choose the right architecture!• Can my existing backend landscape handle the new IoT load?

• Do I have to handle huge amount of events in “real-time”?

• Do I need to filter/aggregate data before invoking existing backend systems?

• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?

• What is the network bandwidth between device/gateway and cloud/backend?

• Centralized or Decentralized IoT solution?

J

Guido SchmutzTechnology Manager

[email protected]

Data & Analytics

IoT Architecture - are traditional architectures good enough?