Streaming Analytics, Data Lakes and PI Integrators...Streaming Analytics, Data Lakes and PI Integrators Matt Ziegler ... • Demonstrate PI Integrator for Business 2017 with Kafka

EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC

Presented by

Streaming Analytics,

Data Lakes and PI

Integrators

Matt Ziegler

Daniele Farris


2

Overview


A singular data system that does all this and more

3

Reduce

maintenance

costs by up to

25%

Eliminate

70% of

breakdowns

Reduce

downtime up

to 50%

Cut

unplanned

outages by

up to 50% Reduce

scheduled

repairs by up

to 12%

Reduce capital

investment by

3-5%


Presentation Goals

• Explore common Enterprise Data Architectures including

– Lambda & Kappa

– Data Lakes

• Clarify the role the PI System plays in each architecture

• Discuss how using the PI System helps maximize the effectiveness of scarce machine learning and analytics talent

• Demonstrate PI Integrator for Business 2017 with Kafka in a machine learning and streaming scenario

4


Data Scientist is the sexiest job of 21st century, but…

5

Source: http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#5481f6037f75


Data Scientist is the sexiest job of 21st century, but…

6

Source: http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#5481f6037f75


7

Critical Assumption – Knowledge workers will have the appropriate knowledge

of processes and equipment to apply the correct context at the time of analysis.


Challenge: Map raw data to the physical world

8

BGE003 FI-111 02T100 AF_FLOW3 02:T103 AT401

ACEDemo.Unit1.Output

TI-178 B352_W778 0_CMP_SVLV_PCT

02F104 CD:F161

GE01_DT 409510395_Wind Speed QI-109 GE01_DT Cooling Fan-711.Feed Rate

1-8.Net

Volume 03LBA32CT0

01-2

DC.SJ.ITLoad.P

WR

TI-145 FR2001

TI-178

GE04_OS

FT9001

FT9001 FR5001

AF_NOISE

DC.SJ.PUE TI-102 DC.Zero DY-108 DC.SJ.C1.Z3.R3.PDU1.PF GE01_A_DT

GE04_Status

02:F101.C

FIC-144 02F100 fasttag

FI-151 0_ENG_AUX_STS

AQUA2-SI005.PV GE02_Energy 0_ENG_MODE_STS FI-151 02T103.Q DC.Z1R

0_CMP_HDR

GE05_Energy C1:14AT5 AC03.Air Flow FeedBin.Cmt Boiler Cold Reheat Pressure

B737_FG117 DC.TimeLoad

D-110.Tank Pressure.PV GE04_DT QI-121 GE03_V_WIN

DC.Rk07R DC.Srv06R GE04_Energy

TI-121 FT9001

FAC.OAK.Power-Kh-Val.PV

fic1001.C GE02_OT

02F102.1HR

AVG BGT001

PI-111 facility_output

DM-05:BW.R

DC.SJ.C1.Z1.R1.Rk06.S2.O03.PWR QI-111FinalProductBin.On

94:GRDIDX.ProdID Boiler-209.Fuel Gas Flow

GE01_A_DT

fic1001.C

02:F101.C

FR2001 TIC-121

Aso AT401 DC.Srv01R

TI-178

asset1_output Active Meters

GE01_A_DT aso AF_NOISE PI-115 DM-05:BW.R

FI-101 Volume

0_CLR_FINAL_OUT_B_TMP F506_E990 339511775_Clear Sky Global Horiz GE01_DT

FI-111

% CO2 GE05_ES T

GE01_CON AlarmTest.Input.

Float32.1

DC.NY.Actual.PWR.day.Tot

80-5.Net Volume GE01_A_DT

45-2.Net Volume

Boiler-209.Fuel Gas

Flow DC.Srv01R

94:GRDIDX.Tr

trigger

AC09.Power

403511195_Wind

Speed

DC.C2Z1.Pwr.Rippl

e GE01_A_DT

AF_NOISE

1-16.Net Volume CB1992_MS

0_CMP_FLOW_TOTAL

FT9001

fasttag

FeedBin.Cmt

DC.Zone1.Number

FT9001

AlarmTest.Input.Float32.1

FT9001

DC.Srv01R Boiler-

125.Fuel Gas

Volume

1-13.Net

Volume B045_FG978

AT401

Anacortes Refinery.Alkylation.Asset

Problems B210_FG005.KPIExcursion

D-110.Tank

Pressure.P

V Boiler Feed

Pump #1

94:BW.R TI-101 F723_E889

369512185-Temp Compressor-439.Feed Rate

DC.CH.DCE FIC-172:210

FI-121 AF_FLOW3

0_ENG_MODE_STS

GE03_Q

DY-131:166 GE01_TD

Boiler-334.Feed Rate

GE04_OSAsset1.Problems QI-122 FI-151

DC.SJ.SiteRealTim

eITLoad.PR FT9001

Weather

Conditions

Relative Humidity: 34%

Current Temp: 85 F

High: 92

Low: 57 F

Wind: 8 mph/N

Crude Furnace

Draft Pressure: -0.5 WC

Stack Temp: 316 F

Oxygen: 2.5%

Outlet Temp: 840 F

Cold Oil Velocity: 6 ft/sec

Crude Desalter

Operating Pressure: 110 psi

Charge Rate: 14 gph

Mix Valve Pressure: 8 psi

Water Rate: 8%

fic1001.C FR5001

AQUA2-TI-201.PV

TIC-181

BGE003 FI-111 02T100

AF_FLOW3 02:T103

ACEDemo.Unit1.Output

TI-178 B352_W778 0_CMP_SVLV_PCT

02F104 CD:F161

AQUA2-SI005.PV GE02_Energy

0_ENG_MODE_STS FI-151 02T103.Q

DC.Z1R 0_CMP_HDR_SUC_PRS

DM-05:BW.R

DC.SJ.C1.Z1.R1.R

k06.S2.O03.PWR

QI-111FinalProductBin.On

94:GRDIDX.ProdID Boiler-

209.Fuel Gas Flow

FI-101 bf5e1d1d-

39c9-4b5b-b3d3-

c2ce05fa3a26 DM-05:BW.R AT401

AT401

DailyTrigger FrqPrbCost_ER

GE01_DT 409510395_Wind Speed

QI-109 GE01_DT Cooling Fan


364511575-AC Power

DY-101 02:F101.C

0_CLR_FINAL_OUT_B_TMP F506_E990

339511775_Clear Sky Global

Horiz GE01_DT

80-13.Net

DY-131 DC.SJ.PUE


GE01_DT

FI-101 bf5e1d1d-39c9-

4b5b-b3d3-c2ce05fa3a26 DM-05:BW.R AT401

4-36.NetVolumeAC04.Air Flow

02T100 03LBB02CT001-2

DC.SJ.SiteRealTim

eITLoad.PR FT9001

02F102.1HRAVG BGT001

PI-111 facility_output

AQUA2-SI005.PV GE02_Energy

0_ENG_MODE_STS FI-151

02T103.Q DC.Z1R 0_CM P_ H D R _SU C _ PRS

403511195_Wind Speed

GE02_Energy

FI-101 bf5e1d1d-39c9- 4b5b-b3d3-c2ce05f


Challenge: Find events in raw data streams

9

What happened

during the event?

Event Attribute Value

Name Ex 20151215-0002

Start 15-Dec-2016

10:35:02

End 15-Dec-2016 11:47:26

Asset Boiler-352

Excursion Type High Violation

Fuel Gas

Flow.Avg 37.12 k sft3/h

myKPI.Max 47.19 bbl/d

Downtime Excursion

Batch


01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

01010101010101

PI System collects, organizes, and enriches data

PI Data

Archive

PI Asset Framework

Asset Analytics

Event Frames

Notifications Automation or

Control Systems

New Sensor

Technology

Remote and

Mobile Assets

PI ProcessBook

PI Manual Logger

PI DataLink

PI Vision

PI Server PI Interfaces & PI Connectors

SQL (OLEDB, JDBC, ODBC)

Web API (REST)

OPC (DA, HDA)

SDK (.NET-based)

PI Visualization Suite

PI System Access


Challenge: Bring trustworthy data to advanced analytics

11

Financial

Inventory

Maintenance

Logs

Data Warehouse

Data Lake

Machine

Learning

Business

Intelligence

Tools

Stream Processing

GIS

Data stores Advanced Analytics Components

PI System can integrate across

advanced analytics architectures

and

deliver authoritative operational data

PI System PI Integrator


Challenge: Bring trustworthy data to advanced analytics

12

Operational

Financial

Inventory

Maintenance

Logs

Data Warehouse

Data Lake

Machine

Learning

Business

Intelligence

Tools

Stream Processing

GIS

Culprits preventing trust:

Raw data replication

Manual edits to data

Data arrives out-of-order

Impact:

Unrealistic predictions

Two databases not synchronized

Analytics disconnected from physics

Incorrect calculations

Combine data to:

• Report

• Slice and dice

• Predict

• Train algorithms

Data stores Advanced Analytics Components


13

Architectures


14

Add speed and scale to

existing architectures


15

Reference design pattern for

Industrial IoT Applications and

Streaming Systems and

Greenfield Data Systems


16

Shift from single vendor integrated

solutions to best of breed customer

integrated solutions

Be aware of the transfer of risk

from vendors to open source, IT,

SIs


PI System – An Integrated Kappa Architecture System

17

400+

Industr

ial

Protoc

ols

Optimized Time

Series Sequential

Store

Integrated Real-

Time Analytics

Critic

al

Event

s

Integrate

d

Visualiza

tion &

Program

ming

Access


Product Portfolio (Infrastructure & Advanced Integrations)

18

PI Connectors &

PI Interfaces

(400+ Industrial

Protocols)

PI Data Archive

(Optimized Time

Series Sequential

Store)

Asset Framework

(AF) Analytics

(Integrated Real-

Time Analytics)

Event

Frames (EF)

(Critical

Events)

PI Vision, PI

DataLink, PI

System

Access

(Integrated

Visualization &

Programming

Access)

PI Server PI Integrators

PI Integrator for Business Analytics

PI Integrator for SAP HANA

PI Integrator for Microsoft Azure

External Systems


19

Roadmap & Positioning


PI Integrators augment the existing set of PI capabilities like

visualization, data access, and analytics with capabilities that

make it easy to interact with non-OSIsoft tool sets.

20


PI Integrators – Climbing the Value Ladder

21

Real-time visibility

Monitoring

Process Optimization

Benchmarking

System Optimization

• BI App (i.e. Tableau, Spotfire, Lumira)

• PI Integrator for Business Analytics

• PI Integrator for SAP HANA

• Machine Learning (Azure ML, R)

• PI Integrator for Business Analytics

• PI Integrator for SAP HANA

• HMI

Disparate assets or interacting one-by-one Interacting with common assets as a fleet

Real-time & historical view across any plant

asset

Fleet-wide performance comparison

Large scale multi-variate analysis

Co

mp

lexi

ty

• PI ProcessBook • PI Coresight • PI Datalink


Business

Intelligence

& Data

Warehouses

Available Today

PI Integrator for Business

Analytics

• Microsoft SQL Server, Oracle

• Hadoop (HDFS/HIVE)

PI Integrator for SAP HANA

Available

Cloud Platforms

• Microsoft Azure

• HANA Cloud Platform (5/2017)

Streaming

Systems

Real-Time GIS

PI Integrator for Esri ArcGIS

• Situational Awareness

• Real-Time Geoprocessing

• Import ESRI features (assets)

Considered (2018)

Stream Systems

• AWS Kinesis

• Esri GeoEvent Server

(Kafka)

2015-2016 2017 Future

New Integration

Patterns

Research

Enable partners and customers

to build applications and interact

programmatically using PI

Integrator Framework.

PI Integrator

Framework

Planned (Q4 2017)

• Process Scale out*

• SSL / HTTPS

Research

Enable business process

orchestration with PI System

data – workflow, asset sync,

transaction-like data, MES

Considered (2018)

More Platforms

• AWS Redshift

• Teradata

Planned (2H 2017)

Stream Systems

• Azure Event Hubs, IoT Hub

• Apache Kafka

• SAP SDS (Available 5/2017)

Planned (2018)

• All Integrators on common

Framework (ESRI)

• Node Scale Out and HA

Research

IoT Platform Integration with 3rd

parties

Aug 2017


23

PI Integrator Concepts


Integration Patterns

Tables

Files

Databases

Streams Other Patterns

Metadata

Programming

On-Demand

Workflow & Transactions

Files

Queues

Messaging

External Analytics Engines


Tables vs Streams

• Business Intelligence

• Human readable

• Batch / Bulk Process

• Normalized data

• Regularly scheduled

• Large data, few messages

• In-line (Streaming) Analytics

• Computer readable

• Specific Data / Targeted Process

• Raw or “Packages” of data

• Triggered

• Small data, many messages


Data Modeling

26

Data Modeling • Reference the entire

scope of the PI System using PI Asset Framework

• Normalize data across similar and dissimilar assets and equipment using templates, categories, and pattern searches

• Enrich datasets with location, status, calculations, and other metadata


Message Designer

27

Message Designer • Variety of message

output formats such as JSON

• Customize the output schema, message triggering, and message content

• Filter erroneous data

• Replay historical data


28

Use Cases


Soft Measurements – Ideal Use Cases for Streaming

• Predicting Soft Measurements

– Deschutes Brewery – Fermentation Transition

– [Pharma Company] – Lights Out Manufacturing

– Quality

• Enabling Data Lakes

– Customize how data is stored & processed

29


Transition occurs between infrequent manual measurements

30

Fermentation

Diacetyl

Rest Maturation

Filling Free Rise Cooling Ready to

Transfer

Emptying

Empty

Constraints • One manual density measurement per vessel every 8-10

hours • Large capital expenditure not an option

Impact • Up to 72 hours lost in production

Options • $750k for inline density meters • Manually predict transition in spreadsheets

9 3 1

2 4

5

6

7

8

Time (hours)

Ap

pa

ren

t D

eg

ree

of

Fe

rme

nta

tio

n

Transition

Too late

Can this be predicted?

Soft Measurement


SQL Data Warehouse

On Premises

Machine Learning

Power BI

INGEST PREPARE DATA SOURCES

ANALYZE PUBLISH CONSUME

Cortana

Web/LOB Dashboards

SQL Data Warehouse

PI System

On Cloud

Azure Data Factory (Orchestration)

Predictions as Future Data (to PI 2015)

PI Integrator for Microsoft Azure

Deschutes Brewery Operationalizing Architecture

31


Automate and Optimize

32

Lab Measurements

• Viable Cell Count (VCC)

• Sugar Concentration

• API Concentration

• …

Online Measurements

• Agitation Speed

• pH

• Spectroscopy

• …

Goals: Optimize Cell Production, Eliminate Coming to the Lab

Technique: Use indirect measurements and machine learning

to predict outcomes of lab measurements


What success looks like for our customers

Customer What

was done

Why

it matters

PI Integrator

Type

Industry

category

Marathon Oil Centralized, trusted

source for IT and OT

data sets

• Decreased time to analyze holistic data set from 3 months

to 3 weeks

• Decreased unplanned well downtime

SAP HANA Oil & Gas

White House

Utility District

Sped up identification

of water leaks across a

large rural area

• Optimized services team’s workflow to save $30,000 per

year

• Saved $900,000 in 2 years by preventing ongoing water

leaks

Esri ArcGIS Water

CEMEX Democratizing data for

decision makers

• Reduced time to begin analysis from months to minutes

• Reduced product variations significantly

Business

Analytics

Metals,

Mining, &

Materials

Deschutes Predicting events in

brewing process

across all brands

• Avoided $750,000 cost to automate density

measurements

• Saving on average 48 hours of production time per batch of

beer

Microsoft

Azure

Food &

Bev

Archer

Daniels

Midland

Prescribe operating

parameters that

extend equipment’s life

• Losing $50,000 every time machine fails, and 28 machines

failing at a high and unexpected rate

• Need advanced analytics approach to analyze 100s of

variables that could be influencing failure

Business

Analytics

Food &

Bev

http://www.osisoft.com/Presentations/The-PI-System-and-SAP-HANA--Operating-at-the-Speed-of-Business/



https://www.osisoft.com/Presentations/Maximizing-Operations-Utilizing-Real-Time-Spatial-Information/




http://www.osisoft.com/Presentations/Transforming-Process-Data-Into-Information--PI-Integrator-for-Business-Analytics/

http://www.osisoft.com/Presentations/Deschutes-Brewery--Reducing-Beer-Production-Time-with-Predictions/

http://www.osisoft.com/Presentations/Using-the-PI-Integrator-for-Business-Analytics--BAx-to-Operationalize-Predictive-Analytics-with-Azure-ML/





Announcing Community Technology Preview

• PI Integrator for Business Analytics 2017 CTP (Open)

• PI Integrator for Microsoft Azure 2017 CTP (RCIP)

34


35

Demo


Reference Architecture for Streaming Analytics

36

Apache Kafka

PREPARE DATA SOURCES

STREAM

PI System PI Integrators Apache Spark

ANALYZE

Predictions as Future Data

Open Source Integration Pattern (Lambda Architecture) Primary Use Cases – • Deliver modeled input data to

CEP calculations and write predictions to operational system of record

• Fill a data like and maintain 1:1 fidelity of raw data or projected data from operational system of record

• Deliver aligned array data at scheduled intervals to R or Spark based compute engines


37


38

Questions

Please wait for the

microphone before asking

your questions

Please remember to…

Complete the Online Survey

for this session

State your

name & company


Matt Ziegler

[email protected]

Product Manager

OSIsoft

Daniele Farris

[email protected]

System Engineer

OSIsoft

39

mailto:[email protected]

mailto:[email protected]


Thank You


Presented by

Documents

Streaming Analytics, Data Lakes and PI Integrators...Streaming Analytics, Data Lakes and PI Integrators Matt Ziegler ... • Demonstrate PI Integrator for Business 2017 with Kafka