Complex Event Processing: Power your middleware with
StreamInsight Mahesh Patel (Microsoft) Amit Bansal
(PeoplewareIndia.com)
Slide 2
Agenda The Value of Timely Analytics The challenges / Scenarios
Introduction to Complex Event Processing Introducing Microsoft
StreamInsight Brief introduction to BizTalk Server Understand the
key scenarios that StreamInsight can unlock for BizTalk Map out the
architectural, integration requirements for BizTalk +
StreamInsight.
Slide 3
The value of timely analytics
Slide 4
Current Products for analytics
Slide 5
Slide 6
Introduction to CEP Complex Event Processing (CEP) is an
architecture style based on the principles of Event Driven
Architecture (EDA). Continuously processing a high-volume stream of
events from various event sources Events are logically grouped in
sequences (typically known as streams) based on a time defined
criteria such as an interval. Event streams are processed through a
series of queries
Slide 7
Capabilities of a CEP Application Event Driven Continuous
Queries Time Sensitive Low Latency High Volume
Slide 8
The value of timely analytics
Slide 9
Scenarios Stream Data Store & Archive Event Processing
Engine Data Stream Asset Specs & Parameters Power, Utilities:
Energy consumption Outages Smart grids 100,000 events/sec Visual
trend-line and KPI monitoring Batch & product management
Automated anomaly detection Real-time customer segmentation
Algorithmic trading Proactive condition-based maintenance Visual
trend-line and KPI monitoring Batch & product management
Automated anomaly detection Real-time customer segmentation
Algorithmic trading Proactive condition-based maintenance Web
Analytics: Click-stream data Online customer behavior Page layout
100,000 events /sec Manufacturing: Sensor on plant floor React
through device controllers Aggregated data 10,000 events/sec
Threshold queries Event correlation from multiple sources Pattern
queries Threshold queries Event correlation from multiple sources
Pattern queries Lookup Asset Instrumentation for Data Acquisition,
Subscriptions to Data Feeds Financial Services: Stock & news
feeds Algorithmic trading Patterns over time Super-low latency
100,000 events /sec Data Stream
Slide 10
Introducing StreamInsight
Slide 11
DEMO CEP using StreamInsight
Slide 12
Traditional messaging applications: High signal every message
is business relevant Rich data schemas sets of related information
Workflow centric and transactional Emerging data sources and
trends: Machine born data is growing at a very rapid rate Not all
of this data is business relevant (low signal/noise ratio) How do
we: Identify and extract business relevant events from streaming
data? Take action on these business insights (line of business,
human workflow, etc) Messages and Events LOB Large Messages Low
Volume High Volume Small Events Web Services B22B SWIFT RFID Stock
Feeds Electrical Grids Operational Data Health Care CRM
Slide 13
Send Port MessageBox Orchestration Receive Port Message Maps
Receive Pipeline Adapter (HTTP) Message LOB Host Send Port Adapter
(File) Map Send Pipeline Component Basic BizTalk Architecture
Component Host Management DBTracking DB Message Agent
Slide 14
Scenarios Good BizTalk + StreamInsight scenarios: Streaming
data/event sources high noise to signal ratio Need to analyze and
extract business relevant events Integrate and transact with line
of business applications.
Slide 15
Scenarios Manufacturing (Oil and Gas) Monitor plant floor
processes Identify business relevant events Transact with LOB apps
Utilities Automated metering handle flood of data from utility
meters Identify business relevant events Transact with LOB apps The
data feeds change, the question changes the core scenario
doesnt!
Slide 16
OSIsoft PI & StreamInsight OSIsoft PI - Industry leading
process data management and historian. Integrating and leveraging
StreamInsight OSIsoft shipping StreamInsight adapter for PI. Stream
data in and out of PI Extract powerful business insights from PI
data Integrate with LOB applications using BizTalk Server
Slide 17
OPC & StreamInsight (with MatrikonOPC) OPC Industry
standard for process automation and data exchange MatrikonOPC
Industry leader in OPC interfaces Developed StreamInsight adapters
for their OPC implementation Extract powerful business insights
from OPC data Integrate with LOB applications using BizTalk
Server
Slide 18
Automated Metering Contoso Utilities is rolling out smart
electrical meters to their customers. Meters report power usage
(wattage, voltage) at a point in time. Data is consolidated into a
back end application used for billing and forecasting. The volume
of data has forced Contoso to only read values from meters every
hour They want to: Monitor the data more closely, compressing the
point stream to an interval stream before loading into the back end
application. i.e. if the data does not change by more than 2% over
a 10 minute window, keep the average value. Capture spikes and
threshold events, route to a process control application for
further inspection.
Slide 19
Automated Metering Collect high volume streaming data from
smart meters Data consists of timestamp, source ID and payload
(power in Watts) Sample operations on the streaming data:
Consolidate stream of points events into 5 minute chunks; record
average, count and standard deviation. Detect anomalies (threshold
10%), collect last 5 minutes worth of data for that meter and all
meters fed from that substation Need to link meter ID with
reference data (location, substation) Results must be integrated
into line of business application
Slide 20
Automated Metering 1 1 2 2 3 3 4 5 4 5
Slide 21
Architecture Message Integration WCF Adapters BizTalk Host
OrchestrationOrchestration Business Activity Monitoring Business
Rules Engine Message Box StreamInsight XmlPayload Output Adapter
QueriesQueries Input Adapters 11 22 33 11 StreamInsight extracts
insights from real time data feeds, feeds aggregate information to
BizTalk. 22 A strongly typed XML schema is inferred from the
StreamInsight query 33 Batched and correlated query results are
delivered to a BizTalk receive port.
Slide 22
StreamInsight is an in-memory event processing engine. Do not
deploy StreamInsight instance on the same hardware as your BizTalk
host instances. Leverage StreamInsight-local MSMQ for message
transfer. Deployment Message Integration
Slide 23
Summary BizTalk Server and StreamInsight together can unlock
powerful scenarios: High volume data processing Message integration
Leverage StreamInsight to: Consolidation and analyze incoming data
streams convert event sets into messages Deliver to BizTalk receive
port Leverage BizTalk Server to: Integrate these insights into line
of Business applications Other potential alignment points: Business
Activity Monitoring (BAM). Tap into BAM event events stream for
rich analytics and event detections Business Rules Engine (BRE).
Leverage BRE to define event pattern rules
Slide 24
2011 Microsoft Corporation. All rights reserved. Microsoft,
Windows, Windows Vista and other product names are or may be
registered trademarks and/or trademarks in the U.S. and/or other
countries. The information herein is for informational purposes
only and represents the current view of Microsoft Corporation as of
the date of this presentation. Because Microsoft must respond to
changing market conditions, it should not be interpreted to be a
commitment on the part of Microsoft, and Microsoft cannot guarantee
the accuracy of any information provided after the date of this
presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR
STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.