The Need for an Event-Driven Platform
2
Analytical results need to reflect important changes in business reality immediately and enable responses to them with minimal latency
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec Tens of thousands of events/sec or more
Query Semantics
Declarative relational analytics
Declarative relational and temporal analytics
request
response
Eventoutput stream
input stream
Relational Database Applications
Financial trading Applications
Scenarios for Event-Driven Applications
Aggregate Data Rate (Events/sec.)
Latency
0 10 100 1000 10000 100000 ~1million
Months
Days
hours
Minutes
Seconds
100 ms
< 1ms
Operational Analytics Applications, e.g., Logistics,
etc.
Manufacturing Applications
Monitoring Applications
CEP Target Scenarios
Data Warehousing ApplicationsWeb Analytics Applications
3
Example Scenarios
4
Dat
a S
trea
m
Stream Data Store & Archive
Event Processing Engine
Dat
a S
trea
m
Asset Specs & Parameters
Power, Utilities:• Energy
consumption• Outages• Smart grids• 100,000
events/sec
Visual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentation Algorithmic tradingProactive condition-based maintenance
Web Analytics:• Click-stream
data• Online
customer behavior
• Page layout• 100,000
events /sec
Manufacturing:• Sensor on plant
floor• React through
device controllers
• Aggregated data
• 10,000 events/sec
• Threshold queries• Event correlation from
multiple sources• Pattern queries
Lookup
Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
Financial Services:• Stock & news
feeds• Algorithmic
trading• Patterns over
time• Super-low
latency• 100,000
events /sec
Standing Queries
Query Logic
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock ticker, news feeds
Event stores & Databases
Pagers &Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
InputAdapter
s
OutputAdapter
sStreamInsight
Engine
Query Logic
Query Logic
StreamInsightApplication Development
StreamInsight Application at Runtime
StreamInsight Platform
What is Project “Austin”?
Analytics
Integrated
• Rich temporal (StreamInsight) and sequential (Reactive Framework) analytics models
• Dynamic, flexible query and data source management experience
• Turn key connectivity for platform data sources and sinks (SQL Azure, Windows Azure Table Storage)
• Integrated with Azure management portal and billing experiences
• Real time data collection from wide variety of connected devices (Sensors, Smart Meters, Servers, Tablets, Phones)
• Standards compliant endpoints (REST, XML, JSON)
• Securable data ingress with data enrichment and transformation (geo-tagging, etc.)
• Multi-tenant Azure service with flexible, elastic capacity for collection and analytics
• Federated scale out collection and analytics
• Distributed service monitoring and tracing
Connected
Scalable
StreamInsight on Azure: “Austin”
Standing Queries
Stream-Insight Query
Prebuilt InputAdapters Austin
StreamInsight Engine
Reactive Query
Stream-Insight Query
StreamInsightApplication Development
StreamInsight Application at Runtime
Prebuilt Output
AdaptersData
Egress Adapter
Data Egress Adapter
RESTfulendpoint
AzureTables
Management Service Monitoring Service
Scalable Data Ingress Adapter
Authentication
Built-in Archive
EventsEvents expose different temporal characteristics
Point in time eventsInterval events with fixed durationInterval events with initially unknown duration
Rich payloads capture all properties of an event
t1 t4t3t2 t5Time
Pa
ylo
ad
/ v
alu
e
ab
cd
e
Event Types
Events in Microsoft’s CEP platform use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fields
Timestamps/Metadata
LongpumpID
StringType
StringLocation
Doubleflow
Doublepressure
… … … … … …
Event Streams & AdaptersA stream is a possibly infinite sequence of events
Insertions of new eventsChanges to event durations
Stream characteristics:Event/data arrival patterns
Steady rate with end-of-stream indication Intermittent, random, or in bursts
Out of order events: Order of arrival of events does not match the order of their application timestamps
AdaptersReceive/get events from the data sourceEnqueue events for processing in the engine
10
Typical CEP QueriesTypical CEP queries require combination of functionality
Complex type describes event propertiesCalculations introduce additional event propertiesGrouping by one or more event propertiesAggregation for each event group over a pre-defined period of time, typically a windowMultiple event groups monitored by the same queryCorrelate event streams Check for absence of activity with a data sourceEnrich events with reference dataCollection of assets may change over time
We want to make writing and maintaining those queries easy or even effortless
StreamInsight Query FeaturesOperators over streams
Calculations (PROJECT)Correlation of streams from different data sources (JOIN)Check for absence of activity with a data source (EXISTS)Selection of events from streams (FILTER)Stream partitioning (GROUP & APPLY)Aggregation (SUM, COUNT, …) Ranking and heavy hitters (TOP-K)Temporal operations: hopping window, sliding window
Extensibility – to add new domain-specific operators
LINQ Query Examples
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };
Join
FilterProject
Grouping
Window
Project &Aggregate
Extensibility SDK
Built-in operators do not cover all functionalityNeed for domain-specific extensionsIntegrate with functionality from existing libraries
Support for extensions in the CEP platform:User-defined operators, functions, aggregatesCode written in .NET, deployed as .NET assemblyQuery operators and LINQ can refer to functionality of the assembly
Temporal snap-shot operator frameworkInterface to implement user-defined operatorsManages operator state and snapshot changesFramework does the heavy lifting to deal with intricate temporal behavior such as out-of-order events
Resiliency
Outages happen in computingPower outages“Patch Tuesday”Human mistakes
Planned and unplanned downtimeSystems need to be “resilient” to outages
Minimize damageBecome operational again quickly
The specific requirements depend on how mission critical your applications is
Resiliency: Timeliness
Timeliness: recover from outages quickly.Goal is simple: as fast as possible.
StreamInsight doesn’t store event data, but it does store query state.
This may be significant.This may be slow to recreate.
Resiliency: Correctness
c) Rough aggregation: get the moving average price of a stock over the last day.• Missing a few inputs will result in inaccurate, but
close results.• Still don’t want to lose a day’s worth of work.
a) Exact equivalence.• The same stream of events, regardless of outage.
b) Equivalent events.• No missed events, and no wrong events, but
duplicates are allowed.
Three Levels:
What is Checkpointing?
Checkpointing saves a query’s state to disk.You control when the checkpoint is initiated.SI takes care of saving out consistent state.
After an outage, StreamInsight can restore this state.
This limits state loss during an outage, speeding recovery.Level of correctness depends on additional work we are able to perform.Recovery process is coordinated by SI.
Checkpointing API
public IAsyncResult server.BeginCheckpoint(Query query, AsyncCallback asyncCallback, object asyncState);
public bool server.EndCheckpoint(IAsyncResult asyncResult);
public void server.CancelCheckpoint(IAsyncResult asyncResult);
When is Checkpointing Useful?
Provides a mechanism to recover from an outage:
To recover from unexpected system failure.To handle expected outages (e.g., patch Tuesday).For machine migration.
Not a panacea:Does not provide uninterrupted service.Does not protect against broken query logic.
Using Checkpoints
We’ll walk through the three progressively-strict checkpointing scenarios:1. State retention.2. Equivalent events.3. Exact equivalence.
Checkpointing
jihgfedc …
jihgfedc …
Enqueue markers into input streams to instructoperators to save their state.
Filling the Gaps
StreamInsight needs help:Missing state since last checkpoint.Missed events during outage.
Solution: replayable adapters.The dance:1. StreamInsight picks a place in the input
stream.2. StreamInsight communicates this to the input
adapter.3. The input adapter replays from the chosen
spot.
A Place in the Stream
0
1
2
3
4
5
6
7
8
Ap
pli
cati
on
Tim
e
hgfedcba …
Physical Stream
0
1
2
3
4
5
6
7
8
Ap
pli
cati
on
Tim
e
0
1
2
3
4
5
6
7
8
Ap
pli
cati
on
Tim
e
0
1
2
3
4
5
6
7
8
Hig
h W
ater
Mar
k
Communicating the State
Input adapter factories can optionally implement one of
IHighWaterMarkInputAdapterFactoryIHighWaterMarkTypedInputAdapterFactory
In a recovery situation, StreamInsight will then call Create with a high-water mark.The factory is then responsible for properly cueing the input.
StreamInsight Design PrinciplesScalability – Aggregate data rate keeps increasing.
Minimum resources impact (co-located).Local computationAvoid flooding the network
ProgrammabilityExtensibility – UserDefinedAggregates, UserDefinedFunctions, UserDefinedOperators.Composability.Developer experience (language, IDE, debugging, supportability)
AdaptablityEasy to integrate via adapters.Portability (servers, edge devices)
34
StreamInsight Architecture
35
Host Process
...
Web Service
Engine
Compiler
Expression / Type
Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adap-ters
Host Process
...
Web Service
Engine
Compiler
Expression / Type Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adapters
Highlights
• Manageability API for query management (i.e. create, start, stop, delete query) and supportability / monitoring of running queries
• Same manageability API for both embedded deployment and web service clients
Management Service
Host Process
...
Web Service
Engine
Compiler
Expression / Type
Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adapters
Compiler & Expressions
Highlights• Standardized IL allows us to implement a
variety of syntactic surfaces over the algebra - e.g., LINQ, CQL, etc.• Allows for domain-specific front-end
languages• Prepared for future extensions
• Compile time type checking and type safe code generation for minimal runtime impact.
• Support for UDF’s, UDAggs, UDOs.• JIT code generation for field references ,
expression evaluation for low latency processing of high event rates.
• Basing on CLR helps leverage – • Code generator, JIT support • Type System• Tools and Libraries (LINQ Expressions,
IDE, etc.)
Host Process
...
Web Service
Engine
Compiler
Expression / Type Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adapters
Highlights• JIT code generation for field references, expression
evaluation because interpreting these references is sub-optimal for low latency processing of high event rates. • Leverage JIT code generation support in CLR runtime for
LINQ expressions.• Bind the query to different deployment environments based
on the metadata.• Event manager is implemented as a combination of
managed and native code in order to minimize overhead and ensure predictable performance.
• Events are read-only and referenced-counted by streams (minimize data copying)
Events & Streams
Host Process
...
Web Service
Engine
Compiler
Expression / Type Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adapters
Highlights• A query is executed by scheduling the individual operators
as they become active.• Operator state transition is managed by the Scheduler.• When an operator becomes active a thread is scheduled for
execution. • Scheduling decision based on priority of the query and other
parameters.• Data flow architecture: reduced coupling and pipeline
parallelism• Operators are affinitized to a thread/core (multi-core
environments) to decrease lock contention and increase caching benefits. Periodic checks and migration for load balancing
.
Query Scheduler
Host Process
...
Web Service
Engine
Compiler
Expression / Type Service
Runtime
Execution Operators
Stream Manager
Event Manager
Query Scheduler
Plan Manager
Synopsis
Command Dispatcher
Management Service
MetadataDiagnostics /
TracingStream OS
Adapters
Highlights• Efficient implementation of
operators that perform incremental evaluation as each event is processed.
• Clean, formal semantics. Leverage relational semantics whenever possible.
• GroupAndApply Operator • Enables parallelism for
scale-up (multi-core).• Groups are dynamically
instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active. (~50M active groups for MSN.com)
XY
Z
Group A,B,C
ApplyApplyApply
Union X,Y,Z
ZZ
Z
YY
Y
XX
X
BB
B
AA
AB
C
CC
C
Execution Operators
The StreamInsight TeamFounded in 2008 based on incubation between MSR and SQL teamsSmall team – by Microsoft standards Roles in Microsoft engineering teams
Program Managers: customer scenarios, functional specs, APIs, project mgmt, evangelismDevelopers: architecture, technical design, product code, unit testsTesters: test breakout, test code, lab runs, release signoff
Using agile development methods
StreamInsight Roadmap
StreamInsight 2.1 (on prem)
Development experienceMajor API overhaul
StreamInsight on Azure (Cloud)
StreamInsight service on Windows AzureCurrently private CTPGA this summer
• Using Scrum to organize and manage schedules• Work organized in sprints/milestones• CTP (Community Technology Preview) after each
milestone – similar to public beta• TAP (Technology Adopter Program) as we get closer to
the planned release
For More Information
StreamInsight download location: http://go.microsoft.com/fwlink/?LinkId=160598 StreamInsight blog: http://blogs.msdn.com/streaminsight/ StreamInsight MSDN documentation: http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx StreamInsight MSDN portal: http://msdn.microsoft.com/en-us/ee476990.aspx