Querying and Monitoring Distributed Business Processes

Querying and Monitoring Distributed Business Processes

Daniel Deutch, Tova MiloTel-Aviv University

ERPERP

HR HR SystemSystem

eComeCommm

CRMCRM

LogisticsLogistics

CustomerCustomer

BankBank

SupplierSupplier

Querying and Monitoring Distributed BPs2

Students Takes

sid=sid

sname

name=“Mary”

cid=cid

Courses

SSN Name Category123-45-6789 Charles undergrad234-56-7890 Dan grad

… …

Select… From… Where…

Students

Optimization

Indexing

Transactions

Files organization

Distribution

...

Data model

Design

Query language

Streams

...

XML

SOAP

WSDL

...


Outline

Introduction to Business Processes

Querying

Monitoring

Summary & Research Directions


Outline


Querying

Monitoring



Logically related activities that, when combined in a flow, achieve a business goal.

Activities may either be local or remote

Operates in a cross-organization, distributed environment

Abstract representation, independent of implementation

Standards facilitate design, deployment, and execution

What is a Business Process?Introduction to BPs


BPs are designed by Non-programmers

Specify combination of functionalities and flow thereof, to solve a complex problem

Example: process an order

The operations/functions are implemented by programmers (programming in the small)

Example: fetch order document

Programming In the largeIntroduction to BPs


Abstract Process Defines the publicly visible behavior of

services that an executable process offers

Executable Process Contain the partner’s business logic behind

an external protocol

Abstract vs. Executable processIntroduction to BPs


Orchestration: Executable process, message exchange sequences are controlled by the orchestration designer.

Choreography: Non-executable protocol for interactions. E.g., the legal sequences of messages exchanged,

guaranteeing interoperability

Orchestration vs.ChoreographyIntroduction to BPs


OrchestrationIntroduction to BPs


Web Service 1

Web Service 2

Web Service 3

Web Service 4

Web Service 5

Web Service n

Company A business process

Local to company A

At company BOn the Web

Web Services Meet BPsIntroduction to BPs


Travel Service

Airlines Websites

ConsolidateResults

Travel request

Confirmation

Hotels Websites

queries

responses

queries

responses

Web-Based Travel Agency BPIntroduction to BPs


Planning

Modeling & Design

Development & Deployment

Execution, Interacting &

Monitoring

Analysis and Optimization

Source: Microsoft BPM Description

BP ManagementIntroduction to BPs


Complex Systems Distributed Settings Interoperability issues Robustness Scale Web interface Legacy systems

Modeling ChallengesIntroduction to BPs


BP market (BPTrends survey)Introduction to BPs


Over 20 million hits in google for “business process”

Over 5 million hits for "business process management"

Vast interest by analysts (e.g. Gartner)

Rapidly growing interest in industry

New Standards

BP Management buzzIntroduction to BPs


2000/05

BPML(Intallio et al)

WSFL(IBM)

BPSS(ebXML)

BPEL4WS 1.0 (IBM, Microsoft)

BPEL4WS 1.1

(OASIS)

WS-Choreography(W3C)

WSCI(Sun et al)

WSCL(HP)

BPEL

XLang(Micorsoft)

2001/03 2001/05 2001/06 2002/03 2002/06 2002/08 2003/01 2003/04 2007/04

WSBPEL 2.0

(OASIS)

2007/06

BPEL4 PEOPLE(Oracle et. Al)

Standards HistoryIntroduction to BPs


Standards stackIntroduction to BPs


Language for specifying BP behavior based on Web Services (WS)

Define BPs as coordinated sets of Web service interactions

Define both abstract and executable processes

Specifies Web services Composition

BPEL in a nutshellIntroduction to BPs


<receive> <reply> <invoke> <assign> <throw> <wait> <sequence> <flow> (parallel) <if> <while> <repeatUntil>

…

Communication-related constructs

Flow-related constructs

BPEL constructsIntroduction to BPs


BPEL is XML-based

In general we could use XML editors for specification design

Infeasible in practice

BPEL as XMLIntroduction to BPs


<process> <!– Definition and roles of process participants -->

<partnerLinks> ... </partnerLinks> <!- Data/state used within the process --> <variables> ... </variables> <!- Properties that enable conversations --> <correlationSets> ... </correlationSets> <!- Exception handling --> <faultHandlers> ... </faultHandlers>

<!- Error recovery – undoing actions -->

<compensationHandlers>...</compensationHandlers> <!- Concurrent events with process itself --> <eventHandlers> ... </eventHandlers> <!- Business process flow --> (activities)*</process>

BPEL as XML (cont.) Introduction to BPs


<if> <condition> bpel:getVariableProperty('shipRequest', 'props:shipComplete') </condition> <sequence> <assign> <copy> <from variable="shipRequest" property="props:shipOrderID" /> <to variable="shipNotice" property="props:shipOrderID" /> </copy> <copy> <from variable="shipRequest" property="props:itemsCount" /> <to variable="shipNotice" property="props:itemsCount" /> </copy> </assign>

If (shipRequest = shipComplete)

{

shipNotice.OrderId = shipRequest.OrderId;

shipNotice.itemsCnt = shipRequest.itemsCnt;

}

BPEL as XML (cont.)Introduction to BPs


Basic activitiesIntroduction to BPs


Flow activitiesIntroduction to BPs


BP specificationIntroduction to BPs


Travel Agency Process FlowIntroduction to BPs


Zoom InIntroduction to BPs


Example Editor (eclipse)Introduction to BPs


Example editor (Oracle)Introduction to BPs


Example editor (IBM)Introduction to BPs


Example editor (Microsoft VS)Introduction to BPs


So far: challenges & solutions for modeling

BPs are hard to analyze, debug, and optimize

Again, due to scale, distributed settings, legacy systems,…..

Good modeling simplifies process specification

But further analysis tools are required

ChallengesIntroduction to BPs


Static Analysis• “What kind of credit services are used (in)directly?”• “How can I buy a plane ticket?”• “Can one get a price quote without giving first credit card info?”

Monitoring• “Notify me when a user hacks the system and get a price quote

without giving his credit card info”

Log Analysis• “Find all logs where a user bought a plane ticket”

Analysis TypesIntroduction to BPs



Querying

Monitoring


Outline


Statically analyze a Business Process “Find all ways in which a user can buy a plane ticket”

Analysis needs• Control flow analysis (Reachability, Cycle Detection, Temporal properties,…)• Structural analysis• Data analysis

• Messages Validation, Pointer analysis, Array bounds analysis,…• Combined (data and flow) analysis

Database approach• Treat BPs as data• Design a query language

Querying Business ProcessesQuerying BPs


Recall that BPEL is XML-based, then… Why not use XQuery?!

Similar arguments to BPs / XML

• Need to handle complex technical constructs

• Complex, unintuitive queries (many joins & recursion)

• No abstraction

• No robustness to changes in BPEL standard

Why not XQuery?Querying BPs


Uniform

Graphical

Scalable

Flexible

Similar to the specification design

Can handle partial information and uncertainty

An Ideal querying/analysis toolQuerying BPs


Finite State Machine

Recursive State Machine

Context Free Graph Grammars

LTL

Monadic Second Order Logic

CTL

CTL*First

Order

Logic

Mu-calculus

Models & Query LanguagesQuerying BPs

Temporal Logic


• Finite State Machines (FSM) for software specification

• Temporal Logic (TL) [Emerson,’90] for querying all possible behaviors

• Very common in software (and hardware) verification

• Typically Linear time evaluation (data complexity)

• Exponential time (query complexity)

First TryQuerying BPs


Finite State Machines (FSM)

• States and transition function (typically no “accepting” state)

• The system configuration is encoded within the states

• Interested in properties of possible traversal over the states

• Temporal Logics express such properties

Finite State Machines (FSM)Querying BPs


• “Flat”– No functions– No recursion

• Cycles allowed• Execution path

by traversal

Payment

CreditCash

Login

Search Reserve

Confirm Cancel

FSM (example)Querying BPs


• Predicates x=0? Was a reservation made?

• Logical operators (and, or, not)

• Queries – Can a reservation be made without relaying a credit

card number? – Must one eventually login if he makes a trip search?

Temporal Logic - IngredientsQuerying BPs


• Quantifiers over execution paths A φ - All: φ holds on all paths starting from the current

state. E φ - Exists: φ holds in at least one path.

• Path-specific quantifiers X φ - Next: φ holds at the next state. G φ - Globally: φ has to hold on the entire subsequent

path. F φ - Finally: φ eventually has to hold (somewhere). φ U ψ - Until: φ has to hold until at some position ψ

holds, and ψ must hold eventually. φ W ψ - Weak until: φ has to hold until ψ holds.

Temporal OperatorsQuerying BPs


• Queries – Can a reservation be made without relaying a credit card

number? E(F(Reserve) and not F(Credit)) – Must one eventually login if he makes a trip search? A (F(login) or not F(search))

• Logics– Linear time Logic (LTL)

• No path quantifiers

– CTL* • Allows path quantifiers

– Mu-calculus • Introduces fix-point operators

First Order

(Monadic) Second Order

Temporal LogicQuerying BPs


1. Scale

2. Expressive Power (Specification)

3. Expressive Power (Query language)

4. (Un)Intuitive Formulation

Features & LimitationsQuerying BPs


• Evaluation is linear in FSM size, but…

• FSM size is huge for real-life specifications

• Call stack• Data• Unfeasible Evaluation

Payment

CreditCash

Login

Search Reserve

Confirm Cancel

1. ScaleFeatures and Limitations


• Bounded Model Checking [Biere et. Al ’99,’03] ,[Clarke et. Al ’04], …

• Summarization [Reps et. Al ’98], [Sagiv et. Al ‘05],…

• Compact representation – StateCharts [Harel ’87] – BDD [Bryant 86’, Lam et. Al ’05]

SolutionsFeatures and Limitations1. Scale


• Graphical tool for designing state machine based specifications

• Extend basic FSM concepts (super-states)

• Simplifies design• UML standard

StateChartsFeatures and Limitations1. Scale


• Data structure that allows compact representation of data with high similarities

• (b1^ b2) V (not(b1) ^ b3)

b1

b3

b2

1 0

BDDFeatures and Limitations1. Scale


• Store commands as db relations – command c is written at location l

• “Open” all possible contexts

• Exploit similarities and represent compactly by Binary Decision Diagrams

• Datalog queries

Relational DB approach [Lam ’05]Features and Limitations1. Scale


• FSMs have limited expressive power

• May yield inaccurate approximation of real-life specification

2. Expressive Power (Specification)Features and Limitations






First

Order

Logic

Models & Query LanguagesExpressive Power (Specification)

Temporal Logic


• A collection of FSMs

• Each with multiple entries & exits

• Some states represent calls to other state machines (or to self)

• An expansion is replacing a call state by a possible implementation

• An execution is a sequence of expansions

Recursive State Machines (RSMs)Expressive Power (Specification)


RSM (example)Expressive Power (Specification)

paymentconfirm Start

Payment

HomePage

cash

search

reserve

paycredit

Call nodes may specify several implementations

Call nodes may specify entry/exit nodes for connection


• Single Exit RSMs – CTL* : Linear data complexity [Benedikt et. Al

‘05]– Mu-calculus: also [Alur et. Al ’07]

• Multiple Exit RSMs– LTL: PTIME– CTL,CTL*: EXPTIME– Mu-calculus: EXPTIME

Evaluation (Temporal Logic)Expressive Power (Specification)






First

Order

Logic

Models & Query LanguagesExpressive Power (Specification)

Temporal Logic


• Extensions of string grammars to graphs

• Labels over graph nodes (VR) or edges (HR)

• Terminal and non-terminal labels

• Derivation rules for non-terminal labels

• No start and end nodes!

• Connection rules, by labels, for the derived sub-graph

Context Free Graph GrammarsExpressive Power (Specification)


Connection relation :

StartPayment connects only to Search

Confirm connects to cash, but not to credit

Context Free Graph Grammars Example

Expressive Power (Specification)

paymentconfirm

Homepage

Start

Payment

HomePage

payment

cash

search

reserve

paycredit


• Depends greatly on the allowed connection relation

• A restricted model defines entries and exits for graphs and is equivalent to RSMs

• Typically Monadic Second Order Queries

• We’ll revisit it later..

Context Free Graph GrammarsEvaluation

Expressive Power (Specification)


• AXML [Abiteboul, M et. Al ’04-’08]• Extension of XML to include embedded Web-

Services calls• Very useful for modeling web-sites• Formally – a restriction of context free graph

grammars to trees• XML query languages

– XPath,XQuery,..

• Practically efficient evaluation

Active XMLExpressive Power (Specification)


• Temporal logics have limited expressive power

• Basically – consider only executions

• Good for behavioral properties

• Can’t capture structural properties

• Bisimulation-invariant

• Example

3. Expressive Power (Queries)Features and Limitations


Behavioral vs. Structural AnalysisExpressive Power (Queries)


Solution?Expressive Power (Queries)





First

Order

Logic

Temporal Logic


• Studied extensively for Context Free Graph Grammars

• Linear in grammar size

• But unfortunately…

• Non-elementary in the query (formula) size

• 2^2^2^…..2 (tower size depends on query size)

• Infeasible for even the smallest queries

Infeasible!Expressive Power (Queries)


4. (Un) intuitive Formulation• Very difficult to express properties of

interest in FO or MSO

• Long and error-prone formulas

• Temporal Logic is more intuitive

• Still, textual and complex, especially for large-scaled analysis

• Existing tools provide inadequate interface

4. (Un)intuitive FormulationFeatures and Limitations


r[n,z](v) = (c[n](v) & r[n,x1](v)?

z(v) | E(v_1) z(v_1) & TC (v_1, v) (v_3, v_4) (n(v_3, v_4) & !x1(v_3)) :

r[n,z](v) & ! (E(v_1) r[n,z](v_1) & x1(v_1) & r[n,x1](v) & !x1(v)))

First Order FormulasTVLA (Sagiv et. Al [04]) syntax

(Un)intuitive formulation


• PReqFullfilledDef1: assert (P_BUTTON_PRESSED -> (~P_REQ_FULFILLED U P_state));

• PReqFullfilledDef2: assert (P_state -> (P_REQ_FULFILLED U P_BUTTON_PRESSED));

• EnterTStateDef: assert ((~T_state & X(T_state)) -> X(ENTER_T_STATE));

• EnterPStateDef: assert ((~P_state & X(P_state)) -> X(ENTER_P_STATE));

• MoveToPPrevDef: assert ((~move_to_p & X(move_to_p)) -> MOVE_TO_P_WAS_SET_TRUE_IN_PREV_SEC);

• MoveToItoTPrevDef: assert( (~move_to_i_to_t & X(move_to_i_to_t)) -> X(X(MOVE_TO_I_TO_T_WAS_SET_TRUE_IN_PREV_SEC)));

Temporal LogicSMV (Clarke et. Al [04]) syntax

(Un)intuitive formulation


• Model problems – expressive power – scale

Pretty much solved by models we’ve seen• Query Language problems

– expressive power – unintuitive formulation

Not solved yet• BPQL to the rescue

Querying BPs Mid-section Summary


Possible SolutionQuerying BPs





First

Order

Logic

Temporal Logic

BPQL


BP patterns (like tree patterns for XML)

Single/double-headed edges (Xpath’s / and //)• edges

• paths of arbitrary length

Single/double-bounded activities:• simple zoom-in • unbounded zoom-in

BPQL QueriesQuerying BPs


local

Q1: used credit card services?Querying BPs


Q2: search without login?Querying BPs


Q3: data flowQuerying BPs


Query: A BP pattern with some transitive nodes & edges transitive

An embedding: a mapping • from: query graphs • to: [possible flows defined by the] BP graphs

A result: image of query graph under an embedding

Answer: all results

Queries and their semanticsQuerying BPs


• Sub-graph homomorphism vs. bisimulation• Both are supported in BPQL

Structural vs. Behavioral SemanticsQuerying BPs


Query with an infinite answer?Querying BPs


Finite representationQuerying BPs


Systems and queries are essentially Context Free Graph Grammars (Recursive state machines)

We basically compute their intersection

Bad news: These are not closed in general under intersection

Good news: • Our systems and queries are sufficiently simple:

PSIZE representation (as a BP) can be computed in PTIME (data complexity!)

Query Evaluation AlgorithmQuerying BPs


So far we have mainly considered flow

Data is important as well

Data: variable values, message exchange,…

Especially interesting in context of Web

Some representative works Web-services analysis Pointer Analysis

And what about data?Querying BPs


[Deutsch et. Al ‘06] • Query language combines LTL and FO• LTL for temporal relationships• FO for data snapshots• Polynomial Data Complexity (restricted versions)

[Fu et. Al ’04] • Guarded automaton for flow query• XPath “guards” relate to data• Reduces problem into “conventional” model checking• Undecidability for general case• Polynomial Data Complexity for bounded message number

Web-services analysisQuerying BPs


[Lam et. Al ’05]• Pointer analysis for sql injections, buffer overflow,…• Analyzes all possible call stack contexts• BDD-based optimization techniques capture

similarities• Efficient run-time, practical system

[Sagiv et. Al ’06]• Shape analysis• Summarization techniques• Efficient run-time, practical system

Pointer AnalysisQuerying BPs


SMV, NuSMV [Clarke et. Al, ’92] bddbddb [Lam et. Al, ’04] TVLA [Sagiv et. Al, ’99] SPIN [Bell Labs,’91] SLAM [Ball et. Al. ’00] Moped [Schwoon ’02] Mops [Chen et. Al ’02] BPQL [Beeri, M, D et. Al ’05]


Context Free Processes

Some SystemsQuerying BPs



Querying

Monitoring


Outline


The aggregation, analysis, and presentation of real time information about activities inside organizations and involving customers and partners (Gartner)

Provide real-time information on executions

What is Monitoring?Monitoring BPs


Imagine you run an auction service…

• Guarantee fair play: notify on too many cancels• Maintain SLA: monitor response time• Promotions: prizes for the x10,000 transaction• Illegal access: notify on buyers attempt to confirm

bids without registering first

Monitoring is crucial for enforcing business policies and meeting efficiency & reliability goals

Why Monitoring?Monitoring BPs


<actionData> <header> <processName> auctionHouse </processName> <instanceId> 517 </instanceId> <sensorTarget> notify_winner </sensorTarget> <timestamp> 2006-05-31T11:32:46.510+00:00 </> </header>

… <activityData> <activityType>invoke </activityType> <evalPoint> completion </evalPoint> …

BPEL XML eventsMonitoring BPs


1. Absorb the stream of events coming from the BP execution engine

2. Process and filter events, selects relevant events data and automatically triggers actions

3. A dashboard that allows users to follow the processes progress, view custom reports, perform analysis,…

Monitoring Systems LayersMonitoring BPs


XML streams management

Complex Event Processing (CEP)

Commercial tools) BAM(

BP-Mon

Existing ApproachesMonitoring BPs


Many works on xml streaming• Query Optimization [Koch et. Al ’04, Viglas et. Al ’02,…] • Security [Altinel, Franklin ’00, Benedikt et. Al, ’08,…]• Updates & Concurrency [Grabs et. Al ’02, Nicola et. Al ’07,…]

Automata-based techniques vastly used

BPEL processes emits XML messages

Use XML streaming engines for monitoring?

XML streams managementMonitoring BPs


Each XML element describes an individual event

Fairly complex XQuery queries (lots of joins)

• Difficult to handle by existing streaming engines…

XML stream engines manage tree-shaped data (vs. DAGs)

XML stream engines expect to receive elements in document order (but we have here parallel flow).

Why not ?Monitoring BPs


Processing of multiple events to identify semantically meaningful combinations.

Studied extensively for Active Databases [Widom ’96], [Payton ’99], [Wolski ’98],…

Recent works in context of BPs [Wu ’06], [Jobst ‘07]

Identify meaningful activities combinations that form a business logic

Still, somewhat low-level

Complex Event ProcessingMonitoring BPs


BAM tools provide solid solutions

But do not incorporate static analysis

BP-Mon is an integrated framework Uses a query language similar to BPQL Graphical & Intuitive Semantics: evaluated over run-time executions (vs.

potential executions in BP-QL)

BP-MonMonitoring BPs


Running ExampleMonitoring BPs


Unfair play (too many cancellations)

As before

Report/ Report*

New

or rep

Query Example (1)Monitoring BPs


Monitor response time of data-store

Sliding window• Time based• Instance based

Output• Xquery like• Group by, having



Illegal bidding (mix of static and run-time analysis)



Intuitive monitoring (looks like BPEL) Easy deployment (implemented as BPEL)

Greedy embedding Automata-based evaluation algorithm Type-based optimization

Some Nice BP-Mon FeaturesMonitoring BPs


• Incrementally extends a greedy matching to one of a larger prefix

• Automaton with pattern nodes as states– Tries to concurrently match the all concrete

patterns of a given pattern– Attempts to match events as early as possible– On failure: backtracks & retries

Complexity: polynomial in the size of the execution log (with the exponent determined by the size of the pattern)

Evaluation algorithmMonitoring BPs


The automaton is compiled into BPEL

May be used on any application server

Guarantees portability

Easy DeploymentMonitoring BPs


Exploit knowledge of specification Infer Irrelevancy & inconsistency using BP-QL

Type-Based OptimizationMonitoring BPs



Querying

Monitoring


Outline


Important aspects of Business Processes• Design• Analysis• Monitoring

Plenty of work on each subject, in many different fields Still missing a real sinergy

• Programming languages, model checking, DB technology• XML Streaming, CEP/BAM• All together…

BPQ: integrated, high-level, intuitive framework for all.

Summary & Research DirectionsAlmost done


Topics for research:• Missing information (partial specifications, logs,…)• Probabilistic Processes• Data values• Interactions• Log mining• Enhanced query language features• Optimizations • Further applications • and more…

Summary & Research DirectionsAlmost done

Documents

Querying and Monitoring Distributed Business Processes