© 2004 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved. Exceptions and Distributed Fault

© 2004 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved.© 2004 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved.

Exceptions and Distributed Fault Exceptions and Distributed Fault ManagementManagementExceptions and Distributed Fault Exceptions and Distributed Fault ManagementManagement

Michael WillersMichael [email protected]@newtelligence.com

Senior Architect, Senior Architect, newtelligence AGnewtelligence AG

Microsoft Security MVPMicrosoft Security MVP

© 2004, 2005 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved.© 2004, 2005 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved.

AgendaAgendaAgendaAgenda

Exception ManagementException Management•Defining termsDefining terms

•How far does the concept carry me?How far does the concept carry me?

Exceptions and TransactionsExceptions and Transactions•Comparison of concepts and scopesComparison of concepts and scopes

Distributed Fault ManagementDistributed Fault Management


What is an Exception?What is an Exception?What is an Exception?What is an Exception?

Definition:Definition:• Logically:Logically:

Violation of semantic contractViolation of semantic contract

• Programmatically:Programmatically: Obstacle in normal flow of algorithmObstacle in normal flow of algorithm An exception is an objectAn exception is an object

• Describes error condition by type and contentsDescribes error condition by type and contents

•Generally: Generally: Error condition that can't be solved at location of Error condition that can't be solved at location of

occuranceoccurance


When do they occur?When do they occur?When do they occur?When do they occur?

Exception can stem from:Exception can stem from:•Coding errorsCoding errors

•Unavailable resourcesUnavailable resources

•Unsolvable conditions for the CLRUnsolvable conditions for the CLR E.g. non-verifiable codeE.g. non-verifiable code

Application cannot continue reasonably in all Application cannot continue reasonably in all casescases•Most CLR exceptions cannot be solvedMost CLR exceptions cannot be solved

ExecutionEngineExceptionExecutionEngineException OutOfMemoryExceptionOutOfMemoryException


Exception Stack WalkException Stack WalkException Stack WalkException Stack Walk

First runFirst run•Starts with throw-statementStarts with throw-statement

•Walk up frame chain until handler is foundWalk up frame chain until handler is found

•Ends when handler is found or stack top is reachedEnds when handler is found or stack top is reached

Second runSecond run•Starts after first run has endedStarts after first run has ended

•Executes finally blocks in each frameExecutes finally blocks in each frame

If handler was found execute it – abort thread If handler was found execute it – abort thread otherwiseotherwise


What is Exception Management?What is Exception Management?What is Exception Management?What is Exception Management?

A strategy to… A strategy to… •Detect exceptionsDetect exceptions

•Collect and convey informationCollect and convey information

•Generate events for external monitoring systemsGenerate events for external monitoring systems

But: Murphy's law does apply here, too: But: Murphy's law does apply here, too: •Exceptions occur when you expect them the leastExceptions occur when you expect them the least

•Exceptions are generated by the CLR, the Exceptions are generated by the CLR, the framework library and the applicationframework library and the application


Non-Scenario for ExceptionsNon-Scenario for ExceptionsNon-Scenario for ExceptionsNon-Scenario for Exceptions

Exceptions indicate irregular conditionsExceptions indicate irregular conditions Do not use them for desired behavior!Do not use them for desired behavior!

// BlackJack-Logic// BlackJack-Logicif (value < 21)if (value < 21){{ DrawAnotherCard(); DrawAnotherCard();}}elseelse{{ // This is misusing the paradigm! // This is misusing the paradigm! throw new MaxPointsReachedException();throw new MaxPointsReachedException();}}

// BlackJack-Logic// BlackJack-Logicif (value < 21)if (value < 21){{ DrawAnotherCard(); DrawAnotherCard();}}elseelse{{ // This is misusing the paradigm! // This is misusing the paradigm! throw new MaxPointsReachedException();throw new MaxPointsReachedException();}}


OverthrowingOverthrowingOverthrowingOverthrowing

Szenario:Szenario:•Method throws exception in case of errorMethod throws exception in case of error

•Method is called very oftenMethod is called very often

•Error condition occurs relatively oftenError condition occurs relatively often

Result:Result:•Many exceptions occurMany exceptions occur

• Performance decreasesPerformance decreases

Solution:Solution:•Refactor method to avoid exceptionsRefactor method to avoid exceptions

• Provide cheap test methodProvide cheap test method


Creating Objects and ExceptionsCreating Objects and ExceptionsCreating Objects and ExceptionsCreating Objects and Exceptions

Creating an object might failCreating an object might fail• Particularly if the object is a larger entity, like a Particularly if the object is a larger entity, like a

serviceservice

Without special precautions you just get an Without special precautions you just get an exceptionexception

Do you want to guard each new() with an Do you want to guard each new() with an exception handler?exception handler?

Provide (static) method to create objectProvide (static) method to create object•Catch exceptions and yield empty referenceCatch exceptions and yield empty reference

Make constructors inaccessible to the respective Make constructors inaccessible to the respective outside!outside!


OvercatchingOvercatchingOvercatchingOvercatching

Szenario:Szenario:•Exception type of catch block too generalException type of catch block too general

Exceptions are caught that cannot be handledExceptions are caught that cannot be handled

Result:Result:•Not handled exceptions disappearNot handled exceptions disappear

Solution: Solution: •Catch more specific exceptionsCatch more specific exceptions


Sieve too coarseSieve too coarseSieve too coarseSieve too coarse

trytry{{

stream = File.Open(name, FileMode.Open);stream = File.Open(name, FileMode.Open);}}catch (IOException e)catch (IOException e){{

Console.WriteLine(“File not found.”);Console.WriteLine(“File not found.”);}}

trytry{{

stream = File.Open(name, FileMode.Open);stream = File.Open(name, FileMode.Open);}}catch (IOException e)catch (IOException e){{

Console.WriteLine(“File not found.”);Console.WriteLine(“File not found.”);}}


trytry

{{

stream = File.Open(name, FileMode.Open);stream = File.Open(name, FileMode.Open);

}}

catch (FileNotFoundException e)catch (FileNotFoundException e)

{{

Console.WriteLine(“File not found.”);Console.WriteLine(“File not found.”);

}}

trytry

{{

stream = File.Open(name, FileMode.Open);stream = File.Open(name, FileMode.Open);

}}

catch (FileNotFoundException e)catch (FileNotFoundException e)

{{

Console.WriteLine(“File not found.”);Console.WriteLine(“File not found.”);

}}

Appropriate Appropriate Appropriate Appropriate


The Great Final(e)The Great Final(e)The Great Final(e)The Great Final(e)

try/finally is justified even without catch!try/finally is justified even without catch! Necessary clean up code should go into a finally Necessary clean up code should go into a finally

block! block! •Even if no exceptions shall be caughtEven if no exceptions shall be caught

Some language constructs do this implicitlySome language constructs do this implicitly• lock or SyncLock with multithreadinglock or SyncLock with multithreading

•using for the Disposal patternusing for the Disposal pattern

try/finally makes the code much more robust!try/finally makes the code much more robust!


Catching Anything?Catching Anything?Catching Anything?Catching Anything?

Scenario for generic catch handlerScenario for generic catch handler

Information is not lostInformation is not lost•Exception can be treated at applicable locationException can be treated at applicable location

Subtle difference to finally clauseSubtle difference to finally clause•Handler is only executed in case of an exceptionHandler is only executed in case of an exception

long originalIndex = obj.currentIndex; long originalIndex = obj.currentIndex;

try { try { obj.SomethingRiskyWithUpdatingTheIndex(); obj.SomethingRiskyWithUpdatingTheIndex();}}catch {catch { // Reset position, regardless of error // Reset position, regardless of error obj.currentIndex = originalIndex; obj.currentIndex = originalIndex;

// Rethrow the (very same(!)) Exception // Rethrow the (very same(!)) Exception throw; throw;}}

long originalIndex = obj.currentIndex; long originalIndex = obj.currentIndex;

try { try { obj.SomethingRiskyWithUpdatingTheIndex(); obj.SomethingRiskyWithUpdatingTheIndex();}}catch {catch { // Reset position, regardless of error // Reset position, regardless of error obj.currentIndex = originalIndex; obj.currentIndex = originalIndex;

// Rethrow the (very same(!)) Exception // Rethrow the (very same(!)) Exception throw; throw;}}


Exception HandlingException HandlingException HandlingException Handling

Local exceptions should be caught if:Local exceptions should be caught if:•Error condition can be solved locallyError condition can be solved locally

•Clean up has to be done before leaving the contextClean up has to be done before leaving the context

•Additional or other information has to be generatedAdditional or other information has to be generated

•Occurence of exception has to be loggedOccurence of exception has to be logged In all other cases the exception should be In all other cases the exception should be

propagated to the callerpropagated to the caller Note: throw SomeException and catch Note: throw SomeException and catch

SomeException should be done on different SomeException should be done on different program levels!program levels!

Performance considerationsPerformance considerations•Minimal effort and no extra cost for cleaning up Minimal effort and no extra cost for cleaning up

objectsobjects

•Resistance is futile:Resistance is futile: Performance is not a reason for not handling exceptionsPerformance is not a reason for not handling exceptions


Options of Exceptions PropagationOptions of Exceptions PropagationOptions of Exceptions PropagationOptions of Exceptions Propagation

Variants:Variants:•Catch, recover, continue Catch, recover, continue

Exception is not propagated all atException is not propagated all at

•Automatic propagation by ignoringAutomatic propagation by ignoring No handling at allNo handling at all try / finally without catchtry / finally without catch

•Catch and re-throwCatch and re-throw Allows for local clean up or additional information Allows for local clean up or additional information

•Catch, wrap, throwCatch, wrap, throw Allows for changing the propagated exception typeAllows for changing the propagated exception type


Exception Interception Exception Interception Exception Interception Exception Interception

Exceptions can be caught just to relay a different Exceptions can be caught just to relay a different oneone

ArrayList.Sort calls IComparable.CompareTo ArrayList.Sort calls IComparable.CompareTo •Exceptions occuring in CompareTo are caught in Exceptions occuring in CompareTo are caught in

Sort and relayed as Sort and relayed as System.InvalidOperationExceptionSystem.InvalidOperationException No matter what the original type wasNo matter what the original type was

• Property InnerException contains original exception Property InnerException contains original exception objectobject

A() B() C()

OriginalException

OriginalException

Wrapper-Exception

Call Call

ThrowThrow


Defining Custom ExceptionsDefining Custom ExceptionsDefining Custom ExceptionsDefining Custom Exceptions

Custom exceptions should be derived from Custom exceptions should be derived from System.ApplicationException System.ApplicationException •Directly or indirectlyDirectly or indirectly

•Use intermediate base class for common Use intermediate base class for common management tasksmanagement tasks

Beware class hierarchy bloat!Beware class hierarchy bloat!•Separating problems out into separate classes only Separating problems out into separate classes only

makes sense if handled separatelymakes sense if handled separately

•Want to write endless cascades of handlers?Want to write endless cascades of handlers?

•But if you do, follow the rulesBut if you do, follow the rules


And nobody sees to it!And nobody sees to it!And nobody sees to it!And nobody sees to it!

There is always an exception that is not caughtThere is always an exception that is not caught• I.e. it is propagated down the call stack up to the I.e. it is propagated down the call stack up to the

endend

Three things will happen:Three things will happen:•Error is not solved in any way, application diesError is not solved in any way, application dies

•User might see nasty message boxesUser might see nasty message boxes

•Developer gets no information about the errorDeveloper gets no information about the error

At least the error must be loggedAt least the error must be logged•With as much information as possibleWith as much information as possible


Handling the UnhandledHandling the UnhandledHandling the UnhandledHandling the Unhandled

Collectinformation

Persistinformation

Send notifcations

Sync or async

Clean up

Exception reachesupmost level

Show necessaryinformation to the user

Sync or async

Notification

Persistant store

AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(LastChanceHandler);

AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(LastChanceHandler);


Important InformationImportant InformationImportant InformationImportant Information

InfoInfo How to get it?How to get it?

TimeTime DateTime.Now + time zone infoDateTime.Now + time zone info

Machine nameMachine name Environment.MachineName Environment.MachineName

ExceptionException Exception.Source, Exception.Source, Exception.Message, Exception.Message, Exception.StackTrace (starts at location of Exception.StackTrace (starts at location of occurrence (throw) and ends at current occurrence (throw) and ends at current location)location)

Exception type Exception type Type.FullName taken from Object.GetType Type.FullName taken from Object.GetType

Call stack Call stack Environment.StackTrace (the complete call Environment.StackTrace (the complete call stack)stack)

Application domain Application domain name name

AppDomain.FriendlyName AppDomain.FriendlyName

Assembly name & -Assembly name & -versionversion

AssemblyName.FullNameAssemblyName.FullName

Thread id Thread id AppDomain.GetCurrentThreadId AppDomain.GetCurrentThreadId

Thread user Thread user Thread.CurrentPrincipalThread.CurrentPrincipal


Who needs which Information?Who needs which Information?Who needs which Information?Who needs which Information?

Save all relevant information about the exception Save all relevant information about the exception Receiver of information can be:Receiver of information can be:

•ApplicationApplication

•UserUser Needs useful description of problemNeeds useful description of problem

•DeveloperDeveloper Needs detailed information in order to fix application Needs detailed information in order to fix application

problem or install automatic recovery mechanismproblem or install automatic recovery mechanism

•OperatorOperator Needs information to fix actual problemNeeds information to fix actual problem


Where to put the Info?Where to put the Info?Where to put the Info?Where to put the Info?

Reliability is most importantReliability is most important Possiblities:Possiblities:

•Windows event log Windows event log

•Central data baseCentral data base

• Text fileText file

•Message Queue Message Queue Usually for custom monitor applicationUsually for custom monitor application


Windows Event Log Windows Event Log Windows Event Log Windows Event Log

Advantages: Advantages: •ReliableReliable

• Tools always availableTools always available

• Framework Library supportFramework Library support

•Most monitoring tools support the event logMost monitoring tools support the event log

•Good integration with WMIGood integration with WMI

•Administrators and many users are accustomed to Administrators and many users are accustomed to itit

•Standard logging methodStandard logging method Synchronisation with other events is relatively simpleSynchronisation with other events is relatively simple

Disadvantages:Disadvantages: •Synchronisation on server farms can be an effortSynchronisation on server farms can be an effort

• Logging on central machine increases failure riskLogging on central machine increases failure risk


Central Data Base Central Data Base Central Data Base Central Data Base

Advantages:Advantages:•Central location for all informationCentral location for all information

•Accessible from almost everywhereAccessible from almost everywhere

• (Powerful) standard reporting tools available(Powerful) standard reporting tools available

•Structure can be chosen application specificStructure can be chosen application specific

Disadvantages:Disadvantages:• If data base is unavailable, information is lostIf data base is unavailable, information is lost

Fallback: Use Windows Event Log, if data base is Fallback: Use Windows Event Log, if data base is downdown

•Synchronisation with other events is difficultSynchronisation with other events is difficult

•Reports and administration must be developedReports and administration must be developed

•Standard monitoring tools must be customizedStandard monitoring tools must be customized


Custom Log FilesCustom Log FilesCustom Log FilesCustom Log Files

Advantages:Advantages:•Maximum flexibilityMaximum flexibility

Disadvantages:Disadvantages:•Development of logging mechanism can be tediousDevelopment of logging mechanism can be tedious

Concurrent access!Concurrent access!

•All administration tasks must be implementedAll administration tasks must be implemented

•Standard monitoring tools not generally usableStandard monitoring tools not generally usable Adapter must be developed or configuredAdapter must be developed or configured Cannot aggregate multiple filesCannot aggregate multiple files


And now we have saved And now we have saved everything…everything…And now we have saved And now we have saved everything…everything…

Who cares?Who cares? Notification is the critical part of Exception Notification is the critical part of Exception

ManagementManagement•Without notification exceptions can go undetected!Without notification exceptions can go undetected!

Most important errors should be reported to Most important errors should be reported to persons in charge immediatelypersons in charge immediately


Monitoring SystemsMonitoring SystemsMonitoring SystemsMonitoring Systems

Typically, a system consists of Typically, a system consists of •Data collection processes Data collection processes

•Storage engine Storage engine

•Analysis console Analysis console

•Additional background processes Additional background processes


MonitoringMonitoringMonitoringMonitoring

Advantages: Advantages: •External solutionExternal solution

No development effortNo development effort

•Monitoring system usually is externally configurableMonitoring system usually is externally configurable Allows for selection ...Allows for selection ...

•Monitor is triggered by eventsMonitor is triggered by events Windows Management Instrumentation (WMI)Windows Management Instrumentation (WMI) System.Management and System.Management and

System.Management.InstrumentationSystem.Management.Instrumentation

Disadvantages:Disadvantages:•Has to be configured – operator know-howHas to be configured – operator know-how

•Must be installed on each machineMust be installed on each machine


Email Notification using SMTP Email Notification using SMTP Email Notification using SMTP Email Notification using SMTP

Send error report immediately to personnel in Send error report immediately to personnel in chargecharge

Advantages:Advantages:•Can be implemented fast and simpleCan be implemented fast and simple

•System.Web.Mail namespaceSystem.Web.Mail namespace

Disadvantages:Disadvantages:•Email can get lostEmail can get lost

• If too many emails occur…If too many emails occur…


Custom Notification SystemCustom Notification SystemCustom Notification SystemCustom Notification System

Custom notification system receives messagesCustom notification system receives messages• Triggers corresponding actionsTriggers corresponding actions

Message queueing is the ideal medium for Message queueing is the ideal medium for messagesmessages

Advantages:Advantages:•Decoupling from application codeDecoupling from application code

•Can be used for multiple applicationsCan be used for multiple applications

•Cost/Use analysis usually positiveCost/Use analysis usually positive

•Extremely reliableExtremely reliable

•Applicable in central and decentral scenariosApplicable in central and decentral scenarios Disadvantages of message queueing:Disadvantages of message queueing:

•Asynchronicity comes with certain problems:Asynchronicity comes with certain problems: Order of messages is not necessarily retainedOrder of messages is not necessarily retained Delay is possibleDelay is possible


Exceptions and ThreadsExceptions and ThreadsExceptions and ThreadsExceptions and Threads

Exceptions are thread-localExceptions are thread-local•Never propagated to parent threadNever propagated to parent thread

Asynchronous delegates support concept for Asynchronous delegates support concept for relayingrelaying•BeginInvoke/EndInvokeBeginInvoke/EndInvoke

•Otherwise: Implement relaying by handOtherwise: Implement relaying by hand

If no relaying is possible or desirable:If no relaying is possible or desirable:• It is crucial that all exceptions are caught on the It is crucial that all exceptions are caught on the

thread on which they occur!thread on which they occur!

• If no action can solve the problem, log it!If no action can solve the problem, log it!


How final is finally?How final is finally?How final is finally?How final is finally?

Watch out for possible exceptions in finally blockWatch out for possible exceptions in finally block• If exceptions are to be expected, handle themIf exceptions are to be expected, handle them

If method will be called in multithreaded scenarioIf method will be called in multithreaded scenario• ThreadAbortException can be thrown by ThreadAbortException can be thrown by

Thread.Abort()Thread.Abort()

•Same effect as internal exceptionSame effect as internal exception

•Start routine should catch all exceptions and log Start routine should catch all exceptions and log themthemvoid ThreadProc()

void ThreadProc() {{ try {try { // Work to do // Work to do }} catch(Exception e) {catch(Exception e) { Log(e);Log(e); }}}}


Limits of Exception HandlingLimits of Exception HandlingLimits of Exception HandlingLimits of Exception Handling

To some extent exception handling can be To some extent exception handling can be considered as transactional processingconsidered as transactional processing• catch blocks can abort or compensate actionscatch blocks can abort or compensate actions

•finally blocks assure clean up and closing operationsfinally blocks assure clean up and closing operations

This is only valid considering single targetThis is only valid considering single target Problematic in distributed scenariosProblematic in distributed scenarios

•Must assure catching of any exceptionMust assure catching of any exception

•Must relay exception to callerMust relay exception to caller May involve some kind of translation (deserialization)May involve some kind of translation (deserialization)

Distributed application = distributed error handling Distributed application = distributed error handling • This means that it must support transactionsThis means that it must support transactions

•Otherwise application's resources will become Otherwise application's resources will become inconsistent in case on an errorinconsistent in case on an error


Try/Catch: Multitude of Execution Try/Catch: Multitude of Execution PathsPathsTry/Catch: Multitude of Execution Try/Catch: Multitude of Execution PathsPaths

Though structured, number of execution paths grow with the order of Though structured, number of execution paths grow with the order of method calls that throw an exception method calls that throw an exception • Multiplied by number of distinct exceptions!Multiplied by number of distinct exceptions!

XDo()

Fix()

YDo()

Fix()

ZDo()

Fix()

tryBlock

catchBlock


try{ X.Do(); try { Y.Do(); try { Z.Do(); } catch (Exception e) { Z.Fix(e); Y.Fix(e); X.Fix(e); } } catch (Exception e) { Y.Fix(e); X.Fix(e); } }catch (Exception e){ X.Fix(e);}

try{ X.Do(); xDone = true; Y.Do(); yDone = true; Z.Do(); zDone = true;}catch (Exception e){ if (zDone) Z.Fix(e); if (yDone) Y.Fix(e); if (xDone) X.Fix(e);

}


What if X erased a file What if X erased a file that cannot be that cannot be unerased?unerased?

What if Y’s work What if Y’s work depends on (future) depends on (future) success of Z?success of Z?

What if an outsider What if an outsider looks at X and Y before looks at X and Y before Z happens (and fails)?Z happens (and fails)?

Try/Catch: State ConflictsTry/Catch: State ConflictsTry/Catch: State ConflictsTry/Catch: State Conflicts

tryBlock

catchBlock

XDo()

Fix()

YDo()

Fix()

ZDo()

Fix()


Two-Phase TransactionsTwo-Phase TransactionsTwo-Phase TransactionsTwo-Phase Transactions

XPrepare()

Abort()

Y

Z

PrepareLoop

CommitLoop

AbortLoop

Commit()

Prepare()

Abort()

Commit()

Prepare()

Abort()

Commit()


What’s the Difference?What’s the Difference?What’s the Difference?What’s the Difference?

Try/Catch (Exception handling)Try/Catch (Exception handling)•Assumes that all “Do’s” work. Assumes that all “Do’s” work.

… … and that, in case, all failures can be undoneand that, in case, all failures can be undone

•Good enough if there’s only one “Do”Good enough if there’s only one “Do” … … or if each piece of work is truly orthogonalor if each piece of work is truly orthogonal

•Big, bad mess for more than one “Do”Big, bad mess for more than one “Do”

Prepare/Commit/Abort (Distributed transactions)Prepare/Commit/Abort (Distributed transactions)•Makes no assumption about outcomeMakes no assumption about outcome

•No work done until it knows all can be doneNo work done until it knows all can be done … … so promises must be made and keptso promises must be made and kept

• Two-Phase protocol may block!Two-Phase protocol may block!


Transaction ManagementTransaction ManagementTransaction ManagementTransaction Management

Services are boundary around stateServices are boundary around state Services interoperate through business Services interoperate through business

transactionstransactions Business transaction <> commit/rollback Business transaction <> commit/rollback

•Not ACID!Not ACID!

Cancel order

Order confirmed

Order cancelled—cancellation fee

Cancel delivery (“Compensation” messages and business logic)

CommittedRollback

ACID transaction boundary

Business conversation boundary


Rethinking Try/Catch: Rethinking Try/Catch: Compensation Compensation Rethinking Try/Catch: Rethinking Try/Catch: Compensation Compensation

Do(): Go Ahead and Do It If You CanDo(): Go Ahead and Do It If You Can Prepare(): Were You Able To Do It?Prepare(): Were You Able To Do It? Commit(): Do Nothing Commit(): Do Nothing Abort():Abort():

•Back out change, often launches compensationBack out change, often launches compensation

•Compensation isn’t necessarily the inverse!Compensation isn’t necessarily the inverse!

Consequences:Consequences:•No locks No locks no bottlenecks no bottlenecks

•All complexity moves into Abort()All complexity moves into Abort()

•Very appropriate for distributed modelsVery appropriate for distributed models

•But: Here you go with complex execution paths But: Here you go with complex execution paths again!again!


Issues with CompensationIssues with CompensationIssues with CompensationIssues with Compensation

Compensation does not necessarily mean: Compensation does not necessarily mean: Restoring the original state!Restoring the original state!•Not all actions are reversibleNot all actions are reversible

•Some confront you with a fee, others with the lawSome confront you with a fee, others with the law

Provides atomicity (more or less), consistency Provides atomicity (more or less), consistency and durability, but no isolationand durability, but no isolation•Other transactions can use committed dataOther transactions can use committed data

Compensating actions are not easy to writeCompensating actions are not easy to write•Must regard possible actions done by other Must regard possible actions done by other

transactionstransactions


Sequential Approach: SSequential Approach: SagasagasSequential Approach: SSequential Approach: Sagasagas

Workflow-like serial transactionWorkflow-like serial transaction•Sequence of transactions that must all complete or Sequence of transactions that must all complete or

all be rolled-backall be rolled-back

Coordinator runs compensatingCoordinator runs compensating transaction(s)transaction(s)• IfIf an error occurs an error occurs

Aborted transaction, unrecoverable failureAborted transaction, unrecoverable failure

Must be able to interleave with other transactionsMust be able to interleave with other transactions• Isolation is not for granted Isolation is not for granted

Sagas need to track progress through stepsSagas need to track progress through steps•Marking savepointsMarking savepoints

• Progress may be implicit in existing durable stateProgress may be implicit in existing durable state

• Progress may need to be recorded as additional Progress may need to be recorded as additional durabledurable state until saga completesstate until saga completes


EndEnd

EndEnd

State 2State 2

A SagaA SagaA SagaA Saga

StartStart

State 1State 1

Book flight & commitBook flight & commit

Book car & commitBook car & commit

CompensateCompensateCancel hotelCancel hotel

CompensateCompensateCancel flightCancel flight

Saga doneSaga done

Saga abortedSaga abortedBook hotel & commitBook hotel & commit

Car reservationCar reservation aborts aborts

Book another hotel & commitBook another hotel & commit

FlightsService

HotelService

CarService


Sagas: Pros and ConsSagas: Pros and ConsSagas: Pros and ConsSagas: Pros and Cons

Only one singular transaction is activeOnly one singular transaction is active•Blocking cannot occurBlocking cannot occur

•No general transaction coordinator is neededNo general transaction coordinator is needed

Very flexible in shaping control flowVery flexible in shaping control flow No concurrencyNo concurrency

•One action (may it be complex) at a timeOne action (may it be complex) at a time

• This means less performanceThis means less performance


Alternative StrategiesAlternative StrategiesAlternative StrategiesAlternative Strategies

CompensationCompensation• ““Execute” / “Compensate”Execute” / “Compensate”

•Execute all actionsExecute all actions

• In case of error compensate each stepIn case of error compensate each step

•No ACID guaranteeNo ACID guarantee

ReservationsReservations• ““Try” / “Confirm” / “Cancel”Try” / “Confirm” / “Cancel”

•All actions are tentative: in good faithAll actions are tentative: in good faith

•No ACID guaranteeNo ACID guarantee


Concurrent ApproachConcurrent ApproachConcurrent ApproachConcurrent Approach

FlightsService

HotelService

CarService

Agent Reserve = DoPrepare = CheckCommit = Book

ReserveFlight

ReserveHotel

ReserveCar

Check Flight

CheckHotel

Check Car

Book Flight

BookHotel

Book Car


Concurrent ApproachConcurrent ApproachConcurrent ApproachConcurrent Approach

FlightsService

HotelService

CarService

Agent Reserve = DoPrepare = CheckAbort = Delete

ReserveFlight

ReserveHotel

ReserveCar

Check Flight

CheckHotel

Check Car

Delete Flight

DeleteHotel

DeleteCar


TxTx11

Coordinator

DebitDebitCustomer Customer AccountAccount

Credit Credit Clearing Clearing AccountAccount

Inter-Bank Inter-Bank NetworkNetwork

TxTx22

TxTx33

Debit Debit Clearing Clearing AccountAccount

Credit Credit Customer Customer AccountAccount

CompensaCompensatortor

TxTx3.13.1TxTx44TxTx55

Inter-Bank Inter-Bank NetworkNetwork

CoordinatorCredit Credit Customer Customer AccountAccount



Solvable ErrorsSolvable ErrorsSolvable ErrorsSolvable Errors

Some errors just go awaySome errors just go away•Database deadlocks or any other locking issuesDatabase deadlocks or any other locking issues

• Timeouts (synchronisation objects, IO)Timeouts (synchronisation objects, IO)

Transaction input must be well definedTransaction input must be well defined Transaction input must be repeatableTransaction input must be repeatable


Tx CoordinatorTx Coordinator



TransactionalTransactionalQueueQueue

LL

TransactiTransactionon

failsfails

Self HealingSelf HealingSelf HealingSelf Healing

Transactional queuesTransactional queues Transactional Transactional

reading from queuereading from queue Recovery is Recovery is

automaticautomatic





TransactionaTransactionall

QueueQueue

TransactioTransactionn

failsfails

Self Deception?Self Deception?Self Deception?Self Deception?

Error occurs once?Error occurs once? Error occurs multiple Error occurs multiple

times?times? Error occurs always?Error occurs always? Messages that Messages that

always cause errors always cause errors are "poisonous"are "poisonous"





TransactionaTransactionall

QueueQueue

TransactioTransactionn

failsfails

Listener puts Listener puts message into message into poison list if poison list if

error is error is encounteredencountered

Into the Poison Cabinet!Into the Poison Cabinet!Into the Poison Cabinet!Into the Poison Cabinet!

Messages that cause Messages that cause errors are put on a errors are put on a poison listpoison list

Errors known to allow Errors known to allow retrials are excepted retrials are excepted from this rulefrom this rule


Dealing with Hazardous Dealing with Hazardous SubstancesSubstancesDealing with Hazardous Dealing with Hazardous SubstancesSubstances

Check poison list after Check poison list after reading from queuereading from queue

If message is on poison list If message is on poison list it is not delivered to it is not delivered to application but entered application but entered into poison queueinto poison queue

This all happens within a This all happens within a transaction!transaction!

Poison queues can exist in Poison queues can exist in several instances and several instances and serve as input for special serve as input for special recovery mechanismsrecovery mechanisms• COM+/ES Queued COM+/ES Queued

ComponentsComponents

Message Message known as known as

poisonous?poisonous?

TransactionalTransactionalQueueQueue

Enter into Enter into Poison QueuePoison Queue


A word on securityA word on securityA word on securityA word on security

Beware of DOS or DDOS!Beware of DOS or DDOS!•Exception handling code writes to an event log or Exception handling code writes to an event log or

sends an administrative alert via emailsends an administrative alert via email

Don‘t tell the attacker too muchDon‘t tell the attacker too much•System configurationSystem configuration

e.g. „This machine has WinXP SP1 installed“e.g. „This machine has WinXP SP1 installed“

•Reason for failureReason for failure e.g. „Unable to locate stuff.txt at c:\secretstuff\docs“e.g. „Unable to locate stuff.txt at c:\secretstuff\docs“


SummarySummarySummarySummary

Watch out for your exceptions!Watch out for your exceptions!• Log them!Log them!

Use exception handling in tightly bound Use exception handling in tightly bound environmentsenvironments

Do not use exception in distributed scenariosDo not use exception in distributed scenarios In complex scenarios use transactions for fault In complex scenarios use transactions for fault

management!management!• Log faults!Log faults!

•Still watch out for exceptions within components!Still watch out for exceptions within components!

•Do not let exceptions leave a component!Do not let exceptions leave a component!

Business transactions are distributed and long-Business transactions are distributed and long-runningrunning•Consider compensation approachConsider compensation approach


Questions? Questions? Questions? Questions?

??

Documents

© 2004 Microsoft EMEA and newtelligence AG – No unauthorized reproduction or distribution permitted. All rights reserved. Exceptions and Distributed Fault