BRE Deep Dive

1

Sponsored & Brought to you by

BRE Deep Dive

Johann Cooper

http://www.twitter.com/JohannCooper

https://nz.linkedin.com/in/johanncooper

http://www.twitter.com/JohannCooper

https://nz.linkedin.com/in/johanncooper

2

Author:

Integration MondayBizTalk BRE Deep Dive

Johann Cooper

Johann Cooper – Principal Integration Consultant

@JohannCooper

Copyright © Datacom Group Limited 2014 Saturday, 18 July 2015

About me• Principal Integration Consultant at Datacom NZ

• Co-author of the book “SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” published by Packt

• Author of the blog “Adventures inside the Message Box”

• Creator of the “BRE Pipeline Framework” CodePlex project

• Author of the White Paper “The A-Y of running BizTalk Server in Microsoft Azure” published by BizTalk 360

• Proud dad

https://www.packtpub.com/networking-and-servers/soa-patterns-biztalk-server-2013-second-edition

https://www.packtpub.com/networking-and-servers/soa-patterns-biztalk-server-2013-second-edition

https://adventuresinsidethemessagebox.wordpress.com/

http://brepipelineframework.codeplex.com/

http://www.biztalk360.com/whitepaper/a-y-running-biztalk-server-microsoft-azure/


Agenda / Goals

1. Basics of BRE

2. Advanced BRE

3. BRE Pipeline Framework

Goal – To illustrate/evangelize how the usage of the BRE and the BRE Pipeline Framework can help you design and develop applications which are flexible to change

Disclaimer – I haven’t used every feature that I’m discussing today in production applications, and in some cases I’m forced to dumb down the topic in the interest of time. Feel free to discuss or counter any of my statements post the presentation


Source code

• All source code (BT 2013R2/Visual Studio 2013) and sample files can be downloaded from the below link

• https://drive.google.com/file/d/0B9cj1Xr--0vFai0xRHI2aXJnbU0/view?usp=sharing

https://drive.google.com/file/d/0B9cj1Xr--0vFai0xRHI2aXJnbU0/view?usp=sharing

https://drive.google.com/file/d/0B9cj1Xr--0vFai0xRHI2aXJnbU0/view?usp=sharing


Why should you use the BRE?

• Decouple your business or processing logic from pipeline/orchestration implementation

• It’s not just for business rules, you can use it for application logic as well

• It’s fast!

• Easy to change at runtime without full deployments

• Is versioned and can be source controlled

• Rules should be easy to understand if vocabularies are created correctly

• It’s part of the BizTalk Server license, maximize your investment

• Even a business analyst can do it

1 - Basics of BRE


Facts

• Discrete piece of information whose characteristics are to be assessed and updated

• Supported types• .NET• XML• Database

1 - Basics of BRE


Vocabularies

• Allow you to abstract away implementation of fact characteristics with friendly terms

• Example vocabulary definition – GetColour : “The primary colour of the cow”• .NET – cow.Color• SQL – select [Color] from dbo.Cow• XML - /cow/color

CowVocabulary (Name – Display Name)• GetColour – The primary colour of the cow• GetLength – The length of the cow in meters• SetCowValue – Update the cow’s value to ${0}

1 - Basics of BRE


Vocabularies

DEMO

1 - Basics of BRE


Rules

• Made up of conditions and actions• String together fact/vocabulary definitions with relevant predicates

• Example rule – Set value for large green cows• Condition

• If the cow’s length in meters is greater than 500 • And the cow’s primary colour is equal to Green

• Action• Update the cow’s value to $1000000000

1 - Basics of BRE


Rules

• You don’t have to use vocabularies but why wouldn’t you? • The below is just as valid.

• Example rule – Set value for large green cows• Condition

• cow.Length is greater than 500 • cow.color is equal to Green

• Action• SetCowValue 1000000000

1 - Basics of BRE


Policies

• A collection of rules

• All conditions will be evaluated first, filling out the agenda, before any rules fire

• Order of conditions being evaluated is non-deterministic

• Order of rule actions firing is non-deterministic unless priorities are specified

• Order of actions firing within a rule is based on order inside the rule

1 - Basics of BRE


Policy stages

1. Match – condition evaluation and adding rules to agenda

2. Conflict resolution – prioritizing sequence of execution

3. Action – Firing actions in the relevant order

1 - Basics of BRE


Assertion

• Making a fact available to a policy

• A policy can only deal with facts it is aware of

• Rules that depend on facts that are not asserted will never fire

• It is possible to enable static support for static .NET method via registry key• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BusinessRules\3.0\

StaticSupport• 0 is default value which implies no static support• 1 implies static support, with the static method being called each time• 2 implies static support with the result of the static method being cached

(only for conditions, not actions) and reused after the first execution.

1 - Basics of BRE


RETE Algorithm

• Very fast, and efficient way of evaluating multiple rules and creating your agenda

• Sacrifices memory for speed

• Breaks apart left and right hand side of conditions into separate node networks and applies the minimal amount of joins to evaluate conditions

• Might be best left to PhDs

1 - Basics of BRE


Rule Priorities

• Priorities can be assigned to rules

• Default value 0

• Priority has nothing to do with conditions, just actions

• Higher the value, higher the priority

• Higher priority rules fire before lower priority rules

• So lower priority rules can override higher priority rules!

1 - Basics of BRE


Policy Caching

• CacheEntries, CacheTimeOut, and PollingInterval can be configured via Registry or BizTalk config file - https://msdn.microsoft.com/en-us/library/aa548008.aspx

1 - Basics of BRE

https://msdn.microsoft.com/en-us/library/aa548008.aspx

https://msdn.microsoft.com/en-us/library/aa548008.aspx


Side Effects

• Once you access an XML or SQL based fact value in a policy, it’s value will be cached and reused in other conditions/actions

• .NET members will not be cached by default

• This is controlled by the sideeffects property, can only be set programmatically or via XML export files, no support in composer

1 - Basics of BRE


Side Effects

• This is a deep topic - further reading

• Charles Young - http://geekswithblogs.net/cyoung/archive/2007/04/09/111169.aspx

• My blog - https://adventuresinsidethemessagebox.wordpress.com/2012/11/20/biztalk-business-rules-policies-not-caching-net-facts-by-default/

1 - Basics of BRE

http://geekswithblogs.net/cyoung/archive/2007/04/09/111169.aspx

http://geekswithblogs.net/cyoung/archive/2007/04/09/111169.aspx

https://adventuresinsidethemessagebox.wordpress.com/2012/11/20/biztalk-business-rules-policies-not-caching-net-facts-by-default/




Rules/Policies/Assertion/Priorities

DEMO

1 - Basics of BRE


Rules testing

• Can right click on policy version in composer and choose “Test Policy”

• Can assert XML files or database connections via wizard

• Testing policy will result in change in state in asserted facts

• Must use a fact creator to assert .NET objects

• There is also a BizUnit 4.0 FactBasedRuleEngineStep test step

2 – Advanced BRE


Rules testing

DEMO

2 – Advanced BRE


Rules traceability

• Can get equivalent information at runtime as you would when choosing “Test Policy”

• Can only do this programmatically, not in orchestration

• Can build your own tracing class based on Microsoft.RuleEngine. IRuleSetTrackingInterceptor

• Alternatively can make use of out of the box Microsoft.RuleEngine.DebugTrackingInterceptor

2 – Advanced BRE


Rules traceability

DEMO

2 – Advanced BRE


Versioning and Source Control

• Can export vocabulary/policy definitions to XML using BRE Deployment Wizard or BizTalk Administration Console

• Can bulk export from BizTalk Administration Console

• Can copy old versions of vocabularies or policies and paste to create a new version and edit

• Orchestration always chooses latest version, but can be explicit from .NET (or ESB Toolkit / BRE Pipeline Framework)

• Can associate policies with an application, but beware that deleting the application is equivalent to deleting the policy

• If deleting a policy from BizTalk Admin Console, if it is the last policy to be associated with a vocabulary then the vocabulary will be deleted as well

2 – Advanced BRE


Versioning and Source Control

DEMO

2 – Advanced BRE


Engine Control Functions

• Can control rules chaining through definitions contained within the Functions vocabulary

• Beware of endless loops, these can consume all the machine’s memory

• This is a topic in itself. We are only scratching the surface

• Assert/Update – Re-evaluate conditions in all rules that make use of the asserted/updated fact. Update will only re-evaluate if the type is used in conditions, not if only in actions (in which case the actions will be left on the agenda)

• Retract/RetractByType – Remove fact from rule engine. RetractByType removes all facts with a matching type while retract only removes a single instance

• Halt – Stop the rules execution (tip – use priority to control when this happens)

2 – Advanced BRE


Engine Control Functions

DEMO

2 – Advanced BRE


Long Running Facts

• Can be implemented using an IFactRetriever interface implementation

• UpdateFacts method can be used to assert/retract one or many facts which will be cached for use until they are retracted

• Can return a token object out of UpdateFacts that will be received into the next firing of the method which will can be used to decide whether to retract/re-assert the long running fact

• FactRetriever is associated with a policy version so no need to assert it from orchestration/.NET code

• Pro Tip – Can be used to assert a .NET helper class and use contained methods within the policy

2 – Advanced BRE


Long Running Facts

DEMO

2 – Advanced BRE


Deployment

• Multiple deployment options• BizTalk MSI• BRE Deployment Wizard• BTDF• BizTalk Administration Console

• With BizTalk MSIs and BizTalk Administration Console you will need to install .NET assemblies into GAC first, restart the Administration Console and then perform the import.

• .NET based vocabularies with long namespace/class names can cause BizTalk MSIs to fail

• Might want to consider deploying policies, restarting Rule Engine Update Service and then starting application

2 – Advanced BRE


BTDF BRE Sections

<PropertyGroup>…<IncludeVocabAndRules>True</IncludeVocabAndRules>…

<PropertyGroup>

<ItemGroup>…

<RuleVocabularies Include="Vocabulary.1.0.xml"> <LocationPath>..\BRE Artifacts\Vocabularies</LocationPath></RuleVocabularies><RulePolicies Include="Policy.1.0.xml"> <LocationPath>..\BRE Artifacts\Policies</LocationPath> </RulePolicies>…

<ItemGroup>

2 – Advanced BRE


Updating published/deployed vocabularies and policies

• Change nStatus to 0 on relevant tables in BizTalkRuleEngineDb database

• You can just publish vocabularies as you would normally

• Won’t be able to publish/deploy a rule that you’ve unpublished from the composer. You’ll need to save it and then set nStatus back to 1

• Be cautious when doing this, and please restrict this to dev!

2 – Advanced BRE


Working out dependencies

2 – Advanced BRE


BRE Pipeline Framework Introduction

• A pipeline component which allows you to execute BRE Policies

• Provides a range of utility vocabularies that help you implement lots of common integration patterns

• Is extensible so you can implement your own utility and corresponding vocabularies

• Makes it easy to assert XML and SQL database facts

• Encourages messaging only application where orchestration would be used for utility rather than orchestrating a process

3 – BRE Pipeline Framework


Some assurances on reliability

• It’s compatible with BizTalk 2010+, same code base/installer

• It’s been heavily unit tested on all compatible versions of BizTalk – 94% code coverage

• Theoretically no reason it won’t continue to work with new versions of BizTalk

• New versions in the 1.x version set will always be backwards compatible

• Close to 1000 downloads (and only about 30 of those are me ), is heavily used in NZ (not just in the consultancy I work for)

• Been successfully used on low latency and high throughput solutions as well as solutions dealing with large messages



Installation + Demo

• Download installer from CodePlex site - http://brepipelineframework.codeplex.com/

• Uninstall previous version (if applicable)

• Install new version

• Restart host instances (if previous version)

• Import OOTB vocabularies that you’re interested in from “C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies”

• Add pipeline component to pipeline toolbox in Visual Studio on dev machines (right click in toolbox and click on “Choose items”)





Some important terms

• ExecutionPolicy – BRE Policy used to assess conditions and update a message’s body or context

• InstructionLoaderPolicy – Optional BRE Policy used to instantiate and assert facts to the ExecutionPolicy

• Used for custom .NET facts, SQL facts or XML facts• Not needed to use OOTB facts

• ApplicationContext – Used to instruct the BRE Policy which pipeline (and optionally pipeline stage) called the Policy



Pipeline component and simple rules

DEMO



Some more important terms

• MetaInstruction • Class that contains helper methods that you can use in conditions or

actions to get details from a message

• Almost all BRE Vocabularies provided in the framework correspond to MetaInstruction, and can only be used in ExecutionPolicy

• Also contains methods you can use in actions that add an instruction

• Instruction• Class that is used to update a message



Pipeline Component Architecture



BREPipelineFramework.SampleInstructions.ContextInstructions Vocabulary + Demo

• Get or set context properties• By enumeration for all out of the box context properties• By explicitly specifying property namespace and name

• Set context property from SSO config store key/value pair or XPath query against message body

• Remove context property



BREPipelineInstructions.SampleInstructions.HelperInstructionsVocabulary

• String manipulation – ToLower/ToUpper, Regex/Substring Find/Replace, StringContains, Concatenate

• Dynamic transformation• Pro Tip – you can use the Context Accessor on a send port as well when

you use this function

• Run XPath queries to extract node name, namespace or value

• Nullify message

• Generate GUID



BREPipelineInstructions.SampleInstructions.HelperInstructionsVocabulary + Demo

• Round up time (great for convoys or rolling log file naming)

• Tracing

• Throw exceptions

• Get Value from SSO Config Store

• Casting



BREPipelineFramework.SampleInstructions.CachingInstructions

• Uses .NET MemoryCache class which is specific to an AppDomain

• Recommendation is to only rely on cached context between a receive and send pipeline on a given two way port. Don’t rely on it across ports as a host instance restart would result in the cache being emptied, and this also wouldn’t work on a BizTalk group with multiple runtime servers

• Cached items automatically expire after 30 minutes, or you can remove them explicitly. By default items will be removed from the cache if the server is under heavy memory pressure but you can override this too



BREPipelineFramework.SampleInstructions.CachingInstructions + Demo

• Cache all or specific context properties to .NET MemoryCache

• Reapply all or specific context properties from cache that is relevant to the current message

• Get cached context property that is relevant to the current message

• Cache or get any custom string for which you must provide a key

• Get a value from an SSO config store and also cache it, subsequent reads will be from the cache



BREPipelineFramework.SampleInstructions.HttpHeaderInstructions + Demo

• Get values of individual inbound HTTP headers

• Set values for individual outbound HTTP headers

• Copy individual inbound HTTP headers to outbound HTTP headers

• Pro tip – Set the BTS.IsDynamicSend context property to true to dynamically set outbound HTTP headers on a static WCF-WebHttp send port

• Along the same veins, support for content negotiation in BizTalk 2013 R2 - https://adventuresinsidethemessagebox.wordpress.com/2015/06/30/dynamic-request-content-and-response-content-negotiation-for-restful-apis-exposed-by-biztalk-server/


https://adventuresinsidethemessagebox.wordpress.com/2015/06/30/dynamic-request-content-and-response-content-negotiation-for-restful-apis-exposed-by-biztalk-server/





BREPipelineFramework.SampleInstructions.PipelineInstructions + Demo

• Execute BizTalk Pipelines• XML Assembler/Disassembler

• XML Validator

• Flat File Assembler/Disassembler

• Note that disassembly of an envelope containing multiple bodies will result in only the first body being passed to the next pipeline component. The BRE Pipeline Framework is not a disassembler (yet)!



BREPipelineFramework.SampleInstructions.XMLTranslatorInstructions

• Modify XML message in a streaming manner

• Add/Replace/Remove namespace/prefix on specified nodes

• Update element/attribute names/values on specified nodes

• Remove element (and optionally all child elements)



BREPipelineFramework.SampleInstructions.PartInstructions

• Get/Set message part properties by part name or index

• Get/Set message part content type/char set by part name or index

• Get part names by index

• Get part count



Asserting XML Facts

• Use SetTypedXmlDocument definition in BREPipelineFramework vocabulary to set the document type (must match document type in XML vocabulary definition)

• Can conditionally set the document type based on ApplicationContext, root node name and namespace, regex queries, string finds etc… These can all be found in the BREPipelineFramework vocabulary and besides ApplicationContext, they can only be used in an InstructionLoaderPolicy

• Once asserted, XML vocabularies can be used in the ExecutionPolicy as per normal

• By default all XML fact based actions will occur before any other instructions, but this behavior can be overridden

• Can only assert one XML fact (since there’s only one XML message body)



BREPipelineFramework.SampleInstructions.XMLInstructions + Demo

• Manipulate XML facts

• Add element/attribute in a specified location• With or without namespace

• With or without value



Asserting SQL Facts + Demo

• Currently only DataConnection based facts are supported, not TypedDataTable or TypedDataRow

• Use AddSQLDataConnection or AddSQLDataConnectionFromSSO definitions in the BREPipelineFramework vocabulary to assert SQL facts in an InstructionLoaderPolicy

• Can assert as many SQL Facts as you want

• Use the SQL Facts as you would normally within your ExecutionPolicy

• Can assert the facts conditionally, as previously described for XML facts



Extensibility - Instruction

• Your instruction class must implement the BREPipelineFramework.IBREPipelineInstruction from the BREPipelineFramework assembly (can be found in “C:\Program Files (x86)\BRE Pipeline Framework\Bin” or GAC)

• Only need to implement one method called execute, similar signature to a pipeline component’s execute method

• void Execute(ref IBaseMessage inmsg, IPipelineContext pc);

• Feel free to add whatever properties and constructors you need to the class

• Modify the message to your heart’s content though keep normal pipeline component considerations in mind (streaming, adding to pipeline context resources etc…)



Extensibility - MetaInstructions

• Derive your MetaInstruction class from BREPipelineFramework.BREPipelineMetaInstructionBase base class found in BREPipelineFramework assembly

• The class will always be instantiated with a default constructor

• Build any helper methods you want with any signatures, and add any properties to the class as well

• You can access the message via base.inMsg and pipeline context via base.pc

• You will not be able to, and should not attempt to modify a message in a MetaInstruction

• You can build methods that instantiate Instruction instances, and then call base.AddInstruction method to add the instruction

• Build vocabularies for methods you want to use in your policy



Asserting your MetaInstruction + Demo

• Two methods of asserting your MetaInstruction

• Add fully qualified MetaInstruction class/assembly name to the CustomMetaInstructionSeparatedList parameter on pipeline component. This is a semi-colon separated list, so you can assert as many MetaInstructions as you want

• You can use the AddMetaInstruction definition in the BREPipelineFramework vocabulary in an InstructionLoaderPolicy to conditionally assert MetaInstructions

• Only use this method if it is really important that you conditionally assert your MetaInstruction. Avoiding having to maintain an InstructionLoaderPolicy is a huge boon

• You can then use vocabularies corresponding to methods in your MetaInstruction just as you would with the OOTB vocabularies



Rules Traceability + Demo

• You can use the TrackingFolder parameter on the pipeline component which will write out rules debug information to a nominated folder, similar to choosing “Test Policy” in the composer

• You can use the CAT Instrumentation Framework Controller to enable Pipeline Component traces. ETW tracing features heavily throughout the BRE Pipeline Framework and will give you deep insight into what is going on as well as execution time



Things I haven’t touched upon today

• You can define constant based vocabulary definitions as well, including lists

• ESB Toolkit BRE Resolvers haven’t been discussed since there has been a recent session on the ESB Toolkit

• Can assert SQL Facts based on TypedDataTable or TypedDataRow instead of DataConnection. If you want to build a long running SQL based fact then you must use these

• Can execute policies via RuleEngine.Execute() instead of Policy.Execute() which allows you to resume halted rules execution

• Complex/repeating XML structures - https://adventuresinsidethemessagebox.wordpress.com/2012/05/23/biztalk-business-rules-and-complex-xml-structures/

https://adventuresinsidethemessagebox.wordpress.com/2012/05/23/biztalk-business-rules-and-complex-xml-structures/




Q&A

• Any questions?

Technology

BRE Deep Dive