Improved Support to Large Orders - Oracle Cloud · –O2A cartridges have been enhanced to take advantage of OSM core product improvements –With appropriate hardware and attention

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Improved Support to Large Orders OSM 7.2.4.1 Enablement Training

November, 2014 Version 1.1

Oracle Confidential – Internal/Restricted/Highly Restricted


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Confidential – Internal/Restricted/Highly Restricted 2


Program Agenda

Overview

Order Automation Concurrency Control

GetOrder & UpdateOrder API enhancements

Other Large Order Improvements

Performance Results

1

2

3

4

5



What are Large Orders?

• Large Order is an order with a sizeable payload and/or complex processing

– Many order lines in an order – hundreds, thousands of order items

– Many fulfillment activities – dozens to many hundreds of order components and tasks

– Heavy amount of data in the OSM data model and/or in order (e.g. O2A)

– Other factors that impact order complexity / processing • Size and complexity of the message payload

• XML parsing and XQuery logic processing

• Amount of fulfillment status & milestone updates



What are Large Orders?

Examples

• B2B Order – an order for a business customer, requiring fulfillment of:

– hundreds of staff, across multiple sites

– Multiple services – IP/Ethernet connectivity, VoIP, VPN services

• Bulk order – operational orders that are managed in bulk. It can be for: – Service readiness – e.g. pre-activation of logical resources

– Billing – e.g. service downgrade or termination due to arrears

Key takeaway: OSM 7.2.4.1 can handle these large orders efficiently

– Large orders can be among the most important to service provider’s business in terms of revenue, margin, and satisfaction of their enterprise customers



Performance of Large Orders Prior to OSM 7.2.4.1

• In versions of OSM prior to 7.2.4.1 the limits to how large an order can process were as follows:

– Before 7.2.4.1, limitation with O2A cartridges is ~250 lines per order

– For non-O2A, the line item upper-bound is typically higher than 250 lines but not unlimited neither

– As OSM is given orders greater that approach or exceed the upper limit, performance decreases to a point where the system becomes unusable and impacts other smaller orders as well • Limitation cannot be solved by more hardware (e.g. increasing memory, cpu, disk, etc.)

• Above limitations were not a problem for B2C or small B2B orders, but for medium/large/x-large B2B or bulk orders, larger order support is needed



Performance of Large Orders With 7.2.4.1

• OSM 7.2.4.1 provides improvements that allow it to process larger orders than ever before, and process them with significant improvements in performance and scalability

– O2A cartridges have been enhanced to take advantage of OSM core product improvements

– With appropriate hardware and attention to system tuning/configuration, OSM 7.2.4.1 can now handle O2A orders of 5000 order lines. Note updated O2A cartridge required

– Custom developed cartridges will also be able to process large orders with 7.2.4.1. Some improvements come without any custom cartridge changes. But to take full advantage of 7.2.4.1 for large orders, some changes to custom cartridges are required.



OSM 7.2.4.1 Large Order Enhancement Summary

• OACC – Order Automation Concurrency Control

– Powerful new feature to manage workload that can be generated by large orders

• UpdateOrder and GetOrder Improvements

– Enhancements to OSM APIs to make critical operations such as fulfillment state calculation and data retrieval super-efficient. The larger the order the bigger the benefits, but significant improvements can be had even for smaller orders.

– Some of these enhancements are automatic, requiring no cartridge changes. Others require some degree of modification to cartridges to take advantage of them. Updates to O2A 2.x cartridges are available that are optimized for 7.2.4.1

• Other OSM enhancements

– Better cache and memory management, faster Order Management UI, etc.



Order Automation Concurrency Control (OACC)



Order A


OSM is a highly concurrent system (for performance)

• Many orders can be processed at the same time

• Independent parts of each order are automatically processed in parallel

OACC is an efficient and configurable way to govern the level of concurrency of order processing in OSM


Background

Order B

Order C



A major motivator is to enable sequential execution of automation plugins from parallel tasks / order components of an order

• Fulfillment state updates must not happen in parallel where race condition can occur

• O2A cartridges have implemented locks at order level, but in large orders, live-locking happens frequently, resulting in transaction timeouts & retries

Another motivator is to constrain resource consumption of certain types of orders at the system level

• A loose analogy is a “rev limiter” on an engine – limits how much order processing can happen at the same time


Background – Motivation



• Policy-driven approach for limiting concurrency within the OSM automation framework – very flexible

• Apply to deployed OSM automation (receiver) plugins globally or selectively

• Concurrency levels can be adjusted


Functional Highlights


Cartridge X v1.0


At any moment, there are many, many automation plugin executions in the server:

• Multiple plugins can be executing within an order

• Many plugins can be executing among orders of a cartridge

• Multiples of that when there are multiple cartridges and cartridge versions running at the same time

An OACC Policy specifies how a certain portion of these plugins are to be concurrency-limited.


OACC Policy

Order A

Order B

Order C

Cartridge X v1.1

Order D

Order E

Order F



An OACC policy can be thought of as a set of multi-process buckets with 3 parameters:


OACC Policy

Element Name Description

Target The automation plugins that are applicable to this policy, a.k.a., the “bucket filter”. •Any of: cartridge namespace, cartridge version, or an XPath (against automationMap.xml)

Scope How the targeted executions are to be grouped by, a.k.a., the “bucket granularity". •Enumerated choice:

•ORDER_ID •CARTRIDGE_AND_VERSION •CARTRIDGE •SERVER

Level Number, of concurrent plugin execution per target & scope. •(“0” or less means unlimited concurrency)

1

2

M …

Level

Target

1

2

M

…

Level

1

2

M

…

Level

Scope



Example 1: Execute all automation plugins in a cartridge sequentially for each order

– Other transactions for the same order cannot start until the current one completes successfully.

• This OACC policy places no restrictions on the number of COM Sales Orders that can be processed concurrently

This is how O2A cartridges improve large order processing in OSM 7.2.4.1, with OACC


OACC Policy

Name Value

Target cartridgeNamespace=OracleComms_OSM_O2A_COM_SalesOrderFulfillment

Scope ORDER_ID

Level 1

All

plu

gin

s o

f [O

2A

Cart

ridge]

Order 123

Order 246

Order 5237



Example 2: Process at a max of 5 automated transactions concurrently for all orders across all versions of a cartridge


OACC Policy

Name Value

Target cartridgeNamespace=MyCartridge

Scope Server

Level 5

All

plu

gin

s o

f M

yC

art

ridge

[Server-wide]



Example 3: Process sequentially for external receiver type plugins listening on the ‘my.queue’ JMS destination for a specific version of a cartridge.


OACC Policy

Name Value

Target cartridgeNamespace=MyCartridge, cartridgeVersion=2.1.0.0.0, pluginSelector=count(receive/jmsSource/from/jndiName=‘my.queue’)>0

Scope Server

Level 1

All

plu

gin

s lis

tenin

g t

o ‘m

y.queue’ i

n

MyC

art

ridge/2

.1.0

.0.0

[Server-wide]



• The level in OACC policy is per Managed Server.

– e.g. In a WLS cluster, example 1 can process multiple O2A orders at the same time (1 per MS)

• Can apply different policies, if needed, to different automation plugins that exist in the system

• Automation plugins that do not match any policies are not impacted

– In example 1 and 2, plugins of other cartridges can run concurrently unlimitedly

– In example 3, all other plugins not in MyCartridge/2.1, or, in it but listens to other JMS queues can run concurrently unlimitedly


OACC Policy



• Two ways to configure OACC

1. Include the OACC policy in your deployed cartridge • Place OACC policy file as resources\automationConcurrencyModel.xml in the cartridge

• Can only target plugins in the enclosing cartridgeNamespace and cartridgeVersion (they are effectively ignored when specified in the policy file)

• When composite cartridge is used, place the file in the cartridge that contains the order definition. All plugins in the solution become governed by it

2. Specify OACC policies via oms-config.xml file • Specify policy file as oms-parameter AutomationConcurrencyModels

• Can be a delimited list of file names


OACC Policy Configuration How-To’s



• Both methods can be used in combination

– (2) provides means to override without cartridge redeploy • (2) overrides (1) with policy of the same name, or different policies with overlapping targets

• The schema of OACC policy file is automationConcurrencyModel.xsd found in OSM SDK

• An XML sample:


OACC Policy Configuration How-To’s

<automationConcurrencyModel xmlns="http://xmlns.oracle.com/communications/ordermanagement/model">

<automationConcurrencyPolicy name=”mypolicy">

<targetPlugins>

<cartridgeNamespace>MyCartridgeNamespace</cartridgeNamespace>

<cartridgeVersion>1.0.0.0.0</cartridgeVersion>

</targetPlugins>

<scope>ORDER_ID</scope>

<concurrencyLevel>1</concurrencyLevel>

</automationConcurrencyPolicy>



• Implementation is based on WLS’ JMS Message Unit-of-Order feature – JMS messages with the same unit-of-order value are delivered sequentially in the

exact order they were published to the same destination

– OACC generates the unit-of-order value by the following: • {SCOPE_VALUE}:UOOID:{random value between 1 and Level}:{policy-name}

– Example: With Scope=ORDER_ID and Level=1 in policy “mypolicy”, plugins running for order 123456 will have unit-of-order of: • “123456:UOODI:1:mypolicy”

– Applies to both oms_events queue and external JMS response queues • oms_events: OACC inserts unit-of-order when publishing JMS message to OACC-targeted plugin

• External JMS destinations: OACC re-publishes to oms_events queue and inserts unit-of-order, when the (response) message is correlated to OACC-targeted plugin


OACC Internal Implementation

http://docs.oracle.com/cd/E13222_01/wls/docs103/jms/uoo.html







OACC has 2 side-effects, due to Unit-of-Order implementation

1.JMS Redelivery – failed transaction of OACC-targeted plugin causes pause, and then redelivery. Other messages of the same unit-of-order are blocked for the pause duration.

– E.g. If scope=SERVER and level=1, no other transaction of the targeted plugins can go through in the Managed Server


OACC Limitation



OACC has 2 side-effects, due to Unit-of-Order implementation

2. Order Priority – JMS message priority is ignored within same unit-of-order

– E.g. If scope=cartridge, then execution of targeted plugins of ALL orders in the same cartridge will not be prioritized based on order priority

– Normally has no downside since by default all tasks for the same order have the same priority. However task priority offset allows this to be adjusted – OACC will effectively mean tasks processed concurrently with the same OACC policy but different priority offset will be processed as if they were the same priority


OACC Limitation



• OACC is a powerful tool that can improve performance through limiting processing concurrency

– OACC is generally not used. Use only when there is a clear requirement that justifies limiting concurrency.

– Do not use OACC to mask OSM system tuning problems (e.g. reduce WLS thread count when too much context switches with too many threads)


Buyer Beware


GetOrder & UpdateOrder API Enhancements




• Intent: Improve performance when processing large orders

– All enhancements are all backwards compatible – there is no mandatory API change

– Applies to both XMLAPI and Web Service. XML Schemas are updated to reflect changes.


Overview

Enhancement Nature GetOrder UpdateOrder Purpose

(Internal Optimizations) Transparent changes Faster / More efficient

ResponseView (New) Optional input parameter

--- Can return order data in UpdateOrder response

OrderDataFilter (New) Optional input parameter

Can filter order data server-side to reduce instance data returned

ExternalFulfillmentState (New) Optional input parameter

--- Can update external fulfillment states more efficiently



GetOrder

– Faster filtering of pivot nodes

– More efficient extraction of order data from OSM’s order cache


Internal Optimizations

UpdateOrder

– Faster updates when doing updates within the context of a pivot node

– No longer fetches entire order from cache when doing order updates –only fetches nodes which are needed to process updates

– Improved efficiency and concurrency on fulfillment state updates

• Efficiencies gained are generally beneficial to all orders, but amplified for large orders


GetOrder API Enhancements

• With large orders, GetOrder requests with <View> element can return a lot of data

– Frequently, the client (automation plugin) is only interested in certain instances of data from GetOrder. But View/QueryTask definition is static.

– /ControlData/OrderItem is a classic example. Automation plugins often want just a certain order item instead, but would end up getting all of them. Plugin then has to parse and search through the haystack to find the needle. This is very expensive especially for large orders.

– Used to filter out unwanted instances of nodes that are included in the static view.

• New optional OrderDataFilter input parameter in GetOrder request –

– Allows 1..n Xpath conditions to be specified that filters the instance data returned

– Supports full, sophisticated XPath 1.0 expressions – not just classic OSM “mnemonic-path-with-index”


OrderDataFilter


GetOrder API Enhancements

<ord:GetOrder>

<ord:OrderId>5219</ord:OrderId>

<ord:View>FulfillmentQueryTask</ord:View>

<ord:OrderDataFilter>

<!—get fulfillment component element with a componentKey=‘foo’

only matching fulfillment component elements are returned,

but all other elements available in the view are returned normally since they

aren’t targeted by the xpath condition -->

<ord:Condition>/FulfillmentOrderManagement/FulfillmentComponent[componentKey=‘foo’]</ord:Condition>

</ord:OrderDataFilter>

</ord:GetOrder>


OrderDataFilter – Example


UpdateOrder API Enhancements

• New optional ResponseView input parameter in UpdateOrder request

– Enables order data to be returned in <Data> element in the response

– If UpdateOrder performs a Fulfillment State update, the response auto-filters to only include OrderItems and OrderComponents instances impacted by it

• New optional OrderDataFilter input parameter in UpdateOrder request –

– Same functionality in GetOrder, but filters for instance data specified in ResponseView


ResponseView and OrderDataFilter



• New optional ExternalFulfillmentState input facet in OrderUpdate request

– When OrderUpdate is to update external fulfillment state, use it to indicate which order-component-order-items have an external fulfillment state change

– Improves performance for external fulfillment state updates (which is a common OrderUpdate use case)


ExternalFulfillmentState



<UpdateOrder.Request> <View>SendASDL</View> <ResponseView>FSUpdateResponse</ResponseView> <OrderID>999 </OrderID> <ExternalFulfillmentStates> <OrderItemOrderComponentFulfillmentState> <ExternalFulfillmentState>COMPLETED</ExternalFulfillmentState> <OrderComponentIndex>1234</OrderComponentIndex> <OrderItemIndex>456789</OrderItemIndex> </OrderItemOrderComponentFulfillmentState> </ExternalFulfillmentStates> <UpdatedNodes> <_root> <ControlData> <Functions> <FulfillBillingFunction> <orderItem> <ExternalFulfillmentState>COMPLETED</ExternalFulfillmentState> <orderItemRef> <serviceName>C_GSM_ADD_SUB</serviceName> <LineId>987654</LineId> </orderItemRef> </orderItem> </FulfillBillingFunction> </Functions> </ControlData> </_root> </UpdatedNodes> </UpdateOrder.Request>


ResponseView and ExternalFulfillmentState – Example


Other Large Order Improvements



Other 7.2.4.1 Performance Improvements

• Improved Order Cache Management

• Improved Automation Plugin Status Logging

• Reduced Memory Usage By Automation Plugin EARs

• Improved load time for data tab of order details page in the Order Management Web Client

These general purpose benefits do not require any adoption/configuration work on your part. They make OSM more efficient and responsive than ever before for orders of all sizes.



Performance Testing Summary



Performance Results

O2A Cartridges 7.2.0 and 2.0.1 are enhanced – includes adoption of all OSM 7.2.4.1 Large Order improvements available

– Uptake of OACC • Note that custom locks at automation plugins in O2A remain in the enhanced cartridges. In other

words, it is not a functional necessity to apply OACC to the updated cartridges.

– Uptake of new GetOrder and UpdateOrder features

– Benefits of other OSM 7.2.4 and 7.2.4.1 improvements such as improved cache and memory management, etc.

Migration script of O2A 7.2.0 and 2.0.1 for above enhancements is available in OSM 7.2.4.1 software – more details in O2A Cartridges enablement deck

O2A Cartridge with OSM 7.2.4.1 Improvements



Performance Results

• OSM 7.2.4.1 with (migrated) O2A 2.0.1

– In COM/SOM Typical Topology

Achieves processing of Large Order with 6000 order line items.

– Great improvement from previous guidance of 250-line item limit

O2A 2.0.1 Large Order Scalability Benchmarking


Large Order (size in order items)

Order Completion Time

(just in seconds)

Heap Size (GB)

1 X 6k-lines 6h 4min 40sec 21880 26

1 X 5k-lines 3h 36min 57sec 13017 20



1 X 2k-lines 32min 28sec 1948 8

1 X 1k-lines 10min 7sec 607 8


Performance Results

• Beware of small overhead when processing large orders along with regular-sized orders

O2A 2.0.1 Large Order Scalability Benchmarking – concurrently with regular orders


Metric Regular Orders without LO

Regular Orders with 1 X 2K Large Order

Difference

Regular Order Lifetime (second) 24.9 26.9 +8%

Large Order Lifetime (second) 4238 4714 +11%

COM Order Completion per Hour 3202 2998 -7%

Transaction per second (TPS) 76 71 -7%

APP CPU 475 598 +26%

DB CPU 1031 1046 0%

IOPS 4458 4286 -4%


Performance Results

• A proof-of-concept solution was built based on O2A 2.0.1 for an existing OSM customer

– Solution is enhanced with large order improvements, environment-tuned, and run in OSM 7.2.4.1

– Objective is to demonstrate large order processing capabilities of OSM 7.2.4.1 can satisfy the customer’s large order volumes

Performance test results of this PoC shows dramatic improvement over OSM 7.2.4 GA for large orders.

– The order size is X-axis is the number of order lines

Large Order PoC with Custom O2A 2.0.1


73 141 271 553 838

Baseline 724ga 0:02:05 0:07:03 0:13:55 1:01:15 2:34:09

LO Improv 7241 0:00:55 0:01:09 0:02:15 0:04:13 0:07:29

0:00:00

0:28:48

0:57:36

1:26:24

1:55:12

2:24:00

2:52:48

hh

:mm

:ss

Order Life Cycle ( Lines vrs Time ) 7241 vrs 724ga

73 141 271 553 838 1116 1500 2144 2504

7241 LO 0:00:55 0:01:09 0:02:15 0:04:13 0:07:29 0:11:42 0:25:40 0:42:12 1:00:16

0:00:00

0:07:12

0:14:24

0:21:36

0:28:48

0:36:00

0:43:12

0:50:24

0:57:36

1:04:48

hh

:mm

:ss

Order Life Cycle (N Lines vrs Time) - 7241

Single Order Test


Performance Results

• Similar dramatic improvements observed in “1-Hour Test” that submit multiple large orders (one

order every 6 seconds), and finish processing at about one hour’s time

– 200 orders with 141 lines

– 70 orders with 271 lines

– 70 mixed orders with 141 and 271 lines. (50% each)

Large Order PoC with Custom O2A 2.0.1


1-Hour Test

200x141 L 70x271 L 35x141 & 271 L

Baseline 724ga 56 67 80

LO Improv 7241 26 17 10

0 10 20 30 40 50 60 70 80 90

min

ute

s

Total OSM Processing Time 7241 vrs 724ga

200x141 L 70x271 L 35x141 & 271 L

Baseline 724ga 1164.9 2523.7 1083.1

LO Improv 7241 73.2 343.3 138.0

0.0

500.0

1000.0

1500.0

2000.0

2500.0

3000.0

seco

nd

s

Average Order Life Time - around 1 hr 724ga vrs 7241


References

• OSM 7.2.4.1 Developer’s Guide, in “Using Automation” Chapter

• OSM 7.2.4.1 Developer’s Guide, in “Using OSM Web Services” Chapter

• OSM 7.2.4.1 XML API Developer’s Guide


Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 42

Documents

Improved Support to Large Orders - Oracle Cloud · –O2A cartridges have been enhanced to take advantage of OSM core product improvements –With appropriate hardware and attention