Orchestrator Studio Performance on Application Interface

Orchestrator Studio Performance on Application Interface Services (AIS) Server

The performance characterization of the AIS Orchestrator Studio

that is available with EnterpriseOne Application Release 9.2 and

Tools Release 9.2.4.3.

June, 2020 | Version 1.01

Copyright © 2020, Oracle and/or its affiliates Confidential – Public

1 TECHNICAL BRIEF | Application Interface Services (AIS) Orchestration Studio Performance | Version 1.01

Copyright © 2020, Oracle and/or its affiliates | Confidential – Public

PURPOSE STATEMENT

This document provides an overview of features and enhancements included in EnterpriseOne Tools Release 9.2.4.3. It is

intended solely to help you assess the business benefits of upgrading to Tools Release 9.2.4.3 and to plan your I.T. projects.

DISCLAIMER

This document in any form, software or printed matter, contains proprietary information that is the exclusive property of

Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle software

license and service agreement, which has been executed and with which you agree to comply. This document and

information contained herein may not be disclosed, copied, reproduced or distributed to anyone outside Oracle without

prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any

contractual agreement with Oracle or its subsidiaries or affiliates.

This document is for informational purposes only and is intended solely to assist you in planning for the implementation

and upgrade of the product features described. It is not a commitment to deliver any material, code, or functionality, and

should not be relied upon in making purchasing decisions. The development, release, and timing of any features or

functionality described in this document remains at the sole discretion of Oracle.

Due to the nature of the product architecture, it may not be possible to safely include all features described in this document

without risking significant destabilization of the code.



TABLE OF CONTENTS

Purpose Statement 1

Disclaimer 1

Executive Summary 3

Overview: Configurating the AIS Server Architecture 4

Architecture and Design of the AIS Server 4

Configuration and Tuning 5

The Value 5

How it works 5

Testing Use Cases 5

Use Case Scenarios Used for Testing 6

Testing Methodology and Metrics 7

Methodology 7

Metrics 7

End User Experience 7

Server Side Metrics 8

Operating System Metrics 8

WebLogic HTML Server Metrics 8

Testing Results 10

Why you should tune first 10

Baseline Test 10

JMeter Metrics 10

Garbage Collection Statistics 11

Orchestration Studio Users 10, 20, and 50 with Background JSON Load 12

Orchestration Studio Performance with Background JSON Load 12

Server Side Response 12

Garbage Collection Statistics 12

Boundary Condition Tests 13

AIS Managed Instance Heap Size 13

Orchestration Studio Login Users 13

Increasing the vCPU 15

Java Garbage Collection Algorithms - Parallel versus G1 GC 16

Oracle Database Configuration – Open Cursors 18

Conclusion 18

Appendix A: Test Configuration 19

Appendix B: Background Load Details 20

Appendix C: use case steps 20



EXECUTIVE SUMMARY In today’s landscape, business requirements are always evolving. IT decision makers are looking for products that improve

their employees’ efficiency throughout the life cycle of an implemented ERP system. The JD Edwards team has been

investigating mechanisms to lower the ongoing total cost of ownership of your JD Edwards solution and deliver on the

promises of more integrated systems, ease of management, and improved feature and functionality.

The following are some challenges that our customers face today:

CHALLENGE SOLUTION

Additional servers needed to support the

EnterpriseOne Application Development

Framework (ADF) and the AIS functionality

This document covers the new implementation of the AIS Server and

more specifically the EnterpriseOne Orchestrator Studio which now

resides on the AIS Server as another managed instance on the HTML

Server component of the EnterpriseOne architecture. The ADF and the

AIS functionality have been merged, creating a more streamlined

implementation of the AIS Server in the EnterpriseOne architecture.

Scalability challenges and how to handle

increased load of the Orchestrator Studio

users

The redesign of the AIS Server, consolidating the ADF and the AIS

functionality into one solution, provides a greater scalable model to the

EnterpriseOne solution.

Understanding the performance impact of a

large number of Orchestrator Studio users

working concurrently on the AIS Server for

processing other activities such as active

notification and orchestration logic

Performance tuning the AIS server heap size, selecting the correct

garbage collection algorith, and making sure Oracle cursor settings

provides a solid foundation for a single AIS server to perform both

Orchestration Studio and respond to notification and orchestration JSON

requests.

Challenges and Solutions

In response to these challenges, the JD Edwards team has developed an integrated solution within the same HTML Server to

support the full range of capabilities of the AIS Server.

For more information, see EnterpriseOne Tools 9.2.4.3 Orchestration Studio Documentation.

https://docs.oracle.com/cd/E53430_01/EOTOS/toc.htm



OVERVIEW: CONFIGURING THE AIS SERVER ARCHITECTURE

The AIS Server provides the interface for communication with various AIS Server clients. The AIS Server provides a REST

interface (HTTP) and uses JSON requests to interact with the EnterpriseOne applications and forms.

This document covers the performance characterization of the new design that is introduced in Orchestrator Studio 9.2.4.0.

The Orchestrator Studio is deployed on the AIS Server instead of on a separate ADF Server and works with EnterpriseOne

Applications Release 9.2 and Tools Release 9.2.4.3. The new design eliminates the additional tasks of installing and

maintaining an ADF Server with its own discrete WebLogic Server.

Architecture and Design of the AIS Server The AIS Server resides on the same EnterpriseOne JAS Server as a separate managed instance. The Orchestrator Studio is

accessed through the AIS Server.

EnterpriseOne Architecture

The EnterpriseOne architecture depicted in the above image has HTML and AIS managed instances on a single WebLogic

Server. In this architecture, the HTML Server is used only to handle the JSON requests initiated on the AIS Server, and not

for processing requests from interactive applications. It is a simplified architecture to concentrate on the testing of strictly

the AIS Server.

In a more normalized architecture, the recommendation is to have two HTML Server configurations: one HTML managed

instance for processing AIS requests and the other HTML managed instance to handle the load of interactive applications.

Generic EnterpriseOne Architecture With Two HTML Servers

The figure above shows the EnterpriseOne architecture with an additional dedicated HTML Server instance for processing

AIS Server requests only. This additional instance is recommended so that the performance of the EnterpriseOne HTML

Server used by the EnterpriseOne web client users is not impacted by the AIS Server requests.



Configuration and Tuning This document provides guidance on the configuration and tuning of the AIS managed instance on the WebLogic Server for

the Orchestrator Studio load and AIS functionality in general.

The following are three tuning methods that can improve performance:

• Increase the heap size on the AIS WebLogic managed instance from a default of 256 MB to 4,096 MB.

• Increase the Oracle database limit for open cursors.

• Change the garbage collection (GC) algorithm specified in the WebLogic Server startup parameters for the AIS

managed instance from parallel to G1 GC.

Heap Size Tuning

Initial tests were executed with a default heap size of 256 MB. Consumption of memory with this default heap size was

inefficient causing a performance bottleneck as the Orchestrator Studio user requests increased. A heap size of 4,096 MB

was configured for the AIS managed instance to provide sufficient memory to handle all the Orchestrator Studio user

requests designed for the testing. More details are discussed in the results section of this document comparing the

256-MB heap size profile with 4,096-MB heap size.

Depending on their specific workload, customers can set their heap size to an appropriate value. Analysis of heap size data,

garbage collection statistics, and other metrics such as end-user response times is a good way to evaluate the performance

impact of bottlenecks created by configuring a heap size that is too small.

Oracle Database Open Cursors

It is important to monitor the Oracle database open cursor limit parameter and set the value high enough to prevent the AIS

processes from running outside open cursors. The open cursor limit will vary depending on AIS traffic and load. Initial load

testing of the Orchestrator Studio on the AIS Server showed that the Oracle Database open cursors gradually increased and

attempted to automatically exceed the Oracle default setting of 300. A setting of 400 or more is used to handle the

increased Orchestrator Studio user load, but this metric should be monitored to determine the range of open cursors used

by the specific implementation of AIS.

As this testing was performed solely on an Oracle database server, a comparison was made with the values of open cursors

for other EnterpriseOne-supported databases such as SQL Server and DB2 UDB. The SQL Database Server has an open

cursor limit of 65,556, which is also the default setting for DB2 UDB. This higher value means that the concern of adjusting

the open cursors is specific to the Oracle database.

Changing Garbage Collection Algorithm

The parallel GC algorithm is used by default with JDK release version 1.8. Starting with JDK 1.9, the G1 GC algorithm will be

the default methodology used for garbage collection. Fortunately, it is backward compatible with JDK 1.8 with a simple

addition to the server start parameters. This document shows how the G1 GC algorithm can be configured to increase

performance by 20%.

The Value As of EnterpriseOne Tools Release 9.2.4.3, the configuration of the AIS Server makes it capable of managing the load of the

Orchestrator Studio users and servicing the JSON requests. Prior to this, the AIS Server only serviced the JSON requests and

a separate server was required to support an ADF Server component, which managed the Orchestrator Studio software

implementation.

How It Works The merged services of the Orchestrator Studio and the functional JSON request services of the AIS Server are now located

on a single AIS managed server instance. This merger of services to the same server limits the number of servers in the

EnterpriseOne architecture and provides the same functionality as before.

TESTING USE CASES The use cases outlined below were designed to recreate typical Orchestrator Studio user activities. These activities include

insert, update, delete, and create actions of a simple orchestration performed through unique user logins to the

Orchestrator Studio residing on the AIS managed instance within the WebLogic HTML Server.



The use case below introduced additional load into the environment to understand the impact on the AIS Server when the

Orchestrator Studio users were running concurrently with the typical AIS JSON functionality, as these functional activities

now reside on the same server.

Orchestration User Activity Use Case:

1. Log in to the Orchestrator Studio.

2. Initiate the creation of a new orchestration.

3. Create a new orchestration rule.

4. Add an existing form request (which was previously created).

5. Delete the form request.

6. Create and define a new form request.

7. Add an existing data request (from an existing data request template),

8. Save the orchestration.

9. Log out of the Orchestrator Studio.

The above use case is the base load for the Orchestrator Studio. Batches of 10, 20, and 50 unique EnterpriseOne users

logged in to the Orchestrator Studio concurrently and performed the above actions to generate the load to simulate a

normal customer environment under load.

While the Orchestrator Studio loads were generated, metrics were simultaneously collected to measure and compare the

performance between the different load levels of 10, 20, and 50 users. The results of these metrics were brought together to

form the basis for the comparison of performance. Detailed analysis of this comparison is provided in the following sections

of this document.

See Appendix C for more detail and screenshots of use case actions.

Use Case Scenarios Used for Testing Tests for different scenarios were performed using the use case actions described in the previous section with the following

configurations to measure performance for a variety of conditions that an Orchestrator Studio application might encounter.

The following scenarios were used in the testing:

1. Baseline tests – 10, 20, and 50 Orchestrator Studio users

2. Baseline tests – 10, 20, and 50 Orchestrator Studio users with additional background load

3. Boundary condition testing included the following configurations to illustrate potential performance bottlenecks:

256 MB versus 4,096 MB

Single versus multiple JDE Users

4 vCPUs versus 8 vCPUs

Parallel GC versus G1 GC

Each boundary condition scenario is described in detail with the metrics collected, in the following sections of this

document. Background load refers to a series of previously defined standard orchestrations that generate bell notifications.

A list of the notifications and orchestrations that formed background load are in the below table:

See Appendix C for more detail and screenshots of use case actions.

NOTIFICATION OR ORCHESTRATION AUTO START

Credit Review Notification 3 minutes

Equipment Down 3 minutes

Invoice Batches in Error 3 minutes

Invoice Batches Out of Balance 3 minutes

Invoice Pay Items Overdue 3 minutes



Large Customer Orders Index 3 minutes

Pending Invoice Batches 3 minutes

Pending Receipt Batches 3 minutes

Receipt Batches in Error 3 minutes

Receipt Batches Out of Balance 3 minutes

Simple Notification for CBM Alerts 3 minutes

Test Case Simple Large 3 minutes

Test Case Simple Shorts 3 minutes

Background Orchestrations

TESTING METHODOLOGY AND METRICS

Methodology

Tests were first categorized as baseline tests, second as baseline tests with load, and finally as boundary condition tests. All

the tests were executed for an hour with a standard warm-up scenario before the actual test. The warm-up test was running

a smaller number of users. This process forces the caching of any recently restarted processes. The goal of the testing was

not to measure any caching activity, but to measure only that activity that represented a steady state of processing.

Metrics

Key performance indicators show whether the application is performing within the expected range of parameters or whether

the performance is below acceptable standards.

The test cases evaluated and explained in this document are a replay of an HTML stream using the JMeter utility. Apache

JMeter is a Java-based open source software designed to load test functional behavior and measure performance. The

metrics that JMeter collects represent the server side response between the communicating servers. Server side metrics

differ from end user metrics in that end user metrics also include the processing that occurs with the browser.

The Orchestrator Studio is an HTML application and as such JMeter is an appropriate tool for collecting these specific

metrics. In JMeter, the simulated replay of the Orchestrator Studio user load can be reproduced.

The following key performance indicators were used in testing the AIS performance including that of the Orchestrator

Studio.

End user experience

This metric was derived through a manual test used to represent the real user experience including browser and

server side performance.

Server side response

This metric is collected by JMeter.

Operating system metrics

These metrics include CPU or memory utilization from the operating system end as well as garbage collection logs.

All these metrics play an important role in identifying the source of any problem and subsequently in troubleshooting.

End User Experience



End user experience serves as a good indicator of an existing performance bottleneck. The end user experience is measured

manually by logging in to the Orchestrator Studio while the load tests are performed and recording any performance

anomalies.

End user experience indicates the response time of the Orchestrator Studio page; the time taken by a page to render the

objects includes the time it takes to respond to any object clicks, saves, rules, form requests, logins, as well as log out

activities. End user experience is a unique metric; it includes the server side response on the EnterpriseOne component and

the time it takes to process that information request through the browser interface.

Server Side Metrics

Server-based performance metrics are collected at the back end of the EnterpriseOne architecture and provide an indication

of the health of the network and communications between the EnterpriseOne server components. JMeter is used to collect

the server side runtime metrics in the tests explained in this document.

Apache JMeter is a Java-based tool that tests performance on static resources, dynamic resources, and dynamic web

applications. It can be used to simulate a heavy load on a server, a group of servers, or network configurations to test the

impact of the load on the EnterpriseOne architecture on which the Orchestrator Studio is deployed, and to analyze overall

performance under different loads. Server side metrics are just one of the many data points based on which performance is

evaluated.

Operating System Metrics

Metrics collection would not be complete without the operating system metrics, namely the processor and memory metrics

that are used as indicators for performance. For the tests discussed in this document, CPU and memory metrics were

collected for the Database Server, Enterprise Server, and HTML Server instances.

This document concentrates on the operating system performance metrics of the HTML Server on which the AIS managed

instance resides. This is the server on which the Orchestration Studio users perform their tasks and the JSON requests for

orchestrations and notifications are received.

Both a 4-vCPU configuration and an 8-vCPU configuration on the HTML Server are analyzed. Increasing server resources

does not always result in better performance. For this project, a higher number of vCPUs on the HTML Server did not show

an impact on performance. Further details are provided later in this document.

WebLogic HTML Server Metrics

The WebLogic HTML Server metrics collected are enabled through the Server Start parameters and are modified in the

WebLogic Administrative Console. The Server Start parameters used for the WebLogic HTML Server under the AIS managed

instance are shown in the below screenshot:



Server Start Parameters For AIS Server

In the above figure, the Server Start tab is highlighted and the arguments shown below are used to start the AIS managed

instance. The AIS managed instance Java process that is initiated handles all the Orchestrator Studio user and JSON

requests for the EnterpriseOne architecture.

This section discusses the garbage collection metrics which helps to measure Java memory heap usage and the importance

of the Java Flight Recorder (JFR) metrics. Although the JFR metrics are not discussed in detail in this document, they are

important to collect if a granular detail of object usage within the Java memory heap needs to be analyzed. For this project

and for the purposes of reporting, JFR metrics are not included.

Server Start parameters in the configuration shown in the above screenshot includes:

Configure the heap size (4,096 MB)

Enabling garbage collection metrics

Setting up the garbage collection algorithm (G1 GC)

Starting the Java Flight Recorder metrics collection

The below table describes each WebLogic Server Server Start configuration parameter in greater detail:

DESCRIPTION SERVER START PARAMETER

Enabling Garbage

Collection

-Xloggc:/u01/OracleJDE/user_projects/domains/jde_domain/servers/ais_server/logs/gc.log

-XX:-PrintGCDetails -XX:+PrintGCTimeStamps

Note: The location of the log file (that is –Xloggc may differ for a customer implementation).

Configuring

Heap Size

-Xms4096m -Xmx4096m

Note: It is recommended that both the minimum and maximum heap sizes be set to the same value.

Specifying the

G1GC Garbage

Collection algorithm

-XX:+UseG1GC

Note: In the absence of this parameter, the default algorithm for JDK 1.8 is parallel

Java Flight

Recorder Setting

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+UnlockCommercialFeatures -

XX:+FlightRecorder -

XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=/tmp/jfr_ais,maxage=3h



WebLogic Server Server Start Configuration Parameters

The Java heap size is the amount of memory allocated for applications running in the JVM and can be altered to match the

workload.

Garbage collection is the process by which Java programs perform automatic memory management. By default, JDK 1.8

uses a parallel algorthim GC if the algorithm is not specifically configured. Starting with JDK 1.9, the garbage collection

algorithm is G1 GC.

Parallel GC is used in all the tests presented in this document except for one test that was performed using G1 GC algorithm

to validate the performance improvement.

The Java Flight Recorder is a powerful tool used to investigate performance issues which does not skew the results with its

own performance overhead. We have collected the JFR logs, but they were not used for the analysis of any of the results

presented in this document. The JFR metrics are mentioned to point out the additional utilities that might be used in

performance analysis.

Note: To enter the JFR setting shown in the above table, you require user credentials to log in to the respective WebLogic

console. These settings are specified in the Arguments field under the Server Start tab.

TESTING RESULTS

Why You Should Tune First Tuning the environment is the best method to avoid performance bottlenecks in the testing process. Failure to tune initially

results in frequent retests of the use cases because the initial tests were performed with sub-optimal conditions.

Many of the initial findings, such as the impact of increasing the heap size and the importance of setting a high Oracle

database open cursor limit, were made as part of the initial testing and tuning of the AIS environment. As such, retesting was

avoided because these configuration requirements were found early in the testing cycle.

Baseline Test The first step in performance analysis is to create a baseline test which creates the foundation for comparing future test

results. The first category of baseline tests that were executed were with 10, 20, and 50 concurrent Orchestrator Studio users

with the heap sizes of 256 MB and 4,096 MB.

The purpose of this test was to determine if increasing the heap size of the AIS managed instance affects the load on the AIS

managed instance. Two metrics—JMeter server side response times and garbage collection statistics— are discussed in the

results section that clearly show the performance impact of the testing scenarios.

JMeter Metrics

The JMeter server side response time on the y-axis of the graph represents the time it took for the Orchestrator Studio user

to complete the use case actions. A detailed description of the use case and the actions performed by the Orchestrator

Studio user are provided in the appendix.

The graph below compares the JMeter server side response times for each of the three use cases of 10, 20, and 50

Orchestrator Studio users with a default heap size of 256 MB and an increased heap size of 4,096 MB.



Graph Representing the JMeter Average With Two Different Heap Sizes

First let us discuss the impact of scaling users from 10 to 20 and the finding derived from testing 50 concurrent Orchestrator

Studio users. Going from 10 to 20 concurrent users, we are seeing a performance improvement of 42 percent with a 256-MB

heap size and an improvement of 16 percent with a 4,096-MB heap size.

The improvement is attributed to performance gains in the AIS managed instance that supports the Orchestrator Studio

caching. When the improvement of caching is achieved the performance of server side response times improves moving

from 20 to 50 concurrent Orchestrator Studio users, as one would expect in an increasing load scenario. This improvement

is observed for both the 256-MB heap size and 4,096-MB heap size managed instance configurations.

The figure above also shows performance improvements in server side responses when the heap size is increased from 256

MB to 4,096 MB. A performance change from 80 percent to 91 percent is observed indicating a major bottleneck at the heap

size with Orchestration Studio user load.

Garbage Collection Statistics

The garbage collection statistics derived from testing 256-MB and 4,096-MB heap sizes lead to conclusions

similar to those derived from the JMeter server side metrics.

The figure below shows the comparison of the garbage collection log file statistics for a 50-concurrent Orchestrator Studio

user test run with two heap sizes, 256 MB and 4,096 MB.

Garbage Collection Graph With Two Different Heap Sizes



The blue lines indicate garbage collection events and the red background indicates the Java heap memory allocation. The

scales of the two graphs are different. In the first graph, the scale is up to 256MB and in the second graph, the scale is up to

4,096 MB. The y-axis scale represents the total heap memory of the JVM process that was allocated by the WebLogic server

at the managed instance start time.

The concluding analysis is that the 256-MB heap size performs poorly in comparison to the 4,096-MB heap size. The

graph for 256-MB shows many garbage collection events. A garbage collection event is a Java action trying to reclaim

unused memory for an active process. Garbage collection events have a major impact on performance—the higher the

number of events the more performance degrades.

Clearly, the 256-MB garbage collection profile requires significantly more events than the 4,096-MB heap size profile. With

fewer garbage collection events, the 4,096-MB heap size profile has a significant improvement in performance.

Orchestration Studio Users 10, 20, and 50 with Background JSON Load To understand how the AIS Server handles the Orchestrator Studio user load and additional background load, tests were

performed with a range of 10, 20, and 50 users with the addition of background load of orchestrations and notifications.

Processing of orchestrations and notifications with JSON requests provides a complete picture of the standard load an AIS

Server can have for a customer. The standard Orchestrator Studio user load was generated using the JMeter utility and the

background load was generated using the AIS Server Quartz scheduler. The purpose of creating an Orchestrator Studio user

load with background processes was to understand the performance of the AIS Server while managing the load created by

other JSON requests, represented by the background load.

Detailed explanations of each background process are in Appendix C. All the tests were performed with a Java heap size of

4,096 MB to show the performance differences related to load increase, given the AIS Server also processes the load of

JSON requests.

Orchestration Studio Performance with Background JSON Load

Server Side Response

The graph below depicts the comparison of the server side average response times recorded for load tests using 10, 20, and

50 Orchestrator Studio users executed with and without the JSON background request.

Graph Representing JMeter Average With And Without Load

The comparison shows that the average response time increased gradually with the additional background JSON load for all

the three scenarios.

AIS Server caching initially has a performance benefit with 10 and 20 Orchestration Studio users; however, that benefit is not

maintained as indicated by the increase in JMeter server side responses for the load of 50 Orchestrator Studio users.

Garbage Collection Statistics



Garbage collection logs were collected and analyzed to understand the differences in Java heap size memory utilization for

the baseline Orchestrator Studio user load with and without additional JSON load created by running orchestrations and

notifications.

The graph below shows the results of a test performed with a load of 50 Orchestrator Studio users with and without JSON

load.

Garbage Collection Graph With and Without Load

The first graph shows the use case scenario of the Orchestrator Studio with a load of 50 users without additional JSON load

and is compared to the adjacent graph that represents a use case scenario with a load of 50 users and with the additional

JSON load.

The graphs are very similar, but the graph with the additional JSON load represents a garbage collection profile that has

significantly more garbage collection events. As was illustrated in the 256-MB versus 4,096-MB garbage collection

comparison, in this use case too, garbage collection events were expensive in terms of the impact on performance.

The garbage collection graphs in the above figures support the findings derived from the JMeter metrics, that is, the

Orchestrator Studio user response time is slower with additional JSON load. The performance is slower because more work

is being done on the AIS managed server Java process on the AIS Server.

The graphs also show that in both cases, sufficient memory has been provided to the Java process heap size to support both

the Orchestrator Studio user load of 50and the additional JSON load.

Boundary Condition Tests A number of boundary conditions required some tuning as part of the initial metrics collection and for initiating the use case

scenarios.

AIS Managed Instance Heap Size

This document has already covered the analysis of the 256-MB and 4,096-MB Java heap sizes for the AIS Server managed

instance configuration; this testing result is only mentioned here as an initial finding in performance tuning and represents a

true boundary condition test.

Orchestrator Studio Logged-In Users

The second boundary condition that was identified is the Orchestrator Studio user login. When concurrent Orchestrator

Studio users log in to the EnterpriseOne application, they may do so using the standard JD Edwards EnterpriseOne user

credential or unique credentials. As this testing involves concurrent Orchestrator Studio logins of 10, 20, and 50 users. An

initial testing revealed that the choice of EnterpriseOne user credential makes a significant impact on performance of the

Orchestrator Studio application.

Orchestration Studio Login Analysis—Server Side Response Analysis



The graph below compares the server side average response time measured by JMeter for a 10-user test for two scenarios—

when each user had the same credential for the Orchestrator Studio login, and when the login was unique for each user.

Graph Representing The JMeter Average With Single And Multiple JDE Users

The graph above shows that there is a decrease of 86 percent, that is a drop from approximately 170 seconds to under 25

seconds, in terms of the server side response time measured by the JMeter utility when a single login—JDE—was used for

the Orchestrator Studio versus 10 unique logins.

The use of a common Orchestrator Studio user is not the normal process for most customers. They use a unique

EnterpriseOne user to log in to the Orchestrator Studio, which is the recommended process, or they use a group of

EnterpriseOne user logins that others may use for accessing the Orchestrator Studio application.

The logistics of how the Orchestrator Studio handles user login sessions, caching, and general management explains the

higher need of resources if a single Orchestrator Studio login is used. Customers should be aware of this fact and avoid this

boundary condition.

Orchestration Studio Login Analysis—Garbage Collection Analysis

The garbage collection log analysis also supports the fact that with a common Orchestrator Studio login, the memory

management of the Java heap size is heavier. The graphs below show the garbage heap size analysis of 10 concurrent

Orchestrator Studio users for two scenarios—a single login and unique Orchestrator Studio logins.

Garbage Collection Graph With Single And Multiple JDE Users



As shown in the figures above, the number of garbage collection events is greater when a single login is used, meaning that

the Java heap had to perform more memory recovery events than performed for the garbage collection profile where

multiple unique Orchestrator Studio users were used.

The graphs show that sufficient memory is available in the Java heap size to handle all the requests to the AIS managed

server Java process, but the number of garbage collection events is greater in the case of the single JDE Orchestrator Studio

user, supporting the slower response times observed from the JMeter metric of server side response time.

Increasing the vCPU

Increasing the number of processors and adding hardware resources to an existing EnterpriseOne architecture is a common

practice to determine the performance impact on EnterpriseOne in general and the Orchestrator Studio application

specifically.

For this boundary condition test, the AIS Server vCPUs were increased from four to eight and JMeter response time and

garbage collection metrics were analyzed. The result showed that no significant performance benefit was gained by adding

resources to the AIS component of the EnterpriseOne architecture.

The performance observed with 50-concurrent Orchestrator Studio users on a four-vCPU configured AIS Server was

compared to the performance observed on the same server with the same user concurrency after increasing the number of

vCPUs to eight.

Increasing Processors on AIS Server—CPU Utilization Analysis

The next set of graphs show the CPU utilization of the EnterpriseOne WebLogic HTML Server where the AIS managed

instance resides. Two tests are performed for a load of 50 Orchestrator Studio users—one using four vCPUs and the other

with eight vCPUs.

CPU utilization is measured during the complete test for the four-vCPU configuration providing a detailed profile of

processor usage.

HTML CPU Utilization With 4 vCPUs

The test with a load of 50 Orchestrator Studio users was repeated with eight vCPUs and the result is illustrated in the graph

below. Similar to the previous graph, the percent of processor utilization is measured for the complete testing process.

HTML CPU Utilization With 8 vCPUs

As seen in the above graphs, the processor utilization is lower for the eight-vCPU configuration than that of the four-vCPU

configuration.



Increasing Processors on AIS Server—Server Side Response Analysis

The below graph illustrates the JMeter metric of the server side response times for the actions of 50 concurrent Orchestrator

Studio users.

Graph Representing The JMeter Average With 4 And 8 CPUs

The graph above shows a 1.25 percent difference in the JMeter metric between the four and eight vCPUs. This difference

does not warrant a recommendation to increase the number of vCPUs for the AIS Server.

Java Garbage Collection Algorithms—Parallel Versus G1 GC

Both garbage collection algorithms are available in the current implementation of Java JDK 1.8.

By default, Java JDK 1.8 uses the parallel GC algorithm and Java JDK 1.9 uses the G1 GC algorithm. Java JDK 1.8 can be

configured to use either algorithm.

This document does not explain the differences between these two algorithms; these are only presented as a boundary

condition test as the choice of algorithm can yield performance benefit for the Orchestrator Studio and the selection of

algorithm in a production environment needs careful consideration. The impact of algorithm change must be evaluated by

each customer individually and the results may not always be what is shown in this document.

Garbage Collection Algorithm—Garbage Collection Profiles

The following graphs depict two test scenarios for 50 concurrent Orchestrator Studio users with a 4,096-MB Java heap size

using parallel GC algorithm for one scenario and G1 GC algorithm for the other scenario.

The average percentage consumption of Java heap memory was calculated and added to the garbage collection profile

graph for an easier comparison of results.



Garbage Collection Graph With Parallel and G1 GC Algorithm

The comparison of the two test scenarios with different garbage collection algorithms leads to the following observations:

G1 GC algorithm has 27.2 percent more memory consumption on an average

Parallel GC algorithm has more garbage collection events

It is inconclusive which algorithm would benefit the performance of the Orchestrator Studio and AIS managed instance.

More information and metrics are required to make that determination.

In the above figure, the parallel GC algorithm has slightly more events. If a larger number of garbage collection events

decreases the performance of the managed instance, then the G1 GC algorithm should be performing better than the

parallel GC algorithm.

The above figure also exhibits that the G1 GC algorithm has a much higher (27.2 percent) memory utilization. This is seen by

the red portion (free heap space) above the blue Java heap usage lines. As long as there is unused capacity in the Java heap

size, performance should increase with the G1 GC algorithm.

Garbage Collection Algorithm—Server Side Performance

As was concluded in the previous analysis, more metrics are required to determine which garbage collection algorithm

should be used for an AIS managed instance when performing Orchestrator Studio activities.

In this next boundary configuration test scenario, the JMeter server side average response time is compared between the

parallel GC and G1 GC algorithms. The comparison will provide a clear answer to the question of which garbage collection

algorithm to use in for an AIS Server managed instance.

Graph Representing The JMeter Average Parallel And G1 GC



The JMeter server side response times observed in the test results indicate that the G1 GC algorithm provides a 20-percent

or approximately an 8-second benefit to the total response time. This test was performed with 50 concurrent Orchestrator

Studio users with the only variable change being the garbage collection algorithm.

Customers are recommended to consider changing the garbage collection algorithm of their AIS managed instance

configuration to derive response time benefit. As the G1 GC algorithm is the default with Java JDK 1.9, it is worth the time to

explore the impact of this algorithm on performance in a customer environment and Orchestrator Studio load by applying

this algorithm in a Java JDK 1.8 environment configuration for testing.

Oracle Database Configuration—Open Cursors

From the metrics that were collected, it was found that the consumption of open cursors in the Oracle database was a

potential bottleneck in testing and resulted in performance degradation.

The Oracle database open cursor setting determines the maximum number of cursors used by a given session to access the

Oracle database. Open cursor limits were reached in the Orchestrator Studio test scenarios.

The Oracle database open cursor limit was being exceeded during testing and it was a concern whether this behavior could

potentially present a boundary condition in the performance of the AIS Server and managed instances supporting the

Orchestrator Studio.

The Oracle database open cursor default limit is 300 and the table below shows the range of cursors consumed during the

testing process. These metrics were collected after the Oracle database open cursor limit was updated to 5,000.

Cursor Utilization

The above table represents the Oracle database consumption of open cursors for the Orchestrator Studio.

Cursor Utilization with Parallel And G1 GC

A similar result was observed with 50 concurrent Orchestrator Studio users regardless of the garbage collection algorithm.

The open cursor setting limit for the other two supported platforms (SQL Server and DB2/UDB databases) is 65,536 by

default. Performance tuning would not be required for those databases.

CONCLUSION The scenarios used in the AIS Server performance characterization under required Orchestrator Studio and JSON request

loads, allowed for the discovery of key tuning values and identification of boundary conditions that customers should

consider when implementing any AIS Server component solution in an EnterpriseOne architecture.

From this performance characterization, it was found that the Java heap size, Oracle database open cursors, and unique

EnterpriseOne user logins were the primary bottlenecks. Moving to the garbage collection algorithm G1 GC should be an

additional tuning parameter.

Configuration of the Oracle database open cursors and the optimal size of the Java heap is dependent on customers’ needs,

and the managed instance load for the Orchestrator Studio users and JSON requests is one of the other factors that must be

considered during implemention of an AIS Server component solution.



APPENDIX A: TEST CONFIGURATION

The JD Edwards EnterpriseOne architecture configured for the testing process discussed in this document is listed below:

JD Edwards EnterpriseOne Enterprise Server:

• Oracle Enterprise Linux 7.5

• Oracle Database 12c (12.2.0.1.0) 32-bit client

• 4 VCPUs x Intel(R) Xeon(R) CPU E5-2697 v2 at 2.70GHz

• 12 GB RAM

Database Server:


• Oracle Database 12c Standard Edition Release 12.2.0.1.0 – 64-bit production

• 4 VCPUs x Intel(R) Xeon(R) CPU E5-2697 v2 at 2.70 GHz

• 12 GB RAM

HTML Server (includes AIS):


• 4 and 8 VCPUs x Intel(R) Xeon(R) CPU E5-2697 v2 at 2.70 GHz

• 12 GB RAM

• WebLogic Server 12c (12.1.3.0) with Java JDK 1.8

• Two managed instances: HTML (4 GB heap size) and AIS (4 GB heap size)

AIS Server:

• Resides as a managed instance on the HTML Server

• Managed instance (4 GB heap size)

Deployment Server:

• Windows Server 2016 Standard

• 4 VCPUs x Intel Xeon CPU E5-2697 at 2.90 GHz

• 16 GB RAM

Server Manager Console:


• 4 VCPUs x Intel Xeon CPU E5-2697 at 2.90 GHz

• 12 GB RAM

Test Controller:

• Windows Server 2012 R2 Enterprise Edition

• 8 VCPUs x Intel Xeon CPU E5-2697 V2 at 2.70 GHz

• 32 GB RAM

• Apache JMeter 5.1.1.r1855137

Software: JD Edwards EnterpriseOne Application 9.2 Update 4 with Tools Release 9.2.4.3



APPENDIX B: BACKGROUND LOAD DETAILS

This section describes the background load that were used for the testing process. Each background load was placed in the

Scheduler of the Orchestrator Studio and started just before the actual test and was stopped right after the test.

The following orchestrations and notifications were used as a part of the background load on the Scheduler:

NOTIFICATIONS AND ORCHESTRATIONS AUTO START

Credit Review Notification 3 minutes

Equipment Down 3 minutes

Invoice Batches in Error 3mins

Invoice Batches Out of Balance 3mins

Invoice Pay Items Overdue 3mins

Large Customer Orders Index 3mins

Pending Invoice Batches 3mins

Pending Receipt Batches 3mins

Receipt Batches in Error 3mins

Receipt Batches Out of Balance 3mins

Simple Notification for CBM Alerts 3mins

Test Case Simple Large 3mins

Test Case Simple Shorts 3mins

Total Row Total Row Text

List of Orchestrations And Notifications Used as Background Load

The notifications in the table above were derived from the same application notification events that are described in the

following technical brief on performance characterization: Oracle JD Edwards EnterpriseOne Notifications.

The additional load that was generated using the orchestrations and notifications is indicated by the number of jobs

executed on the EnterpriseOne application which is shown below as the bell notification:

EnterpriseOne Application Displaying the Notifications Icon

https://www.oracle.com/webfolder/technetwork/tutorials/jdedwards/White%20Papers/NotificationsPerformanceWhitepaper.pdf



APPENDIX C: USE CASE STEPS

This section illustrates the user activitiy performed while executing the designed use cases.

Orchestrator User Activity Use Case:

1. Log in to the Orchestrator Studio.

2. Create a new orchestration.

3. Add a new orchestration rule.

4. Select Standard.



5. Define the rule to be greater than 100.

6. Add an existing orchestration form request.

7. Delete the previously created form request

8. Add a new form request

9. Define the order of execution.



10. Define the form request.

11. Add an existing data request.

12. Select the data request.

13. Save the orchestration.



14. Log out of the Orchestrator Studio.

Validation of Orchestration and Rule

The basic orchestration, rule, and form requests that were manually created in the Orchestrator Studio or were a result of a

use case execution, can be validated using the Work with User Defined Objects (P98220U) application.

1. Access Work with User Defined Objects (P98220U) application.

2. Search for Orchestrations.

3. Search for Rule.

4. Search for Service Requests and query the form description.

5. Search for Service Requests and query the data request description.



CONNECT WITH US

Call +1.800.ORACLE1 or visit oracle.com.

Outside North America, find your local office at oracle.com/contact.

blogs.oracle.com facebook.com/oracle twitter.com/oracle

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without

notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties

and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed

either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without

our prior written permission.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of

SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered

trademark of The Open Group. 0120

Disclaimer: This document is for informational purposes. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing

decisions. The development, release, timing, and pricing of any features or functionality described in this document may change and remains at the sole discretion of

Oracle Corporation.

https://www.oracle.com/

https://www.oracle.com/corporate/contact/

Documents

Orchestrator Studio Performance on Application Interface