sysco.no

Administration of SOA Suite – a few tips

Jon Petter Hjulstad

SYSCO AS

OUGN April 3rd, 2014

sysco.no

Information about me• Jon Petter Hjulstad

• Dept Manager for Middleware at Sysco

• 17 years experience with Oracle Products

• Focusing on WLS, SOA Suite, BPM Suite ++

• SOA Partner Community Award 2012

• Twitter: jphjulstad

• Blog: http://sysco.no/blogg/

Info

sysco.no

Information about SYSCO• IT company established 2004

• Continuous growth, over 100 employees

• Operations, development, consulting in technology and economics

• Competence in database technology, middleware

• Special focus in the energy sector

• Engineered Systems Partner of the Year 2014

• Haugesund, Bergen, Stavanger, Oslo, Ølen, Stord, Lima

Info

sysco.no

SYSCO Middleware

Info

• Certified Consultants

• Architecture, Proactive Support

• Developing: SOA, BPM, Java

• Testing, Tuning, Fixing

• Installation, Patching, Monitoring

sysco.no

Energy

•Lyse

•Eidsiva Energi

•Agder Energi

•BKK

•Hafslund

•SKL

•NTE

•Haugaland Kraft

•Gassco

Public Sector

•Statens Innkrevingssentral

•Stavanger kommune

•Haugesund kommune

•Karmøy kommune

•50+ other municipalities

•Sjøfartsdirektoratet

•Several counties

Shipping

•DeepOcean

•Solstad

•Grieg Star Shipping

•Westfal Larsen

•Aibel

•Knutsen OAS

Media

•TV2

•Altibox

•Haugesunds Avis

•TV Haugaland

•Radio102

•Stavanger Aftenblad

Others

•GE Money Bank

•CGI

•Enoro

•Karmsund Havnevesen

•Marine Aluminium

•FKH

•Ungt Entreprenørskap

Customers and industries

Customer portfolio

sysco.no

Overview

• Introduction

• Patching

• Monitoring / Health Check

• RDA

• Purging

• Tuning

sysco.no

What is SOA administration ?

• «Not Development»

– The way NoSQL is defined – «everything but SQL»

• You get a SOA installation, and some SOA apps – what now ?

• Keeping the wheels running - uptime

• Installing, provisioning, monitoring, tuning, checking logs, patching for security,

patching for bugs ..

• How do we make changes with low risk and without breaking SLA ?

• Are you ready ?

Introduction – SOA Administration

sysco.no

SOA running on Weblogic

Introduction – Fusion Middleware

sysco.no

Enhanced My Oracle Support

• Is not a lot better!

• More interactive, useful

information centers,

advisors.

• Patching experience

is not yet like Windows

Update

Example:

Oracle Fusion Middleware (FMW) SOA 11g Infrastructure Database: Installation, Maintenance, Administration and Purging Guide (Doc ID 1384379.1)

Introduction – MOS

sysco.no

Why patch?

• Software is never without bugs

• Others may have experienced problems you have not seen yet

• Changes in load / type of applications may trigger new problems

• You may have problems that you have not yet discovered

• Why not patch: A patch may give other problems (Software is never without bugs)

Patch Management

sysco.no

How do we keep Software up to date?

• What Products ?

– WLS, JVM, SOA, OSB, DB

• New versions coming – 11.1.1.4, 11.1.1.5, ...

• Patch Bundles – SOA 11.1.1.7.1, 11.1.1.7.2

• CPU – Security focused Quarterly updates from Oracle

– Release dates known upfront

Patch Management

sysco.no

What type of patches exist?New name Previous Used Name(s) Description

Interim Patch PSE, MLR, Exception releaseOne-Off, x-fix, Hotfix,Security One-Off

Fixes available tocustomers who cannot wait until the next patch set

Bundle Patch Maintenance Pack, Service Pack, MLRs, Cumulative Patch, Update Release

An iterative, cumulative patch that is issued between patch sets

Security Patch Update (SPU) Critical Patch UpdatesCPUs

An iterative, cumulative patch consisting of security fixes. Formerly known as Critical Patch Update.

Patch Set Updates (PSUs) Patch Set Updates are used to patch WLS only. Patch Set Updates - quarterly

Patch Management

sysco.no

Should we keep SW up to date all the time?

• No! Upgrades cost

• Being the first may be a risk

– Being the last may also not be the best strategy

• Do we need new functionality ?

• Is security a strong consern ?

• Are your problems fixed in newer version?

• Is current version supported ?

Patch Management

sysco.no

Patch OPatch

• Opatch is the main patching tool for patches within one version

• Patch 6880880

• A plethora of versions– For FMW 11g – you want 11.1

• Latest: OPatch version 11.1.0.10.3 for Oracle software releases 11.1.0.x (March 2014)

• Easy unzip into MW-home

• FMW 12c will also use OPatch to patch WLS

• PS! SOA+OSB has 3 Opatch-directories

Patch Management

sysco.no

Patch JVM

• Using JRockit as example

• Major and Minor versions are released on OTN - Installers

• Patch Set releases are available via MOS

– Distributed as zip files

• Customer patches available via a Service Request

– Contact Customer Support obtain these one-off patches

• Plan how to do it – New installer, new path (or use same)

Patch Management

sysco.no

Patch WLS

• Master Note on WLS Patch Set Updates (PSUs) Note 1470197.1.

Patch Management

sysco.no

Patch SOA

• SOA 11g: Bundle Patch Reference (Doc ID 1485949.1)

Patch Management

sysco.no

Patch OSB

• OSB 11g: Bundle Patch Reference (Doc ID 1499170.1)

Patch Management

sysco.no

Patch according to CPU

• May involve several products

Patch Management

sysco.no

Patch according to CPU

• Smaller patches

Patch Management

sysco.no

What other patches?

• Recommended Patch Advisor

• Search ..and you may find

Patch Management

11.1.1.7

11.1.1.6

sysco.no

A sample strategy

• Patch for Bundle Patches

• Patch according to CPU

• Patch for bugs that is relevant for your use (ex clustered env, use DVM)

• Install new version in separate environment – run same tests there as

in todays environment

• Or: Use patching advice in EM 12c

Patch Management

sysco.no

What to monitor ?

• CPU, Disk, Logs, Memory

• Throughput

• Composite instances failed

• Locked threads

Monitoring

sysco.no

Tools for middleware management

• There are a lot of things to monitor

Monitoring

EM FMW Control

EM 12c

WLS Admin Console

JR Mission Control

WLDF

WLST

BAM

For Farm with a single WLS Domain. Composite lifecycle management, Instance Tracking, Performance Metrics, Fault and Message Recovery

Extends FMW Control for multiple farms. SOA/WLS Management Pack. Historical data, Alerts and Notifications

Monitor and manage the WLS domain and Managed Servers. Configure JDBC, JMS, JTA, Security, Deploy Applications etc.

JVM monitoring and management. Flight recorder for troubleshooting and diagnosis.

Generate and Analyze diagnostic data from WLS instances and your applications. Watches and Notifications. Integrate with JFR for troubleshooting.

Create/manage/monitor WLS Domains via Jython based WLI. WLS Admin Console/Configuration Wizard on a command line.

Real-Time Business dashboards. Correlate Business Indicators with System Metrics by integrating with EM 12c

sysco.no

Recommended health checks

• Monitor cpu, memory usage, disk

• Monitor database growth

• Check for relevant patches

• Check for problems in logs

• Investigate incidents

• Investigate if solutions provided were effective

• RDA

Monitoring

sysco.no

What is RDA ?

• RDA is what Support will tell you to give them when you register a

SR on MOS

– Prepare so you avoid ping-pong in your SR

• RDA is a selection of files and statistics which are relevant for

diagnosting problems in your install

• Current version is RDA 8

Monitoring - RDA

sysco.no

What can you get from RDA?

• Resolve Problems Faster!

Use RDA(Doc ID 1498376.1)

• Data collection

• Health checks

Monitoring - RDA

sysco.no

Health checks

• RDA - Health Check / Validation Engine Guide (Doc ID 250262.1)

Monitoring - RDA

sysco.no

Start using RDA

Monitoring - RDA

sysco.no

How to update RDA?

• OPatch format of RDA is available for RDA 8.02 and higher.

• This format is only suitable for FMW 11g and 12c product stacks.

• OPatch will backup an existing RDA and install the new version in its

place.

Monitoring - RDA

sysco.no

What is purging ?

• SOA Database is growing

• Purging is removing old instances that are not needed

– You can control growth, is OSB an alternative in some cases ?

– Or you must not write all instances to DB

• Not purging may result in bad performance

• You need a purging strategy

– How many instances is produced per day?

Purging

sysco.no

How to prepare for purging ?

• Get to a newer release, og patch – see note ID 1384379.1

• Look for relevant patches

• Monitor database growth / table

statistics

Purging

sysco.no

Purge via console or script

Purging

• Parallel

• Useful for medium [ ~500 GB total to

~10 GB daily] to large installations

• Resource intensive – hence run during off hours.DOP indicates number of parallel threads.

• For large installs [>~500 GB total or

> ~10 GB daily], recommended to run a hybrid of parallel purging + dropping partitions, using optimal thread counts.

• Partitioning• Dropping partitions is the fastest

way to purge.

• Time-Range Partitioning Done

at the component level.

• Decide time-range based on

average and max composite

lifetime.

• Run this in conjunction with

parallel purging for a mixed

load.

• Looping

• Useful for small installations [~100k rows]

• Loops and runs until max_runtime loop parameter is exceeded.

• Not resource intensive, but slow performance.

• All purgeable instances are deleted.

sysco.no

More about database

• Mark Simpson – Understanding the SOA Infra Database

• Including some scripts

• Count rows in relevant tables -

example last week

• Sample scripts in books

http://www.soa247.com/article.php?id=78&subcat=43

Purging

sysco.no

Why tune ?

• A faster solution may remove some pain

• Tuning is not simple – do small shanges, check if response is faster

– Create relevant tests

• There are some quick wins, and books / best practices out there

Tuning

sysco.no

What can be tuned ?

• Database

• OS

• WLS

• Engines

• Application code – good/bad patterns

• Do you have the right architecture for your task?

Tuning

sysco.no

FMW Control – Tuning SOA

• Configure SOA-Infra Properties

– Audit/Instance Tracking(CPU,DB), Payload Validation (CPU)

• Configuring Service Engine threads

– BPEL - Threads(Concurrency), Audit (DB), Validation (CPU)

– Mediator - Parallel/Resequencer Threads (Concurrency),

Audit (DB), Sleep Intervals/Metrics Level (CPU)

• Configure Adapter Properties

– MaxRaise/TxnSize(DB), Number of Threads (Concurrency)

• All this Accessible as MBeans - Update via WLST

Tuning

sysco.no

Tuning tip I - EM

Tuning

• Issue: EM dashboard loads slowly / hangs on login.

• Step 1: Set default time duration for aggregate queries.

• Step 2: Disable auto-fetch of aggregated metrics.

sysco.no

Tuning tip II - EM

• A new parameter is introduced in in 11.1.1.6

• It speeds up login on the Enterprise manager by using cached results when available.

• It is described in note 1423893.1 on MOS.

• It involves adding the parameter

oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS = true

• Ref: http://blog.whitehorses.nl/2014/02/05/improving-login-performance-on-ofm-emc-11-1-1-6/

Tuning

sysco.no

Tuning tip III - SOA

• Tuning Composite Audit Level

• Three levels: off - production - development

• Adjust component>>composite>>engine>>SOA infra

• dev vs prod – database 2X-3X

• dev vs prod – speed down 40%

• What do you loose? Less info for debugging

• Ref SOA Suite 11g Administrator's Handbook

Tuning

sysco.no

Can this be automated?

Tuning

sysco.no

Relevant books / articles

MOS: Performance Tuning and Trouble-

shootingfor SOA Suite Integration 11g

(Doc ID 1366419.1)

SOA Suite 11g Database Performance Tuning White Paper:

http://www.oracle.com/technetwork/middleware/soasuite/learnmore/psrsoadbperformance-1919499.pdf

Tuning

sysco.no

SOA Management Pack Enterprise Edition

• EM 12c contains history and can monitor several servers

• Business Transaction Management

– Discovery of transaction flows across the enterprise

– Leverages transaction content/payload for business visibility

– Exception Management

• Performance & Diagnostics

– Enterprise scale multi-domain management

– Specific framework/metadata knowledge

– SOA 10g (BPEL/ESB), SOA 11g (BPEL/Mediator/DS/HW)

– OSB (Business/Proxy Svc, Pipelines)

• Lifecycle Management

– Configuration Management: SOA infrastructure and applications

– SOA domain and application provisioning

– SOA Patching support

EM 12c SOA Management Pack

Performance & Diagnostics

Lifecycle Management

Business Transaction

Management

Production Test

Development

sysco.no

Summary

• .. one more: Use scripting

• This was just a small selection, no time for deep-dive

• Being pro-active will give you better days

• Find ways to monitor and give you alerts

• Do patch, do check health periodically

• Build competence – and do things a little smarter every week

sysco.no

Thanks for attending!

• Feel free to contact me!

• Twitter: jphjulstad

• Email: jonpetter.hjulstad@sysco.no