1

<Insert Picture Here>

Troubleshooting and Tuning WebLogic

Jeffrey WestSr. Principal Product Manager

3

<Insert Picture Here>

Program Agenda

• General Troubleshooting Tools & Tips• JRockit JVM• JRockit Mission Control• JRockit Flight Recorder• SOAP UI & JRMC/JRF Demo

4

Troubleshooting & Tuning Areas of FocusWhere to start?

• Application• Application Server / Middleware• Database• Operating System• Machine• Storage• Network

Some people start here

My application is fine!

(probably not)

I have the best developers!

(really? in the whole world?)

Well, they’re really good!

(how much are you paying them?)

5

The Near-Ideal ‘Performance Funnel’

• In a well-designed distributed application, I/O becomes the bottleneck

• You really want each layer of the stack to have the smallest possible impact on the performance of your application

Machine

Database

Middleware

Application

Network

Storage

Operating System

6

The ‘Performance Funnel’: Where you start

• It’s OK - This is where you are supposed to start

• First, make your application work RIGHT

• Then, make it work FASTER

• Only a very mature team of developers will produce code that doesn’t need much performance tuning

Machine

Database

Middleware

Application

Network

Storage

Operating System

7

Steps along the way…

• As you open the flood gates, performance tuning is required down the stack

• Each iteration will likely yield a new bottleneck

Machine

Database

Middleware

Application

Network

Storage

Operating System

8©2011 Oracle Corporation

Oracle Overall Strategy

• Deliver a complete, open, integrated stack of hardware, infrastructure, database, middleware, and business applications

• Exploit processor, systems, storage, and networking trends to deliver breakthrough innovations by combining Oracle software with Sun hardware

• Integrate components of Oracle’s software stack to provide unique value to customers

9

<Insert Picture Here>

Troubleshooting Tools

10

Where there’s smoke…

• Application– Java Profilers– Thread Dumps– Custom performance logging

• Application Server – # of JVM’s– Heap / GC– Threads– JMS– JDBC– EJB’s

• Operating System– Swap Space– Open Files

• Database– Sessions & Connections– Tablespace Separation (LOB, IDX,

ROWDATA)

• Machine – RAM– Cores– CPUs

• Storage– RAID Levels & Performance

• RAID-0 – best all-around performance, no redundancy

• RAID-1 – Full rendundancy, single-disk performance

• RAID-3 – best for large datasets (sequential read/write)

• RAID-5 – Most common, versatility and redundancy

• RAID0+1 – RAID-0 performance with RAID-1 redundancy

• Network– Speed/Duplex of:

• Machine• Load Balancer• Switch

11

Our Focus Today

• Application– Java Profilers– Thread Dumps– Custom performance logging

• Application Server – # of JVM’s– Heap / GC– Threads– JMS– JDBC– EJB’s

• Operating System– Swap Space– Open Files

• Database– Long Running SQL– Sessions & Connections– Tablespace Separation (LOB, IDX,

ROWDATA)

• Machine – RAM– SWAP– Cores– CPUs

• Storage– RAID Levels & Performance

• RAID-0 – best all-around performance, no redundancy

• RAID-1 – Full rendundancy, single-disk performance

• RAID-3 – best for large datasets (sequential read/write)

• RAID-5 – Most common, versatility and redundancy

• RAID0+1 – RAID-0 performance with RAID-1 redundancy

• Network– Speed/Duplex of:

• Machine• Load Balancer• Switch

} JRockit Flight Recorder

JRockit Mission Control

12

<Insert Picture Here>

JRockit JVM

13

The JRockit JVM

• Written for server-side applications– Machines with a lot of physical memory– Applications run for a long time

• Trades Memory and start-up time for performance– May load slower than HotSpot, but performs better in the long

run

• No support for Java WebStart• JRockit supports (Java SE) UI applications– Due to the short-lived nature of UI applications HotSpot might

be a better choice

© 2009 Oracle Corporation – Proprietary and Confidential

<Insert Picture Here>

WebLogic Monitoring & Troubleshooting

15

RetrieveExposeProcessCapture

Overview of the WebLogic Diagnostics Framework

Log File Event Collector

MBean Property Harvester

Code Instrumentor

Archiver

Watcher

JRockit Flight Recorder

Buffer

Accessor

Notifier

JRockit Mission Control

•WLST API

•JMX API

•WLDF Dashboard

•SNMP Trap

•JMS Message

•JMX Notification

•SMTP Email

•Diagnostic Image

Pull

Push

What is the JRockit Flight Recorder?

• New in JRockit R28• “Circular buffer” in JRockit JVM that

stores diagnostic data• Always on

• New data comes in and is stored, old data dropped off

• Built-in integration with JRMC• Replaces JRMC Runtime Analyzer and

Latency Analyzer

• Very low/near zero overhead• Uses data already used by JVM

• Data can include events from the JVM and from any other event producer• WebLogic Server (WLDF)

• Fusion Middleware (DMS)

New Data

Old Data

Time

• What it is designed for?• Provide diagnostic information in running production systems• Look back in time to see what happened after a crash• Capture most recent activity to enable analysis leading up to

an issue• Capture data from all levels JVM, WLS, DMS, etc…• Offline/offsite analysis can be done using the JRMC GUI• JRockit dumps capture information to assist in crash-analysis

• What it is not designed for?• Large event payloads or very high volumes of events• Long history• Not a replacement for Debug logging or the server logging

JRockit Flight Recorder Use Cases

18

WebLogic/Flight Recorder Integration Goals and Points

Integration Goals Integration Points

• View and correlate WebLogic and JVM events

• Make problem identification easy

• Enable easy configuration

• Leverage diagnostics in WebLogic Server

• Enhance usefulness of Flight Recorder

• WebLogic JFR events

• Custom WebLogic event viewer in JRMC

• Available OOTB

• On/off/volume control

• Automatic JFR recording control

• WLDF Watch Rules and Notifications

• Remote Diagnostics Agent

1

2

3

19

WebLogic Diagnostic Events for Flight Recorder Integration

• Point-cuts woven into WebLogic to generate diagnostic events– Events are propagated to the

Flight Recorder

• Logical event generators: low, medium, and high

• Very low performance impact– Low – 0.81 decrease in throughput– High – 3.83 decrease in throughput

• Events are throttled to avoid high overhead– Request is tagged for event generation– All events for a specific context ID are

captured regardless of throttling– Target number of reqs/events per

second, checked every 2 seconds, throttling readjusted

1

20

Easy Configuration with WLDF Volume

• Simple configuration: enabled and configured using a volume control per server: Off (default), Low, Medium, High

• Each level has a predetermined data set– Determined through consultation with dev groups and Support– Data set was adjusted based on performance testing– High volume includes user info and stack traces, which are expensive data gathering operations

• Volume also controls “WLDF Recording” in JFR– WLDF has it’s own recording independent of other “recordings” in JFR– At startup (WLDF volume = Off), WLDF recording created, but all events turned off– When WLDF volume set to Low or higher, WLDF events and default JVM events are generated– DMS leverages the same JFR recording

2

21© 2010 Oracle Corporation

Proprietary and Confidential

WLDF Low Volume Events

• WLDF Logging Snapshot• WLDF LogRecord Snapshot• WLDF WLLogRecord Snapshot• Connector Activate Endpoint• Connector Deactivate Endpoint• Connector Inbound Transaction Rollback• Connector Outbound Connection Error• Connector Outbound Destroy Connection• Connector Outbound Register Resource• Connector Outbound Release Connection• Connector Outbound Reserve Connection• Connector Outbound Transaction Rollback• Connector Outbound Unregister Resource• EJB Business Method Post Invoke• EJB Business Method Pre Invoke• EJB Pool Manager Post Invoke• EJB Pool Manager Pre Invoke

• JDBC Connection Rollback• JDBC Statement Execute• Servlet Invocation• Web Application Load• Web Application Unload• Webservices JAXRPC Client Request• Webservices JAXRPC Client Response• Webservices JAXRPC Dispatch• Webservices JAXRPC Request• Webservices JAXRPC Response• Webservices JAXWS Endpoint• Webservices JAXWS Request• Webservices JAXWS Resource

22© 2010 Oracle Corporation

Proprietary and Confidential

WLDF Medium Volume Events

• EJB Home Create• EJB Home Remove• EJB PoolManager Create• JDBC Connection Close• JDBC Connection Commit• JDBC Connection Create

Statement• JDBC Connection Prepare• JDBC Connection Release• JDBC Connection Reserve• JDBC Data Source Get

Connection• JDBC Driver Connect• JDBC Statement Creation

• Servlet Execute• Servlet Request Run• Servlet Request Dispatch• Servlet Request• Servlet Filter• Servlet Async Action• Servlet Context Execute• Servlet Response Write Headers• Servlet Response Send• Servlet Stale Resource• Servlet Check Access• JMS BE Consumer Log

23© 2010 Oracle Corporation

Proprietary and Confidential

WLDF High Volume Events

• EJB Database Access• EJB Business Method Post Invoke

Cleanup• EJB Pool Manager Remove• EJB Replicated Session Manager• EJB Timer Manager• JDBC Transaction Commit• JDBC Transaction End• JDBC Transaction Get XA

Resource• JDBC Transaction Is Same RM• JDBC Transaction Prepare• JDBC Transaction Rollback• JDBC Transaction Start

• JTA Transaction Commit• JTA Transaction End• JTA Transaction Prepared• JTA Transaction Prepare• JTA Transaction Start• Servlet Request Overload• Servlet Request Cancel• Servlet Context Handle Throwable

• Plus stack traces and User IDs

24

Automated Recording Capture

• Integrated with the WebLogic Diagnostic Image and Watch and Notification system– Enables capture based on system state, event – capture during event; no need to

replicate– Watch for stuck thread count, heap size increase, available memory, etc.– Set up notifications: capture WLDF image– Diagnostic image capture spurs JFR file generation; JFR file included in diagnostic image– Includes full JFR data from all event generators

• Integration with Oracle Support’s Remote Diagnostics Agent– Generates WL Diagnostic Image

WLDF

Trigger JFR Buffer Write

Capture JFR in Diag Image

3

25

WebLogic Experimental Plug-in for Mission Control 3

26

WebLogic Experimental Plugin

27

<Insert Picture Here>

WebLogic Tuning

28

Areas of Focus

• JVM’s and Machines• Threads / Work

Managers• JMS• JDBC• EJB

29

The timeless battle with Garbage Collection

• How do you slice & dice your CPU and memory?

• More JVM’s with smaller heap… or fewer JVM’s with bigger heap?

• Bigger Heap = longer pause times for GC

• Ultimately depends on your application – can you tolerate longer GC times?

30

JVM’s and Machines

• Machines:– 8GB RAM– Dual-CPU, Quad-

Core

• Why run only one JVM per machine?

• Run multiple JVMs!

512MB: Admin

2GB: OS, etc

2GB: MS-1

2GB: MS-3

2GB: MS-2

2GB: OS, etc

2GB: MS-4

2GB: MS-5

31

Threads / Work Managers

• Define a work manager for each of your applications• WebLogic will self-tune the work managers for the

best performance• You can define:– Context Classes for prioritizing requests– Minimum Thread Constraint– Maximum Threads Constraint

• If you have an application that makes a WS call back to an application on WLS, the applications should have different Work Managers (ESB)

32

JMS

• Connection Factories– Enable Load Balancing– Disable Server Affinity

• Persistent Storage– File Stores offer the best performance– JDBC stores offer ease of failover

• Distributed Destinations– Spread the work across the cluster– Partitioned Distributed Topics eases event architectures

33

JDBC

• Use JNDI Datasources!– Don’t use application-level connection pools – you can’t monitor

them holistically

• Number of Threads / Connection Pool Size– Max Threads Constraint associated with connection pool– Monitor & Minimize waiters

• Max Size = Initial Size, disable Shrinkage– Less latency for handling peak loads

• Prepared Statement Cache– Tune it so you have a high (+90%) hit rate– Write your applications to use prepared statements

34

Common Application ‘mistakes’

• Read a properties from a file on each request– Poor performance – File I/O, doesn’t scale– Try storing your properties in a database and/or caching them

in the JVM

• New JAXB context for each parse event– Create one singleton context and reuse

• Too many session variables– Clear a variable when you don’t need it anymore

35

36

Base Content Slide

• First-level bullet– Second-level bullet• Third-level bullet– Fourth-level bullet• Fifth-level bullet

37

Example of TitleExtending to Two Lines

• First-level bullet– Second-level bullet• Third-level bullet– Fourth-level bullet• Fifth-level bullet

38

Alternate Title with SubheadArial Size 18 (Gray)

• First-level bullet– Second-level bullet• Third-level bullet– Fourth-level bullet• Fifth-level bullet