Software Performance Engineering

© 2006 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Software Performance Engineering

Oldrich Kepka, PCoE SOA Performance Architect

Created: Mar 2008

PCoE Site - http://rndportal.mercury.co.il/sites/PCoE/default.aspx 2

Agenda

• About PCoE Group• Introduction to SPE• SPE applied to Software Development Lifecycle

−Performance Risk Assessment and Performance Objectives

−Architecture, Design Guidelines and Code Review

−Performance Modeling

−Performance Testing

−Deployment Guidelines• Q&A


Performance Center of Excellence

• HP Group

− responsible for implementing full SPE lifecycle.

− responsible for knowledge about performance collecting• Currently 5 teams

− AD

− AM

− PPM

− SOA

− Opsware

− ?


Introduction

• Key Components

− Performance Team

− Development

− Product

− SPE Methodology

− Tools

− Knowledge• Key Objectives

− Increase business revenue by ensuring the system can process transactions within the requisite timeframe.

− Eliminate late system deployment due to performance issues

− Eliminate avoidable system rework due to performance issues

− Avoid additional and unnecessary hardware acquisition costs

− Reduce increased software maintenance costs due to performance problems in production

Proactive instead of Reactive


Software Performance Engineering Building Blocks, Roles & Life Cycle

Engineering for Performance

Performance Objectives

(Response Time, Throughput, Resource Utilization, Workload)

Measuring, Testing, Tuning

Measuring: Response Time, Throughput, Resource Utilization, Workload

Testing: Load Testing, Stress Testing, Capacity Testing

Tuning: Network, System, Platform, Application

Roles

Managers

Performance Engineers

Architects

Developers

Testers

Administrators

Life Cycle

Requirements

Design

Develop

Test

Deploy

Maintain

Performance Modeling

(Scenarios, Objectives, Workloads, Requirements, Budgets, Metrics)

Architecture, Design Guidelines & Code Review

(Principles, Practices, Patterns & Anti Patterns, Coupling & Cohesion, Communication,

Concurrency, Resource Mgmt, Caching, State Mgmt, Data Structures, Algorithms)

Performance Risks Assessment

(Project wise, Technical wise)

Deployment Guidelines

(Benchmarking, Capacity Planning, Configuration Tuning)


Software Performance EngineeringOverlay on Software Life Cycle

Software Development Life Cycle

Requirements Design Develop Test Deploy & Maintain

Performance Risks & Objectives


Architecture, Design Guidelines & Code Review

Measuring, Testing, Tuning

Deployment Guidelines


• SDLC Phase: Requirements, Design• Goals

− Identify high level risks during requirements/design phases, mitigate or manage them

− Start discussion on topic of performance early

− Reduce problem space

• Mostly subjective evaluation in these categories:− Business Criticality

− Execution Frequency

− New Architecture elements or frameworks

− Scheduling or Batch Nature

• Risk Assessment Ranking− Low

− Medium

− High

Performance Risks & ObjectivesPerformance Risks Analysis


• HP SOA Systinet Example

Performance Risks & ObjectivesPerformance Risks Analysis cont.

Requirement

Description Status Risk Detail

F01 New Repository API Implementing

High link

F02 Single EAR Depl. Done Low link

F03 J2EE Based Security Done Low link

F04, F102 Atom-based REST Implementing

Medium

link


• SDLC Phase: Requirements• Pre-requisites

− Performance Risks Analysis− Performance Requirements

• Goals − Define measurable performance objectives that are aligned with customer needs and agreed

upon all stakeholders

• Two approaches− Absolute goals

• Analyze customer environments (Workload Characterization)• Determine the right workload for the testing environment.

− Workload consists of: number of users, concurrent active users, data volumes, key scenarios and transaction volumes

• Define objectives for: Response Time, Throughput, Resource Utilization• Suitable for new projects

− Relative goals (based on historical data)• Group existing operations to sets:

− Based on response time: fast, medium, slow, asynchronous− Based on resource utilization

• Then assign each operation desired group• The main advantage is that we need not deal with target environment, which is sometimes

impossible.

Performance Risks & ObjectivesPerformance Objectives


• Absolute objectives example from BAC 7.5 (just in terms of TRT)

Note: These numbers usually expressed in 90% quantile.

• Relative objectives example from HP SOA Systinet 2 (just in terms of TRT)

− Old operations

− New or updated operations

Software Performance EngineeringPerformance Objectives (cont.)

Module Business Process Operation OK [s] Minor [s]

Critical [s]

MyBAC Brows_My_BAC Select_Page 20 40 60

Maximize_Portlet 6 16 30

Maximize_Portlet_Back 6 16 30

Module Business Process Operation TRT

All Login_Logout Login Medium

Logut Medium

Policy Manager Browse_BP Click_Search Slow

Module Business Process Operation TRT

All Fultext_search Simple_Fultext_Search Slow


• SDLC Phase: Design• Pre-requisites

− Performance Objectives− Performance Risk Analysis

• Goals− Ensure application key design choices takes performance principles under consideration

• Practices− All System Architects, Software Designers and Performance Engineers participate− System Architects and Software Designers takes performance into account when creating designs− Performance Engineer reviews their work and participate on architecture meeting− Prototyping

Architecture, Design Guidelines & Code ReviewPerformance Design Guidelines


• SDLC Phase: Develop• Pre-requisites

− Performance Objectives− Performance Risk Analysis

• Goals− Ensure that code is written so that it uses resources effectively.

• Practices− Usually done when some sensible unit of code is finished.− Done by Performance Engineer in co-operation with Software Designer .− Output – set of defects

Architecture, Design Guidelines & Code ReviewCode Review


• SDLC Phase: Design, Develop

• Goals− Educate development about performance aspects of technologies used.

− Force development to think about things important for performance.

− Learn development from their own mistakes.

− Get feedback form development.

• Practices− Bi-weekly meetings

• Frameworks and libraries used• Code Review, Testing etc. results• Discussion

Architecture, Design Guidelines & Code ReviewDevelopment Education


• SDLC Phase: Design, Develop, Test• Pre-requisites

− Performance Objectives, Architecture Design, Performance Risk Analysis, Workload Analysis

• Goals − Characterize and understand the system’s behavior so that effective decisions may be made.− Orchestrate design, develop and test phases to ensure that performance goals are met.

• Process1. Identify key scenarios2. Identify workloads3. Identify budget

• Budget are your constraints – execution time, resource utilization• For example peak CPU utilization 75% and memory consumption 4GB

4. Identify performance objectives5. Identify processing steps

• Itemize your scenarios6. Allocate budget

• Assign execution times to steps− If you don’t know how, divide it equally. Later on you will correct your assignment.

• Assign resource utilization7. Evaluate – modify your design, reevaluate requirements, change budged allocation8. Validate – prototypes and performance measuring9. Repeat 4. 5. 6. until goals are met.



• SDLC Phase: Develop, Test

• Goals − Identify how well application performs in relation to performance objectives

• Key Types of Performance Testing− Performance Test

• Load Test− Behavior under normal and peak load.− Endurance Test (Longevity Test) – MTBF and MTTF

• Stress Test− Behavior when pushed beyond normal or peak load− Synchronization issues, race conditions and memory leaks− Spike Test – behavior under repeated short peaks

• Capacity Test− To plan future system growth.− To identify scaling strategy.

• Additional Concepts− Unit Test− Component Test− Smoke Test

Measuring, Testing & TuningPerformance Testing


• SDLC Phase: Test, Deploy, Maintain

• Goals − Identify how the application performance stands in relation to defined performance

goals − Helps to identify the bottlenecks that affect application performance

• Practices− Measure System and Platform metrics− Measure Network− Profiling− Analyze log files− Instrument the application

Measuring, Testing & TuningPerformance Measuring


• SDLC Phase: Develop, Test, Deploy, Maintain

• Goals − identification and systematic elimination of bottlenecks to improve performance

• Inputs− Good product and environment knowledge.

• Categories for Tuning− Application. Includes the application configuration parameters.

− Platform. Includes the host operating system, together with Web and database servers

− System. Includes the hardware resources on a server. These resources include CPU, memory, disk I/O, and network I/O.

• Process1. Run test and measure relevant metrics2. Result analysis (JVM behavior, thread pools, connection pools, DB wait events etc.)3. Propose parameter modification based on results4. Modify test environment accordingly5. Run test again6. Compare results (iterate)

Measuring, Testing & TuningPerformance Tuning


• SDLC Phase: Test, Deploy

• Goals − Assess the relative performance of an application in particular environment.− Equip field personnel with valuable information to facilitate product sales

• Details− Base for customer environment validation.− Characterize HW requirements− Proving capabilities in comparing to competitors− Promoting sales

• PCoE teams provide benchmarking kits for their customers.

Deployment GuidelinesPerformance Benchmarking


• SDLC Phase: Test, Deploy, Maintain

• Goals− Planning for growth and forecasting peak usage periods in order to meet system

and application capacity requirements

• Practices− Transaction cost analysis

• Compile a user profile• Execute discrete tests• Measure the cost of each operation• Calculate the cost of an average user profile• Calculate site capacity• Verify site capacity

−Predictive analysis• Collect performance data • Query the existing historical data• Analyze the historical performance data• Predict the future requirements

Deployment GuidelinesCapacity Planning


• HP LoadRunner• HP Analysis• HP Diagnostics• YourKit• Idea 7.0• Oracle AWR reports• Oracle Remote Diagnostic Agent• Our own tools

Software Performance EngineeringTools


Software Performance EngineeringQ&A

Documents

Software Performance Engineering