View
228
Download
0
Category
Preview:
Citation preview
© 2012 Collaborative Consulting
Enterprise Software Performance Engineering
Presented By: Walter Kuketz
Scaling your career..
2 2012 Collaborative Consulting
Software Performance matters - everywhere
DTG
Facebook IPO
3 2012 Collaborative Consulting
Who is responsible for end-to-end system performance?
• When a website slows down?
• When the marketing team runs a big event?
• When the Storage subsystem slows down, which
slows down the database, which slows down the
application servers, and slows down the iPad?
Where do the performance goals
come from?
4 2012 Collaborative Consulting
Agenda – Building systems that scream…
Review the current challenges of being a performance engineer
Overview of the five knowledge areas of Software performance engineering body of knowledge
Roles and career path overview
Case study – 10X event (Communicate to Sr. Mgt.)
5 2012 Collaborative Consulting
No clear standards in the market for PE
Performance Engineer Performance tester
Capacity planner
Performance Architect
Test automation technician
Performance analyst
Performance Data engineer
Troubleshooting Benchmarks
?
?
?
?
?
?
? Software Engineer
Performance test
6 2012 Collaborative Consulting
Opportunity to take SPE to the next level..
Define the profession of performance
engineering Define profession
Establish a body of knowledge for
performance engineering
Establish Body of
Knowledge
Establish a common language for roles,
responsibilities, and practice areas using
the form of Knowledge Areas
Establish
common
language
Define the competencies shared across
the knowledge areas
Define
competencies
Capture common techniques for sharing
across the industry, or your company
Capture common
techniques
7 2012 Collaborative Consulting
The SPE Body of Knowledge K
now
ledge a
reas
Planning, coordination, information sharing, and control
SDLC &
architecture
(SA)
Application
performance
management
(APM)
Problem
detection
and
resolution
(PDR)
Underlying competencies and roles for delivering
Techniques supporting the knowledge areas
Performance
validation and testing
(PVT)
Capacity planning
(CP)
For the Enterprise
8 2012 Collaborative Consulting
The Enterprise PE team: There must be a practice leader
Operations & PT
Architecture and
production
Business direction
Development teams
Communication
9 2012 Collaborative Consulting
The Enterprise PE team: There must be a practice leader
A shared service and clearing house for all performance, scalability and
capacity planning information. Transforms performance information to
business knowledge and advantage.
Communicates across the teams to share information
For instance, the production workload and the
performance test workload – are they in sync?
Make sure the architecture team knows how the
new release, or newest build behave in production.
The business risks are clearly identified and
communicated. Application risk profile.
Can provide performance guidelines to the
developers, must influence the unit testing process.
Operations & PT
Architecture and
production
Business direction
Development teams
10 2012 Collaborative Consulting
PE: Knowledge areas
PVT
Performance testing and
validation
CP
Capacity planning
APM
Application performance
management
PDR
Problem detection and
resolution
SA
Software development
lifecycle and architecture
11 2012 Collaborative Consulting
PE: Software development lifecycle and architecture
PVT
Performance testing and
validation
CP
Capacity planning
APM
Application performance
management
PDR
Problem detection and
resolution
SA
Software development
lifecycle and architecture
12 2012 Collaborative Consulting
PE: SA (software development lifecycle)
Define Design Develop Functional
test
Performance
test Deploy
Waterfall methodologies
Need to plan for
performance testing
Build code to performance
requirements
Define
Design
Build Test
Deploy Agile
methods
Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?
13 2012 Collaborative Consulting
Performance budgets
PROXY
Datacenter
WEB
MQ/ESB
DCOM
CORBA
SERVER
APP SVR
LDAP
Web
Services
Code runs
everywhere.
Respond to the user
in 2 seconds.
Do your developers know how much time they have for their piece of the transaction?
14 2012 Collaborative Consulting
SA
Software development
lifecycle and architecture
PE: Performance testing and validation
CP
Capacity planning
APM
Application performance
management
PDR
Problem detection and
resolution
PVT
Performance testing and
validation
15 2012 Collaborative Consulting
PE: PTV (performance testing and validation)
What is changing? Where does it need to be validated? What (all) is being measured?
The project: Why are we testing and what are the success criteria?
Prerequisites:
Business goals and non-functional requirements are defined
Workload and scenarios defined
Environment defined (where are we testing)
Application in a stable state
Configuration management and release management ready
Core performance team identified
Supporting team committed
16 2012 Collaborative Consulting
• Deep understanding of
the application being
tested
• Deep understanding of
the technical
architecture of the
application
• Understanding of the
business area
supported by the
applications
Business and
application area
knowledge
• Performance engineers
involved in design
sessions
• Understand the nature
of the data required for
testing
• Able to understand
technical gaps to meet
requirements not
supported by the tool
Early involvement
with the applications
team
• Well-understood
workload model and
characterization
• All components in the
test are monitored and
reported on
• A defined triage process
for troubleshooting
• A process to manage
the inevitable
unplanned tests and
test archive exists
Test execution
• Testing environment is
a close approximation
of production, including
database size
• Differences between
test and production are
clearly known
• Release and
configuration
management in place
and procedures
enforced
Environment
Organization: Performance team value
You still need to explain to the rest of the organization what you do
Performance team value
17 2012 Collaborative Consulting
SA
Software development
lifecycle and architecture
PE: Capacity planning
PVT
Performance testing and
validation
APM
Application performance
management
PDR
Problem detection and
resolution
CP
Capacity planning
18 2012 Collaborative Consulting
PE: CP (capacity planning)
The right system at the right price!
Workloads drive system transactions, which drive resource utilization
Determine
capacity
requirements
Analyze
current
capacity
Plan for
future • Forecast workloads
• Plan for usage
• Track utilization of
existing systems
• Track workload
• Define system to
support workloads
• Test
• Define service
levels
19 2012 Collaborative Consulting
Capacity: Use of resources
PROXY
Datacenter
WEB
MQ/ESB
DCOM
CORBA
SERVER
APP SVR
LDAP
Web
Services
The overall system has a capacity; each component has a capacity.
20 2012 Collaborative Consulting
SA
Software development
lifecycle and architecture
PE: Application performance management
PVT
Performance testing and
validation
CP
Capacity planning
PDR
Problem detection and
resolution
APM
Application performance
management
21 2012 Collaborative Consulting
Five elements of APM (Gartner)
Monitoring the performance of complex distributed applications
End user experience measurement – browser/mobile
device
Create a model of the run-time environment (discovery)
Profile the performance and behavior of user-defined
transactions
Performance metrics from each of the
applications/systems technical components (Webserver,
App server, Database, etc.)
Application performance management database
1
2
3
4
5
22 2012 Collaborative Consulting
Mobile
Track key user experience metrics
Network latency
Browser rendering time
Total end-user response time
Renderin
g
Mobile Real User Monitoring
23 2012 Collaborative Consulting
And see every hop of the transaction
Mobile
App server
Web server
Proxy
server
Database
Mainframe
Data warehouse gateway
Dynamic Transaction
Path Detection
Renderin
g
24 2012 Collaborative Consulting
SA
Software development
lifecycle and architecture
PE: Problem detection and resolution
PVT
Performance testing and
validation
CP
Capacity planning
APM
Application performance
management
PDR
Problem detection and
resolution
25 2012 Collaborative Consulting
When problems occur
Flash web site events (all at once)
New code released causes problems
Infrastructure upgrade
Consolidation of applications and servers
New workload model (batch and online)
26 2012 Collaborative Consulting
Cause and Effect
27 2012 Collaborative Consulting
SPE Competencies
Career path
28 2012 Collaborative Consulting
PE roles: The team – many possible roles
Managing and developing the core team members
Technical Project manager Performance Architect
Sr. Performance Engineer
Performance Engineer
Sr. Test automation technician
Test automation technician
Performance data engineer
Sr. Capacity planner
Capacity planner Workload and capacity
Modeler
29 2012 Collaborative Consulting
Underlying competencies
Analytical
thinking
problem solver
Application and
tool knowledge
Workload
model
development
Statistics
Test data
management
Project
management
Communication
skills
Behavioral
characteristics
Test execution
process Forecasting
Business
Knowledge
Monitoring and
tuning
Underlying
competencies
30 2012 Collaborative Consulting
Technical competencies
Technical
competencies
Code
profiling
tools (Static analysis,
diagnostics)
Database
monitoring
tools
End to end
performance
tools
Deep dive
tools (HP Diagnostics,
CA Wily, etc,
Jprobe, etc)
Test design
and
execution
tools
Comprehensive
results collection
and reporting
Resource
utilization
system
level
Queuing
and
modeling
tools
Statistics and
Distributions
Test data
creation
DB tuning
SQL tuning
31 2012 Collaborative Consulting
PE Techniques
Brainstorming Data modeling
Creating a workload model Performance testing
Triage approach Static code analysis
System metrics collection Logging and
instrumentation
32 2012 Collaborative Consulting
Career path: Performance engineer foundation
Grow your breadth of capabilities
Grow your
leadership and
communication
Performance
Engineer
Performance
testing
Lead
technical
role
Workload
models
Communication
Production
triage
Technical
architecture
Additional
tech
competency
Core
technical
competency
33 2012 Collaborative Consulting
Upcoming Marketing event
34 2012 Collaborative Consulting
How do these Knowledge areas work together (10X)
Forecast the
workload for
the event
Measure the current
workload and
resource utilization
Plan and execute
PT for forecast
workload with
support for
troubleshooting
SA PVT
APM PDR
CP CP
Architecture and
Tune application
based on results
Responsibilities
35 2012 Collaborative Consulting
Questions?
Thank you!
wkuketz@collaborative.com
36 2012 Collaborative Consulting
• Development methodologies
Agile
Iterative
RUP
Relational Data models
Object oriented
• Implementation concepts
Swing
Hibernate
JMS
Web Services
Windows
RESTful services
SOAP
• Integrated Development Environments
Eclipse, Windows, SQL Developer
37 2012 Collaborative Consulting
• Operating systems
Windows Platform
Unix and all its varient
iOS
Z/OS
• Database platforms
Oracle
Microsoft
DB/2
• Application servers
Microsoft
Jboss
Oracle AS
Weblogic
Websphere
• Application frameworks
.NET
J2EE
Spring
Struts
• Programming languages
Java
C#
XML
C/C++
COBOL
Ruby/
• Reporting technologies
• Integrations technologies
• Software managements technologies
38 2012 Collaborative Consulting
CP: Workload model
Step Transaction – by product Percentage
1 Login 100
2 Search product 100
3 View product detail 100
4 Add to shopping cart 50
5 Confirm purchase method 50
6 Shipping information 50
7 Send confirmation email 50
CP
The CP team uses this information to forecast the
new workload 10X.
39 2012 Collaborative Consulting
Review current Production workload, and the under lying system
utilization.
• Average workload
• Current peak workload
CP: Analyze current capacity
System utilization 100%
Response
time
Service level
CP
40 2012 Collaborative Consulting
• Use the workload model from CP
• Enable APM during testing
• Work with the Application Architect
PVT: planning, executing, and analysis
Methodology is iterative and tactical
Review infrastructure
& architecture
Define business
activity profiles &
service levels
• Identify risk areas
• Review configuration
settings, topology, & sizing
• Define points of
measurement
• Types & numbers of
users
• Business activities &
frequencies
Iterate testing & tuning
• Test data generation
• Create test scripts
• User & transaction profiles
• Infrastructure configuration
Design & build tests
PVT
Recommended