Upload
eric-proegler
View
475
Download
0
Embed Size (px)
Citation preview
Interpreting and Reporting Performance Test Results
ERIC PROEGLER
• From Mountain View, California• @ericproegler• Performance Engineer, (soasta.com)• 13 years in (back-end) performance, 20 in software• Speak Easy Mentor (speakeas.ie)• AST Board (associationforsoftwaretesting.org)• WOPR Organizer (performance-workshop.org).
…Think “CAVIAR”C ollectingA ggregating V isualizing I nterpretingA ssessingR eporting
For Actionable Results…
Collecting• Objective: Gather all results from test that• help gain confidence in results validity• Portray system scalability, throughput & capacity
• provide bottleneck / resource limit diagnostics
• help formulate hypotheses
Aggregating• Objective: Summarize measurements using
• Various sized time-buckets to provide tree & forest views
• Consistent time-buckets to enable accurate correlation
• Meaningful statistics: scatter, min-max range, percentiles
• Multiple metrics to confirm (or invalidate) hypotheses
Visualizing• Objective: “forest view” of metrics
relative to load• Turn barrels of numbers into a few pictures• Vary graph scale & summarization granularity to
expose hidden facts• ID load point where degradation begins• ID system tier(s) where bottlenecks appear, limiting
resources
Visualizing• Errors over load (“results valid?”)• Bandwidth throughput over load (“system
bottleneck?”)• Response time over load (“how does system scale?”)
• Business process end-to-end• Page level (min-avg-max-SD-90th percentile)
• System resources (“how’s the infrastructure capacity?”)• Server cpu over load• JVM heap memory/GC• DB lock contention, I/O Latency
Interpreting• Objective: Draw conclusions from
observations, hypotheses• Make objective, quantitative observations from
graphs / data • Correlate / triangulate graphs / data• Develop hypotheses from correlated observations • Test hypotheses and achieve consensus among tech
teams• Turn validated hypotheses into conclusions
Interpreting• Observations:
• “I observe that…”; no evaluation at this point!• Correlations:
• “Comparing graph A to graph B…” – relate observations• Hypotheses:
• “It appears as though…” – test with extended team; corroborate w/other information (anecdotal observations, manual tests)
• Conclusions:• “From observations a, b, c, corroborated by d, I conclude
that…”
Monitor resources liberally, provide (and annotate!) graphs selectively: which resources tell the main story?
Assessing• Objective: Turn conclusions into
recommendations• Tie conclusions to test objectives – were objectives
met?• Determine remediation options at appropriate level –
business, middleware, application, infrastructure, network
• Perform agreed-to remediation, Re-test
Assessing• Recommendations:
• Should be specific and actionable at a business or technical level
• Should be reviewed (and if possible, supported) by the teams that need to perform the actions
• Should quantify benefit, if possible cost, and risk
• Final outcome is management’s judgment, not yours
Step 1: *DO NOT* press “Print” of tool’s default Report
•Who is your audience? •Why do they want to see 50 graphs and 20 tables? What will they be able to see?
•Data + Analysis = INFORMATION
Reporting
•Step 2: Understand What is Important•What did you learn? Study your results, look for correlations.
•What are the 3 things you need to convey?•What info is needed to support these 3 things?•Discuss findings with technical team members: “What does this look like to you?”
Reporting
•Step 3: So, What is Important?•Prepare three paragraph email summary
•Prepare a 30 second Elevator Summary for when someone asks you about the testing
•More consume these than any test report
•Get feedback
Reporting
•Step 4: Preparing Final Report•Your primary audience is usually executive sponsors and the business.
•Write the Summary for them:• Language, Acronyms, and Jargon • Level of Detail• Correlation to business objectives
Reporting
•Step 5: Audience (cont.)•Rich Technical Detail within:
•Observations, including selected graphs• Include Feedback from Technical Team•Conclusions •Recommendations
Reporting
Reporting•Step 6: Present!
• Remember, no one is going to read the report.• Gather audience: executive, business, and technical.
• Present your results: Help shape the narrative. Explain the risks. Earn your keep.
• Call to action! Recommend solutions
…Think “CAVIAR”C ollectingA ggregating V isualizing I nterpretingA ssessingR eporting
For Actionable Results…