Performance Engineering and Proven PracticesAugust 9, 2016

Brought to you by

Vivit Performance Engineering SIG

Leaders: Todd DeCapua, Petar Puskarich,

Paul Shovlin and Chris Trimper

http://bit.ly/VIVITPE

Hosted By

Petar Puskarich

Sr. Leader

Performance Engineering

Panelists

Brandon Teets

Lead Performance Engineer

Nationwide Insurance

Petar Puskarich

Sr Leader

Performance Engineering

• This “LIVE” session is being recorded

Recordings are available to all Vivit members

• Session Q&A:

Please type questions in the Questions Pane

Housekeeping

Webinar Control Panel

Toggle View Window between

Full screen/window mode.

Questions

Today’s Agenda

Join us for an informative presentation covering several best practices in the Performance Engineering field

Topics to be covered include:

• Staffing

• Tooling

• Visibility

• Organizational Structure

• Process Integration (Agile, Waterfall, Hybrid)

• Technical Considerations

We hope you may find some new ideas to implement with your team. The field of Performance Engineering has changed rapidly over the last few years due to technology and perception of the field by the business areas. The presentation today will cover several areas that have proven successful in this new era of engineering.

Staffing & Organizational Structure

Distributed CentralizedCenter of Excellence

(CoE )

PE

App 1

App 2

App 3

App 4

App 5

App 6

PEApp 1

App 2

App 3 App 4

App 5

App 6 PE

Solution Design

Reqs

Dev

FunctionalPerformance Testing

Deploy

Monitoring

PE

PE

PE

PEPE

PE

Distributed

Pros

• Dedicated PE to application/dev line (reserved allocations)

• More likely to be considered part of the team and overall SDLC

• Hands on PE cross training and poly-skilling may occur with the line

• Potentially become a SME for the dedicated application, issues/root cause diagnosis may occur quicker due to learned tribal knowledge

Cons

• Often only team member on the line that knows “effective” PE

• Tech stack barriers and boundaries for performance feedback (Cultural)

• Commonly switching between Functional and Performance testing, becomes difficult to separate those disciplines personally and within organization

• Frequencies of available work may vary depending on project/release cycles

Centralized

Pros

• PEs are grouped together, allocated per request

• Peer reviews, cross engineering, unique approaches and technical knowledge shared within team

• Team observations and support available rather than one individual PE dedicated to a dev line

• Transform into a “Service” model

• Exposure to ‘X’ amount of applications and solutions rather than dedicated to small subset

Cons

• Being requested outside of dev lines can result in poor project allocation

• Demand in work can drastically change both +/-

• Often at the end of the SDLC, very little time for effective PE

• Easy for the mindset to transition to quantity from quality due to backlog of work

• Working environment can feel Reactive vs. Proactive

Center of Excellence (CoE)

Pros

•Performance finally has an enterprise presence and can be consumed as a framework within all tech domains

•PEs can also become PE practitioners/evangelists (similar to agile/SCRUM masters for Agile practices)

•Each dev line is able to identify proper KPIs, SLAs, and User Acceptance criteria with end user experience always in mind

•“Performance Ambassadors” within each line can be established for maintaining the performance mindset with each line, these our SPOCs

•Embeds nicely into the DevOps adoption where we are handing “back” certain pieces of performance testing directly to the developers sooner

•Supports the shift-left movement allowing the much needed performance insight earlier in the lifecycle, especially one week dev sprints (1 month release = 4 agile sprints)

•Opens the opportunity to integrate with other professionals that heavily contribute to performance such as Ops (ITPA), Network, Modern Web/Mobile, Continuous Delivery (CI/CB), Monitoring (APM) = Performance Ecosystem

Cons

•Not heavily practiced, learning curves

•Cultural concerns and embracing a “new” wave in PE

•Enterprise constraints in supported technologies or tools that support this transition

•Not having a complete team of passionate engineers willing to step out of the load testing comfort zone and become PE Practitioners readily available for performance adoption and enterprise root cause analysis

•Difficult to develop the PE Readiness Model involving understanding all apps, enterprise SDLCs, and frameworks proving there is a ROI to CoE

•Effectively handing certain PE pieces to application teams and showing the value in doing so

•Not many known “C” level leaders throughout the industry dedicated to these cultural changes, the PE team and organization have to remain together to be successful in proving the ROI

Poll Question #1

• How is Performance Engineering structured within your Organization?• Distributed

• Centralized

• Center of Excellence

• Other

• Wish I knew.. Still trying to figure that out.

PE Process Integration

Waterfall Agile Hybrid

Performance Engineering Lifecycle

• Validating performance scripts

• Smoke Testing

• Global workload document

• AUT Hosts (Cluster & JVMs)

• Test(s) creation and modeling

• Execute performance tests

• Conduct Analysis

• Tuning & continuous feedback

• APM Trending

• Root cause/reporting

• Sign off and retrospective

• Business Process Navigations

• Transaction/NFR Mapping

• Scripting Business Processes

• SPOC

• Workloads

• Tests (Goals)

• Milestones

Requirements Scripting Phase

Test PrepTesting/Analysis

Consumability of Work

Performance Team - Waterfall

Requirements Design Development Testing Release

• Engaged and assigned per project from beginning to end

• Often falls under the “Dedicated” organization alignment model

• Remains SPOC for application/project team, no context switching

• Has a designated back up for project handoff situations

• Exposed to more “leader” and PM experience per application

• Gains valuable SME and tribal knowledge of the application seeing the application flow through the whole PE Lifecycle

Fail Often

Fail Fast

Performance Team - Agile

• Backlog, Iteration +1…, Current Iteration

• If work is Blocked and no follow update provided, cancellation process begins to set scope on removing blocker (avoid PE retainer)

• PEs based upon availability pick up new work, reach out to customer, NOT ASSIGNED

• Consumable work is prioritized and place in appropriate iteration columns to be picked up by PE

• Tracking team throughput, velocity driven commitments, weekly cadence meetings (Estimation, IPM & Retrospective) while maintaining 1 week sprints

• When work is blocked, follow up date is assigned and new consumable work is pulled into the iteration

Requirements

ScriptingTest Prep

Testing & Analysis

Request

Backlog

(6 weeks+)

Requirements

Requirements

Requirements

Requirements

Requirements

Requirements

Requirements

Iteration +1

Requirements

Scripting

Scripting

Test Prep

Analysis

Current Iteration

Requirements

Scripting

Test Prep

CoW

PEPE

PE PE

PEPE

Entr

y G

ate

Consumed

BLOCKED

Engagement Cancelled & Application Notified

48 Hours +Nearing Cancellation

24 Hours +Waiting for response

Current DayBlocked Email Sent

Application Team

No Date

Blocked Email

Hybrid Agile PE

Waterfall “Monthly” Release

No

Fo

llow

Up

Dat

e

Poll Question #2

• What methodology best describes your current team of PEs or an individual/group of PEs stacked for Performance Engineering?

• Waterfall

• Agile

• Hybrid

• Other

• No Idea….

Technical Considerations

Tools

Hybrid Infrastructure

(On/Off Premise)

Cloud Load Testing

HPE LoadRunner

Apache JMeter

HPE StormRunner Load

ORACLE Application Testing Suite

HPE Performance Center

Neotys NeoLoad

WebLOAD

Appvance

SOASTA CloudTest

OctoPerf

IBM Rational Test Workbench

Perfecto Mobile

HPE Mobile Center

BlazeMeter

TestPlant eggPlant Performance

SoftLogica WAPT

Original Software Qualify

SmarteSoft Testing Suite

MicroFocus Data Express

SmartBear LoadComplete

Enterprise Technologies

Time to Market

Business Value & ROI

Tools & Visibility

TERTRA

Hybrid Infrastructure

AUT 1AUT 2

AUT 3

AUT 4AUT

LG 1

LG 2

LG 3

On Premise

inTRAnet applications (private internal apps)

No artificial performance impacts (TRT Overhead)

from outside FW emulations (load

balancing, monitoring, pipe routing, etc.)

Easier culture adjustment for initial PT when environment is in

companies control

Infrastructure costs reduced up to 50%,

remaining infrastructure meant for TRA testing

Cloud/SaaS

inTERnet applications (external, .com),

Geographic Testing

Latest and greatest release for your tenant, fast

turnaround for upgrades

Hold vendor accountable for infrastructure support,

upgrades, SLOs/Rs/As

More likely for integrations, beta, design

partners with other industry technologies

ScalableEase of

Use

Geographic

Testing (NV)

Speed Integration

Cloud Load Testing

Integration - allows your agile testing and development teams to run performance tests as part of their builds in an easy, automated manner.

Continuous Integration (CI) servers such as Jenkins, Hudson, or TeamCity.

Ease of Use - enables anyone on your application delivery team can perform load tests—including developers, QA specialists, and project managers.

Speed - with the cloud-based resources you can quickly create and maintain tests to accelerate the velocity of your software development project.

Scalable - Don’t waste time and energy managing your testing infrastructure. It’s automatically created on demand to meet the needs of your specific test, saving your team time and money.

Geographic Testing - Emulate load scenarios distributed across different cloud regions located in multiple data centers. This empowers you to accurately test and optimize applications for many network performance conditions disbursed across the U.S., even globally.

Poll Question #3

• In regards to performance infrastructure, if known what is the current setup for your infrastructure responsible for performance testing?

• On Premise

• Cloud/SaaS

• On Premise & Cloud/SaaS

• Other

• Don’t know, just as long as my tests run!

More Information

HPE Software Performance Engineeringhttp://www.HPE.com/Software/PE

TechBeacon Performancehttp://techbeacon.com/performance

Blogs: LoadRunner and Performance Centerhttp://bit.ly/HPE-LRPCBlog

Vivit Worldwide: PE SIGhttp://bit.ly/VIVITPE

Questions & Thank you

http://bit.ly/VIVITPE

Vivit WorldwidePerformance Engineering

Special Interest Group