40A Sevcik APM and Apdex · Peter Sevcik NetForecast, Inc. 955 Emerson Drive Charlottesville, VA 22901 [email protected] 434 249 …

Session 40A Application Performance and How Apdex Makes it Better

Application Performance and How Apdex Makes it Better

CMG International ConferenceSan Diego, CaliforniaDecember 5, 2007

Peter SevcikNetForecast, Inc.955 Emerson DriveCharlottesville, VA 22901

[email protected] 249 1310

Session 40A

©2007, Apdex Alliance, Inc. All rights reserved. Slide 2

Apdex Symposium

Apdex Case Studies45A4:00-5:00

Setting Performance Objectives Using Apdex44A2:45-3:45

Measurement Tools and Reports43A1:15-2:15

Apdex Process41A9:15-10:15

Application Performance and How Apdex Makes it Better40A8:00-9:00

We thank the Contributing Members for their financial support of the Alliance



Outline

BSM = TSM + APMApdex

© 1990-2007 Peter Sevcik and NetForecast, Inc., All rights reserved.


Application Delivery Complexity

WANsAccess

Firewalls& IDS

LANSwitches& Routers

LoadBalancing

System

Servers VirtualMachines

ResourcePools

app

OS

OS

OS

Applications

Technology Silos

Single User Traffic Flow Flow Processing Point

What Matters: Task = User hits “Enter” to system “Responds”What Matters: Task = User hits “Enter” to system “Responds”



Many Columns

The datacenter now houses:Access system (firewalls, intrusion detection)LAN (many layers of Gigabit Ethernet switches)Load balancers (server directors, TCP shaping, SSL offload)Web serversApplication serversDatabase serversSAN (storage area network)Storage

However, many of the servers are in fact operating within virtual machines. So there is the added complexity of:

Virtual machinesVM resource pools

Let us not forget the:Client (any kind of machine running any kind of browser)Network (private or Internet)


Many Rows

Tasks: Users interact with an application one task at a timeFrom the Apdex Specification:Task time is measured from the moment the user enters an application query, command, function, etc., that requires a server response to the moment the user receives the response such that they can proceed with the application. Often called the “user wait time” or “application response time”.

Turns: Tasks comprise many turns and a lot of payloadEach application client-server software interaction needed to generate a user response or task. These software-level client-server interactions add to the time it takes for the software to complete a task. A turn is a client-server request-driven round-trip. Often called application “chattiness”.Web-based B-B application turns have grown

– 1995: 20 turns per task– 2007: 88 turns per task– Tasks are on a 13% compound annual growth rate—with no end in sight



Application Profiles Sample

1

10

100

1,000

Payload per Task (Bytes)

App

licat

ion

Turn

s pe

r Tas

k (c

ount

)

1,000 10K 100K 1M 10M

CIFS

SAPGUI

SAPWeb

MAPI

WEB2002

WEB2007


How Applications Are Delivered

TechnologiesResourcesSilosAssetsColumns

Many ColumnsA typical user may touch 12 subsystems to execute a task

FlowsTrafficTurns (app round trips)User ExperienceRows

Many RowsA typical task may require 10-100 turns to execute a task



Visualizing How an Application Works for One User(Rows and Columns)

12 Technologies

30 T

urns

Each column representsthe device used withineach technology

Each row representsa turn within eachtask within the flow


Two Views of Performance

12 Technologies

30 T

urns

PerformanceDelivered Note:

Each taskrequires10 turns,or rows

Every column deliveredthe same performance

This task hadsignificantlypoorer performancewith 4-times thenumber of reds

TaskA

TaskB

TaskC



Real World Examples

Commercial flights have strict flight rules. The FAA tracks every plane throughout the flight.

Private pilots fly in and out of general aviation terminals that are well maintained. They do not have to file a flight plan.

Airline Service

The card company keeps track of your buying habits. It calls you when there is a suspicious charge and cancels the charge.

The credit card company checks your credit, processes debits and payments, keeps a strict set of rules on how you must pay.

Credit Card Service

FedEx provides on-line 24-hour detailed tracking of your package from door-to-door.

The postal service keeps their offices open and their carriers are punctual. You post a parcel and it arrives at the other end.

Package Delivery

The lights are synchronized to maximize traffic flow and each car gets through the city faster.

Each light is operating, no roads are congested, all must be well.

City Traffic Lights

RowColumn


Service Delivery Frameworks

ITIL and ITSM (Columns)The IT Infrastructure Library® (ITIL) describes the organisation of IT resources to deliver business value, and documents processes, functions and roles in IT Service Management (ITSM)ITSM (IT Service Management) is focused on managing technologiesand software not user flows

Application Performance Management – APM (Rows) is missing in ITIL and ITSM

APM is the art and science of making applications run well by linking application performance goals to business objectivesApplied during ongoing operations of a business applicationAPM provides process and tools to manage user flowsAPM is required for an end-user experience SLA



ITIL Processes Related to APM

Service OperationEvent ManagementIncident ManagementRequest FulfillmentProblem ManagementAccess ManagementOperational Activities of Processes …– Change Management– Configuration Management– Release and Deployment Management– Capacity Management– Continuity Management– Availability Management– Knowledge Management– Financial Management for IT Services

Continual Service ImprovementService Level Management

Minimal set of processesthat relate to ongoingAPM delivery quality

ITIL v3 published in May 2007comprises 5 volumes:

Service StrategyService DesignService TransitionService OperationContinual Service Improvement


ITIL Service Delivery Applied to Flows

Incident ManagementLoss of user access, loss of service within a geographic region, slow performance, software incompatibility (client and server failing to communicate), missing cookies, can’t acquire address or credentials, etc.

Availability ManagementAll authorized user access methods (wireline, cell service, WiFi) are working, client-server connections can be made, all devices on a flow path are operating, SSL keys are installed and certified, alternate routing, etc.

Capacity ManagementSufficient bandwidth for each flow, QoS and precedence handling, load balancing, traffic control, sufficient TCP connection pools, latency within application needs, proper application acceleration techniques applied, etc.

Service Level ManagementFlow characteristics are known and supported, user response time supports the business function, voice services meet quality standards, videoconferencing supports business function, etc.



Different Reports on the Same Infrastructure

Availability

Incident

Service LevelService Level

Availability

Incident

Service Level

Availability

CapacityCapacityCapacity

Incident

Per technology, device,subsystem, system

Per flow, application,user group, location

CoreDevices

AndSystems

TechnologyService Management

ApplicationPerformance Management

Availability

Incident

Service LevelService Level

Availability

Incident

Service Level

Availability

CapacityCapacityCapacity

Incident


Availability

The Views Diverge With Sophistication

TechnologyService Management

ApplicationPerformance Management

Incident

Capacity

Srvc Level

Incident

Availability

Capacity

Srvc Level

CommonAttributes

UserCentric

AssetCentric



Diverging Granularity

CommonAttributes

UserCentric

AssetCentric

Utilization ofeach Device

Actual load

Predictedaggregate load

Utilization ofeach subsystem

(memory, disk, etc)

Actual traffic

Source-Destinationflows by location

Individualuser flows

(resp time, VoIP quality, etc)

Mor

e G

ranu

larit

yDetail helps

capacity planningand ROI

Detail helpslink to business

and SLA


Stages of Maturity

Reactive DiagnosisDiagnostic tools, problem triage, fault identification

Proactive InterventionOngoing measurement, trend analysis, resource planning

Quality WarrantyService objectives are stated

Portfolio ManagementManaging many business applications as a group



APM Value Framework

Reactive Diagnosis

Proactive Intervention

QualityWarranty

Portfolio Management

Service LevelCapacityAvailabilityIncident


BSM – Business Service Management

APMApplication Performance Management

Flows

TSMTechnology Service Management

Technologies

Service LevelCapacityAvail-

abilityIncident

Reactive Diagnosis


QualityWarranty



abilityIncident



Stereoscopic View

Stereoscopic views of ITITSM assumes that if service is available, the user experience is satisfactoryAPM assumes that if response time is satisfactory, the service is availableEach image is convincing but each is incomplete

Good BSM needs bothITSM is a necessary but insufficient service modelAPM can’t exist without a good TSM foundation

The 3-D effect adds clarity and new information not

seen in each image alone


BSM Framework Supported

APMApplication Performance Management

Flows

TSMTechnology Service Management

Technologies


abilityIncident

Reactive Diagnosis


QualityWarranty



abilityIncident

ApdexApplies

AppSLA

More Valu

e

More Valu

e

Should be balanced



Outline

BSM = TSM + APMApdex

© 1990-2007 Peter Sevcik and NetForecast, Inc., All rights reserved.


Today’s Problem:Many Numbers, Little Insight

Measured Response Time (seconds)

App A App B App C App D App E

Day Average 6.0 12.5 3.1 8.4 2.0

BestHour 5.0 6.8 2.8 4.1 1.7

Worst Hour 18.6 18.9 8.6 19.3 6.5

95th

Percentile 8.1 17.3 10.7 12.9 9.5

Which application is in trouble?



Example: 100 Numbers

Start with what you haveYour measurement tool produced 100 samples

The samples areSingle applicationUser-level response time measurementsOne hour period of observation

Is the application operating well?18.1022.5021.3319.69

5.202.652.241.80

1.873.463.343.28

14.831.266.0215.00

1.777.572.205.85

3.857.986.382.38

2.772.491.367.50

4.115.331.742.76

1.464.531.7630.54

30.082.952.703.74

16.373.832.326.09

3.223.691.992.02

3.424.644.903.87

2.4815.352.131.64

2.352.203.675.60

10.462.151.461.61

23.566.216.397.38

4.294.903.542.78

3.993.283.787.37

2.943.738.8314.55

11.9813.204.752.99

3.849.768.4412.56

4.752.224.5016.67

2.513.3313.2559.55

54.503.3616.896.45

1


Numbers Beget Numbers

0

5

10

15

20

25

30

35

40

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60

Incremental Time Period (sec)

Num

ber o

f Sam

ples

in T

ime

Perio

d

secAverage 7.6Median 3.9Mode 4.8Standard Deviation 9.495th Percentile 22.5Minimum 1.3Maximum 59.6

Now you have 137 numbers.Can you answer the question,“Is the application operating well?”

2



Apdex Defined

Apdex is a numerical measure of user satisfaction with the performance of enterprise applicationsIt defines a method that converts many measurements into one number

Uniform 0-1 scale, 0 = no users satisfied, 1 = all users satisfied

Standardized methodIt is a comparable metric across all applications, andAcross enterprises


Deconstructing Application Transactions

Session = Period of time that a user is “connected” to an application

Start theapplication

End or suspendthe application

Task = Each interaction with the application during the session

Type orchoose

Enteror click

Systemresponds

Userwaits

User readsor thinks Type Wait Read

Enteror click

Systemresponds

Process = A group of user interactions that accomplish a goal

Get new email, add an employee, check on inventory status, etc. Idle



Deconstructing Application Transactions (con’t)

Turn = Each application client-and-server software interaction needed to generate a system response

Protocol = Each TCP Open, ACK, retransmission, etc, required to operate a Turn and move Payload

Packet = Each packet as seen on the wire in support of the above

Wait


The Task Defined

Task response time is the elapsed time required for an application system to respond to a human user input such that the user can effectively proceed with the process they are trying to accomplish

Time when the user is waiting in order to proceedUser feels the responsiveness of the applicationLong Task time makes the user less productive

The Task is what a user cantime with a stopwatch



How Users View Application Task Performance

SatisfiedUser maintains concentrationPerformance is not a factor in the user experienceTime limit threshold is unknowingly set by users and is consistent

ToleratingConcentration is impairedPerformance is now a factor in the user experienceUser will notice how long it is taking

FrustratedPerformance is typically called unacceptableCasual user may abandon the processProduction user is very likely to stop working


0%

2%

4%

6%

8%

10%

12%

14%

16%

0 20 40 60 80 100 120Load Time of a Typical Business Page (sec)

Prob

abili

ty o

f Exp

erie

ncin

g th

e Ti

me

Example

52% Satisfied

42% Tolerating

6% Frustrated



How Apdex Works

Start with a sufficient number of Task measurement samplesTarget response time “T” defines the satisfied zone (0-T sec)

T is shown as a subscript of all Apdex values (for example 0.80T )

Count the number of samples within three performance zonesSatisfied, Tolerating, Frustrated

Tolerating count

ApdexT = Total samples

Satisfied count 2+

GivenTarget response time T andSufficient response time measurement samples

Then

NoteFrustrated samples are not in numeratorbut are counted in total samples

Index0 = Failure; 1 = Perfection (all users satisfied)


Putting it All Together

Tolerating

ApdexT= Total samples

Satisfied 2+

Frustrated

Satisfied

Tolerating

Good

Fair

Poor

Una

ccep

tabl

e

0.00T

0.50T

1.00T

0.85T

0.94T

0.70T

ExcellentReport Group:ApplicationUser GroupTime Period

Existing TaskResponse TimeMeasurement

Samples

T1

2

3

45

6F

1. Define T for the applicationT = the application target time (threshold between satisfied and tolerating users).F = threshold between tolerating and frustrated users is calculated (F = 4T).

2. Define a Report Group (details available are tool dependent).3. Extract data set from existing measurements for Report Group.4. Count the number of samples in three performance zones.5. Calculate the Apdex formula.6. Display Apdex result (T is always shown as part of the result).



The Apdex View of the 100 Numbers

User productivity is impaired if the application responds in more than 8 seconds

T = 8 sec

Apdex for the 100 measurements = 0.878

The application barely providing “Good”performance

100 numbers = 0.878

4

Good

Fair

Poor

Una

ccep

tabl

e

0.00T

0.50T

1.00T

0.85T

0.94T

0.70T

Excellent


0%

2%

4%

6%

8%

10%

12%

14%

16%

0 20 40 60 80 100 120Load Time of a Typical Business Page (sec)

Prob

abili

ty o

f Exp

erie

ncin

g th

e Ti

me

Apdex Highlights the Long-Tail

52% Satisfied

42% Tolerating

6% Frustrated

Major eCommerce site ($4B annual on-line sales)North American broadband users accessing the San Francisco data center

This site had an average Keynote responsetime of 4 seconds, so it looked like all was wellBut: Apdex = 0.7310 = Fair



Apdex Benefits

ProcessForces a process by which the enterprise becomes focused on the important performance management issues

SimplicityConverts mountains of existing response time measurements into asimple value that can be easily understood by non-technical managers

Business LinkageOffers a clear picture of how well the IT infrastructure is really performing in support of specific business objectives

Open StandardProcesses and results that can be applied across industries and applications


Getting Started with Apdex

Start measuring and reportingPick any application and use any tool you can– 17+ vendors can supply the data necessary to calculate Apdex reports– Currently 5 vendors generate Apdex reports within their product

Getting some data is better than no dataIt serves as a foundation for planning and budgeting

Full benefit of Apdex requires a processInvolve key stakeholdersFormal dialog on Apdex termsBenchmark your APM capabilities before and after an Apdex pilotPlan for application SLAs based on ApdexSet the stage for continual APM quality improvement



Apdex Methodology

Gather

• Measure• Gather baseline data• Test Apdex Parameters

A T

1) Work withusers to set T

2) Work with businessmanagers to define theservice objective for A

Agree Improve

• Apdex reports & trends• Improve performance

where needed• Continual performance

& process improvement


The Apdex Alliance

Apdex Alliance is a non-profit industry allianceOpen collaborative approachwww.apdex.orgContributing Member – Annual dues

Supporting Member – Free– Individual interested in applying Apdex within their organization– Currently more than 700 members

Apdex is FREEYou can use Apdex in your organization or a commercial productYou or your organization do not need to join Apdex to use ApdexYou join to learn more and be plugged into the community



Leveraging the Apdex Alliance

Contributing MembersAre promoted as sponsors of the AllianceReceive support in implementing Apdex reporting

All Members (Contributing and Supporting)Receive a quarterly email newsletterHave access to formal documents and news on the web siteCan participate in the Apdex Exchange (Google Group) discussion board– All members can choose to join the discussion board– You can receive all postings, email summaries, or only see material

when you log in– There are more than 100 people in the Exchange

The numbers:7 Contributing, 700 Supporting, 100 in the Exchange


Apdex Alliance Member Survey

2007 Web-based survey70 responses

Number of employees15% less than 50037% 500 to 10,00048% more than 10,000

Manufacturing

IT Product Manufacturing

FinancialServices

Government

TransportationEducation & Research

Healthcare

Carriers& Utilities

Business & IT Services

Other



Who Should Lead the Apdex Alliance?

Vendors, 4%Enterprises, 11%

Balanced, 86%


Apdex In Use

No Plans, 26%

Likely in 1 Year, 29%Experimenting, 21%

Pilot Project, 9%

In Regular Use, 10%

Our StandardReport, 5%

45%UsingApdexNow

45%UsingApdexNow



The Big Picture

ManageAsset

Efficiency

TSMTSM

APMAPMDeliverUserValue

Thank You

Articles and reports on performance measurement, analysis, and management are available for free at www.netforecast.com

Information about Apdex and joining the Apdex Alliance is at www.apdex.org

Documents

40A Sevcik APM and Apdex · Peter Sevcik NetForecast, Inc. 955 Emerson Drive Charlottesville, VA 22901 [email protected] 434 249 …