AppTransaction Xpert

8/10/2019 AppTransaction Xpert

http://slidepdf.com/reader/full/apptransaction-xpert 1/171

Introduction to ACE Analyst

Lecture Manual



NOTICES

Product Information

Lecture Manual for: ACE Analyst

Software Release: 16.0.T PL0

Technical Support

Telephone: 240.497.1200

Facsimile: 240.497.1064

E-mail: [email protected]

Web: http://www.opnet.com

OPNET Technologies, Inc. (“OPNET”) is the sole owner

of all proprietary rights in this Lab Manual (the “LabManual”) and the OPNET IT Guru software (the“OPNET Software”).

Terms and Conditions of Use

Access to and use of the Training Manual and/or theOPNET Software is restricted to authorized users whoare employees or direct contractors of a Licensee whohas accepted the terms and conditions of OPNET’s

Software Master License Agreement. As a condition ofhaving access to or use of the Lab Manual and/or the

OPNET Software all users agree to be subject the termsof OPNET’s Software Master License Agreement. Users

shall not copy or reproduce, in whole or in part, or

permit access to or disclose the Lab Manual or the

OPNET Software to any other person without the

prior written consent of OPNET.

Liability Statement

The terms and conditions governing the licensing ofOPNET Software consist of those set forth in the written

contracts and license agreements between OPNET and itscustomers. No representations in this Lab Manual orstatements of capability, suitability for use, accuracy or

performance, shall be a warranty by OPNET unless

contained in a written agreement signed by OPNET.

OPNET DISCLAIMS ALL EXPRESS OR IMPLIED

WARRANTIES AS TO THE LAB MANUAL ANDTHE OPNET SOFTWARE INCLUDING IMPLIEDWARRANTIES OF MERCHANTABILITY ANDFITNESS FOR A PARTICULAR PURPOSE.

Destination Control Statement

This Lab Manual and the OPNET Software weremanufactured in the United States by OPNET. The initialexport of this Lab Manual and the OPNET Software, andany subsequent relocation to another country shall complywith United States law relating to the export of technical

data, equipment, software and know-how.

Restricted Rights Legend

USE BY AGENCIES OF THE UNITED STATES

GOVERNMENT: For purposes of use of this LabManual and OPNET Software by agencies of the UnitedStates government (or those authorized to use of this LabManual and OPNET Software on behalf of an agency),the OPNET Software is “Commercial Computer

Software” within the meaning of DFARs 227.7202 and252.227-7014(a)(1) and FARs 52.227-19. Use,reproduction and disclosure is subject to DFAR and FAR

provisions relating to Commercial Computer Software

and the License Agreement between OPNET and theagencies using OPNET Software. OPNET software is

provided with a RESTRICTED RIGHTS NOTICE forgovernment use.

Trademarks

OPNET, OPNET Technologies, Inc., OPNET Modeler, ITGuru, OPNET Development Kit, NetBiz and IT Guru aretrademarks of OPNET Technologies, Inc. All other

trademarks used in the Lab Manual are the property of their

respective owners and used herein for identification purposes only.

CONFIDENTIAL INFORMATION

DO NOT DISCLOSE, FORWARD, DISTRIBUTE,

SHARE, OR MAKE COPIES OF THIS DOCUMENT INWHOLE OR IN PART. This document containsconfidential information and may contain information thatis proprietary, privileged, and/or exempt from disclosure

under applicable law. This document is intended for theexclusive use of the person to whom it is disclosed. If youare an unauthorized person, you are hereby notified thatany viewing, copying, disclosure or distribution of this

information may be subject to legal action. Allunauthorized persons must immediately destroy the originaldocumentation without making any copies or furtherunauthorized disclosure.



Introduction to ACE Analyst

Lecture Manual



NOTICES

Product Information

Lecture Manual for: ACE Analyst

Software Release: 16.0.T PL0

Technical Support

Telephone: 240.497.1200

Facsimile: 240.497.1064

E-mail: [email protected]

Web: http://www.opnet.com

OPNET Technologies, Inc. (“OPNET”) is the sole owner

of all proprietary rights in this Lab Manual (the “LabManual”) and the OPNET IT Guru software (the“OPNET Software”).

Terms and Conditions of Use

Access to and use of the Training Manual and/or theOPNET Software is restricted to authorized users whoare employees or direct contractors of a Licensee whohas accepted the terms and conditions of OPNET’s

Software Master License Agreement. As a condition ofhaving access to or use of the Lab Manual and/or the

OPNET Software all users agree to be subject the termsof OPNET’s Software Master License Agreement. Users

shall not copy or reproduce, in whole or in part, or

permit access to or disclose the Lab Manual or the

OPNET Software to any other person without the

prior written consent of OPNET.

Liability Statement

The terms and conditions governing the licensing ofOPNET Software consist of those set forth in the written

contracts and license agreements between OPNET and itscustomers. No representations in this Lab Manual orstatements of capability, suitability for use, accuracy or

performance, shall be a warranty by OPNET unless

contained in a written agreement signed by OPNET.

OPNET DISCLAIMS ALL EXPRESS OR IMPLIED

WARRANTIES AS TO THE LAB MANUAL ANDTHE OPNET SOFTWARE INCLUDING IMPLIEDWARRANTIES OF MERCHANTABILITY ANDFITNESS FOR A PARTICULAR PURPOSE.

Destination Control Statement

This Lab Manual and the OPNET Software weremanufactured in the United States by OPNET. The initialexport of this Lab Manual and the OPNET Software, andany subsequent relocation to another country shall complywith United States law relating to the export of technical

data, equipment, software and know-how.

Restricted Rights Legend

USE BY AGENCIES OF THE UNITED STATES

GOVERNMENT: For purposes of use of this LabManual and OPNET Software by agencies of the UnitedStates government (or those authorized to use of this LabManual and OPNET Software on behalf of an agency),the OPNET Software is “Commercial Computer

Software” within the meaning of DFARs 227.7202 and252.227-7014(a)(1) and FARs 52.227-19. Use,reproduction and disclosure is subject to DFAR and FAR

provisions relating to Commercial Computer Software

and the License Agreement between OPNET and theagencies using OPNET Software. OPNET software is

provided with a RESTRICTED RIGHTS NOTICE forgovernment use.

Trademarks

OPNET, OPNET Technologies, Inc., OPNET Modeler, ITGuru, OPNET Development Kit, NetBiz and IT Guru aretrademarks of OPNET Technologies, Inc. All other

trademarks used in the Lab Manual are the property of their

respective owners and used herein for identification purposes only.

CONFIDENTIAL INFORMATION

DO NOT DISCLOSE, FORWARD, DISTRIBUTE,

SHARE, OR MAKE COPIES OF THIS DOCUMENT INWHOLE OR IN PART. This document containsconfidential information and may contain information thatis proprietary, privileged, and/or exempt from disclosure

under applicable law. This document is intended for theexclusive use of the person to whom it is disclosed. If youare an unauthorized person, you are hereby notified thatany viewing, copying, disclosure or distribution of this

information may be subject to legal action. Allunauthorized persons must immediately destroy the originaldocumentation without making any copies or furtherunauthorized disclosure.



2/10/

PNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All

hts reserved. OPNET and OPNET product names are trademarks of OPNET Technologies,c.

Introduction to AppTransaction

Xpert (ACE Analyst)

test

OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Day 1

Corporate Overview

• Founded in 1986

• Publicly traded (NASDAQ: OPNT)

• HQ in Bethesda, MD

• Approximately 600 employees

• Worldwide presence through direct offices and channel

About OPNET Technologies, Inc.®

partners

• Global distribution agreement with Cisco

Best-in-Class Solutions and Services

• Application Performance Management

• Network Performance Management

• Network R&D

Strong Financial Track Record

• Long history of profitability

test• Trailing 12-month revenue of over $100M

• Approximately 25% of revenue re-invested in R&D

Broad Customer Base

• Corporate Enterprises

• Government Agencies/DoD

• Service Providers

• Network Equipment Manufacturers

3 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.



2/10/



Customers (partial list)

Energy / UtilitiesGovernmentFinancial Services Insurance

Healthcare / B io / Pharmaceut ical Manufac turing Media / Communications Services

test

Retail / Consumer Products Transportation / Logistics

OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.4 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

OPNET’s APM Xpert Suite

End-user experience and networkmonitoring and analytics

Deep application componentmonitoring and analytics

Deep transaction tracing,analysis, and prediction

Passive databaseperformance monitoring

test

5 OPNET Confidential – Not for release to third parties

© 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Network and serverdiscovery and monitoring

“What's in your Application?”



2/10/



OPNET is an IACET Authorized Provider

OPNET Technologies, Inc. has been approved as an Authorized Provider by the

International Association for Continuing Education and Training (IACET), 1760 OldMeadow Road, Suite 500, McLean, VA 22102. In obtaining this approval, OPNET

has demonstrated that it com lies with the ANSI/IACET 1-2007 Standard which is

internationally recognized as the Standard for training organizations.

As a result of their Authorized Provider status OPNET is authorized to offer IACET

Continuing Education Units (CEUs) for its programs that qualify under the

ANSI/IACET 1-2007 Standard.

test


Continuing Education Units

What is IACET?

IACET is a Standards Developing Organization accredited by the American Standards Institute

(ANSI) and is an authorizing body for education providers that meet the American National

Standard for continuing education and training (ANSI/IACET 1-2007 Standard). The IACET

standard is what learners seek for quality and is the core of thousands of educational programs

worldwide. (see: www.IACET.org)

What is an Authorized Provider?

The IACET Authorized Provider (AP) status means that an organization adheres to IACET’s

internationally recognized ANSI/IACET 1-2007 Standard for Continuing Education and Training.

Only Authorized Providers may award IACET Continuing Education Units (CEUs).

test

a s a

The Continuing Education Unit (CEU) was created by IACET as a measurement of continuing

education. One (1) IACET CEU is equal to ten (10) contact hours of participation in an organized

continuing education experience under responsible sponsorship, capable direction, and qualified

instruction.




2/10/



Earning CEUs for this Course

Introduction to AppTransaction Xpert

– This course is worth 1.3 CEUs

•To receive these CEUs, you must

– Attend the entire course

– Fill out the feedback form

– Score 80% in a test at the end of class

•Your test is scored in the next 10 business days

• If successful, you will receive a certificate denoting your CEUs in the mail

test

•Your completion of the course is stored in our database for later retrieval as

needed


Let’s get to know each other…

• Name ?

• What company you work for ?

• Which Department ?

• What experience do you have with OPNET products?

• Did you know we changed the names… !!??

test




2/10/



Course Structure

Day 1

Introduction to AppTransaction Xpert

Day 2

Advanced AppTransaction Xpert

•About OPNET

•Visualizations in AppTransaction Xpert

•Troubleshooting application

performance

•Predicting application behavior before

•Advanced Topics

•Understanding Delay Components

•Parallel Effects, User think Time

•Troubleshooting TCP applications

test•Capture and import strategies

•WAN Acceleration


Class Format

• In general, four phase instruction

– Short lecture sections

– Demonstrations

– Related hands-on labs

– Discussion of lab applications

• Class is interactive – don’t hesitate to ask questions

• Please fill out the feedback form online @ eval.opnet.com

test – Give us information on your current network and application concerns

– Provide suggestions on how we can improve our products

– Let us know how we can improve this training class




2/10/



Agenda

Introduction

- Licensing Options- Overview, Use Cases

- Basic Workflow

Using AppTransaction Xpert to visualize applications

- AppTransaction Xpert Visualizations

- Lab 1: Visualizing Application

Using AppTransaction Xpert to diagnose problematic applications

- AppDoctor

Reporting capabilities of AppTransaction Xpert

test

- eports

- Lab 2: Application Performance Troubleshooting

Using AppTransaction Xpert to perform performance predictive studies

- QuickPredict and QuickRecode

- Lab 3: Predictive Analysis


Licensing Options

• Download and administer licenses using the License Manager

• Licensed on concurrent user basis

• Online license management

(http://www.opnet.com/support/license_management.html)

• Licensing options

– Standalone vs. Floating (license server)

– Loanable Licenses

test




2/10/



Overview of AppTransaction

Xpert

test


AppTransaction Xpert Solution

• AppTransaction Xpert Plus

• AppTransaction Xpert Standard

• IT Guru with the following modules

– ACE

– ACE Decode Module

– Advanced ACE Capabilities

• Starting with version 16.0, the different offerings of AppTransaction Xpert (that is:

AppTransaction Xpert Plus, AppTransaction Xpert Standard, and AppTransaction

Xpert Module) have been combined into a single installer.

test

Important— Existing users are not losing any AppTransaction Xpert functionality

because of this change. Functionality is determined by licenses.




2/10/



What is AppTransaction Xpert... ?• Application Characterization Environment

• Capture application traces

• sua ze App cat on ynam cs

– Application layer analysis

– Network layer analysis

• Diagnose performance problems

– Minimize troubleshooting time

test• Validate solutions

– Predict application behavior under different network conditions

– Evaluate performance impact of application design decisions

– Validate infrastructure readiness for new deployments

• Generate reports


When to use AppTransaction

Xpert ??

(Use Cases)

test




2/10/



Use Case 1

Troubleshooting application performance –

• Users of a critical application are reporting very bad response times

– Network engineers are reporting no issues on the network

– Application engineers are reporting no issues with software

• Result: Finger pointing !!

Wasted time

test


Use Case 2

Testing changes of the enterprise environment –

• Relocation of key processing data center from US to China

• Management needs to know the implications of move

– Impact on application performance and network utilization?

– Will additional resources be required?

• Result: Need to produce accurate performance projections and network

test planning requirements




2/10/



Use Case 3

Application pre-deployment analysis –

• Purchase of a new 3-party application system

• Management needs to know how the new system will perform

– Application response times

– Network utilization

• Result: Need to eliminate the need for trial and error deployment of

testapplication. Expensive and time consuming.


Basic Workflow

test




2/10/



What does AppTransaction Xpert tell you … ?

• “Why is my application slow?”

– User’s desktop running slow?

– Server dela s?

– Non-optimal protocol settings (TCP and others)?

– Heavy congestion over WAN?

– Too little bandwidth?

– Application too chatty?

• “Can this new application be deployed over our network?”

test – Expected transaction response times?

– Impact other critical applications?

– Bandwidth requirements?


How does AppTransaction Xpert tell you this...?

• Reports, graphs, and tables

• Visualization of application behavior

• Diagnosis of bottlenecks and performance problems

• Predictive analysis of same traffic on different network

• Input to other OPNET products for further analysis

test – Application traffic analysis

– Capacity planning studies




2/10/



Inputs to AppTransaction Xpert

• Packet traces from the following sources – ACE Capture Agents

–

– Cisco NAM Blade

– WAN Accelerator

– F5 BIG-IP

– Sniffer

– UNIX (no agent installed)

– Tcpdump, windump ..etc

test• Basic characteristics of network (Recommended)

– Tier locations

– Bandwidth between tiers

– Latency between tiers


Application Troubleshooting Workflow

Identify performance issues

Capture Packet Traces

O tional: Create IT Guru

Visualize, Diagnose, and Predict

O tional: Detailed test

network model

Generate Reports

Analysis




2/10/



AppTransaction Xpert

Visualizations

test


AppTransaction Xpert Visualizations

• Data Exchange Chart – Shows timeline of data sent

• Tier Pair Circle – Shows summary statistics

• Tree View – Shows Transaction Analysis

• Decodes – Shows Application layer headers

test• Graphs – View time-based graphs of application statistics




2/10/



Data Exchange Chart (DEC)

• Timeline of application

messages and network

packets between tiers

• Slope indicates time to cross

network

• Color indicates payload size

of message

test


DEC: Message Groups

• When messages and packet

are too frequent to be displayed

individually, a solid block is shown

• Block is a histogram showing the

size of messages in groups

• Zoom in or use tooltip to see more

detail

test




2/10/



DEC: View Options•Application Chart Only

Shows every application message – several packets may comprise a message

• Network Chart Only

Shows every individual packet in the trace file, including TCP acknowledgements

• Application and Network Charts

Shows both charts in the same window

test Application Messages

Network Packets


DEC: Show Dependencies

• Triangles appear at largest

chunks of delayClick “Show Dependencies”

• White = delay at host

• Orange = user think time

• Purple = network delay

test

LegendLegend




2/10/



DEC: Application Characteristics

• Quiet Time

– User think time?

– Slow client computer?

– Slow server?

– Not seeing the whole picture?

§Tip: See what happened before and

a ter silence

test

Tip: Check Network Chart for

more details

Large periods of silence


DEC: Application Characteristics (cont.)

• Chatty Application

– Lots of back-and-forth communication

(Application Turns)

Lots of small messages

(orange and yellow)

test

Groups of messages close together




2/10/



Application Turn

• Counted on the Application Layer (Application Messages)

• Every turn has to cross the network latency which adds up to the total

Latency Delay

2

Client

test

1 3

Server

Latency


DEC: Application Characteristics

• One-way data transfers

– Dark blue arrows

–

Tip: Check the Network Packet

Chart for more details

test

Zoom to see the arrow heads




2/10/



DEC: Application Characteristics (cont.)

• Lots of irrelevant

traffic – Do not try diagnose

you removedirrelevant data

test

Tip: Go to Tier PairCircle and deleteunrelated traffic


Tier Pair Circle

• Summary of traffic between all tiers

– A high-level view of the

application task

– Is there extraneous traffic?

• Right-click to exclude

unwanted traffic

test

Tip: If you don’t see the protocol you expect, then right-click to re-recode traffic on the

fly




2/10/



Tier Pair Circle: Functionality

• Choose statistics to view

• Right-click a tier or

conversation to exclude

/delete it or all others

• Show only traffic containing

a specific protocol

• Find To Statistics to sort

testconversations by relevance


Tier Pair Circle: Statistics

• Pull down to select statistics to

view:

test




2/10/



Tree View

• Tree pane: One-line summary of tier pairs, connections and frames

• Table pane: Detail statistics by tier pairs, connections and frames

• Timeline pane: Overview of messages over the time of the task

test

Tree Pane Table Pane Timeline Pane


Tree View: “View by:” Options

• Tier Pairs – Network Packets

• Tier Pairs – Application Transactions

• Application Transactions

Group by:

test




2/10/



Tree View: Sorting

• Click a column header to sort the table by descending values.

• Click the table header again to sort by ascending values.

test


Tree View : Customize the Table Pane

• Choose View > Choose Table Columns…

• Right-click on the column header and click “Choose TreeView

Columns…”

• Select the network statistics to appear in the table

test




2/10/



Decode Viewer

ETH IP TCP HTTP ETH

• What are decodes?

– Analysis or description of the bytes at each layer of a packet

test• What can you learn from decodes?

Detailed protocol information

Information about data being transferred

Ex: TCP gives Sequence number and Acknowledgement number


Embedded Decode Viewer

• Show / Hide the

embedded decode viewer

by clicking and

• Select traffic in the top

pane and see

corresponding decodes in

the bottom pane

• Right-click on selectedtest

frames to export to

spreadsheet

• Click the plus sign to

expand layers of decodes




2/10/



Tree View to DEC

• Display Context-Based Traffic in Data Exchange Chart —

– Double-click on traffic for a tier pair in the Timeline pane to display the

corresponding context- based traffic in the Data Exchange Chart.

test


Graphs

• Time varying statistics

– Advanced diagnosis of network

and TCP problems

test




2/10/



Graphs (cont.)

• Select graphs to display

• Options:

– Specify bucket width for

precision of graph

– Embed graphs in DEC for

easy comparison against

messages in DEC

test – Overlay similar graphs or

overlay all

• Select “Show”


Graphs: Bucket Width

• What is a bucket width?

– Statistics can be averaged over

time rather than shown as

individual data points

– A bucket is the duration of data

you want to average, like 100 ms

or 3 seconds

• Same trace, same statistics,

different bucket widths:test – 100 ms

– 3000 ms (3 seconds)




2/10/



Graphs: Embedded Statistics

• Time-aligned statistics with the Data Exchange Chart

test


LAB 1

LAB 1

Visualizing

Application

test




2/10/



Lab 1: Visualizing Application

• Overview

– The ACE trace file represents a three-tier Oracle application. You will

explore some of the different visualizations in AppTransaction Xpert that

you can use to troubleshoot and analyze the application.

• Starts on page 1 in your lab manual

test


Lab 1: Conclusion

AppTransaction Xpert provides many different views of a transactionto understand the interaction of the application on the network

• Data Exchan e Chart – Shows timeline of data sent

• Tier Pair Circle – Shows who talks to whom and summary statistics

• Tree View – View the network packets in order, with a decode summary – Transaction analysis

• See the timeline of transactions

•test – Drill down into TCP, IP, or even application layer headers to understand

application behavior • Graphs

– View time-based graphs of throughput, congestion, and other statistics




2/10/



AppDoctor

test


What is AppDoctor … ?

• Invoke AppDoctor – AppDoctor (menu) – AppDoctor (icon)

• AppDoctor Tabs

– Summary of Delays

– Executive Summary

– Diagnosis

– Statistics

– Protocol-specific

test




2/10/



AppDoctor: Summary of Delays

• Pie chart that breaks down the

total response time into

processing, network, and

para e e ects

• Most useful view in

AppTransaction Xpert when

troubleshooting applications

• Quickly points you to what is

testthe problem, or at least what is

definitely not the problem

• Must be used on a single

application task


Summary of Delays: Processing Effects

• Tier Processing delay

– Time spent processing on each tier

test




2/10/



Summary of Delays: Network Effects

• Latency

– Delay caused by the physicaldistance between tiers

– Common problem in chattyapplications

• Bandwidth

– Delay caused by transmission(i.e. putting the bits onto thewire)

• Protocol

– Could be non-optimal TCPsettings

–

test

bottlenecks

– Check Statistics tab for values

• Congestion

– Delay caused by other traffic inthe network


Summary of Delays: Parallel Effects• Delay incurred when multiple types of delay (such as tier processing,

bandwidth, and latency) occur simultaneously

test




2/10/



AppDoctor – After Network Definition (16.0 )

test

BEFORE AFTER


Summary of Delays: Drill-Downs

• Double-click any section of the pie chart for extra information

• Drill-downs give more information that can help diagnose problems

• Ex: Bandwidth drill-down shows throughput graphs and link bandwidth

test




2/10/



AppDoctor: Executive Summary

• Gives overview information for application task

– Total transaction time – Total application data transferred

– ng e grea es cause o e ay

– Tiers in application

– Bandwidth and latency at each tier

test


AppDoctor: Diagnosis• Displays common

application, network,and server problems

potential bottlenecks

• Various characteristicsof the ACE file arecompared withthresholds

• Click on an celltest

(especially on bottlenecks) for extrainformation




2/10/



AppDoctor: Statistics

• Tabular view of key

application characteristics

• “ ”

the components of delay

• Other statistics include

– Application message statistics

– Network packet statistics

– TCP statistics

testTip: Use it with Diagnosis tab to

quantify bottlenecks

Tip: You can export to spreadsheet


AppDoctor: Protocol-Specific Tabs

• Protocol-specific information,

statistics, and diagnoses

• Appear automatically when

certain protocols are detected in

an ACE file:

– HTTP

– Citrix

– CORBA

– test




2/10/



Right-Click Operations in AppTransaction Xpert

• Graph Statistics such as throughput or congestion

• Exclude or permanently delete packets and messages

• Include packets and messages

• Rename a tier or lookup a tier name with DNS

• Re-code traffic

• Zoom in and out in the DEC

• Show protocol decodes for selected traffic

• Export decodes for selected frames in the decode viewer

test


Take-Away Points

AppTransaction Xpert provides an analysis of the main application bottlenecks that is easy to use and to understand. The diagnosis is displayedand broken down into components of delay. The different components are:

• Processing Effects

• Network Effects

– Latency

– Bandwidth

– Protocol (with recommended options)

test

– ongest on

• Other Effects

– Parallel Effects: Combination of any processing effects and networkeffects that occur at the same time




2/10/



Reports

test


Reporting Motivation

• Quick and easy to use

• Support your findings

• Recording and dissemination of all

analysis results to all interested

stake holders

• Documentation of application

behavior for future referencetest




2/10/



Report Selection - Type and Format

• Very simple and quick – select report type for output (from pull down menu, or icon

button)

test


Report Options

• Select report options and generate report

– Report sections, title (statistics)

– Report directory (output), Report Server (optional)

test




2/10/



ACE – MS Word Report

test


ACE – Web Report URL

Home Dir

(all reports)

test




2/10/



ACE – Multiple Transaction Report

Trans 1

Trans 2

test


ACE – Difference Report

test




2/10/



ACE – Spreadsheet Export Reports

Comparison Report

Tree View Statistics

AppDoctor Statistics

test


• Report template — Defines the previously-defined templates used

to generate the report. You can modify and customize report templates.

The standard templates:

ACE – Report Template

– ApplicationDeveloperReport.ace.rt.txt

– ManagementReport.ace.rt.txt

– NetworkEngineerReport.ace.rt.txt

– concise_report.ace.rt.txt

– detailed_report_ace.rt.txt

– standard_report_ace.rt.txt

•test

– Provides security, allows access by only registered users

– Provides automatic ageing of reports




2/10/



LAB 2

LAB 2

Application

Performance

Troubleshooting

test


Lab 2: Application Performance Troubleshooting

• Overview

– Response time problems for remote users of new web application

– Remote site connected via a 256-Kbps Frame Relay circuit

– Need to determine cause of the delay (network or web server)

– Traces have been captured at both server and client

• Objectives

– Visualize application in ACE

– Identify the cause of the application delay

– Use Transaction Analysis to identify problem with application

test• Starts on page 9 in your lab manual




2/10/



Lab 2: Conclusion

• Conclusions

– The web page takes approximately 24 seconds to download

– Bandwidth and congestion are the biggest sources of delay

– Server delay is not a problem

– Majority of traffic flow is from the server to the client

– Web page consumes almost all of the 256-Kbps circuit bandwidth

– Largest transaction is a download of a 352 KB graphic

test – This single graphic takes 18 seconds to download


Performance Predictive Studies

The use cases for predictive studies:

• Predict application behavior under different network conditions

– Problematic applications

• Evaluate performance impact of application design decisions

– Modified applications

– Problematic application

•test




2/10/



Prediction in AppTransaction Xpert

• What can predictive analysis tell you?

– Response time of application under different circumstances

– Will adding bandwidth help?

– Is this application sensitive to latency?

– Can we roll this out to remote users?

• What sort of studies will it help with?

– Troubleshooting performance problems

– Testing a new application

test – Deploying a current application to new users

– Server relocation studies


Application Deployment Workflow

Identify performance issues

Capture and import into ACE

Create report and prediction

scenario templates

Use QuickPredict to predict

performance

testpt ona : se u t - ser

QuickPredict for basic capacity

planning

Optional: Use IT Guru for

network and server capacity

planning83 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.



2/10/



Application Deployment Workflow

• Optional: Use simulation for further studies

– Discrete Event Simulation to predict response times or perform server

capacity planning

– Flow Analysis for network capacity planning

– Simulation covered later in course

test


QuickPredict

test




2/10/



QuickPredict

• Simplified simulation engine

• Helps study effects of the network on the application

• Questions QuickPredict can answer

– How much bandwidth would it take to get a 4 second response time?

– Would increasing the TCP Window improve response time?

– What happens to response time if latency increases?

– How sensitive is this application to packet loss?

– What would the response time be if the link was 65% utilized by other

traffic?

test – What if I had a 10% WAN Acceleration in my Network – NEW !!


QuickPredict: Outputs• Two different output styles

– QuickPredict Bar Charts shows components of delay for different scenarios

– QuickPredict plots response time over range of network conditions

test




2/10/



QuickPredict Bar Charts: Scenarios

• Click to add multiple scenarios

• Right-click to remove scenarios

test


QuickPredict Bar Charts: Options

• Put parameters on slider

– Drag slider back and forth to

dynamically update bar chart

• Rename scenarios

• Symmetric Bandwidth

– Set asymmetric values for

bandwidth

• Label delay categories

• Load and save parameter settings

test




2/10/



QuickPredict Bar Charts: Options – NEW

Introduce WAN Acceleration• Byte Reduction

• Turn Reduction

• ompress on

• Caching

test


QuickPredict Bar Charts: Tabular Results

• View detailed results for each scenario in the Tabular Results tab

test




2/10/



QuickPredict: Response Time Graphs

• Shows sensitivity of task response time to network characteristics:

– Bandwidth

– Latency

– Packet Loss

– Link Utilization

– TCP Window Size

– WAN Acceleration

test


QuickPredict: Advanced View

• Choose network path to modify

• Choose an X-axis parameter and the range of values over which to plot

response times

• Select values for each of the other parameters

• Click Update Graph

• Overlay multiple curves on same graph

• Load and save templates

test




2/10/



QuickPredict: Table View

• Modify parameters on all network paths in the task

–Red text indicates X-axis parameter

–Use Set X-axis to specify X-axis parameter

–Load and save templates

test


QuickRecode

test




2/10/



QuickRecode

• Changes high level application characteristics

– Shows sensitivity of task response time to application characteristics:

• Application turns

•Total application bytes sent

•Total processing time in tiers

– Use with QuickPredict to see how response times are affected

test


QuickRecode: Use Cases

• Use Cases

– Performance benefits

– Changing database access

to reduce chattiness

– Demonstrate effect of faster

app server

Hint: Use QuickRecode to

prove effect of optimizing

application

test




2/10/



LAB 3

LAB 3

Predictive

Analysis

test


Lab 3: Predictive Analysis

• Overview

– Deploying a new application to several remote locations

– Application has client querying an Oracle database

– Application has been deployed to pilot site (in same city as database) and

poor performance is already apparent

– Goal is to determine if the application deployment will be successful

– If not, what can be done to improve performance?

• Objectives

– Perform a deployment study using QuickPredict Bar Charts

– Use AppTransaction Xpert to drill down into the app and see what’s going

teston

– Use QuickRecode to test a possible fix





2/10/



Lab 3: Conclusion

• Always try to determine “goal” or “objective” of any problematic application

– Is there an SLA expectation

– What is the user expectations

– What are other users, departments, experiencing

• Additional useful troubleshooting information to determine

– Is the problem reproducible or is it transient

– Are all transactions of the application behaving the same

test


Agenda Capture Methodologies

Capture Data Sources

- New Capture Manager Options

Usin A Transaction X ert to erform acket ca tures

- Capture Types

- Lab 4: Capture Process

Using AppTransaction Xpert to filter packet captures

- Capture Filters

Using AppTransaction Xpert to import packet captures

- Import Process

test

- a : ent y ng er ormance ro ems n n tere races




2/10/



Capture Methodologies

test


Basic Capture Methodology• Define and understand the problem at hand

– What is slow?

– Is it the login or another specific task?

– How often does it occur?

• What is a task?

– Isolate an “atomic” Transaction

– Loading single web page

– From when user clicks “Submit” to screen refresh

• Plan your capture – Simultaneous or single-sided

– Which tiers are involved in the application?

test• Aim for as “clean” a packet trace as possible

– Keep measurement as simple as possible

– Filter out unrelated traffic

– Capture one occurrence of one task at a time

– Perform capture numerous times to help eliminate anomalies




2/10/



Atomic Transaction Example (Task)

• Web Application of a Bank

– Load Login Page• Open a Web Browser

• Type URL of the Bank

• “Enter”

• Login Page Loads

– Login

• Type User credentials

• “Login”

• Home Page shows up

– Check Savings Account

• Mouse/cursor around

• “ ”

Red arrow = Start Capture

Blue arrow = Stop Capture

test

• Savings Accounts Page Loads

– Logout

• Call up Logout screen

• “Submit”

• Logged out page



test




2/10/




• Packet traces are the primary data inputs

• AppTransaction Xpert supports a wide variety of packet sources

– ACE Capture Agents

– ACE Live

– ACE Enterprise Management Server

– Agentless Capture Options – NEW!

• F5 Load Balancers

• UNIX, Linux platforms

test• WAN Accelerators

• Cisco NAM

– Wireshark and other Winpcap and Libpcap sources• TCPDump, Snoop


Agentless Capture Options – NEW

• No software installation required

• Use the AppTransaction Xpert Capture Manger to control captures remotely

via SSH

• Supported on platforms that support TCPDump

– TCPDump file remains on machine until capture is completed, then deleted after

retrieval

• Included in the AppTransaction Xpert Plus license

test




2/10/



Agentless Capture Options – NEW (cont.)

• F5 Load Balancers

– Excellent way to follow a transaction to the proper server – Perform successful trace file merging across firewalls

–

• UNIX-Based Operating Systems

– Supported Platforms

• Linux

• Solaris SPARC

• Solaris Intel/AMD

• AIX

test• FreeBSD

• HP-UX Itanium

• Macintosh PowerPC

• Macintosh Intel

• True64


Agentless Capture Options – NEW (cont.)

• WAN Acceleration Devices

– Support for Riverbed and Cisco

– Restore visibility of the WAN

– More information can be found in session 1471 - Monitoring and

Troubleshooting Applications in a WAN Accelerated Environment with

ACE Live™ and AppTransaction Xpert

test




2/10/



Capture Data Sources – AppResponse Xpert

(ACE Live)• New seamless 1-click integration

– Let ACE Live do the filtering for you

• Easily pull traces from alert snapshots

o actua app cat on ssues

test


Capture Data Sources – AppForensics Xpert(ACE EMS)• ACE Enterprise Management Server

– Provides web-based extension to Continuous Capture functionality

– Supports both End-User Driven Capture workflows, as well as Server-Driven

test

– End Users report application slowness

• Trouble ticket is created with trace files of the incident attached

• Troubleshooters claim problem reports and troubleshoot on their schedule




2/10/



AppForensics Xpert – Web Capture Retrieval

test


Capture Data Sources – AppForensics Xpert• Server-Driven Capture Workflow

– Panorama provides forensic actions based on threshold violations within the

application servers

– The trace file forensic action automaticall enerates a trace that is accessed

through AppForensics Xpert

– For more detail, attend session 1482 – Deploying an Enterprise-Wide Packet

Capture Architecture with ACE EMS and ACE Live

test




2/10/



Capture Data Sources – ACE Agents

• Benefits of using ACE Agents

– Unlicensed

– Support for major platforms (Windows, HP-UX, AIX, Solaris, Linux, Mac OS X,

– Lightweight, Secure, and Stable

– Silent Install capability

– Server Statistics collection (Windows)

– Enhanced Filtering (NEW)

– On-Demand and Continuous Capture modes

test


Additional Data Sources

• AppTransaction Xpert can import a variety of additional data

– Identify these sources during capture to ensure collection

• From OPNET products

– Windows ACE Capture Agents can collect Lightweight Server Statistics

– OPNET Panorama

• Server statistics and performance metrics

• Java transaction tracing

• Third-party sources

– HP GlancePlus

– Perfmontest – Custom data in comma

saved (.csv) format




2/10/



Capture Lightweight Server Statistics

• Capture Windows performance data

– ACE Capture Agent generates two files

– Application traffic (*.appcapture) file

– Windows performance data (*.appcapture.da) file

Display internal server metrics

•Percent Processor Time

•File Read Bytes Per Second

•System Calls Per Second

•test

•File Write Bytes Per Second


AppInternals Xpert (Panorama) Server Statistics

• Capture Server Statistics from AppInternals Xpert

– Use with on-demand captures

– Thousands of available metrics

• For more information:

– Session 1461- OPNET Panorama® Essentials: Multi-Tier Application Monitoring,

Correlation, and Troubleshooting

test




2/10/



AppInternals Xpert Transaction Trace

• AppInternals Xpert can create a trace of all activity through the Java

Virtual Machine (JVM) or .NET CLR

– Provides a call-by-call sequencing (or call stack) of all activity within your

app cat on

• This transaction trace can be imported into AppTransaction Xpert to explain

what is occurring on a given system during long processing delays

test


Capture Manager

• Capture Manager control parameters

test




2/10/



Capture Agent Settings

test


Capture Types

test




2/10/



On-Demand Capture

•Methodology –User knows exactly when a problem is going to occur and

–Deploy and configure ACE Capture Agents

–Start the capture

–User re-creates the problem

–Stop the capture

– test


On-Demand Capture (cont.)

test




2/10/



On Demand Capture Process: Adding an Agent

test


On Demand Capture Process: Specify AgentSettings

test




2/10/



Capture Process: Start Capture

test


Capture Process: Finish Capture

test




2/10/



Capture Process: Open in Transaction Analyzer

test


Scheduled Capture

•Useful for reproducible problems during off hours

•Methodology

–User knows when a problem will occur • For example, after-hours batch process or database backup

–Schedule a capture• Choose one-time or repeating capture

• Set the schedule for a capture

• Specify schedule for download

test – mpor race or ana ys s




2/10/



Scheduled Capture (cont.)

test


Scheduled Capture (cont.)

Scheduled capture can create one-time,

daily, weekly, or monthly capture

schedules

test

Scheduled capture utility can optionally

create a one-time, daily, weekly, or

monthly schedule to download trace files




2/10/



Continuous Capture

•Useful for troubleshooting intermittent problems•Methodology

–User experiences intermittent problem

–Start continuous capture

–User calls help desk

• Problem occurred again around 9:30

–Preview capture

test

–

–Import trace for analysis

•Agent details –Same capture agents as on-demand capture


Continuous Capture (cont.)

test




2/10/



Continuous Capture: Preview/Download

test


LAB 4

LAB 4

Capture

Process

test




2/10/



Lab 4: Capture Process

• Overview

– Configure a capture agent

– Perform an on-demand capture

– Open in Transaction Analyzer

• Skills covered

– Learn how to configure a capture agent

– Understand how to use agent filters

– Capturing from the network

test• Starts on page 32 in your lab manual


Lab 4: Conclusion

• Configured capture agent for capture

– Located and specified host IP Address

– Specified capture mode

– Set up a filter

• Performed a capture

– Started the agent

– Performed a transaction

– Stopped the agent

– Brou ht the ca ture file into Transaction Anal zer test




2/10/



Capture Filters

test


Filtering Unrelated Traffic

• You want to capture the transaction of interest – AppTransaction Xpert lets you filter out irrelevant traffic

• Options to filter during capture, during the import or after the import

– The earlier the better

• Why filter during capture? – Only capture the specific transaction – Smaller trace file to process => easier/faster import

• Why not filter during capture? – Sometimes you want to see the whole picture – If you’re not 100% sure about application path

test• Recommended methodology – Capture without filters for short duration – Examine trace – Determine best filter to use – Capture subsequent tasks with filter




2/10/



Filter: During Capture

Users can filter out or only allow traffic based on a specific protocol

Users can filter specific hosts

and/or ports for incoming or

outgoing traffic

test

Users can elect to exclude or

allow broadcast and multicast

traffic

ew er ng p ons nCapture Agent 3.9 Use wildcards and slash notation

Use DNS names

Port ranges

Broadcast and multicast can beexcluded independently


Filter: During Capture (cont.)

• Preview in Continuous Capture

– Select the IP addresses, conversations and time ranges of interest

• Great for troubleshooting intermittent issues

test




2/10/



Filter: During Import

•What is Trace Explorer? – Graphical protocol analyzer

– Used to hel filter unrelated traffic

during import process

•Why would you use it? – Graphically filter a capture file

– Allows for fast read of large trace

files

– You want to know what you’re

lookin at

test


Filter: After Import• Once capture file has been imported, users

can still filter irrelevant traffic

– Right-click on any tier pair

conversation, application message,

app cat on transact on or networ

packet

• Option to Exclude or Permanently

Delete

– Filter Wizard

• Remove unidirectional, non-IP, UDP,

broadcast traffic

– Capture File Filter

test

• Use to remove traffic by host, port,

protocol




2/10/



Import Process

test


Import Overview

• Now that the data has been captured, it must be imported into


• Review what has been captured

– Do you have a single capture file?

– Do you have more than one capture file?

– Are you capturing WAN accelerated traffic?

– Is the capture file filtered?

• If not, use Trace Explorer

test• Greatly simplified for 16.0 !!




2/10/



Import Changes in 16.0

• Import process has been greatly simplified

– 15.0 required answering a series of questions about tier locations, bandwidths, and

latencies

– For 16.0, we focused on asking as few questions as possible

• Filter, view statistics or diagnoses, or explore your transaction immediately

• Locations, bandwidths, and latencies can still be specified to break down Network

Transfer effects into Bandwidth, Protocol, and Congestion

• Most importantly, you can change these values whenever you need to

• Import is now broken down into use cases

– Single packet trace

–

test

– Merge traces from a WAN-accelerated environment

– Batch import

– Filter into Trace Explorer


Import a Single Capture File

test




2/10/



Merging Capture Files

• What is a merge?

– Two or more traces of an application transaction can be combined

– Creates unified view of application

– Accurately determines the send and receive times of each packet

• Benefits and use cases

– Allows analysis of multi-tier transactions where all traffic does not

go through one location

– Improves analysis in a production environment with congestion issues such as

packet loss and retransmissions

test


Merging Capture Files (cont.)

• All traces must be of the same transaction

• All must have been captured at the “same time”

• Easier to import if there is some common traffic between traces

test




2/10/




File name

test

Load previously saved importdetails (tier names, tier locations,

bandwidth, latency,…)



• Replace IP addresses with friendly names (can use DNS)

test




2/10/



After Opening/Merging

• Now what?

– Filter down to a single transaction or remove user think time

– Open AppDoctor

• Note that there are only Processing and Network effects identified

– Specify bandwidth and latency to break out individual network effects

test


AppDoctor

Bandwidth, protocol and

congestion are all

included in one category

named “Network

Transfer”

To break down the various

test

components o etwor

Transfer”, click the

“Refine Network

Effects…” button




2/10/



Refine Network Effects

Specify the local and

remote bandwidths

here

testTier locations and

latencies are

automaticallydetected, but can be

changed if needed


Specify Bandwidth and Latency

• Minimum bandwidth

– Capacity of the slowest link

– Should be a known factor about your network

• Minimum one-way latency

– PathProbe provides an estimate

– Half the value returned by ping

– Auto-calculated in a trace merge

– In man cases, A Transaction X ert can estimate latenc of a sin le-sidedtest

trace




2/10/



Specify Bandwidth and Latency (cont.)

• Quiz

– Bandwidth of each segment is known

– Ping reports 46ms RTT

test• What would you specify for bandwidth and latency between the Client and

the Server during import into AppTransaction Xpert ?

Bandwidth = 1544 Kbps Latency = 23ms


AppDoctor – After Network Definition

test

BEFORE AFTER




2/10/



LAB 5

LAB 5

Identifying

Performance

Problems in

test


Lab 5: Identifying Performance Problems inUnfiltered Traces

• Overview

– Open a single side capture

– Filter traffic

– Refine network effects

– Analyze transaction

• Skills covered

– Step through the import process in AppTransaction Xpert

– Understandin the im ortance of bandwidth settin stest





2/10/



Lab 5: Conclusion

• Imported a single-sided capture file

– Imported capture file

– Filtered irrelevant traffic

– Estimated locations

– Entered local and remote bandwidth

– Estimated latency

test


Introduction to AppTransaction

Xpert

test


Day 2



2/10/



What to Expect in Advanced AppTransaction

Xpert Topics

• Mix of lecture and lab

• Advanced class

– Address network concepts important to using AppTransaction Xpert

– Dive into “the science” behind ACE

– Highlight new ACE features

– Discuss TCP

•

test

– Participate

– Ask questions


Prerequisites

• Prerequisites

– Reviewed yesterdays material

– Reviewed session 1441

– Equivalent experience

• You should already understand the mechanics of ACE

– Capturing a trace

– Importing traces

– Generating reports

test• You should already recognize “vanilla” problems

– Chatty applications

– Insufficient bandwidth

– Slow servers

– …




2/10/



Agenda

How AppTransaction Xpert Calculates and Predicts Components of Delay

- Lab 1: Calculating Network Delays

- Congestion Delay

- Protocol Delay

- Lab 2: Understanding Components of Delay

Understanding Parallel Effects

- Parallel Effects

test

- User Think Time

- Lab 3: Understanding Parallel Effects


How AppTransaction Xpert

calculates and predicts

com onents of Delatest




2/10/



Components of Application Delay

• The following are key components of delay in a transaction:

– Processing

• Client

• Server(s)

– Network

• Protocol (TCP WIN, Nagle’s, etc.)

• Bandwidth

• Latency (baseline, dominated by network hardware and ultimately ruled by C)

• Congestion (including queuing)

• Packet loss

test – Application itself

• Think of the top two topics as “resources”. How does the app use them?

– Efficient/inefficient code

– App turns/Chattiness


Review of Network Delays

• Packets experience a complex series of

– Network hops

– Technologies

– Queuing delays

– …

T1 OC3

test




2/10/



Apps See the Network as a “Black Box”, So Does

AppTransaction Xpert• For most troubleshooting, complexity can be simplified

• Think of entire network as a single “equivalent” circuit

• Once packet leaves source, the delay can be separated into 3 categories

T1 OC3

– Bandwidth delay

– Latency delay

– Congestion (queuing) delay

• What about Protocol?

– Protocol delay is a function of the sender/receiver

– Packet Loss is handled by the protocol

test

Bandwidth, Latency,

Congestion


Bandwidth Delay

• Time it takes to stream all bits onto circuit

• This number varies with packet size

• Can be calculated from packet size and circuit bandwidth

HeadersData

2000 bits

2000 bits/second

test…

What is the bandwidth delay for this packet?




2/10/



Calculate Serialization Delay

How long does it taketo serialize this packet

on to the wire?

Serialization and BW

are the same thing

test• If you know the bottleneck bandwidth across a link and the size of each

packet, you know how long it takes to get across the link due to serialization

delay (ie: Bandwidth delay)

• Note: we are ignoring latency at this point


Latency Delay

• Time it takes for signal (1st bit) to travel across network

– Distance / speed of light

– Normal delays caused by device forwarding

• This is a constant number for all packets

7 ms

1 ms

10 ms

1 ms

10 ms

1 ms1st bit

10 ms

testWhat is the latency for this circuit?

What value would “ping” produce?




2/10/



Application Turns

• A turn is a change in the flow direction of application data

• Turns magnify circuit latency

• An application with no turns would only experience latency once

• An application with 99 turns would experience latency 100 times

• Simple equation for total latency delay

– Latency Delay = circuit latency * (turns +1)

– This equation only works on sequential applications

No A lication Turns Man A lication Turns

test


Congestion Delay

• Queuing on devices

• Usually caused by other traffic

• Can vary from packet to packet

Forwarding rate = 1 packet / ms

packet

test 5 packets in forwarding queue

What is the congestion delay for this device?

What is the total delay across this device?




2/10/



Determine Queuing/Congestion

Packet should have taken

t s t me . . .

. . . but ended up

taking this time

test

• If a packet should take x time but takes y time, y – x yields

queuing/congestion delay


Example Network Delay

• Bandwidth = 1,000 bps

• Latency = 5 seconds

5 seconds

30 seconds

Latency delay

Bandwidth delayClient

T = 0

30,000 bits

10 20 30 40 50

test

15 seconds

Congestion delay

Server

50




2/10/



Packet Trains

• Stream of packets that are sent together

• Arrive at destination spread out

• Many TCP segments of a larger application message

• Spreading caused by limited bandwidth of bottleneck link

T = 0 10 20 30 40 50 60

test

10 20 30 40 50 600


Calculating Delay for Packet Trains

• Treat packet trains like 1 big packet

• Same rules apply

– First—establish delay caused by bandwidth (stream bits onto wire)

– Second—add latency (last bit of the packet train getting across)

– Third—anything left over is congestion

Application Layer

testTransport Layer

Network Layer




2/10/



Calculating Delays—Advanced

• Bandwidth = 1,000 bps

• Latency = 5 seconds

+ 10

+ 5

5

Bandwidth Delay

Latency Delay

Congestion Delay

+ 15

+ 5

+ 305

5

ClientT = 0 10 20 30 40 50 60 70 80 90 100

test

Tier Processing (Server)

Total Delay = 45+15+20+10+10 = 100

10

10

Server

5,000 bits 10,000 bits

8010 20 65 100


Pie Chart

• AppTransaction Xpert creates a pie chart that shows all of the components of delay in one

screen

• Shows the total impact on response time due to each individual effect

• Eachwed e is the maximumbenefit that a sin le u radewill rovide

Bandwidth

Latency

Congestion

Client

Server

Bandwidth 45

Latency 15

Congestion 20

Client 10

Server 10

test




2/10/



Network Effects in App Doctor

• AppTransaction Xpert 16 does not require fully specifying network data for

import

• Creates a “Network Effects” pie slice

test


Network Effects

• Specifying Bandwidth will cause AppTransaction Xpert to recalculate the

values for the components of delay

test• For more information, see

Session 1442: Application Capture

and Import Strategies with AppTransaction Xpert




2/10/



Calculating Delays

• Two ways to calculate the Pie Chart

– Adding up (add the components of delay)

– Squeezing down (reduce effects one at a time)

ClientT = 0 10 20 30 40 50 60 70 80 90 100

test

Server 8010 20 65 100


Review of Network Delays

• Different factors can affect network delay

– Bandwidth

– Latency

– Congestion

• Some factors are constant and some are variable

• Network delay is not always due to lack of bandwidth

• Important that you understand fundamentals to interpret ACE results

test




2/10/



LAB 1

LAB 1

Calculating

Network

Delays

test


Lab 1: Calculating Network Delays

• Overview

– Apply knowledge of sources of delay to a real application

• Objectives

– From the raw statistics calculate total bandwidth delay

– From the raw statistics calculate total latency delay

– Compare results with ACE analysis


test




2/10/



Response Time = 26.03

Network Data = 3,199,760 Bytes

Size of link = 1,544,000 Kbps

Lab 1: Conclusion

Bandwidth Delay = (3,199,760*8)/1,544,000 = 16.58

Percentage of Bandwidth = 16.58/26.03 = 63.7%

Turns = 337

Latency = 24.23ms

Latency Delay = 24.23*(337+1) = 8.18 seconds

test

ercen age o a ency = . . = .


Calculated that Bandwidth caused 63.7% of response time

Calculated that Latency caused 29.8% of response time

Lab 1: Conclusion

test




2/10/



Why didn’t the math work out perfectly?

Some amount of congestion in trace Some amount of protocol delay in trace

Lab 1: Conclusion

quat on n t account or - rect ona tra c

Our simple math was very close

Important to perform this type of “sanity check”

test


Congestion Delay

test




2/10/



Congestion Delay

It is somewhat amazing that congestion can be calculated Very difficult to measure and calculate manually

can measure e e ec even w ou see ng e cause

ACE provides 2 important pieces of information forcongestionActual congestion delay for each packet

The cumulative effect of congestion on the response time

test


Key Concept

Trace file includes 1 piece of time information

Time the packet arrived at the capture probe

ACE includes 2 pieces of time information for each packet

Time the packet left the sending side

Time the packet arrived at the receiving side

1 piece of time information per packet

test

2 pieces of time information per packet




2/10/



Key Concept

ACE knows both the send and receive time

Delay analysis requires that both values are known

How does ACE derive the second piece of time information?

Trace Merge

Single Trace Adjust

test


Trace Merge

Each trace shows what happened locally

Once clocks are aligned, draw the packets from send to receive

Client

Trace captured at client

side

B A C C* D

ClientT = 0 2 3 4 5 61 7 8 9 10

A C C*B DT = 0 2 3 4 5 61 7 8 109

test

Server

race cap ure a server

side

B A C* DServer

0 2 3 4 5 61 7 8 9 10

B D A C*

0 2 3 4 5 61 7 8 109




2/10/



Single Side Adjust

With a single trace, ACE only knows what happened locally

Must estimate what happened on network and at other side Based on bandwidth and latency values set during import

es ma es w en e pac e was sen

ACE estimates when the packet will arrive

Client

Trace captured at client

side

ClientT = 0 2 3 4 5 61 7 8 9 10

A B C C* E

T = 0 2 3 4 5 61 7 8 9 10

testB A C C* E

Server

0 2 3 4 5 61 7 8 9 10


Trace Merge

What happens if the import parameters are incorrect

Bandwidth is set too low

Latency is set too high

Might create false packet crossings

This will confuse analysis

Client

Trace captured at client side

ClientT = 0 2 3 4 5 61 7 8 9 10

A B C D E

T = 0 2 3 4 5 61 7 8 9 10

testB A C D E

Server

0 2 3 4 5 61 7 8 9 10




2/10/



Key Concept

With BOTH send and receive time ACE knows Actual packet delay

Order of how packets were sent and received

What would happen if we graphed the delays?

testTotal Packet Delay


Packet Size vs. Delay Graph

• Graph the packet size vs. total packet delay

D e l a y ( s e c )

Congestion

Slope is bandwidth*

test

Packet Size (bits)

Pure latency

* Inverse of bandwidth (1/BW)




2/10/



Packet Size vs. Delay Graph (cont.)

• Graph Packet Size vs. Total Packet Delay

D e l a y ( s e c )

test

Packet Size (bits)


Viewing Congestion in ACE

• Several good ways to view congestion in ACE

– AppDoctor diagnosis

– Summary of Delays

– Congestion drill-down

– Data Exchange Chart

test




2/10/



Congestion in the AppDoctor Diagnosis

• Specifically marks category as a Bottleneck

test


Congestion in the Summary of Delays

• Shows the cumulative effect of the congestion on response time

• How much slower was the application because of congestion?

test




2/10/



Drilling into Congestion

test• Drilling into congestion can show if the packets are getting queued or

otherwise slowed down


Drilling into Congestion (cont.)

test• Look for “plateaus”

• Always keep in mind the scale of the chart!

• Is delay uniform or random?

– Is there any encryption on the link?




2/10/



Congestion in the Drill-Down

• Packet Congestion graph

• Retransmission graph

test


Congestion in the DEC

• Network chart layered with Packet Congestion graph

test




2/10/



Congestion Review

• Congestion Delay is cause by other traffic on the network

• Always better to merge traces from congested circuits

– Congestion values for a trace merge are precise

– Congestion values for a single-side adjust are approximations

• Several ways to see congestion and its effect on the application

• Imagine “squeezing” the congestion out of the trace to calculate its

contribution to overall delay

test


Protocol Delay

test




2/10/



Protocol Delay

• Caused by the artificial throttling of traffic

• Packets are prevented from even getting onto the network

• Caused mainly by TCP

test


Why Does TCP Exist?

• TCP offers reliable data transport to applications

– Application data is guaranteed to be delivered

– Packets that are “lost” by the network will be retransmitted

• TCP makes applications behave in a network-friendly manner

– Throttles the rate at which application sends data

– Ensures that no single user consumes all of the bandwidth

• Both can impact application performance

test




2/10/



What Does Protocol Delay Look Like?

• Protocol delay prevents packets from even getting on the network

Application Layer

Transport Layer

Network Layer

Protocol DelayNo Protocol Delay

test

0 2 3 4 5 61 7 0 2 3 4 5 61 7 8 9 10 11 12 13

Protocol

Delay


Observe Protocol Behavior

Protocol dela is when the

sending side delays or

throttles sending of data (this

is different than processing

delay)

test• Observe how data leaves in stages indicating Protocol Delay

• Observe other attributes like TCP window size

• Our example here is ridiculously simplified, but real protocol delay is

observable in a real trace




2/10/



Forms of Protocol Delay

• Critical that you understand mechanisms of TCP

• Several different TCP mechanisms cause can protocol delay

– TCP windowing

– Slow Start

– Frozen window

– Nagle’s algorithm

– Retransmissions

– Out of sequence packets

test


Viewing Protocol Delay in ACE

• Several ways to view Protocol Effects

–AppDoctor diagnosis

– ummary o e ays

–Protocol drill-down

–TCP Visualization in DEC

test




2/10/



Protocol Delay in the AppDoctor Diagnosis

• Specifically marks category as a Bottleneck

test


Protocol Delay in the Summary of Delays

• Double-click on slice to get more detail

test




2/10/



Protocol Delay Drill-Down

• Graphs dynamically appear based on AppDoctor Diagnosis

• Users can manually add or remove graphs

• Protocol Delay

– TCP In-Flight Data

– Retransmissions

– Out of Sequence Packets

– TCP Nagle’s Delay

– TCP Advertised Receive Window

test


Protocol Delay Review

• Protocol delay results from the artificial throttling of traffic

• Several different TCP mechanisms can cause it

• Prevents the packet from even getting onto the network

• Can often be calculated without a trace merge

• Imagine “squeezing” the protocol effects out of the trace to calculate its

contribution to overall delay

test




2/10/



LAB 2

LAB 2

Understanding

Components of

Delay

test


Lab 2: Understanding Components of Delay

• Overview

– User is transferring files with Microsoft file browser

• Transfer time is very slow

• Want to understand cause of delay

• Objectives

– Identify and understand the sources of delay

– Compare slow transfer to fast transfer


test• Homework

– Continue with homework sections of lab manual if you have time




2/10/



Lab 2: Conclusion

• File transfer is slow

• Primary source of delay is protocol/congestion• Some “potential” problems with TCP but no smoking gun• Look at packet delay graph to evaluate congestion

• Significant congestion on WAN circuit – Base latency is 25ms – Some packets take 125 ms to cross the network

test


Lab 2: Conclusion

• Congestion is primary cause of delay

• Removing other traffic on network fixes problem

– Reduces congestion

– Response time decreases

• Might be a case for QoS

Significant Congestion No Congestion

test




2/10/



Parallel Effects

test


The “Mysterious” Gray Wedge

• AppTransaction Xpert shows “parallel effects” as a gray pie wedge

test – Transaction has more than one effect taking place at one time

– HTTP is an example of a transaction that exhibits parallelism




2/10/



Parallel Effects Drill-Down

• Can show the individual effects with their combined parallel effects together

• Solid bars are the individual effects

• Striped bars are the parallel effects

• Optimizing both components simultaneously will give the benefits of the

individual effects AND the parallel effects

T = 0 10 20 30 40 50 60 70 80 90 100

Response Time

Bandwidth 60%

Latency 10%

Server 20%

Parallel 10%

test

ara e ec s o an w an a ency

Parallel Effects of Server and Latency



• Parallel delays may seem counterintuitive at first

• It is a real effect and does happen in the real world

• Real Life Parallel Effects example that you will remember!

– Reading a paper while eating breakfast

– Cooking a meal (dessert in the oven while steak is on the grill)

• Would be difficult to split these into simple 2 category pie charts

– Incorrect to say grilling steak was 50% and dessert was 50% ?

– Suggests that eliminating dessert would cut dinner preparation in half

– Probably more like 25% steak, 25% dessert, 50% both

– Life is aralleltest




2/10/



Types of Applications

• Applications fall into two main categories –Transactional

• rong cause an e ec r e a ons p e ween pac e s

•Each application message is dependent on the one before it

•Examples

– Database queries

– Web transactions

– File transfers

–Asynchronous

test• No strong cause and effect relation between packets

•Examples

– Voice – Citrix


Types of Applications

• Transactional application can be further divided

–Sequential

• ng e execu on pa

•Application is only doing one thing at a time

–Parallel

•Two or more simultaneous execution paths

•Application is doing multiple things at a time

test






2/10/



AppDoctor Individual Effects

Response Time 6 secServer 2 sec (33%)

Client 2 sec (33%)

• Summary of Delays for a sequential transaction

Bandwidth 2 sec (33%)

test


AppDoctor Parallel Effects

• Summary of Delays for a parallel transaction

• Server is now processing while simultaneously sending the large message

test




2/10/



AppDoctor Parallel Effects (cont.)

How does AppDoctor handle multiple effects occurring at the same time?

Response

Time

6 sec

Server 2 sec

test Client 2 sec

(33%)

Parallel

Effects

2 sec

(33%)The "Parallel Effects" categoryMixed wedge shows total impact due to server processing and bandwidth occurring in parallel

Because there may be many mixed wedges, ACE shows these as gray in the Pie Chart


Understanding Parallel Applications

• Calculating delays for sequential transactions is fairly easy

– Each slice of time can be attributed to a single cause

– All the slices add up to the total

• Calculating delays for parallel transactions can be complex

– A slice of time may be attributed to multiple sources

– Only the squeezing method works here

• Parallel Effects are subtle and not always easy to visualize

• Parallel Effects do not always occur just because delays overlap

• Aren’t you glad you have AppTransaction Xpert to do this for you

test




2/10/



Parallel Effects Example

• Parallel

– Two independent connections occur in parallel

– Both must complete for the application to complete

Client

Conn. B

A

B B B

test

Server Server

Conn. A



• Slide Conventions

– Show independent connections as parallel DECs

– Keep in mind that these are happening on top of each other

– Obvious now why you can’t just add delays up

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Connection A

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Connection B




2/10/




• Parallel applications with squeezing method

– Perform experiments that squeeze out each category of delay

• Color associated with that delay would disappear

• Response time of transaction may decrease

– Record the difference from each experiment

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100



• What would happen if bandwidth became infinite?

– Green would be squeezed out

– Total response time of the transaction would change to 40 seconds

– Bandwidth is responsible for 60 seconds of response time (60%)

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100




2/10/




• What would happen if latency became zero?

– Red would be squeezed out


– Latency is responsible for 10 seconds of response time (10%)

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100



• What would happen if the server became infinitely fast?

– Yellow would be squeezed out


– The server is responsible for 20 seconds of response time (20%)

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100




2/10/




• Something interesting just happened

– Bandwidth caused 60%, Latency caused 10%, Server caused 20%

– The individual sources of delay are only responsible for 90%

– What caused the remaining 10%?

• Delays that are not removed by a single source are Parallel Effects

Bandwidth

Bandwidth 60%

test

Latency

Server

ParallelServer 20%

Parallel 10%



• Squeeze out 2 effects at the same time

– Bandwidth = 60%

– Latency = 10%

– Server = 20%

– Parallel Effects = 10%

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B




2/10/




• Remove the Server and Latency together

– Server by itself is 20%, Latency by itself is 10%

– The pair is responsible for 35 seconds of response time (35%)

– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B



• Remove Bandwidth and Latency together

– Bandwidth by itself is 60%, Latency by itself is 10%

– The pair is responsible for 75 seconds of response time (75%)

– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously

Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

Server Server

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B




2/10/



Parallel Effects Drill-Down

• Can show the individual effects with their combined parallel effects together

• Solid bars are the individual effects• Striped bars are the parallel effects

•

Bandwidth 60%

Latency 10%

Server 20%

AND the parallel effects

Response

Time

T = 0 10 20 30 40 50 60 70 80 90 100

Parallel Effects of Bandwidth and Latency

testParallel 10%

Parallel Effects of Server and Latency


Parallel Effects Summary

• Delays that can not be removed by a simple change of one parameter

• To reduce, you must change at least two parameters simultaneously

• These delays are not bad

– Application is efficient

– Application is doing multiple things at the same time

testBandw idth

Latency

Server

Parallel




2/10/



User Think Time

test


Capturing a Good Transaction (Task)

• Important to get an “atomic” transaction

– Atomic transaction is the smallest transaction that you can capture effectively

– Capture should be from edge of user input to screen refresh

– Eliminate “PEBCAK”

test




2/10/



Atomic Transaction Example

• Modify Customer Record

– Login

• Call up the Login screen•

• Type information

• “Submit”

– Look up record

• Call up correct screen/menu

• Type information

• “Submit”

– Modify data

• Mouse/cursor around

“modifying a customer

record is slow”

• Need to find out what is

involved in transaction

test• Type information

• “Submit”

– Logout• Call up Logout screen

• “Submit”

Red arrows indicate the key part

of the transaction to capture


Identifying and RemovingUser Think Time

• Always get as close as you can to a perfect “Atomic” transaction as described

in the Ca ture Section

• However, sometimes you just can’t get the perfect capture

• “User think time” enters the trace

• AppTransaction Xpert deals with think time

– Auto detects when calling up AppDoctor

– Allows user to explicitly specify think time

test




2/10/



Removing User Think Time

AppTransaction Xpert automatically identifies that the trace contains user think

time. This dialog comes up when AppDoctor is selected


testalso allows the user to

pick any dependency on

the client (first talker)

and categorize it as user

think time


Removing User Think Time

Original Analysis

test

User Think Time

Removed




2/10/



LAB 3

LAB 3

Understanding

Parallel Effects

test


Lab 3: Understanding Parallel Effects

• Objectives

– Explore an application transaction that has significant parallel effects

• Trace is fairly simple

• Trace is somewhat artificial

– Practice visualizing the squeezing method to understand the behavior

– Use Drill-down and QuickPredict to understand the interactions


test




2/10/



Lab 3: Conclusion

• Parallel delays may seem counterintuitive at first

• It is a real effect and does happen in the real world

• This example is exaggerated to make it easy to visualize

test


AgendaOverview of TCP

- TCP Basics

- TCP Windowing

- Lab 4: Examining TCP Behavior

rou es oo ng

- Multi-User QuickPredict

- Bandwidth-Delay Product

- Lab 5: Troubleshooting a Slow File Transfer

- Detecting Packet Loss

- Frozen Window

- ’test

- Discrete Event Simulation

Instructor Demo: Nagle’s Algorithm using DES

References

- What’s New ?

- Wrap up255 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.



2/10/



TCP Basics

test


What is TCP?

• Transmission Control Protocol

• Transport layer protocol

• Layered between IP and application layer

• Typical applications such as

– HTTP (Web)

– POP, IMAP, SMTP (e-mail)

Application Application

TCP

IP

test – Telnet, FTP, and more

Data Link




2/10/



Why Does TCP Exist?

1) TCP offers reliable data transport to applications

• User data is guaranteed to be delivered

2) TCP makes applications behave in a network-friendly manner

• TCP may throttle the rate at which it sends user data

Both impact application performance

test


Which Applications Don’t Use TCP?

• TCP adds overhead

• TCP limits throughput

• Not all applications need what TCP provides

– VoIP – Voice over IP

– Video Conferencing

– SNMP – Simple Network Management Protocol

– Network Gaming Protocols

– Domain Name System (DNS)

– Network Time Protocol (NTP)

test




2/10/



Motivation – Why Is TCP Important?

Example TCP/IP Stack

SOAP SOAP

IP

FIBER

IP

ETHERNET

IP

ETHERNETNetwork Managers

TCP TCP

SSL SSL

test

• TCP is in a key location of networked applications’ protocol stacks – Frequently misunderstood – This makes people and software that understand TCP valuable


Review: ACE Application Message Chart

Timeline

Application

message

groups

Application

message

arrows

Tiers

test

Red packets have no payload (usually acknowledgements, SYNs, or FINs)

Packet

sizes






2/10/



Closing a Connection

• Four-way handshake to close connection

• “I have no more data to send.”• Typical example

client

273 274

connection closed at client

test

server

272A

275

D

connection closed at server


Closing a Connection (cont.)

• Often only three segments used

• Second FIN also is an acknowledgement for first FIN (B&C in one packet)

• TCP Resets are an

alternate way to

terminate a connection 86

A

87

B&C

88

D

test




2/10/



SEQ and ACK Numbers

• Ensures that data was received

• Sending TCP gives each segment a sequence number

– SEQ of segment K is SEQ of segment K-1 plus length of segment K-1

• Receiving TCP acknowledges reception by replying with ACK

– ACK number is the SEQ number of the next expected byte

• SEQ and ACK numbers are in

bytes, not packetsReceiver

SEQ=100 LEN=10

Sender

testACK= ?

SEQ = ? LEN=55

110

110

K-1 K

Seq # K-1 Seq # K


SEQ and ACK Quiz

• Which frame(s) is 36 ACKing?

• Which frame(s) is 44 ACKing?

test




2/10/



TCP Acknowledgements

• Receiver normally ACKs every other packet

– Inefficient to ACK every packet

• What if only one packet is received?

– Delayed-ACK timer must expire

– 100-200 ms (OS determines exact delay)

– Usually does not impact total application performance

m e n t

test

D a t a

A c k n o

w l e d g

268 OPNET Confidential –Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Review: TCP Basics

• TCP is a connection-based protocol

• It uses a three-way handshake for opening connections

• Each side closes the connection separately

• SEQ and ACK numbers ensure reliable delivery

test




2/10/



TCP Windowing

test


TCP Windowing

• TCP window size is the maximum amount of data TCP will allow

outstanding (“in-flight”)

• If window size is less than or equal to outstanding data, TCP will not send

data

• In trace below, server is waiting for ACKs before sending more data

Client

test

Server




2/10/



TCP Windowing

• TCP uses the minimum size of the three windows

– Congestion control window

• Changes during connection

• Internal to sending TCP

• Not visible in real world

• Lives in the “mind” of the sender

– Receive buffer

• May change during connection

• User-configurable

• Visible as WIN= in decodes

test – Send buffer

• Controlled by sending

application

• Not visible in decodes

• Rarely the bottleneck

Congestion Control Window

Receive Buffer

In-Flight Data


TCP Congestion Control Window andSlow Start

• Estimate of how much data the network can handle

• Sending TCP increases the congestion control window each time it receives

an ACK

– Congestion control window increases exponentially every round-trip time

• Actually stored in bytes: 1460, 2920, 5840, 11680

• When TCP first starts transmitting data, it goes slowly

– Sender starts by sending one segment, then waits for ACK

test




2/10/



TCP Congestion Control Window in Slow Start

CWND = 1Data (S1)

Sender Receiver

CWND = 2

CWND = 3

ACK (S2)

Data (S3)

Data (S2)

Data (S5)

Data (S4)

Data (S6)

ACK (S4)

test

=

CWND = 7

ACK (S8)

Data (S7)

Data (S8)Data (S9)

ACK (S10)

Data (S10)

ACK (S12)

Data (S11)Data (S10)


Bulk Data Transfer

• Congestion window is greater than receive window

• TCP sends data as soon as it receives acknowledgements

• In-flight data is constant

• In-flight data = outstanding data = unacknowledged bytes

• Data transfer limited by receive window

test

When you see a pattern like this, blame TCP275 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.



2/10/



Review: TCP Windowing

• TCP limits the amount of outstanding data

• Minimum of three windows is used

– Send window: How much data the sender can handle

– Receive window: How much data the receiver can handle

– Congestion control window: How much data the network can handle

• Congestion window

– Initially small (1, 2, or 4 segments)

– Grows exponentially during slow start

test


LAB 4

LAB 4

Examining

TCP Behavior

test




2/10/



Lab 4: Examining TCP Behavior

• Use AppTransaction Xpert to examine TCP behavior

• You will see

– Connection opens

– Slow start

– Congestion control window

• Raise your hand, and a TA will assist you with any questions


test


Lab 4: Conclusion

• TCP uses SEQ and ACK numbers to ensure reliable delivery

• Minimum of three windows (congestion, receive, and send) determines

whether TCP sends more data

• Comparison of bulk and chatty applications

– Window size and bandwidth determine throughput for bulk applications

– Latency determines throughput for chatty applications

test




2/10/



When to Suspect TCP as a Bottleneck

• You should suspect TCP of causing problems when:

– You cannot explain why an application is running slowly

– The servers and network are adequately provisioned

• Servers have low load

• Sufficient bandwidth

• Links not congested

– The ACE Live™ Response Time Composition Chart shows

• Large Payload Transfer (Xfer) Times

• Large Retransmission Delays

– AppTransaction Xpert ’s AppDoctor Summary of Delays tab shows

test• Large Network Transfer Effects

• Large Protocol delays

– AppTransaction Xpert ’s AppDoctor Diagnoses point to• One or more TCP-related bottlenecks or potential bottlenecks


Troubleshooting TCP with ACE Live- Excellent for focusing troubleshooting, monitoring TCP health

• Where to start looking, where and when to capture

• Measures connection resets, packet loss, long response times, and more

ACE Live Response Time Composition Chart

– Lar e Retransmission Dela

• May be sign of packet loss

• May also be unfiltered duplicate packets

– Large Payload Transfer Times

• May be TCP-related

• May also indicate bandwidth, congestion or even server processing causes

test




2/10/



Troubleshooting TCP with AppTransaction Xpert

• AppDoctor provides quick way to check for TCP problems

• Suspect TCP if the Summary of Delays shows

– Large Network Transfer Effects

– Large Protocol delays

• Use diagnosis tab to identify which

specific TCP-related issue affects

your trace

– Bottlenecks

– Potential bottlenecks

test


Understanding Network Transfer Effects

• Combination of three effects related to transferring data

– Bandwidth, Congestion, Protocol

• Bottleneck bandwidth is needed to separate these phenomena

– Refine Network Effects key to identification of delays due to the TCP protocol

test




2/10/



How to Identify TCP Problems

• Look at the AppDoctor Diagnosis

tab

– Effect of Protocol

– Effect of Network Transfer

– Last 7 stats deal with TCP

– Helps target the specific

TCP issue

• Look at decodes for long

application transactions

test

• oo n e ne wor

chart at the start and

end of long delays


How to Identify TCP Problems (cont.)

• Double-click in the Summary of Delays for more information

• Use separate drilldown screens for Protocol and Congestion

test




2/10/



How to Identify TCP Problems (cont.)

• Congested networks also can cause protocol delays

• TCP implements congestion avoidance• Single-sided captures of receiving data may make congestion look like protocol delay or tier processing

test


Review: Analyzing TCP Behavior

• Look for TCP problems in traces with large amount of protocol delay

• Congestion delays also induce protocol delays

• AppDoctor (forensic)

– Separately diagnoses protocol and congestion

– Diagnoses several TCP problems

• QuickPredict (what if?)

– Separately models protocol and congestion

– Can simulate changing TCP Window Size

test




2/10/



Review: ACE QuickPredict

• Quick simulation engine – Combination of analytical and discrete event simulation

– Uses analytical models of TCP behavior • Initially shows a pristine view of network • • Outputs graphs of response time vs. one of the following:

– Bandwidth – Latency – Packet loss – Link utilization – TCP receive window size – NEW: WAN Acceleration parameters

test• QuickPredict Bar Charts supports

asymmetric window sizes


Review: QuickPredict Input and Output

• Y axis is always response time

• X axis is a range of values for one of:

– Bandwidth, Latency, Packet Loss, Link Utilization, TCP Window size,

and WAN Acceleration ( NEW!)

Change values of other variablesto see impact on response time

test




2/10/



Multi-User QuickPredict

test


Multi-User QuickPredict (MUQP)

• Single-user QuickPredict considers one client only

• MUQP allows you to have more than one

– Client and client site

– Applications (either ACE traces or ACE Whiteboard traces)

– Data centers

• Output

– Response time (including

– delay components)

– Utilization

• Resultstest – Average

– N-th percentile




2/10/



Inputs to MUQP

• Deployed applications (defined in ACE and/or ACE Whiteboard files)

• Network topology (number of client sites, clouds, and data centers)

• Tier-pair parameters (bandwidth, latency, link utilization, window size, and

packet loss)

• Application deployment options (number of transactions per hour for each

client site)

test


Outputs of MUQP

• "Average-case" response times for applications

• "Worst-case" results for access links (utilization) and applications (total

response time and components of tier processing, latency, protocol, and

congestion delay)

test




2/10/



Instructor Demo of MUQP

test


Bandwidth-Delay Product

test




2/10/




• Bandwidth-delay product — Defined as amount of data (in bytes) that can besent before ACK is received for the first unacknowledged bit of data

– 2 x one-way latency x bandwidth of slowest link • To “fill the pipe,” TCP window size must be equal to bandwidth-delay product• If window is too small, TCP must wait for acknowledgment before sending new data• Example below

– Bandwidth: 1,544 kbps = 193 KB/sec – Latency: 250 ms = 0.250 sec – BWDP: 2 x .25 x 193 = 96.5 KB – Receive Window: 8 KB (1/12 ideal size) – Throughput limited to: 130 kbps (1/12 link speed)

test


Capacity = .096 s × 125 KB/s = 12 KB (8 segments)


Sender Receiver

-

96 ms

test

1 Mbps




2/10/



LAB 5

LAB 5

Troubleshooting

a Slow File Transfer

test


Lab 5: Troubleshooting a Slow File Transfer

• Overview

– Computer manufacturer moving bulk data from Chicago to Taipei has dedicated

T1 without any packet drops

– Not seeing expected response time

– You are a network engineer

– Your manager: “It must be the network”

– Contemplating (expensive) link upgrade

• Objective

– Determine the cause of slow

response time

test – Propose and test a solution





2/10/



Lab 5: Conclusion

• Simple TCP configuration change improved performance

• Over same time period:

– Large receive window

test

– Small receive window

• What is the downside of making this change?


Changing the Receive Window

• How do I change the receive window on my computer?

– Windows

• http://www.microsoft.com/technet/network/tcpip/default.mspx

• http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip

2k.mspx

• Google search: windows tcp/ip implementation details

– Solaris

• http://www.sean.de/Solaris/soltune.html#water

– Linux

• http://www.speedguide.net/read_articles.php?id=121

test• Default for modern operating systems is 64 KB

– Maximum advertised receive window value in TCP packet is 64 KB

– How do we handle today’s large capacity , high latency pipes?

• TCP Window Scaling




2/10/



TCP Window Scaling

• TCP allows for WIN= field to be scaled by a power of 2

– Decodes will show WIN<<2=262140 – WIN<< x means multiply field by 2 x

•

– Both sides must agree and understand this option

test

• What if a packet trace does not contain the opening of a connection? – AppTransaction Xpert cannot detect the presence of TCP Window scaling


Detecting Packet Loss

test




2/10/



TCP and Packet Drops

• Where do packets get dropped, and why?

– Media collisions

– Wireless signal interference

– Congested networks

• Endpoints must react to overloaded routers

– Drops are a form of indirect communication between routers and senders

• Routers have multiple strategies for keeping queue size in check

– Tail drop

– Random Early Detection (RED)

test


Packet Loss and Retransmissions

Packet is dropped by the network and retransmitted

Time between when it should have arrived and when it did arrive

Controlled by TCP timers

oun e as ro oco e ay

Retransmission also invokes congestion control

Slows down throughput

Counted as Protocol Delay

ClientT = 0 2 3 4 5 61 7 8 9 10

A B B

test

Server

0 2 3 4 5 61 7 8 9 10

Protocol Delay




2/10/



Drops and Retransmissions

• TCP must make sure all bytes of a stream are received

– TCP guarantees delivery by retransmitting packets presumed dropped

• Two methods b which TCP will retransmit a dro ed acket

–Retransmission timeout

• Sending TCP schedules a timer for outstanding packets

• Timer is cancelled when the packet is acknowledged

• If timer expires, packet is presumed missing, re-sent

–Triple duplicate ACKs

• ACK number indicates next expected packet

test• If receiving TCP does not get the expected packet, it ACKs every packet

• If receiving TCP does not get the expected packet, ACK number does not

increase.

• If sending TCP receives acknowledgements with the same ACK number, it

knows a packet was likely lost


TCP One Loss: Retransmission TimeoutSender Receiver

Data S104

Data (S103)

Data (S102)

Data (S101)

ACK (S103)

ACK (S105)

Data (S108)

Data (S107)

Data (S106)

Data (S105)

ACK (S106)

test

Data (S106)

ACK (S109)

Retransmission

timeout

ACK (S106)




2/10/



TCP One Loss: Triple Duplicate ACK

Sender Receiver

Data (S103)

Data (S102)Data (S101)

Triple duplicate

Because 104 was

not received,

Receiver does not

change ACK

number

Data (S104)

ACK (S104)

Data (S107)

Data (S106)

Data (S105)

ACK (S104)

ACK (S104)

Data (S108)

test

retransmit

Data (S104)

ACK (S109)

ACK (S104)


Cost of Loss Detection• Retransmission timeout

– There are not enough ACKs for triple duplicate ACK – Cost: Timer (often large) plus time to send 1 packet – Congestion window decreases to one segment

• Triple-duplicate ACKs – Cause fast retransmit – Cost: Time to send 1 packet

Triple-duplicate ACK

Retransmission

test

• Notice the send times Retransmission




2/10/



Review: Detecting Packet Loss

• Two mechanisms

– Retransmission time-out (RTO)

• Costly

• Always works

– Triple-duplicate ACK

• Relatively efficient

• Does not always happen

• Chatty applications rarely “qualify”

• Food for thought

– Why acknowledge every packet during loss?

test – Why not quintuple-duplicate ACK?

– Why not single-duplicate ACK?


Frozen Window

test




2/10/



TCP Frozen Window

• Receiver stores packets in receive buffer

• Advertised receive window “WIN=” reduced by data in buffer

• Normal applications remove data quickly

• What happens if:

– A server is handling too many clients?

– What if the server is running other applications?

– What if a another thread is hogging the CPU?

test


TCP Frozen Window (cont.)

• As WIN decreases, buffer fills up, sender cannot send data

• Connection becomes “frozen”

• Sender will send probe packets to see if window size has increased

test




2/10/



TCP (Almost) Frozen Windows

• As WIN decreases, sender cannot send data

• What if the WIN is less than MSS?

• Sender could send data, but often does not

• Detection of “chilly windows” new to AppTransaction Xpert 16.0

test


Drilling into Protocol (cont.)

test

• Frozen Windows:

– Receiving WIN goes to zero

– Sending side sends “probe” packets




2/10/



Nagle’s Algorithm

test


Nagle’s Algorithm

• Nagle’s Algorithm

– TCP mechanism to reduce sending of many tiny messages

– Sender may not send data unless a full, MSS-sized segment can be sent

OR there is no unacknowledged data

• Delayed Acknowledgement

• Interaction of these two mechanisms sometimes causes problems

• Ridiculous situation where both sides wait for the other to do something

Client

test

B

Server

A B

200 ms317 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.



2/10/



Nagle’s Algorithm Example

• Client sends segment A (less than MSS)

• Client wants to send segment B, but needs to wait for ACK

• Server is waiting for another segment before sending ACK

Eventually, server sends ACK

Client then sends segment B

Process repeats…Client

test

B

Server

A B

200 ms318 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Nagle’s Algorithm Example (cont.)

• What causes the delay for this application?

test




2/10/



Nagle’s Algorithm and Delayed ACK

• To eliminate delays when sending consecutive small messages:

– Rewrite the application

• Sending multiple small consecutive messages is inefficient

• Sometimes you can’t change the application

– Disable Nagle’s algorithm

• Done within the application

• Effects only the application changed: (TCP_NODELAY socket option)

• Not always possible to change an existing application

– Acknowledge every packet

• Practical real world alternative

test• On Windows: http://smallvoid.com/article/winnt-nagle-algorithm.html

• Warning: Applies to every application on the machine!

– Interacts with congestion control window, affecting bulk applications


References

• TCP in OPNET – Online documentation – Overview PDF

• Model Library/Standard / TCP Model Usage Guide

– n ow ng ott enec on ne ocumentat on

• Applications in OPNET – Several whitepapers including SMARTE

• Help / Product Documentation / Modeling Methodologies / Methodologies and CaseStudies

• http://www.psc.edu/networking/projects/tcptune/ – Setting system wide TCP window sizes – Listing of TCP features for each operating system

• http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.html

test

–• http://www.sean.de/Solaris/soltune.html

– Tuning your TCP/IP stack on Solaris

• TCP/IP Illustrated, Volume I, Richard Stevens




2/10/



Take-Away Points

• Windowing causes TCP to wait before transmitting application data

• Bulk data-transfer throughput often limited by TCP windowing

• TCP frequently does not use the full available bandwidth

• Packet loss can cause significant reduction in throughput

• QuickPredict is a fast, easy, accurate way to predict application response time

• Use DES to model and study detailed behavior

• Simulate using the proper TCP machine settings

Next step: Run homework labs for a greater understanding of TCP

test


What’s New ??

test




2/10/



What’s New in AppTransaction Xpert ??

• 64-Bit Support

– Access more that 4GB of memory on 64-bit machines which allows you analyze

much larger capture files

• Windows 7 Professional (32-bit and 64-bit)

• Capture Enhancements

– Manage captures from ACE Live appliances within the Application Capture

Manager. New ACE Live tab to quickly download and preview data

– ACE Capture Agent 3.9 Ability to filter the traffic data by time range, IP

addresses, and/or IP conversations.

– Capture File Filters Greater flexibility in specifying hosts and ports.

test• Support for Riverbed Transparent Mode

– The AppTransaction Xpert import now supports capture files from Riverbed

accelerators running in full or port transparency mode


What’s New in AppTransaction Xpert (cont.)

• Streamlined Import Process

– Quickly open capture files through the user interface without specifying any information

(such as location, bandwidth, latency, etc.)

• Bandwidth Latenc and Locations can bes ecified as needed , ,

– Automatic Filtering of Duplicate Packets and TSO (TCP Segmentat ion Offloading)

– Simplified Manual Merge Workflow

• QuickPredict Enhancements - WAN Acceleration Support

– Support for WAN optimization technologies by allowing users to predict behavior for

compression, caching, and application acceleration

test

na ys s n ancemen s

– Automatic detection of HTTP on non-standard ports (thus eliminating the need to perform

redecodes or modifying config files)

– Improved analysis for packet sl icing and missing packets

– HTTP-specific visualization of object response times in TreeView




2/10/



References

test


Additional Documentation• Online at:

www.OPNET.com / Support / Support Center

–Documentation

–Release Notes

• Also look at: http://userforums.opnet.com

–Info on ALL OPNET products

–Discussion Groups

test – nts tr c s




2/10/



OPNET Training Courses

•OPNET provides Instructor-led training for all of our products. –OPNET Trainin Facilities

• Bethesda, Cary, Dallas, Nashua, Santa Clara

–On-site at YOUR facilities –On the Web via WebEx

•For AppTransaction Xpert : –2-day Instructor-led-training

test•Register for courses online at:

www.opnet.com/Training


OPNET Certification Program

Become recognized as an expert user of OPNET products…

Certification covers relevant topics such as:

•

• Configuration

• Product Selection

• Product Operation

• Networking Knowledge

U on successful com letion:test• Letter

• Certificate

• Individual certification ID

• Valid for 2 years




2/10/



AppTransaction Xpert Online Training

test


OPNET Web Pages - Training Resource

test




2/10/



OPNETWORK is the largest event of its kind,

attracting thought leaders from industry, government,

test

332

RSVP at: www.opnet.com/opnetwork2010

,

advance best practices for:

• Application Performance Management

• Network Engineering, Operations, and Planning• Network R&D


Course Evaluation

We’ll use this information to improve our courses and teaching methods. Pleaseenter as much comment material as you’d like… the more info you add, the better we’ll get!

Point your browser to:

http://eval.opnet.com

Please enter Instructor and Course info…

– Course: Introduction to AppTransaction Xpert

– Course Date:

– Instructor Name:

FREE

Stuff !!

test

Thank You in advance for your participation!




2/10/

test


Documents

AppTransaction Xpert