Advanced Troubleshooting the Cisco …d2zmdbbm9feqrf.cloudfront.net/2011/las/pdf/BRKAPP-3003.pdfSyslog's, SNMP, … ARP, DHCP relay High-Availability ACL Compilation ACE30 data plane

BRKAPP-3003

Advanced Troubleshooting the Cisco Application Control Engine

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 2

Session Agenda

ACE Architecture

Discuss the Architecture

Functions of control plane and data plane

Common debugging commands

Packet Capturing and logging

Traffic Forwarding on ACE

Admin Context and ACL Merge

Flow Management

Connection Handling on ACE

Layer 4/7 Troubleshooting and Performance

Health Monitoring on ACE

High Availability on ACE


ACE Architecture


ACE20 Module Hardware Architecture

Switch

Fabric

Interface

16G

SSL

Crypto

10G

2G

Console

port

Sup

Connect

100M

Control

Plane

Network

Processor 1

Network

Processor 2

10G10G

Classification

Distribution

Engine

(CDE)


ACE30 Module Hardware Architecture

Switch

Fabric

Interface

16G

2G

Console

port

Sup

Connect

100M

Control

Plane

8G

Daughter Card 1

NP1 NP2

8G

Daughter Card 2

NP3 NP4

Classification

Distribution

Engine

(CDE)


2x 700MHz MIPS

1 GB Memory

Control Plane Software

Supervisor

Connection

DBUS

16 Gbps

Bus

RBUS

EOBC

Cisco

ASIC

100 Mbps 8 Gbps

8 Gbps

1 Gbps

ACSW OS

60Gbps switching Capacity

IPv4, IPv6 Classifications

TCP Checksum

Generation/Verification

Variable Load Distribution

Daughter Card 1

16 Gbps

CEF720 Linecard

20 Gbps

20 Gbps

Switch Fabric

ACE30 Detailed Hardware Architecture

CPU

Classification Distribution

Engine (CDE)

NetworkProcessor

1

Verni FPGA

DRAM 4 GB

DRAM 4 GB

NetworkProcessor

2

shared memory

Daughter Card 2

NetworkProcessor

3

Verni FPGA

DRAM 4 GB

DRAM 4 GB Network

Processor4shared memory

Cavium Octeon CN5860 (OcteonPlus)

16 core, 600 MHz CPUs with 4G DRAM 32k iCache, 16k dCache, 2MB L2 cache

On chip support for Encryption/Decryption Coprocessors for Compression/Decompression


Data Traffic vs Management Traffic

ACE30 Control plane architecture is very similar to ACE20

Device control

Configuration manager (CLI, XML API, SSH, …)

Server health monitoring (native probes, TCL scripts)

Syslog's, SNMP, …

ARP, DHCP relay

High-Availability

ACL Compilation

ACE30 data plane architecture is very similar to ACE 4710

Connection management

TCP termination

Access lists

NAT

SSL Offload

Regular expression matching

Load Balancing & forwarding


Common Debugging


Common Debugging

Show commands on the Catalyst 6500 Supervisor

show version

show clock

show module

show power

show asic slot <n>

show interface TenGigabitEthernet <n>/1

show interface TenGigabitEthernet <n>/1 trunk

show svclc vlan-group

[no] power enable <module>

show svclc module <n> traffic

Show fabric utilization detail

Make sure the module status is OK

VLAN‘s used by ACE must be

configured in the MSFC


Common Debugging

Show commands available on ACE

show version

show cde health

show ft group status

show ip int br

show int vlan <n>

show arp

show service-policy

show serverfarm

show rserver

show probe

show conn

show stat

show ip traffic

show resource usage

show np 1 me-stats “-s norm”

show np 1 me-stats “-s norm –M1”

System Information

L2, L3

Performance,

Resources

Debugging

Flows

L4, L7

This provides the DELTA

If incorrect version, check ‗boot‘ parameter


Show Module from the Catalyst 6500 Supervisor

cat6k#show mod

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 1 Application Control Engine 10G Module ACE20-MOD-K9 SAD12345678

2 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD12345L44

5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD12345D5L

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0001.0002.0003 to 0001.0002.000a 2.4 8.7(0.22)ACE A2(2.3a) Ok

2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok

5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD123456N2 1.3 Ok

5 MSFC3 Daughterboard WS-SUP720 SAD123455VE 2.1 Ok

Mod Online Diag Status

---- -------------------

1 Pass

2 Pass

5 Pass

Module status shows OK


Verifying Version and Licenses

ACE/Admin# show version

Cisco Application Control Software (ACSW)

<snip>

Software

loader: Version 12.2[121]

system: Version A2(2.3a) [build 3.0(0)A2(2.3a)

system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3a.bin

installed license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9

Hardware

Cisco ACE (slot: 1)

cpu info:

number of cpu(s): 2

cpu type: SiByte

cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz

cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz

Installed Licenses


Available System Memory and Uptime

ACE/Admin# show version – Continuation of output

[...]

memory info:

total: 827128 kB, free: 335372 kB

shared: 0 kB, buffers: 3540 kB, cached 0 kB

cf info:

filesystem: /dev/cf

total: 1014624 kB, used: 529472 kB, available: 485152 kB

last boot reason: NP 2 Failed: NP ME Hung

configuration register: 0x1

ACE kernel uptime is 7 days 23 hours 42 minute(s) 25

second(s)

Displays ACE module uptimeUseful information in

case of system reload


ACE File System

Use the dir command to view directory listing for files

ACE/Admin# dir ?

core: Directory or filename

disk0: Directory or filename

image: Directory or filename

probe: Directory or filename

volatile: Directory or filename

The internal File system is mapped as below

/mnt/cf - Image:

Also the following compressed file systems are used

/TN-HOME = disk0:

/TN-CONFIG = Startup config

/TN-LOGFILE = Internal Storage for audit logs

/TN-CERTKEY-STORAGE : internal storage for Cert and Keys

/TN-COREFILE = core:


ACE File System

Load debug plug-in to access ACE file system

Startup configuration located at /mnt/cf/TN-CONFIG

ACE will generate / fix any missing or corrupted file systems during boot

When to use the format command?

If you receive the following error

Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!!

ACE/Admin# write memory

ERROR!config filesystem is not mounted on compact flash


Working with Core Files

If ACE creates a core file you can locate the files in the core directory

All cores files are stored in dir core: (core names are self explanatory)

ACE/Admin# dir core:

99756 Apr 1 17:57:05 2011 ixp2_crash.txt

13047 Apr 1 17:56:59 2011 loadBalance_core_log.tar.g

ixpx_crash.txt will have some details on the core dump

If it is a kernel crash , then a file named crash info will be available in core

Show version will show last reload reason


System Logging


Logging Features

Each virtual context generates logs independently and sends to specified destinations

Syslog server, console, buffer, SNMP station, etc..

Rate limiting of syslog messages is recommended. Never log to the console using level 7

ACE can log connection setup/teardown at the connection speed

Access-List deny entries are logged

Use the terminal monitor command to display log message when not using console

Useful commands to troubleshoot syslog issues:

show logging statistics show logging history

show logging queue

Make sure logging queue size is set properly


Basic Configuration to Enable Logging

Enable logging on the ACElogging enable

logging timestamp

logging monitor 4

logging trap 4

logging buffer 4

logging history 4

logging queue 1024

no logging message 111008

It is recommended to disable or change the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command

To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages to the syslog server


Real-Time ―TCP Dump‖



Supportability and analysis of load balanced traffic is a major requirement in today's load balanced environment

ACE can capture real-time packet information for the network traffic that passes through it

The attributes of the packet capture are defined byan ACL

The ACE buffers the captured packets, and you can copy the buffered contents to a file in flash memory on the ACE or to a remote server

User can also display the captured packet information on your console or terminal; capture can also be exported and viewed using Ethereal or Wireshark



To enable the packet capture on ACE use the capturecommand

capture c1 interface vlan 211 access-list FILTER bufsize 64

Buffer in Kbytes

(can be circular)

Pre-defined ACL to

identify relevant traffic

Interface to apply

capture

One capture session per context

Capture triggered at flow setup

Capture configured on client interface where flow is received



ACE can capture traffic based on a configured access-list and interface

Follow the following procedure to capture traffic on ACE:

1. Specify an ACL

2. Capture on an interface or globally

access-list FILTER line 10 extended permit tcp any any eq www

capture c1 interface vlan 211 access-list FILTER

Show capture status show status and buffer size

ACE/Admin# show capture c1 status

Capture session : c1

Buffer size : 64 K

Circular : no

Buffer usage : 1.00%

Status : stopped



Start the capture on the ACE

ACE/Admin# capture c1 start

23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58:

172.16.11.190.443 > 209.165.201.11.1180: S

1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460>

(ttl 255, id 2401, len 44, bad cksum 0!)

23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54:

172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408

(ttl 255, id 2402, len 40, bad cksum 0!)

ACE/Admin# capture c1 stop

To copy the packet capture to disk0: use the copy capture

ACE/Admin# copy capture c1 disk0: c1

Maximum buffer size is 5MB of data


Traffic Forwarding on ACE


ACE Load Balancer Policy Lookup Order

There can be many features applied on a given interface, so feature lookup ordering is important

The feature lookup order followed by data path in ACE is as follows:

1. Access-control (permit or deny a packet)

2. Management traffic

3. TCP normalization/connection parameters

4. Server load balancing

5. Fix-ups/application inspection

6. Source NAT

7. Destination NAT

The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface


Checking VLAN Configuration

Show interface provides you with valuable information ACE/Admin# show interface vlan 248

vlan248 is up

Hardware type is VLAN

MAC address is 00:16:36:fc:b3:36

Virtual MAC address is 00:0b:fc:fe:1b:02

Mode : routed

IP address is 172.16.10.21 netmask is 255.255.255.0

FT status is active

Description:WAN Side

MTU: 1500 bytes

Last cleared: never

Alias IP address is 172.16.10.23 netmask is 255.255.255.0

Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0

Assigned on the physical port, up on the physical port

499707 unicast packets input, 155702918 bytes

1485258 multicast, 5407 broadcast

0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops

497610 unicast packets output, 46804782 bytes

6 multicast, 8201 broadcast

0 output errors, 0 ignored


MAC Addresses

Virtual MAC (VMAC) is used for the alias IP, VIP address

Alias IP and Virtual IP (VIP) are associated with a VMAC only if high availability is configured

Active context responds to ARPs for alias IP with VMAC

One unique VMAC per FT Group 00:0b:fc:fe:1b:XX(XX=FT group number in hex)

Packets destined to the VMAC are blocked on standby context


MAC Addresses

The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids

Use the show interface internal iftable to locate the VMAC

Each ACE supports 1,024 shared VLAN‟s, and uses only one bank of MAC addresses randomly selected at boot time

ACE‟s may select the same address bank so avoid this conflict use the shared-vlan-hostid command


Admin Context Resource Reservation



If Admin context is not configured correctly, Admin could be starved of all resources

Default resource class in ACE has no minimum allocation

When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc

Highly recommended to put some safeguard in place to ensure that the Admin context always receives at least a small percentage of resources



Shows starved resources and drops for throughput

ACE/Admin# show resource usage context Admin

Allocation

Resource Current Peak Min Max Denied

-------------------------------------------------------------------------------

Context: Admin

conc-connections 9 9 0 0 0

mgmt-connections 2 12 0 0 0

proxy-connections 0 0 0 0 0

xlates 0 0 0 0 0

bandwidth 0 4715 0 0 3704068

throughput 0 4247 0 0 3704068

mgmt-traffic rate 0 468 0 125000000 0

connection rate 0 7 0 0 8

ssl-connections rate 0 0 0 0 0

mac-miss rate 0 1 0 0 0

inspect-conn rate 0 0 0 0 0

acl-memory 26816 26880 0 0 0

sticky 0 0 0 0 0

regexp 0 0 0 0 0

syslog buffer 1024 4096 0 1024 0

syslog rate 0 7 0 0 118

No resources reserved



Suggesting the following reserved resources for Admin

resource-class Admin

limit-resource conc-connections min 5.00 max equal-to-min

limit-resource mgmt-connections min 5.00 max equal-to-min

limit-resource rate bandwidth min 5.00 max equal-to-min

limit-resource rate ssl-connections min 5.00 max equal-to-min

limit-resource rate mgmt-traffic min 5.00 max equal-to-min

limit-resource rate conc-connections min 5.00 max equal-to-min


Access-Control Lists and ACL Merge


ACL Merge Process and Enhancements

ACL merge is responsible for merging all the features and generating a single merged list for any given interface. ACL compiler is responsible for programming the merged list into MTrie data structure – “Fast retrieval of data”

ACL memory usage has been optimized to better support incremental changes

The new implementation provides a consistent ACL memory usage during system boot up time and during incremental changes after the system comes up

This feature also provides an early detection of failure if the configuration needs more ACL resources than available

Also, note ACL masks are in 255.255.x.x format (not 0.0.y.y)


View Total Action Nodes

Use the show np 1 access-list resource to view action nodes

ACE/Context3# show np 1 access-list resource

ACL Tree Statistics for Context ID: 3

=======================================

ACL memory max-limit: None

ACL memory guarantee: 0.00 %

MTrie nodes(used/guaranteed/max-limit):

6 / 0 / 262143 (compressed)

2 / 0 / 21999 (uncompressed)

Leaf Head nodes (used/guaranteed/max-limit):

3 / 0 / 262143

Leaf Parameter nodes (used/guaranteed/max-limit):

7 / 0 / 524288

Policy action nodes used: 4

memory consumed: 4696 bytes resource-limited 128 bytes other

4824 bytes total.

min-guarantee: 0 bytes total.

max-limit: 78610432 bytes total, 0 % consumed


Connection Handling in ACE


Flow Management

Level of Flow Processing Type of Processing Function

Layer 3 and Layer 4 Balance on first packet Basic Load Balancing

Applies to TCP/UDP for layer 4 rules Source IP Sticky

Applies to all other IP protocols TCP/IP Normalization

Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules based on first

request (URL LB)

Buffer request, inspect, LB Cookie Sticky (Persistence)

Create Hardware Shortcut Generic TCP Payload Parsing

Layer 7 Re-proxy TCP Splicing + ability to parse

subsequent HTTP requests within

the same TCP

HTTP Layer 7 rules with HTTP

1.1 connections keepalive

(“persistence rebalance”)

Layer 7 Full-Proxy Fully terminate clients connection SSL Offload

TCP re-use

HTTP 1.1 Pipelining

Protocol Inspection (FTP,SIP)


Internal Mapping of TCP/UDP Flows

TCP and UDP Flows = 2 X Internal Half Flows

ACE/Admin# show conn

conn-id np dir proto vlan source destination stat

-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+

9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB

6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB

Client IP:port VIP Address

Server IP Returning half flow

automatically created for

both TCP and UDP flows

INIT, SYNACK,

ESTAB, CLOSED

SYN_SEEN, SYN_SEEN,

ESTAB, CLOSED

Non TCP shows as ―--‖

Use conn-id

to track flow

through ACE

Check the

Network

Processor


Troubleshooting Connections

Use the show stats connection command to show connections statistics

Use the clear stats connection command to clear these counters

ACE/Admin# show stats connection

+------------------------------------------+

+------- Connection statistics ------------+

+------------------------------------------+

Total Connections Created : 288232

Total Connections Current : 2

Total Connections Destroyed: 283404

Total Connections Timed-out: 892

Total Connections Failed : 3934

Note: ACE does not destroy connection. These connections are closed correctly!!!


Troubleshooting Connections

Use the show stats loadbalance command to view the load balance statistics

To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command

ACE/Admin# show stats loadbalance

+------------------------------------------------------------+

+------- Loadbalance statistics ----------------------+

+------------------------------------------------------------+

Total version mismatch : 0

Total Layer4 decisions : 0

Total Layer4 rejections : 0

Total Layer7 decisions : 24

Total Layer7 rejections : 0

Total Layer4 LB policy misses : 0

Total Layer7 LB policy misses : 0

Total times rserver was unavailable : 0

Total ACL denied : 0


Troubleshooting Individual Connections

Use the NP and connection ID from ‟show conn‟ command to view the front-end and back-end connection statistics using show np <#> me-stats “-c <connection ID> -v”

ACE/Admin# show np 1 me-stats “-c 4096 –v”

+------------------------------------------------------------+

+------- Individual connection statistics -------------------+

+------------------------------------------------------------+

Connection ID:seq: 4096[0x1000].2

Other ConnID : 8194[0x2002].14

Proxy ConnID : 0[0x0].0

Next Q : 0[0x0]

10.1.1.22:23 -> 12.2.2.14:8739 [RX-NextHop: TX] [TX-NextHop: CP]

Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No

L3 Protocol : IPv4 L4 Protocol : 6

Inbound Flag : 0

Interface Match : Yes

Interface MatchID:24

……… <snip>


Troubleshooting Individual Connections

To further debug and check if the traffic pattern matches the correct rule, the following command can be used: show np 1 access-list trace vlan <inbound vlan> in protocol <IP protocol #> source <source IP> <source port or „0‟> destination <destination IP> <destination port>

ACE/Admin# show np 1 access-list trace vlan 10 in protocol 6

source 10.10.10.1 0 destination 10.20.30.40 80

<snip> <look for NAT pool ID, vserver ID, etc.>

src nat 0x0 dst nat 0x0 vserver 0x66 fixup 0x0

<snip> <vserver ID here is 0x66 or 102 decimal>

Now, the internal vserver ID 102 can be looked up in the config: ACE/Admin# show cfgmgr internal table l3-rule | inc 102

102 224 249 0 0 DATA_VALID

Internal Policy Map # is 224 and Class Map # is 249: ACE/Admin# show cfgmgr internal table policy-map | inc 224

224 MyPolicy9 0 DATA_VALID

ACE/Admin# show cfgmgr internal table class-map | inc 249

249 MyClass4 0 DATA_VALID


Troubleshooting VIPACE/Admin# show service-policy client-vips detail

Status : ACTIVE

Description: -

-----------------------------------------

Interface: vlan 211

service-policy: client-vips

class: VIP-HTTPS

VIP Address: Protocol: Port:

172.16.11.190 tcp eq 443

loadbalance:

L7 loadbalance policy: HTTPS-POLICY

VIP Route Metric : 77

VIP Route Advertise : DISABLED

VIP ICMP Reply : ENABLED-WHEN-ACTIVE

VIP State: INSERVICE

curr conns : 22 , hit count : 22

dropped conns : 0

client pkt count : 0 , client byte count: 0

server pkt count : 0 , server byte count: 0

max-conn-limit : 0 , drop-count : 0

conn-rate-limit : 0 , drop-count : 0

bandwidth-rate-limit : 0 , drop-count : 0

L7 Loadbalance policy : HTTPS-POLICY

class/match : class-default

LB action :

primary serverfarm: backend-ssl

backup serverfarm : -

hit count : 22

dropped conns : 0


Troubleshooting Serverfarm

Use this command for checking server status and load

ACE/Admin# show serverfarm HTTPS-FARM detail

serverfarm : HTTPS-FARM, type: HOST

total rservers : 4

active rservers: 4

description : -

state : ACTIVE

predictor : ROUNDROBIN

failaction : -

back-inservice : 0

partial-threshold : 0

num times failover : 0

num times back inservice : 0

total conn-dropcount : 0

----------connections-----------

real weight state current total failures

---+---------------------+--------+---------------------+-----------+-----

-

rserver: linux-1

192.168.1.11:0 8 OPERATIONAL 10 1000 1

max-conns : - , out-of-rotation count : -

min-conns : -

conn-rate-limit : - , out-of-rotation count : -

bandwidth-rate-limit : - , out-of-rotation count : -

retcode out-of-rotation count : -


Layer 7 Troubleshooting


Layer 7 Policy Hits

Expanding the show service-policy using the detail option to provide hit count for layer 7 matches

ACE/Admin# show service-policy client-vips detail

Status : ACTIVE

Description: -

-----------------------------------------

Interface: vlan 211

service-policy: client-vips

<snip>

L7 Loadbalance policy : pslb

class-map : curl1

LB action :

serverfarm: s1

hit count : 3

dropped conns : 0

class-map : curl2

LB action :

serverfarm: s2

hit count : 0

dropped conns : 0

Shows hit count for layer 7

load balanced policy


Match URL Hit Count

show service-policy url-summary option to provide visibility on which match http url‟s are getting hits

ACE/Admin# show service-policy url-summary

Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: VH-01

match http url /ACCOUNTING/.* hit: 42

Service-Policy: VIRTUAL-HOSTING-02 L3-Class: WEB-SSL L7-Class: VH-02

match http url /BUSINESS/.* hit: 93

match http url /SALES/.* hit: 102

match http url /SPECIAL/.* hit: 67

match http url /BUSINESSOBJECTS/.* hit: 78

match http url /CUSTOMERS/.* hit: 84

Use the show service-policy <service-policy-name> class-map <L3-class map-name> url-summary for more granularity


Troubleshooting HTTP

To effectively troubleshoot HTTP use the show stat http commandACE/Admin# show stats http

+------------------------------------------+

+-------------- HTTP statistics -----------+

+------------------------------------------+

LB parse result msgs sent : 6288 , TCP data msgs sent : 9143

Inspect parse result msgs : 0 , SSL data msgs sent : 6041

TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19

SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0

Drain msgs sent : 3107 , Particles read : 37917

Reuse msgs sent : 1539 , HTTP requests : 3145

Reproxied requests : 0 , Headers removed : 1549

Headers inserted : 1598 , HTTP redirects : 2

HTTP chunks : 0 , Pipelined requests : 0

HTTP unproxy conns : 0 , Pipeline flushes : 0

Whitespace appends : 0 , Second pass parsing : 0

Response entries recycled : 3032 , Analysis errors : 0

Header insert errors : 15 , Max parselen errors : 0

Static parse errors : 9 , Resource errors : 0

Invalid path errors : 0 , Bad HTTP version errors : 0


Troubleshooting HTTP Cookies

ACE parses HTTP requests for cookies with the name given in the configuration and can skip a certain number of bytes and look for another specific number of bytes.

If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value.

If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm

ACE can parse HTTP headers (includes cookies) up to 64KB (default header max parse length is 4KB)

Make sure that sticky timeout (note this is more like an idle timeout) matches the session timeout on the application


Troubleshooting TCP Reuse

When using TCP connection reuse,"Connection: keep-alive" is inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early

User needs to configure Source NAT in the policy map when using TCP connection re-use

Use the show stats http | include Reuse counters to check if see if TCP Reuse is in effect

ACE/Admin# show stats http | include Reuse

Reuse msgs sent : 1 , HTTP requests : 4

„sh conn detail‟ will also show information about server side connection reuse


Troubleshooting HTTP Compression


HTTP Compression Overview

ACE uses Cavium Octeon zip engine

Implement deflate block as defined in RFC 1951

History buffer is supported to achieve better compression ratio

Support two output file formats. GZIP (RFC1952) or X-

GZIP (RFC2616) and ZLIB (aka DEFLATE) RFC1950

Compression is used with HTTP connection only

Compression only supports HTTP 1.1 protocol

No decompression support

Mainly for text based content (html,xml…)

Feature available on ACE 4710 and ACE30


ACE Compression Traffic Flow Example

2. ACE rewritesClient‟s request

GET / HTTP/1.1

Accept-Encoding: gzip,

deflate

1. Request before ACE

GET / HTTP/1.1

Accept-Encoding: identity

Request after ACE

4. ACE Inspects response

HTTP/1.1 200 OK

Content-type: text/html

Content-Encoding: deflate

Transfer-Encoding:

chunked

6. Response after ACEServer sends uncompressed HTTP payload of 5963 bytes

7. Client receives compressed HTTP payload 2577 bytes

Cisco ACEClientLAN

HTTP/1.1 200 OK

Content-type: text/html

Content-Length: 5963

3. Response before ACE

5. ACECompresses

Response

Server

WAN


Debugging HTTP Compression

Check the following for errorsFrom client side:

1. Accept-Encoding is not present or has invalid type

2. User-Agent is being excluded from the configuration

3. HTTP version is not 1.1 or higher

From server side

4. Invalid HTTP response header

5. HTTP response code not 200

6. Content type is not allowed

7. Content length is too small

8. Chunk encoding has invalid format

Additional details using show np x me-stat “-s http”

Get request from client:

GET HTTP/1.1

Host: www.yahoo.com

User-Agent: Mozilla/5.0 Windows; U; Windows NT 5.1;

Accept: text/html,application/xhtml+xml,

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive


Troubleshooting Secure Socket Layer (SSL)


Troubleshooting SSL

Configuration of SSL on ACE is relatively simple. However if you experience an issue, how to troubleshoot?

Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify command

ACE/Admin# crypto verify RSA2048.key RSA2048.cert

Keypair in RSA2048.key matches certificate in RSA2048.cert

Check the size and location of the key. Use the show crypto key command

ACE/Admin# show crypt key all

Filename Bit Size Type

-------- -------- ----

RSA2048.key 2048 RSA


Troubleshooting SSL

Review the certificate details. Use the show crypto certificate command

ACE/Admin# show crypto certificate cisco-sample-cert

Certificate:

Data:

Version: 3 (0x2)

Serial Number:

ad:e4:e2:f1:50:b7:ce:bd

Signature Algorithm: sha1WithRSAEncryption

Issuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=AAAA, CN=SSL-TEST

Validity

Not Before: Apr 3 09:50:55 2009 GMT

Not After : Apr 1 09:50:55 2019 GMT

Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=AAAA, CN=SSL-TEST

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

RSA Public Key: (1024 bit)

Modulus (1024 bit):

00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:

26:af:7a:05:49:ed:8d:93:3b

Exponent: 65537 (0x10001)


Troubleshooting SSL – CRL Download

Check to make sure you can download the CRL

ACE/Admin# show crypto crl test2 detail

test2:

URL: http://119.60.60.23/test.crl

Last Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTC

Total Number Of Download Attempts: 1

Failed Download Attempts: 0

Successful Loads: 1 Failed Loads: 0

Hours since Last Load: 0 No IP Addr Resolutions: 0

Host Timeouts: 0 Next Update Invalid: 0

Next Update Expired: 0 Bad Signature: 0

CRL Found-Failed to load: 0 File Not Found: 0

Memory Outage failures: 0 Cache Limit failures: 0

Conn failures: 0 Internal failures: 0

Not Eligible for download: 3 HTTP Read failures: 0

HTTP Write failures: 0

To look for all best-effort CRLs in the system and their

download status, use show crypto crl best-effort

http://119.60.60.23/test.crl


Advanced SSL Debugging

This command provides the current crypto statistics ACE/Admin# sh np 1 me-stats "-s crypto”

Crypto Statistics: (Current)

------------------

Internal Error: 172 0

ARC4 operations: 376572 0

TCP msgs received: 285260 0

APP msgs received: 235151 0

Nitrox messages forwarded to XScale: 381041 0

SSL ctx allocated: 47758 0

SSL ctx freed: 47758 0

SSL received bytes: 61070430 0

SSL transmitted bytes: 283256220 0

SSL received application bytes: 7679113 0

SSL transmitted application bytes: 275120867 0

SSL received non-application bytes: 53391317 0

SSL transmitted non-application bytes: 3292887 0

Bulk flush operations: 95037 0

ME records sent to XScale: 285808 0

ME records received from XScale: 47723 0

ME hw responses: 471516 0

First segments received: 47400 0

Handshake failure alert: 94 0

CM close: 446 0


Advanced SSL Debugging

The show stats crypto server command provides statistics of the SSL handshake

ACE/Admin# show stats crypto server

+---- Crypto server termination statistics -----+

+------- Crypto server alert statistics --------+

+--- Crypto server authentication statistics ---+

+------- Crypto server cipher statistics -------+

+------ Crypto server redirect statistics ------+

+---- Crypto server header insert statistics ---+

Other useful commands:

show crypto hardware (applies only to ACE20)

show crypto cdp-errors




Fundamentals for ACE probing

ACE probes are fundamental to the system. It is key to not oversubscribe the ACE health monitoring system

Note both the primary and standby ACE send out probes; also, the interface IP is used and not the alias IP

Some key probe parameters:

faildetect <number of consecutive failed probes>

interval <time interval between probes>

open <timeout for completing 3-way TCP handshake>

receive <timeout for receiving a probe response from the server>


Fundamentals for ACE probing

Use the show resource internal socket to determine how many sockets ACE has open. This is an Admin command

ACE/Admin# show resource internal socket

Application MaxLimit Current Creates Frees

--------------------------------------------------------------

SYSTEM 4000 0 0 0

CRITICAL 50 0 0 0

AAA 256 0 0 0

MGMT 256 0 0 0

XINETD 512 1 12 11

HEALTH_MON 2500 532 193494 192962

USER_TCL 200 0 0 0

SYSLOG 256 10 14 4

VSH 256 0 0 0

OverAll - 650 194812 194162

Non Reg App Usage: 107


Health Monitoring Process

If you see probe related issues, check the health monitoring process. The show proc cpu command provides useful information

ACE/Admin# show proc cpu

CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%

PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process

972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr

HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is consuming CPU

ACE/Admin# show proc cpu | inc hm

CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%

PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process

987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm

988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm

989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm

990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm



Use the show probe detail command to determine the status of the probe or possible last failure

ACE/Admin# show probe detail

--------------------- probe results --------------------

probe association probed-address probes failed passed health

------------------- ---------------+----------+----------+----------+-------

rserver : CAS1

10.7.53.55 24 24 0 FAILED

Socket state : CLOSED

No. Passed states : 0 No. Failed states : 1

No. Probes skipped : 0 Last status code : 403

No. Out of Sockets : 0 No. Internal error: 0

Last disconnect err : Received invalid status code

Last probe time : Wed Nov 25 18:48:16 2009

Last fail time : Wed Nov 25 18:25:16 2009

Last active time : Never


High Availability on ACE


High Availability Basic building blocks

FT PEER

Only one FT peer per ACE device

1:1 peer relationship

FT GROUPOne FT group per ACE virtual context

FT VLAN

Designated VLAN between the redundant peers

All HA related traffic sent over this VLAN

FT VLAN can be trunked between two Catalyst 6500 Chassis

Should not be used for normal traffic

Admin Context

Context A

Context B

Context A

Context B

ACE2 (FT PEER)

FT VLAN

FT Group 2

FT Group 3

ACE1 (FT PEER)

FT Group 1


High Availability Control Traffic

TCP Connection between FT Peers

State Machine (Election, Preempt, Relinquish)

Configuration sync

State Sync for ARP

Heartbeats between FT peers

Heartbeats are sent over UDP

Monitors the health of the peer

Heartbeat interval and count are configurable


FT Heartbeats

If heartbeats missed is increasing, heartbeats are not reaching the peer. Possibility for both ACE‟s to go Active/Active

ACE/Admin# sh ft stats

HA Heartbeat Statistics

------------------------

Number of Heartbeats Sent : 1095573

Number of Heartbeats Received : 1092586

Number of Heartbeats Missed : 2987

Number of Unidirectional HB's Received : 2640

Number of HB Timeout Mismatches : 0

Num of Peer Up Events Sent : 1

Num of Peer Down Events Sent : 1

Successive HB's miss Intervals counter : 0

Successive Uni HB's recv counter : 0


ACE High Availability State Machine

STANDBY_BULK State

ARP Sync (knob to turn on/off)

Connection Table Sync

Sticky Database Sync (knob to turn on/off)

STANDBY_HOT State

Standby FT group member is ready to take over

Incremental Configuration Sync from Active to Standy

Incremental State Sync from Active to Standby

STANDBY_COLD State

Due to error during Config Sync or SSL certs mismatch

No Config or State Sync happens from Active to Standby‟

STANDBY_WARM State

Major version mismatch between peers (example 2.x and 4.x)


ACE High Availability State Machine

Mismatch in software version

FT Peer may become INCOMPATIBLE

Could result in ACTIVE ACTIVE state on both FT group members

Mismatch in Virtual Context Licenses

Configuration Sync (all types) for Admin context is disabled

State Sync for Admin context will continue to happen

For matching user contexts – Configuration State Sync will work

Mismatch in Other Licenses

Configuration and State Sync will work

After switchover, new Active will handle traffic as per its licenses


Preventing Active-Active Scenarios

When no heartbeat is received, ACE can use the Query Vlan to check the HA status

ACE tries to do a ping to the destination via the Query VLAN

If ping fails, the Standby will transition to the ACTIVE state

If ping succeeds, the Standby will transition to a STANDBY_COLD state

To configure a query interface, enter the following:

ACE/Admin(config-ft-peer)# query-interface vlan 110


More debugging commands


Additional Debugging

Some more ACE debugging commands

show np <#> me-stats -cpu

show np <#> me-stats –Q

show np <#> me-stats “-s fp”

show np <#> me-stats “-s tcp”

show np <#> me-stats “-s icm”

show np <#> me-stats “-s ocm”

show proc cpu

show netio stats

Show service-policy summary


6

Receive 25 Cisco Preferred Access points for each session evaluation you complete.

Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don‟t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

Complete Your Online Session Evaluation

http://www.ciscolivevirtual.com/


7

Visit the Cisco Store for Related Titles

http://theciscostores.com

http://theciscostore.com/


Recommended Reading

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 79

Thank you.


Appendix and Additional Troubleshooting Information


Additional Information

Layer 4 flow setup

Layer 7 flow setup

TCP Connection States


Layer 4 Flow Setup

SYN

SYN_ACK

Shortcut

ACK

Shortcut

Data

Shortcut

Data

Shortcut

Matches Existing

Flow

Rewrites L2/L3/L4

Matches VIP

Selects Server

Rewrites

L2/L3/L4

Basic Load Balancing

Source IP sticky

TCP/IP Normalization


Layer 7 Flow SetupClient Connects to “L7” VIP

SYN

Starts

BufferingACK

Data

ACK‘s Client Packets

Keeps Buffering

Matches VIP w/L7

logic

Chooses SEQ #

Replies w/SYN_ACK

HTTP L7 rules on first request

(cookie sticky, URL parsing, …)

Generic TCP payload parsing


Layer 7 Flow Setup—ContinueACE Establishes Connection to Server

Data

SYN_ACK

Empties Buffer

Sends Data to Server

Acts as Client

Does Not Forward

SYN_ACK

Parses the Data

Selects Server

Initiates TCP


Layer 7 Flow Setup—Continue ACE Splices the Flows (UNPROXY)

ACK

Data

Shortcut

ACK

Shortcut

Data

Shortcut

Matches Existing Flow

Rewrites L2/L3/L4

and SEQ/ACK

Does Not Forward ACK

Ready to

Splice the Flows


Layer 7 Flow SetupACE Reproxies the Connection

ACK

Data

ACK

Data

Shortcut

…ACK

…Shortcut

Shortcut

ShortcutData

REPROXY

ACK‘s GET & Buffer…

HTTP L7 rules with HTTP 1.1

connection keepalive

(―persistence rebalance‖)


Layer 7 Flow SetupACE Acts as a Full Proxy

Fu

ll Pro

xy

Ind

ep

en

den

t clie

nt &

se

rve

r co

nn

ec

tion

s

SYNSYN_ACK

ACK

DataGET/HTTP 1.1

ACK SYN

SYN_ACKACK

Data—GET

ACK

ACKData

DataHTTP/1.1 200 OKHTTP/1.1 200 OK

Client connection Server connection

… …

SSL offload

TCP re-use

Protocol inspections

HTTP 1.1 pipelining


TCP Connection States

L4 TCP Connections

SYNSEEN (Client „SYN‟ received)

INIT (Server side half flow initialized)

SYNACK („SYN ACK‟ sent by server)

ESTAB (Client and Server; TCP Handshake completed)

L7 TCP Connections

SYNSEEN (Client „SYN‟ received)

ESTAB (Client side TCP Handshake completed; „SYN ACK‟ sent by ACE, Client ACK received)

ESTAB (Server side TCP Handshake completed from ACE after L7 data received from the client and parsed)

CLOSED (Client or Server „FIN ACK‟ followed by ACK)

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 89

Thank you.

Documents

Advanced Troubleshooting the Cisco …d2zmdbbm9feqrf.cloudfront.net/2011/las/pdf/BRKAPP-3003.pdfSyslog's, SNMP, … ARP, DHCP relay High-Availability ACL Compilation ACE30 data plane