The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri · 2015-03-27 · #ATM15 | The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri March 2015 @ArubaNetworks

#ATM15 |

The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri

March 2015

@ArubaNetworks

CONFIDENTIAL © Copyright 2015. Aruba Networks, Inc. All rights reserved

2 #ATM15 |

WLAN Design, Configuration and Troubleshooting Tips by TAC

@ArubaNetworks

3 CONFIDENTIAL © Copyright 2015. Aruba Networks, Inc. All rights reserved

#ATM15 |

•  Segmental Troubleshooting •  AP Stability and System profile Optimization •  Optimize load on processes •  Datapath Debugging •  Deployment Tips

Aruba OS

@ArubaNetworks


#ATM15 |

Transition Content

#1

Segmental Troubleshooting

@ArubaNetworks


#ATM15 |

Transition Content


Segmental Troubleshooting gains. # Faster root cause analysis.

# One time Data Collection

# Bring focus on the smallest segment in the network within our control.

@ArubaNetworks


#ATM15 |

Transition Content


Where do we start if we are unsure of the exact cause of the current issue being faced?

@ArubaNetworks


#ATM15 |

Transition Content


User show tech-support user mac <Mac Address>

tar logs user mac <User Mac > tech-support

User Debugging Logging Level debugging user-debug <Mac

Address>

@ArubaNetworks


#ATM15 |

Transition Content


AP

show ap tech-support ap-name <Name of AP>

show ap debug counters

show ap bss-table ap-name

show ap debug system-status ap-name

@ArubaNetworks


#ATM15 |

Transition Content


Controller show tech-support

tar log tech-support

Outside world Debugging for Specific process/Sub-cat.. (Explained)

Pcap

show interface gigabitethernet <slot/module/port>

Network Diagram

Note: Show tech-support <filename> Store output in file.

@ArubaNetworks


#ATM15 |

Transition Content

Segmental Troubleshooting.. Processes

show process monitor statistics

Process Monitor Statistics

Name State Restarts Allowed Restarts Timeout Value Timeout Chances Time Started

/mswitch/bin/dbstart PROCESS_RUNNING 8 0 240 3 Sat Feb 28 21:31:55 2015

/mswitch/bin/packet_filter PROCESS_RUNNING - 0 240 3 Sat Feb 28 21:31:56 2015

Mdns , httpd_wrap , Authmgr ,STM , WMS , cfgm , dhcp

@ArubaNetworks


#ATM15 |

#2

AP Stability and System Profile Optimization

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization AP System Status.

Campus and Remote APs have similar challenges to stay connected to the controller.

Health Check of the AP is vital, since it can trigger client and controller anomalies.

Show AP debug system-status ap-name <Name of AP>

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization •  Reboot Information

DHCP/Controller/Keep Alive miss •  Rebootstrap Information

Date Time Reason (Latest 10)

LMS Change/Heartbeat Miss •  HA Failover Information

Date Time Reason (Latest 10)

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization •  Recent Control Messages from AP to Controller

Date Time Message Description

Sun Mar 1 12:29:49 2015(164 secs ago): SENT REQ type=KEEPALIVE len=45 peer=10.163.196.72 seq_num=4567 num_attempts=1 rtt=0 secs

•  Rebootstrap LMS

•  Crash Information

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization •  CPU and Memory Usage

Timestamp CPU Util(%) Memory Util(%) 2015-03-01 12:32:27 2 24

•  Peak CPU Util in the last one hour Timestamp CPU Util(%) Memory Util(%) 2015-03-01 12:19:25 3 24

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization Heartbeat Stats of Serving Controller Heartbeats Sent Sent Seqnum Heartbeats Received Rcvd Seqnum MTUs sent Misc sent Measurement Duration

2690183 25824 2667575 25824 22607 0 since last rebootstrap

2690193 n/a 2667575 n/a 22607 0 total since bootup

Interface counters Interface Rx_pkts Rx_errors Rx drops Tx_pkts Tx_errors Tx_drops Resets

wifi0 3209433 16822381 2230363 236918 61 0 0

wifi1 4096977 2070224 4095468 2242763 58 0 11

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization MTU Discovery Probes Responses Last Sent Last Rcvd

45214 22607 2712890 2712890

Switch MTU, 1500

Ethernet bonding SlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings

Autoneg Speed (Mbps) Duplex Iface

0 eth0 UP ACTIVE 0 on 1000 Full eth0

eth1 DOWN STANDBY 0 on 10 Half eth1

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization Controller Information Item Value

Primary LMS 10.163.196.72

Backup LMS 10.163.196.71

AP to Active Controller Message Information Item Value

AP state REGISTERED

Power Status Operational State : Unknown

Current HW State POE-AT: No restrictions

@ArubaNetworks


#ATM15 |

AP Stability and System Profile Optimization MTU Discovery Probes Responses Last Sent Last Rcvd

45214 22607 2712890 2712890

Switch MTU, 1500

Ethernet bonding SlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings

Autoneg Speed (Mbps) Duplex Iface

0 eth0 UP ACTIVE 0 on 1000 Full eth0

eth1 DOWN STANDBY 0 on 10 Half eth1

@ArubaNetworks


#ATM15 |

AP Stability Optimizations

@ArubaNetworks


#ATM15 |

AP Stability Optimizations

Heartbeat DSCP: Assign a DSCP value to AP heartbeats to prioritize heartbeats traveling over low-speed links. The supported range is 0-63, and the default value is 0.

Bootstrap threshold: Number of consecutive missed heartbeats on a GRE tunnel (heartbeats are sent once per second on each tunnel) before an AP rebootstraps. On the controller, the GRE tunnel timeout is 1.5 x bootstrap-threshold; the tunnel is torn down after this number of seconds of inactivity on the tunnel.

SAP MTU: Maximum Transmission Unit, in bytes, on the wired link for the AP.

Spanning Tree: Select this checkbox to enable the Spanning Tree protocol.

@ArubaNetworks


#ATM15 |

# 3

Optimize load on processes

@ArubaNetworks


#ATM15 |

HTTPD

Stress on the webserver can be because of number of sessions in the initial role for guest access. This is either because of large certificate (Key Length 2048 or 4096) that are used by the server or a large number of devices (phones/Tablets with APPs) that generate HTTP/HTTPS sessions and get re-directed to the web-server.

show web-server profile

Web Server Configuration

Parameter Value

SSL/TLS Protocol Config tlsv1

Captive Portal Certificate GUEST-AUTH

User session timeout <30-3600> (seconds) 3600

Maximum supported concurrent clients <25-320> 75

Enable WebUI access on HTTPS port (443) true

Enable bypass captive portal landing page false

@ArubaNetworks


#ATM15 |

HTTPD show web-server statistics

Web Server Statistics:

Current Request Rate: 1 Req/Sec

Current Traffic Rate: 1 KB/Sec

Busy Connection Slots: 7

Available Connection Slots: 68

Total Requests Since Up Time: 284

Total Traffic Since Up Time: 1122 KB Avg.

Request Rate Since Up Time: 1 Req/Sec Avg.

Traffic Rate Since Up Time: 6144 Bytes/Sec

Server Scoreboard: _____________KKKKKK_W_____________

Scoreboard Key: _ - Waiting for Connection, s -

Starting up R - Reading Request, W - Sending Reply K - Keepalive, D - DNS Lookup C - Closing connection, L - Logging G - Gracefully finishing, I - Idle cleanup of worker . - Open slot with no current process

@ArubaNetworks


#ATM15 |

STM and WMS

STM

Station Management is responsible for all AP information and Station information. This process can be over run if there is,

# Aggressive polling from Airwave/SNMP servers for Wlan tables.

# Network wide AP reboot and bootstraps

# AP debug scripts run from the controller.

WMS

IDS/IPS events and frequent AP bootstraps could lead to WMS being busy. WMS is actively looking for RF information of WiFi devices(Rogue/Valid/Interfering).

@ArubaNetworks


#ATM15 |

Mitigation

Use AMON

WMS Offload to Airwave.

Reduce SNMP polling or increase the polling period

Disable WMS functionality if you do not require IDS/IPS functionality.

@ArubaNetworks


#ATM15 |

# 4

Datapath Debugging

@ArubaNetworks


#ATM15 |

Datapath Monitoring

Show datapath utilization show datapath utilization

Datapath Network Processor Utilization

| Cpu utilization during past |

Cpu | 1 Sec 4 Secs 64 Secs |

10 | 99% | 99% | 99% |

11 | 0% | 0% | 0% |

12 | 0% | 0% | 0% |

13 | 0% | 0% | 0% |

14 | 0% | 0% | 0% |

15 | 0% | 0% | 0% |

16 | 0% | 0% | 0% |

show datapath frame 10 |SUM/| | | |

|CPU | Addr | Description Value |

+----+------+-----------------------------------------------------+

| 10 | [00] | Allocated Frames 1040|

| 10 | [01] | Max Allocated Frames 2208 |

| 10 | [03] | Unknown Unicast 147074970|

| 10 | [34] | Flood Frames 1506164167|

+----+------+-----------------------------------------------------+

| 10 | [00] | Rx Frames 635394472|

| 10 | [01] | Rx Bytes 1864525959|

| 10 | [02] | Tx Frames 1240985989|

+----+------+-----------------------------------------------------+

@ArubaNetworks


#ATM15 |

Datapath Bandwidth Management show datapath bwm Datapath Bandwidth Management Table Entries

Type Id Bits/sec Policed Bytes Bytes Flags CPU Status

---- ---- --------- ---------- ------- ----------- ------- ------- ------

0 1 20000000 0 78125 0/0 9 ALLOCATED

0 2 4000000 0 15625 0/0 9 ALLOCATED

0 3 160000000 0 624890 0/0 9 ALLOCATED

0 4 4000000 0 15625 0/0 9 ALLOCATED

0 5 2000128 0 7813 0/0 9 ALLOCATED

0 6 2000128 0 7813 0/0 9 ALLOCATED

0 7 2000128 0 7813 0/0 9 ALLOCATED

Firewall: Rate limit CP untrusted ucast traffic Enabled 20 Mbps

Rate limit CP untrusted mcast traffic Enabled 4 Mbps

Rate limit CP trusted ucast traffic Enabled 160 Mbps

Rate limit CP trusted mcast traffic Enabled 4 Mbps

Rate limit CP route traffic Enabled 2 Mbps

Rate limit CP session mirror traffic Enabled 2 Mbps

Rate limit CP auth process traffic Enabled 2 Mbps

@ArubaNetworks


#ATM15 |

CP and DP Packet Capture

•  Wifi -Client

packet-capture datapath wifi-client aa:aa:aa:aa:aa:aa all

•  VIA client/RAP packet-capture datapath ipsec <peer-ip>

•  Generic traffic to controller packet-capture controlpath tcp/udp 4343

@ArubaNetworks


#ATM15 |

# 5

@ArubaNetworks

Deployment Tips Missing optimizations


#ATM15 |

Deployment Tips

Honey Comb Pattern

Wireless

Local Probe Threshold = 25

Transmit Power of AP 5Ghz Min Tx – 12 Max Tx – 15

2.4 Ghz Min Tx – 6 Max Tx – 9

Avoid Asymmetric RF

The difference between minimum and maximum Tx power on the same radio should not be more than 6dbm

DMO Enable

Basic and Beacon rate

802.11a 5Ghz – 24

802.11g 2.4Ghz – 12

80 Mhz Channel bonding - DFS Channels

@ArubaNetworks


#ATM15 |

Deployment Tips.. Contd

GRE Stripping IP - VRRP for LMS and Stripping IP

Jumbo Frames - Enabled

802.3at

Airgroup

Dot1x

OKC

Validate PMK ID

802.11r/k/v

EAPOL Rate Optimization

@ArubaNetworks


#ATM15 |

Deployment Tips… ASE

https://ase.arubanetworks.com/

@ArubaNetworks


#ATM15 |

Deployment Tips..

ASE for troubleshooting

@ArubaNetworks

36 #ATM15 |

Network Services AirWave ClearPass

@ArubaNetworks


#ATM15 |

Transition Content

# 6

ClearPass Platform: System Cleanup Options

@ArubaNetworks


38 #ATM15 |


•  Free disk space threshold is a config in Cluster Wide Service Parameter. Default 30%

•  A system cron job runs every hour and checks the disk utilization. If the free space falls below the configured threshold, an alert is logged into the system. NOW in addition, the following aggressive cron cleans up anything more than 1 day old in version 6.5 of CPPM

•  Log database records

•  Core files

•  System load monitor files

•  Application and system log files

•  Auto and manual backup files

•  Stored reports

•  Expired guest accounts

•  Audit records


39 #ATM15 |



40 #ATM15 |


We also introduced some new CLI commands –  Check on disk-space and memory usage - “show sysinfo”… –  system cleanup [# of days to retain] **This is an on-demand task


41 #ATM15 |


•  Same command function also exist in the GUI –  Remember this is an on-demand task


#ATM15 |

# 7

@ArubaNetworks

ClearPass Platform : Graphite


#ATM15 |

•  Graphite is a new reporting tool to compliment Insight in CPPM from 6.3 version.

•  Graphite runs on every node irrespective of standalone or cluster and statistics can be viewed from any node.

•  Performance monitoring Display is disabled by default and should be enabled manually and set access permission levels accordingly.

•  To access Graphite data, use the URL https://<CPPM IP Address>/graphite



#ATM15 |

•  Make sure Performance monitoring is enabled from GUI



#ATM15 |

•  Setting up access to Graphite from CPPM UI



#ATM15 |

•  We can allow or deny any networks to access Graphite for a node or cluster.

•  Make sure stats collection is set true True under Service parameters.



#ATM15 |



#ATM15 |



#ATM15 |

Transition Content

# 8

ClearPass : Upgrade Utility Tool

@ArubaNetworks


50 #ATM15 |


•  The Cluster Upgrade Tool is a simple user interface that automates the upgrade procedure for a ClearPass cluster.

•  When the upgrade is initiated, no manual actions are required until all selected nodes have been upgraded.

•  The Upgrade Tool is not available while the publisher is rebooted and migrating the Configuration Database.

•  The Upgrade Tool will not detect nodes that were upgraded manually without the tool.

•  If a configured standby publisher node was manually upgraded without the tool, the Upgrade Tool will not restore the state of the standby publisher configuration.


51 #ATM15 |


•  The Cluster Upgrade Tool is released as a patch update. It can be downloaded and installed either through Policy Manager’s Software Updates portal, or from the Aruba Support portal.


52 #ATM15 |


•  Log in to Policy Manager on the publisher node and go to Administration > Agents and Software >Updates > Software Updates.

•  When the installation is complete, the Admin service will be restarted. You do not need to reboot.


53 #ATM15 |


•  Before you begin the upgrade, the upgrade image must be present on the publisher node of the cluster.

•  Download the upgrade image to the publisher under Software updates


54 #ATM15 |


•  To monitor the progress of the other nodes in the cluster, wait until the database migration is complete and then log in to the tool again.

•  Change the url to https://CPPM IP Address/upgrade

•  We should see all the subscribers status that are in sync.

•  The list of subscribers will be present and subscriber upgrades will go in parallel.

•  ‘Start Upgrade’ to start the upgrade on the servers.


55 #ATM15 |



56 #ATM15 |



57 #ATM15 |


•  Check the logs for each node by ‘View Logs’ next to each node and we can see the progress of patches and upgrades from publisher.


58 #ATM15 |

ClearPass : Upgrade Utility tool


#ATM15 |

# 9

@ArubaNetworks

AirWave – VisualRF Performance Tips


#ATM15 |

•  Sometimes we see VisualRF takes long time to show up new AP’s to deploy on the floor plans.

•  We can manually force VisualRF to poll the AP’s to get new AP or existing AP’s updated details.

•  NO need to restart VisualRF to show up new AP’s.

•  Change the url in AMP to https://<Airwave IP Address>/visualrf/poll_aps_now.xml



#ATM15 |

•  In 8.x moved to HTML5 for VisualRF UI for faster UI interaction with backend.

•  Whilst we are improving the features on the new UI, there are some features which were present in flash and not in HTML5.

•  In VisualRF > Setup page, we can switch between HTML5 and flash so that we can take advantage of options present in both version

•  Switching between HTML5 and Flash version is easy with below URL without refreshing VisualRF.

•  Change the URL to

https://Airwave IP Address/site?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725

from

https://Airwave IP Address/vrf?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725



#ATM15 |

AirWave – VisualRF Performance tips


#ATM15 |



#ATM15 |

•  We can change ‘vrf’ to ‘site’ on any page for VisualRF URL’s to switch to flash mode from HTML5 mode.

At times we see Heat maps not showing/updating properly in VisualRF

•  We can resize the floor plan to same size so that the grid calculation happens and heat maps will be re-drawn.

•  No need to restart VisualRF for heat maps to update.



#ATM15 |

•  Unlock the floor plan and go to properties and ‘Measure’

Airwave – VisualRF Performance Tips


#ATM15 |

•  Select the distance, click ‘OK’ and ‘Save’ without changing the distance, this will trigger floor plan to recalculate the heatmpas.

Airwave – VisualRF Performance Tips


#ATM15 |

Transition Content

# 10

Airwave – Tips for Data Retention Settings

@ArubaNetworks


68 #ATM15 |


•  Data in AirWave is primarily stored in 2 formats:

•  Postgres - an open source, relational SQL database. Usually, when you see data in tables, that data is stored in Postgres.

•  RRD Files - used for storing data that's displayed in time-sequence graphs (i.e, client count over the last year, bandwidth used over the last month). There can be many thousands of RRD files on a single AirWave server. One benefit of RRD is that its files have a fixed size. As data is inserted to an RRD file (like by an AirWave monitoring process), it does not grow. A downside of this is that the file starts using storage space as soon as it is created.


69 #ATM15 |


•  We can set data retention settings under AMP Setup > General page under the section ‘Historical Data Retention’.

•  Client Association and VPN Session History. This setting has a bearing on how much history we can show in the association history on the client historical table and how much data can be included in the user session data.

•  Its recommend to keep high because the data is useful


70 #ATM15 |

Airwave – Tips for Data Retention settings

•  Inactive Client and VPN User Data. This setting determines how long we keep the information on every client that has ever connected to the network.

•  This impacts how long we keep RRD files. Keeping it low can save disk space.


71 #ATM15 |


•  Client data retention Interval : This influences how much historical data you can see for each client in the graphs, for example the signal quality, usage graphs on the client detail/diagnostic page.

•  It's very important to keep this low, like in the 14-31 days range. •  This is especially important in public wi-fi deployments that will have lots of unique users. •  This setting controls what size RRD files are created to store per-user historical signal, usage,

goodput, health and other metrics. •  Keeping it low doesn't impact device, group and folder-level monitoring, and it doesn't have any

negative impact on reports. It only impacts the graphs on the client Detail and Diagnostic pages.


72 #ATM15 |


•  By default Rogues are kept forever which will impact the overall system performance and for RAPIDS page load.

•  It also impacts VisualRF for Rogue calculation if it has thousands of Rogue devices.

•  This is especially important in public wi-fi deployments that have open SSID and lot of nearby devices are detected as Rogues

•  Setting the value to low as 14 days will greatly help.


73 #ATM15 |


•  Airwave by default has 20+ Reports which runs daily. •  Keep Reports that are needed for the environment and delete/disable the default reports. •  Report retention setting can be costly in high dense environments especially for disk space. •  This increase the nightly backup file size, nightly maintenance time and report generation time. •  Exporting the reports via .csv or .pdf or emailing them is a good option. •  Keeping the retention value will have the pickled client tables not to grow huge in size and makes

report generation faster.


#ATM15 |

Questions

@ArubaNetworks

THANK YOU

75 #ATM15 | @ArubaNetworks

Documents

The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri · 2015-03-27 · #ATM15 | The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri March 2015 @ArubaNetworks