Cisco Application Virtual Switch Troubleshooting Guide ... · Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x) First Published: 2016-07-01 Last Modified:

Cisco Application Virtual Switch Troubleshooting Guide, Release5.2(1)SV3(2.x)First Published: 2016-07-01

Last Modified: 2018-02-08

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000 800 553-NETS (6387)Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITEDWARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITHTHE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain versionof the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDINGANYOTHERWARRANTYHEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS"WITH ALL FAULTS.CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OFMERCHANTABILITY, FITNESS FORA PARTICULAR PURPOSEANDNONINFRINGEMENTORARISING FROMACOURSEOFDEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUTLIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERSHAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, networktopology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentionaland coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnershiprelationship between Cisco and any other company. (1110R)

© 2016-2018 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

http://www.cisco.com/go/trademarks

C O N T E N T S

C H A P T E R 1 Cisco ACI Fabric and Cisco AVS Overview 1

Cisco ACI Fabric Overview 1

Cisco AVS Overview 2

About the Cisco AVS and the VMware vCenter 4

Cisco AVS in a Multipod Environment 5

Required Software 6

C H A P T E R 2 Overview of Troubleshooting 7

About the Troubleshooting Process 7

About Best Practices 7

Common Troubleshooting Tasks 8

Troubleshooting Basics 8

Troubleshooting Preliminary Steps 8

Verifying Ports 9

Verifying Layer 2 Connectivity 9

Contacting Cisco Customer Support 9

Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support 10

Manually Generating Log Files for Cisco Customer Support 10

Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI 11

Collecting and Exporting Cisco AVS Log Files Using the Basic GUI 11

Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLI 12

C H A P T E R 3 Installation and Configuration 15

Verifying Your VMware License 15

Recovering from a Cisco AVS Creation Failure 16

C H A P T E R 4 Adding Hosts 17

Ensuring That a VTEP vmknic Is Added for Valid Host 17

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x) iii

C H A P T E R 5 OpFlex 19

Enabling OpFlex 19

OpFlex Connection Sequence 21

Recovering from an OpFlex Failure Due to a Certificate Issue 23

C H A P T E R 6 Ports, Endpoint Groups, and Layer 2 27

About Data Paths 27

Diagnosing Port Activity 28

Troubleshooting Unavailable Ports 29

Checking Port Synchronization Using Port Counters 30

Troubleshooting Endpoint Groups 31

Recovering from Endpoint Group Creation Failure 32

Debugging Isolated Endpoints Within an EPG 32

Verifying That Policy to Deny Intra-EPG Communication Is Enabled 32

Tracking Statistics for Isolated Endpoints in an EPG 33

Troubleshooting Layer 2 Switching 33

C H A P T E R 7 Port Channels 37

Port Channel Overview 37

Verifying Port Channels 37

Troubleshooting Port Channel Creation 41

C H A P T E R 8 Switched Port Analyzer 43

About the Switched Port Analyzer 43

Viewing the Switched Port Analyzer Configuration 44

Troubleshooting the Switched Port Analyzer 45

C H A P T E R 9 Distributed Firewall 49

Distributed Firewall Flow Logging Troubleshooting 49

vemcmd show dfw flows {all|unreported} 49

vemcmd show dfwdenyflows {all|ltl_number} 51

vemcmd show dfwslflows {all|ltl_number} 51

vemcmd show dfw globals 52

vemcmd show dfw globals ltl ltl_number 52

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x)iv

Contents

vemcmd show dfw connection stats 52

vemcmd show dfwflows ltl ltl_number 52

vemcmd dpa show dfwlog config 53

C H A P T E R 1 0 System Troubleshooting 55

VEM Commands 55

Cisco AVS Troubleshooting with vemcmd show Commands 57

vemcmd show Commands for Troubleshooting 58

Cisco AVS Health Status 60

Faults Monitored for Cisco AVS 60

Viewing Faults for Cisco AVS 61

Viewing All Cisco AVS Faults by Domain or Type 61

Viewing Cisco AVS Faults by Type 62

Troubleshooting Cisco AVS Faults 63

Switch OpFlex Channel is down 63

Switch VMM Domain Config isn't downloaded 64

Host Process has crashed 64

Host Kernel has crashed 64

Port Link is down 65

VTEP IP isn't assigned 65

VTEP isn't pinned to PNIC 66

PNIC isn't pinned to VTEP 66

Port Attach isn't acked 66

Port Detach isn't acked 67

Port is quarantined 68

Port EPG isn't downloaded 68

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x) v

Contents

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x)vi

Contents

C H A P T E R 1Cisco ACI Fabric and Cisco AVS Overview

• Cisco ACI Fabric Overview, page 1

• Cisco AVS Overview, page 2

• About the Cisco AVS and the VMware vCenter, page 4

• Cisco AVS in a Multipod Environment, page 5

• Required Software, page 6

Cisco ACI Fabric OverviewThe Cisco Application Centric Infrastructure (ACI) includes Cisco Nexus 9000 Series switches with theApplication Policy Infrastructure Controller (APIC) to run in the leaf/spine ACI fabric mode. These switchesform a “fat-tree” network by connecting each leaf node to each spine node; all other devices connect to theleaf nodes. The APIC manages the ACI fabric. The recommended minimum configuration for the APIC is acluster of three replicated hosts. The APIC fabric management functions do not operate in the data path ofthe fabric. The following figure shows an overview of the leaf/spin ACI fabric.

Figure 1: ACI Fabric Overview

The ACI fabric provides consistent low-latency forwarding across high-bandwidth links (40 Gbps, with a100-Gbps future capability). Traffic with the source and destination on the same leaf switch is handled locally,and all other traffic travels from the ingress leaf to the egress leaf through a spine switch. Although thisarchitecture appears as two hops from a physical perspective, it is actually a single Layer 3 hop because thefabric operates as a single Layer 3 switch.

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x) 1

The ACI fabric object-oriented operating system (OS) runs on each Cisco Nexus 9000 Series node. It enablesprogramming of objects for each configurable element of the system.

The ACI fabric OS renders policies from the APIC into a concrete model that runs in the physical infrastructure.The concrete model is analogous to compiled software; it is the form of the model that the switch operatingsystem can execute. The figure below shows the relationship of the logical model to the concrete model andthe switch OS.

Figure 2: Logical Model Rendered into a Concrete Model

All the switch nodes contain a complete copy of the concrete model. When an administrator creates a policyin the APIC that represents a configuration, the APIC updates the logical model. The APIC then performs theintermediate step of creating a fully elaborated policy that it pushes into all the switch nodes where the concretemodel is updated.

The Cisco Nexus 9000 Series switches can only execute the concrete model. Each switch has a copy ofthe concrete model. If the APIC goes off line, the fabric keeps functioning but modifications to the fabricpolicies are not possible.

Note

The APIC is responsible for fabric activation, switch firmware management, network policy configuration,and instantiation. While the APIC acts as the centralized policy and network management engine for thefabric, it is completely removed from the data path, including the forwarding topology. Therefore, the fabriccan still forward traffic even when communication with the APIC is lost.

The Cisco Nexus 9000 Series switches offer modular and fixed 1-, 10-, and 40-Gigabit Ethernet switchconfigurations that operate in either Cisco NX-OS stand-alone mode for compatibility and consistency withthe current Cisco Nexus switches or in ACImode to take full advantage of the APIC's application policy-drivenservices and infrastructure automation features.

Cisco AVS OverviewThe Cisco Application Virtual Switch (AVS) is a key part of the Cisco Application Centric Infrastructure(ACI). It is a distributed virtual switch that offers different forwarding and encapsulation options and extendsacross many virtualized hosts and data centers defined by the VMware vCenter Server.

The Cisco AVS is integrated with the Cisco ACI architecture as a virtual leaf and is managed by the CiscoAPIC. The Cisco AVS implements the OpFlex protocol for control plane communication.

This section provides an overview of the Cisco AVS.

The Cisco AVS supports two modes of traffic forwarding: Local Switching mode, formerly known as Fexdisable mode; and No Local Switchingmode, formerly known as Fex enable mode. You choose the forwardingmode during Cisco AVS installation.

Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x)2

Cisco ACI Fabric and Cisco AVS OverviewCisco AVS Overview

Local Switching Mode

In Local Switching mode, all intra-EPG traffic is locally forwarded by the Cisco AVS, without the involvementof the leaf. All inter-EPG traffic is forwarded through the leaf. In this mode, the Cisco AVS can use eitherVLAN or VXLAN encapsulation—or both—for forwarding traffic to the leaf and back. You choose theencapsulation type during Cisco AVS installation.

Beginning with Cisco AVS Release 5.2(1)SV3(2.5), you can configure a single VMM domain in LocalSwitching mode to use VLAN and VXLAN encapsulation. Previously, encapsulation was determined solelyby the presence of VLAN or multicast pools, and you needed to have separate VMM domains for EPGs usingVLAN and VXLAN encapsulation.

If you choose VLAN encapsulation, a range of VLANs must be available for use by the Cisco AVS. TheseVLANs have local scope in that they have significance only within the Layer 2 network between the CiscoAVS and the leaf. If you choose VXLAN encapsulation, only the infra-VLAN needs to be available betweenthe Cisco AVS and the leaf. This results in a simplified configuration and is the recommended encapsulationtype if there are one or more switches between the Cisco AVS and the physical leaf.

Figure 3: The Cisco AVS in Local Switching Mode


Cisco ACI Fabric and Cisco AVS OverviewCisco AVS Overview

No Local Switching Mode

In No Local Switching mode, all traffic is forwarded by the leaf. In this mode, VXLAN is the only allowedencapsulation type.

Figure 4: The Cisco AVS in No Local Switching Mode

About the Cisco AVS and the VMware vCenterThe Cisco Application Virtual Switch (AVS) is a distributed virtual switch that extends across many virtualizedhosts. It manages a data center defined by the vCenter Server.

The Cisco AVS is compatible with any upstream physical access layer switch that complies with the Ethernetstandard, including Cisco Nexus switches. The Cisco AVS is compatible with any server hardware listed inthe VMware Hardware Compatibility List (HCL).

The Cisco AVS is a distributed virtual switch solution that is fully integrated within the VMware virtualinfrastructure, including VMware vCenter for the virtualization administrator. This solution allows the networkadministrator to configure virtual switch and port groups in order to establish a consistent data center networkpolicy.


Cisco ACI Fabric and Cisco AVS OverviewAbout the Cisco AVS and the VMware vCenter

The following figure shows a topology that includes the Cisco AVS with the Cisco Application PolicyInfrastructure Controller (APIC) and VMware vCenter.

Figure 5: Sample Cisco AVS Topology

Cisco AVS in a Multipod EnvironmentThe Cisco AVS can be part of a multipod environment. Multipod environments use a single APIC cluster forall the pods; all the pods act as a single fabric.

Multipod environments enable a more fault tolerant fabric comprising multiple pods with isolated controlplane protocols. They also provide greater flexibility in full mesh cabling between leaf and spine switches.

Cisco AVS does not require any additional configuration to operate in a multipod environment.

For detailed information about multipod environments, see the following documents on Cisco.com:

• Cisco Application Centric Infrastructure Fundamentals

• Cisco APIC Getting Started Guide

• Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

The following features are not supported for Cisco AVS with multipod in the Cisco APIC 2.0(1.x) release:

• L3 Multicast

• Storage vMotion with two separate NFS in two separate PODs

• ERSPAN destination in different PODs

• Distributed Firewall syslog server in different PODs


Cisco ACI Fabric and Cisco AVS OverviewCisco AVS in a Multipod Environment

Required SoftwareThe following table shows the versions of software you need to install for Cisco Application Virtual Switch(AVS) to work with the Cisco Application Policy Infrastructure Controller (APIC), VMware vCenter, andVMware ESXi hypervisor:

DescriptionComponent

Cisco AVS is supported in Release 4.2(1)SV2(2.3) and later releases.However, Release 5.2(1)SV3(1.5) or later is required if you want touse Distributed Firewall and Microsegmentation with Cisco AVS.

Cisco AVS software

See the Cisco AVS Release Notes for compatibility information.However, version 1.1(1j) or later is required with Cisco AVS5.2(1)SV3(1.5) or later if you want to use Distributed Firewall andMicrosegmentation with Cisco AVS.

Cisco APIC

Cisco AVS is compatible with release 5.1, 5.5, or 6.0 of VMwarevCenter Server.

VMware vCenter

Cisco AVS is supported as a vLeaf for the Cisco APIC with release5.1 and later releases of the VMware ESXi hypervisor.

When you choose a Cisco AVS VIB, you need to choose theone compatible with the version of VMware ESXi hypervisorthat you use. ESXi 5.1 uses xxix.3.1.1.vib, ESXi 5.5 usesxxix.3.2.1.vib, and ESXi 6.0 uses xxxx.6.0.1.vib.

Note

VMware vSphere bare metal

Cisco AVS is supported in VSUM Release 1.0 and later releases.Cisco Virtual Switch UpdateManager (VSUM)


Cisco ACI Fabric and Cisco AVS OverviewRequired Software

C H A P T E R 2Overview of Troubleshooting

This chapter contains the following sections:

• About the Troubleshooting Process, page 7

• About Best Practices, page 7

• Common Troubleshooting Tasks, page 8

• Troubleshooting Basics, page 8

• Contacting Cisco Customer Support, page 9

About the Troubleshooting ProcessTo troubleshoot the Cisco Application Virtual Switch (AVS), follow these general steps:

1 Gather information that defines the specific symptoms.

2 Identify all potential problems that could be causing the symptoms.

3 Systematically eliminate each potential problem (from most likely to least likely) until the symptomsdisappear.

About Best PracticesBest practices are the recommended steps you should take to ensure the proper operation of your network.We recommend that you follow these best practices:

• Maintain the same Cisco Application Virtual Switch (AVS) release across all network devices.

• Refer to the release notes for your Cisco Application Virtual Switch release for the latest features,limitations, and bugs.

• Verify and troubleshoot any new configuration changes after implementing the change.


Common Troubleshooting TasksUsing a given a set of symptoms on a network, you should be able to diagnose and correct softwareconfiguration issues and defective hardware with minimal disruption to the network. With help from thisguide, you can perform the following common troubleshooting tasks:

• Identify key Cisco AVS troubleshooting tools.

• Obtain and analyze protocol traces using Switched Port Analyzer (SPAN) or Ethanalyzer on the CLI.

• Identify or rule out physical port issues.

• Identify or rule out switch module issues.

• Diagnose and correct Layer 2 issues.

• Obtain diagnostic data for use by the Cisco Technical Assistance Center (TAC).

Troubleshooting BasicsThis section introduces questions to ask yourself when you are troubleshooting a problem with the Cisco AVSor connected devices. Use the answers to these questions to identify the scope of the problem and to plan acourse of action.

This section includes the following topics:

• Troubleshooting Preliminary Steps, on page 8

• Verifying Ports, on page 9

• Verifying Layer 2 Connectivity, on page 9

Troubleshooting Preliminary StepsTo discover a network problem, use the following general network troubleshooting steps:

Before You Begin

By answering the questions in this section and the following subsections, you can determine the paths youneed to follow and the components that you should investigate further.

Answer the following questions to determine the status of your installation:

• Is this a newly installed system or an existing installation? (For example, is it a new host, switch, orVLAN?)

• Has the host ever been able to see the network?

• Are you trying to solve an existing application problem? (For example, is the device too slow, is thelatency too high, the response time excessively long, or did the problem occur only recently?)


Overview of TroubleshootingCommon Troubleshooting Tasks

•What changed in the configuration or in the overall infrastructure immediately before the applicationsstarted to have problems?

Step 1 Gather information about the problems in your system. See individual sections for instructions on how gather information.Step 2 Verify Layer 2 connectivity. See Verifying Layer 2 Connectivity, on page 9.Step 3 Verify the configuration for your end devices (storage subsystems and servers).

Verifying PortsIn addition to gathering software-configured port information, answer the following questions to verify physicalport integrity:

• Are you using the correct media (such as copper, optical, or fiber)?

• Are the media broken or damaged?

• Are you checking a physical Ethernet port? If so, are you looking at the server, or are you looking at anupstream switch?

Verifying Layer 2 ConnectivityAnswer the following questions to quickly eliminate common problems with Layer 2 connectivity:

• Are the necessary interfaces in the same VLANs?

• Are all ports in a port channel configured the same for speed, duplex, and trunk mode?

Contacting Cisco Customer SupportIf you are unable to solve a problem after using the troubleshooting suggestions in this guide, contact a Ciscocustomer service representative for assistance and further instructions. Before you call, have the followinginformation ready:

• Version of the Cisco AVS software that you are running.

• Version of the VMware vSphere (ESXi) and vCenter Server software that you are running.

• Contact phone number.

• Brief description of the problem.

• Brief explanation of the steps that you have already taken to isolate and resolve the problem.


Overview of TroubleshootingVerifying Ports

Collecting and Exporting Cisco AVS Log Files for Cisco Customer SupportCisco Customer Support might ask you to provide log files from Cisco AVS.

In releases earlier than Cisco AVS Release 5.2(1)SV3(1.10), you issue a vem command to generate CiscoAVS log files into a tar file that you can provide to Cisco Customer Support.

In Cisco AVS Release 5.2(1)SV3(1.10) and later, you can use the APIC GUI or NX-OS style CLI to collectand export Cisco AVS log files to a designated remote server. However, be aware of the following:

• In Release 5.2(1)SV3(1.10), you can use the APIC GUI or NX-OS style CLI to collect and export logfiles if you are using IPv4 addresses, provided that the destination server supports IPv4 addresses.However, if you are using IPv6 addresses, you must issue a vem command to generate a .tar file.

• In Release 5.2(1)SV3(1.15) and later, you can use the APIC GUI or NX-OS style CLI to export log filesif you are using IPv4 addresses or IPv6 addresses, provided that the destination server supports IPv4 orIPv6 addresses.

For instructions, see the following sections in this guide:

• Manually Generating Log Files for Cisco Customer Support

• Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI

• Collecting and Exporting Cisco AVS Log Files Using the Basic GUI

• Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLI

Cisco recommends that you do not mix Cisco APICGUI configurationmodes (Advanced or Basic).Whenyou make a configuration in either mode and change the configuration using the other mode, unintendedchanges can occur. For example, if you apply an interface policy to two ports using Advanced mode andthen change the settings of one port using Basic mode, your changes might be applied to both ports.

Caution

Manually Generating Log Files for Cisco Customer SupportYou can use a vem command to generate Cisco AVS log files in a .tar file that you can send to Cisco CustomerSupport when asked.

Step 1 Log in to the Cisco AVS host.Step 2 Enter the following command: vem-support all

The resulting log files are generated in a tar file with the following format:cisco-vem-<year>-<monthday>-<time>-<hostname>.tgz.

Example:/tmp # vem-support -h/bin/vem-support [options] all

Options:-i Interactive; hit a key to continue each step-v Verbose; prints each command before executing


Overview of TroubleshootingCollecting and Exporting Cisco AVS Log Files for Cisco Customer Support

-z Dry run; just print what would be done-t <dir> Specify a directory where the data will be gathered.

Default: /tmp/tmp # vem-support allGenerated /tmp/cisco-vem-2016-0201-1657-localhost.cisco.com-module1.tgz/tmp # ls -ltotal 24588-rw-r--r-- 1 root root 15993309 Feb 1 16:58cisco-vem-2016-0201-1657-localhost.cisco.com-module1.tgz

Step 3 Copy the .tar file to wherever you need in order to send it to Cisco Customer Support.

Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI

Before You Begin

You must make sure that the destination server supports IPv4 or IPv6.

The OpFlex communication channel between Cisco APIC and the Cisco AVS host must be in active state.

Step 1 Log in to Cisco APIC, choosing Advanced mode.Step 2 Go to Admin > Import/Export.Step 3 In the Import/Export navigation pane, expand the Export Policies folder.Step 4 Right-click the AVS TechSupport folder and choose Create AVS TechSupport.Step 5 In the Create AVS TechSupport dialog box, in the Name field, enter a name.Step 6 In the Export Destination area, choose a destination from the drop-down list or create one if you have not already done

so.Step 7 In the AVS area, click the + icon.Step 8 From the drop-down list, choose the AVS hosts from which you want to collect log files.Step 9 Click UPDATE and then click SUBMIT.

The AVS TechSupport work pane displays the new policy.Step 10 Right-click the new policy, and choose Collect TechSupports.Step 11 Click the Operational tab to view the status of the collection.

What to Do Next

Retrieve the exported log files from the remote destination.

Collecting and Exporting Cisco AVS Log Files Using the Basic GUI

Before You Begin




The OpFlex communication channel between Cisco APIC and the Cisco AVS host must be in active state.

Step 1 Log in to Cisco APIC, choosing Basic mode.Step 2 Go to Admin > TechSupport/Core Export.Step 3 In the TechSupport navigation pane, right-click the AVS TechSupport folder, and choose Create AVS TechSupport.Step 4 In the Create AVS TechSupport dialog box, in the Name field, enter a name.Step 5 In the Export Destination area, choose a destination from the drop-down list or create one if you have not already done

so.Step 6 In the AVS area, click the + icon.Step 7 From the drop-down list, choose the AVS hosts from which you want to collect log files.Step 8 Click UPDATE and then click SUBMIT.

The AVS TechSupport work pane displays the new policy.Step 9 Right-click the new policy, and choose Collect TechSupports.Step 10 Click the Operational tab to view the status of the collection.

What to Do Next


Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLIThis section explains how to use the NX-OS style CLI to collect and export Cisco AVS log files. The examplecommands shown use an IPv4 address; if you are using Cisco AVS Release 5.2(1)SV3(1.15) or later, you canuse IPv4 or IPv6 addresses if the destination server supports them.

For information about accessing and using the NX-OS style CLI, see the Cisco APIC Getting StartedGuide.

Note

Before You Begin


Step 1 Define the destination on a remote server to receive exported Cisco AVS log files, providing the IP address, port, protocol,username, and filepath for the destination.

Secure Copy Protocol (SCP) is the only protocol supported for exporting Cisco AVS logfiles.

Note

Example:

apic1# configureapic1(config)#apic(config)# techsupport remote dest1 10.0.0.1 8000 scp username /tmp/<CR>

Step 2 Press Enter, enter a password, and then exit configuration mode.



Example:apic(config)# techsupport remote dest1 10.0.0.1 8000 scp username /tmp/Destination password: *********apic1#(config)# exit

Step 3 Export the Cisco AVS log files, providing the host ID number and remote destination.

Example:apic1# trigger techsupport host 167819358 remotename dest1

Step 4 View the progress of the export.

Example:apic1# show techsupport host 167819358 statusNodeid : 167819358Collection Time : 2015-10-16T19:03:43.528+00:00Status : preInitDetailed status : Waiting to be scheduled.

It can take several minutes for the export to complete. You might see status from previous times you enteredthe command.

Note

What to Do Next






C H A P T E R 3Installation and Configuration


• Verifying Your VMware License, page 15

• Recovering from a Cisco AVS Creation Failure, page 16

Verifying Your VMware LicenseYou can verify that your ESXi server uses the VMware Enterprise Plus license. This license includes thedistributed virtual switch (DVS) feature, which allows you visibility into the Cisco Application Virtual Switch(AVS).

Before You Begin

• Ensure that you are logged into vSphere Web Client on the VMware vSphere (ESXi) server.

• Ensure that you are logged into the Cisco AVS.

Step 1 From vSphere Web Client, choose the host whose Enterprise Plus license you want to check.Step 2 Examine the Enterprise Plus licensed features:

a) Click the Configuration tab.b) Choose Licensed Features.

Step 3 Verify that the following features are included in the licensed features:

• An Enterprise Plus license.

• The DVS feature.

Step 4 If your ESXi server does not have an Enterprise Plus license, upgrade your VMware license to an Enterprise Plus licenseso that you can have visibility into the Cisco AVS.


Recovering from a Cisco AVS Creation FailureAfter you create a VirtualMachineManager (VMM) domain on the Application Policy Infrastructure Controller(APIC) GUI, configuration mistakes can prevent a distributed virtual switch (DVS) from being created. Ifyou do not see DVS creation on the vCenter GUI, perform the following steps in the APIC GUI:

Step 1 Ensure that the correct attachable entity profile is associated with the VMM domain that you used to create the CiscoAVS in vCenter.

Step 2 Ensure that the correct vCenter credentials are associated with the VMM domain that you used to create the Cisco AVSin vCenter.

Step 3 Ensure that the correct vCenter credential profile is associated with the vCenter controller profile for the VMM domainthat you used to create the Cisco AVS in vCenter.

Step 4 Ensure that the correct data center name is entered in the vCenter controller profile associated with the VMM domainthat you used to create the Cisco AVS in vCenter.

Step 5 Ensure that the correct vCenter IP address is entered in the vCenter controller profile associated with the VMM domainthat you used to create the Cisco AVS in vCenter.


Installation and ConfigurationRecovering from a Cisco AVS Creation Failure

C H A P T E R 4Adding Hosts

This chapter contains the following section:

• Ensuring That a VTEP vmknic Is Added for Valid Host, page 17

Ensuring That a VTEP vmknic Is Added for Valid HostWhen you add valid and invalid hosts to the Cisco Application Virtual Switch (AVS), the Virtual ExtensibleLAN (VXLAN) tunnel endpoint (VTEP) virtual kernel NIC (vmknic) might not be added for the valid hosts.(A valid host has vSphere Installation Bundle [VIB] installed; an invalid host does not.) When you try toinstall valid and invalid hosts, vCenter returns a Simple Object Access Protocol (SOAP) error.

Complete the following steps to ensure that VTEP vmknics are added for valid hosts.

Step 1 Log in to vCenter.Step 2 Choose Home and then Hosts and Clusters.Step 3 Choose the Host, click theConfiguration tab, clickNetworking underHardware, and then click vSphere Distributed

Switch.Step 4 ClickManage Virtual Adapters.Step 5 In theManage Virtual Adapters dialog box, click Add.Step 6 In theAdd Virtual Adapters dialog box, make sure that theNew virtual adapter radio button is selected and then click

Next.Step 7 In the vSphere Distributed Switch area, click the Select port group radio button, choose vtep from the drop-down

list, and then click Next.Step 8 Click the Obtain IP settings automatically radio button and then click Next.Step 9 Click Finish.Step 10 Wait for about a minute, choose the VTEP vmknic in theManage Virtual Adapters dialog box, and verify that the

VTEP vmknic received a DHCP address in the IP Address field.


What to Do Next

Verify that OpFlex comes online by entering the following command on the valid host's console and examiningthe output: vemcmd show openflex

The status should be 12 (Active).


Adding HostsEnsuring That a VTEP vmknic Is Added for Valid Host

C H A P T E R 5OpFlex


• Enabling OpFlex, page 19

• OpFlex Connection Sequence, page 21

• Recovering from an OpFlex Failure Due to a Certificate Issue, page 23

Enabling OpFlexOpFlex is the control protocol between the vLeaf and the Application Policy Infrastructure Controller (APIC).It is automatically enabled when the Cisco Application Virtual Switch (AVS) is added to the distributed virtualswitch (DVS) in vSphere Web Client. If OpFlex is not enabled, virtual machine (VM) ports fail to come up.If this situation occurs, use the following procedure.

One of the Virtual Extensible LAN (VXLAN) tunnel endpoint (VTEP) virtual kernel NICs (vmknics)providesthe OpFlex connection between the Cisco APIC and the Cisco AVS in addition to load balancing data traffic.If you accidentally delete a VTEP vmknic, you can lose the connection between the Cisco APIC and the CiscoAVS. However, if you have more than one VTEP vmknic, one of the others will take over and provide theconnection after a short TCP timeout.

Step 1 On an ESXi hypervisor console, verify that OpFlex is online by completing the following steps:a) Enter the following command:

vemcmd show opflexThe system shows the runtime status of OpFlex.

Example:avs-instance# vemcmd show openflexStatus: 12 (Active)Channel0: 12 (Active), Channel1: 0 (Discovering)Dvs name: comp/prov-VMware/ctrlr-[mininet]-O3-Vcenter/sw-dvs-2923Remote IP: 10.0.0.30 Port: 8000Infra vlan: 2FTEP IP: 10.0.0.32Switching Mode: LS


Encap Type: VXLANNS GIPO: 228.1.1.1

A status of 12 (Active) indicates that OpFlex is online. Any other status indicates a problem. See OpFlexConnection Sequence, on page 21 for more information about possible OpFlex status values.

Note

The system provides other useful information, including the remote and fabric tunnel endpoint (FTEP) (leaf) IPaddresses, the infra VLAN, and the switching mode.

b) If OpFlex is not running, restart the Cisco AVS by removing and readding the host to the DVS.c) If restarting does not fix the problem, call the Cisco Technical Assistance Center (TAC).

Step 2 Verify that the Virtual Ethernet Module (VEM) agent is running.a) Enter the following command:

vem status

The system shows the status of the VEM agent and which VEM modules are running, as shown in the followingexample:

Example:avs-instance# vem statusVEM modules are loadedSwitch Name Num Ports Used Ports Configured Ports MTU UplinksvSwitch0 128 9 128 1500 vmnic0DVS Name Num Ports Used Ports Configured Ports MTU Uplinksmininet 1024 13 1024 1500 vmnic2

VEM Agent (vemdpa) is running

b) If the VEM is not up, call the Cisco TAC.

Step 3 Verify that the Virtual Extensible LAN (VXLAN) tunnel endpoint (VTEP) acquired a valid DHCP IP address.a) Enter the following command:

esxcfg-vmknic -l

Example:avs-instance# esxcfg-vmknic -lInterface Port Group/DVPort IP Family IP Address Netmask BroadcastMAC Address MTU TSO MSS Enabled Typevmk0 Management Network IPv4 10.30.13.55 255.255.254.0 10.30.13.255f8:72:ea:a4:97:0a 1500 65535 true STATICvmk0 Management Network IPv6 fe80::fa72:eaff:fea4:970a 64f8:72:ea:a4:97:0a 1500 65535 true STATIC, PREFERREDvmk1 11 IPv4 10.0.28.93 255.255.0.0 10.0.255.25500:50:56:61:ef:11 1500 65535 true DHCPvmk1 11 IPv6 fe80::250:56ff:fe61:ef11 6400:50:56:61:ef:11 1500 65535 true STATIC, PREFERRED

Alternatively, you can check the VTEP port group in the vCenterGUI.

Note

b) If the VTEP is not configured properly, remove the host from the DVS and add it back in as described in Step 1a.c) If removing and readding the host does not correct the problem, call the Cisco TAC.

Step 4 Verify the switch opaque data.a) Display the switch opaque data by entering the following command:

vemcmd show sod

Example:avs-instance# vemcmd show sodSwitch Opaque Datadata-version 2.0control-protocol open-flexopen-flex port 8000


OpFlexEnabling OpFlex

open-flex ipaddr 10.0.0.30ftep ipaddr 10.0.0.32dvs-name comp/prov-VMware/ctrlr-[mininet]-O3-Vcenter/sw-dvs-2923profile dvportgroup-2925 encap vlan 2profile dvportgroup-2925 capability open-flexprofile dvportgroup-2924 port-channel activeprofile dvportgroup-2924 mtu 9000Switch opaque data is bootstrapping information for the host that is given through vCenter by the APIC.

The system shows the port channel status as one of the following:

• active

• passive

• mac-pinning

• static

The status should be the same as configured on the APIC as shown in the example.

b) If the switch opaque data is incorrect (for example, the port channel is not in the state specified in the configuration),call the Cisco TAC.

Step 5 Verify the uplinks by completing the following steps:The uplinks and the VTEP should be in the forwarding state. The port channel local target logic (PC-LTL) number ofthe uplink port should be nonzero. The corresponding port channel port also should be in the forwarding state.

a) Enter the following command and examine its output:vemcmd show port

Example:avs-instance# vemcmd show portLTL VSM Port Admin Link State Cause PC-LTL SGID Vem Port Type19 Eth1/3 UP UP FWD - 561 0 vmnic249 UP UP FWD - 0 0 vmk1561 Po1 UP UP FWD - 0

The example output shows that the uplink port (here named vmnic2) and the VTEP (vmk1) are in the forwardingstate. This is indicated by Admin UP, Link UP, and State FWD. The example also shows a port channel in the FWDstate and that the PC-LTL number of the uplink port is 561. The system also shows the ports used by each connection.

b) If the port data is incorrect (for example, a port is not in forwarding state), call the Cisco TAC.

OpFlex Connection SequenceYou can use the vemcmd show openflex command to view the state of the OpFlex connection.

Normal OpFlex Connection Sequence

The following table describes the normal sequence for a successful OpFlex connection. The messages appearquickly one after the other and end with the oplex_CHANNEL_ACTIVE (12) message.


OpFlexOpFlex Connection Sequence

DescriptionOpFlex State

This state is the initial Opflex state.opflex_CHANNEL_DISCOVERING (0)

The Opflex client sends a DISCOVERmessage to theanycast IP address. (The anycast IP address can befound by entering vemcmd show sod; it is theopen-flex ipaddr.)

opflex_CHANNEL_SEND_DISCOVER (1)

The DISCOVERmessage (reply) is received from theleaf.

opflex_CHANNEL_RECV_DISCOVER(2)/opflex_CHANNEL_DISCOVERED (3)

The client sends a CONNECTmessage (similar to theSEND_DISCOVERmessage but this is for the actualOpFlex connection) after the initial discovery phase.

opflex_CHANNEL_CONNECTING (6)

The client sends a HELLOmessage to the IFM on theleaf. This is the first message after the SSL session isup for IFM.

opflex_CHANNEL_SEND_HELLO (7)

The ID and the FUNC data are sent to the leaf in thesestates.

opflex_CHANNEL_SEND_ID(9)/opflex_CHANNEL_SEND_FUNCTION (10)

Once the ACKs are received, the state moves toACTIVE.

opflex_CHANNEL_CONNECTED (11)

OpFlex is UP.opflex_CHANNEL_ACTIVE (12)

Solutions to OpFlex Channel State Issues

OpFlex can get stuck during discovery and connection. The following table lists the abnormal states that canbe viewed in the output of the vemcmd show openflex command, their causes, and solutions.

SolutionCauseOpFlex State

Check the network. Ifthere are no networkissues, reboot the leaf.

A network issue probably has occurred.opflex_CHANNEL_SEND_DISCOVER(1)

Call the Cisco TAC.The client code has a problem.opflex_CHANNEL_RECV_DISCOVER(2)

Call the Cisco TAC.SSL keys might be missing.opflex_CHANNEL_DISCOVERED(3)

Call the Cisco TAC.An IFM version mismatch due tosoftware incompatibility has occurred.

opflex_CHANNEL_VERSION_MISMATCH(4)


OpFlexOpFlex Connection Sequence

SolutionCauseOpFlex State

Check that the vmknic isup and has a valid IPaddress and that theanycast IP address isreachable by a ping.

Network connectivity is down betweenthe vLeaf and the leaf switch.

opflex_CHANNEL_DISCONNECTED(5)/opflex_CHANNEL_CONNECTION_ATTEMPT(14)

Call the Cisco TAC.The images are incompatible.opflex_CHANNEL_SEND_ID(9)/opflex_CHANNEL_SEND_FUNCTION(10)

Call the Cisco TAC.The contents of the ACK areinconsistent.

opflex_CHANNEL_INACTIVE (13)

Call the Cisco TAC.There is a difference in the distributedvirtual switch (DVS) ID between theleaf switch and the vLeaf.

opflex_CHANNEL_INVALID_DVS(15)

Recovering from an OpFlex Failure Due to a Certificate IssueWhen you upgrade earlier releases of Cisco ACI to 1.2(2g), OpFlex might fail to come up on your ESXi hoston Cisco AVS because of a certification problem. You can recover your setup by following the procedure inthis section.

The recovery procedure requires that you perform tasks on VMware vCenter and the Cisco AVS host, CiscoAPIC, and leaf and spine switches in the Cisco ACI fabric.

Before You Begin

For root access into Cisco APIC and Cisco ACI switch nodes, you need to open a service request withCisco Technical Assistance Center (TAC).

Note

Make sure that the certificates on Cisco APIC, Cisco AVS, and the leaf and spine switches are truly mismatched.To do so, log in to Cisco APIC, each of the leaf and spine switches, and the Cisco AVS and enter the followingcommands:

• Cisco APIC: (as admin)acidiag verifyapic

Examine the output, verifying that the system time on the Cisco ACI fabric falls between the "NotBefore" and "Not After" times within the output. If the Cisco ACI fabric system time is after the "NotAfter" time, contact Cisco TAC for the next remediation steps.

• Cisco APIC (as root):cksum /securedata/vssl/server.*

• Leaf and spine switches: (as root)cksum /securedata/vssl/server.*

• Cisco AVS hosts (as root):cksum /tmp/server.*


OpFlexRecovering from an OpFlex Failure Due to a Certificate Issue

Export the configuration in Cisco APIC with Advanced Encryption Standard (AES) encryption. See the CiscoACI Basic Configuration Guide and Importing and Exporting Configuration Files for information about usingconfiguration file encryption.

Step 1 Log in to VMware vCenter using the VMware vSphere Client.Step 2 Remove VMs from port groups created for the Cisco AVS host.Step 3 Remove vmknics from the Cisco AVS.Step 4 Remove Hosts from the DVS.Step 5 Remove the Cisco AVS from VMware vCenter.Step 6 Log in to the Cisco AVS host as root.Step 7 Remove certificate files if they exist.

Example:[root@localhost:~] rm /tmp/server.*

Step 8 Log in to the Cisco APIC as root. (Requires TAC.)Step 9 Remove the ODev key file.

Example:root@apic1:~# rm /data/odev_keys_created

Step 10 Clean the configuration data.

Example:root@apic1:~#acidiag touch clean

Step 11 Reload Cisco APIC.

Example:root@apic1:~#reboot

Step 12 Repeat Step 8 through Step 11 on each of the other Cisco APICs.Step 13 On one of the leaf or spine switches in the fabric, log in as root.Step 14 Remove the ODev key file if it exists.

Example:leaf1# rm /data/odev_keys_created

Step 15 Clean the configuration data.

Example:leaf1# setup-clean-config.sh

Step 16 Reload the switch.

Example:leaf1# vsh -c 'reload'

Step 17 Repeat Step 13 through Step 16 on each of the other leaf and spine switches in the fabric.Step 18 Log in to Cisco APIC in a web browser.Step 19 Complete the fabric membership.

See the sections for registering unregistered switches in the Cisco APIC Getting Started Guide.



http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/basic-config/b_ACI_Config_Guide.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/basic-config/b_ACI_Config_Guide.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Using_Import_Export_to_Recover_Config_States.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/getting-started/1_2_x/b_APIC_Getting_Started_Guide_Rel_1_2_1.html

Step 20 Create an import policy with AES encryption.Step 21 Import the configuration into Cisco APIC.

What to Do Next

Verify the certificates for Cisco APIC, the leaf and spine switches, and Cisco AVS hosts. Complete thefollowing steps:

1 Log in to Cisco APIC and each fabric switch as root. (Requires TAC.)

2 Verify the certificates in Cisco APIC and on the fabric switches, as shown in the following examples:root@apic1:~# cksum /securedata/vssl/server.*leaf1# cksum /securedata/vssl/server.*

The keys in Cisco APIC and on the switches must be the same. They will appear as follows: xxxxxxxx812 /securedata/vssl/server.crt,xxxxxxxx 637 /securedata/vssl/server.csr,xxxxxxxx 889 /securedata/vssl/server.key

Note

3 Create or import configurations for the interface policy, switch profile, and VMM domain.

4 In VMware vCenter, add the Cisco AVS host to the DVS.

5 Verify the certificates on the Cisco AVS host, as shown in the following example:[root@localhost:~] cksum /tmp/server.*

6 Repeat Step 5 on the other Cisco AVS hosts.

The keys in Cisco APIC and the Cisco AVS hosts must be the same.Note

7 Verify that OpFlex is active.





C H A P T E R 6Ports, Endpoint Groups, and Layer 2


• About Data Paths, page 27

• Diagnosing Port Activity, page 28

• Troubleshooting Unavailable Ports, page 29

• Checking Port Synchronization Using Port Counters, page 30

• Troubleshooting Endpoint Groups, page 31

• Recovering from Endpoint Group Creation Failure, page 32

• Debugging Isolated Endpoints Within an EPG, page 32

• Troubleshooting Layer 2 Switching, page 33

About Data PathsBefore a switch can use interfaces to relay frames from one data link to another, you must define thecharacteristics of the sending and receiving interfaces. The configured interfaces can be Ethernet (physical)interfaces and virtual Ethernet (vEth) interfaces.

On the virtual side of the switch, three layers of ports are mapped together:

Virtual NICs

VMware has two types of virtual NICs.

• The virtual NIC (vnic) is part of the virtual machine (VM) and represents a VM virtual port thatis plugged into the virtual switch.

• The virtual kernel NIC (vmknic) is used by the hypervisor for management, vMotion, iSCSI,Network File System (NFS), and other network access needed by the kernel. This interface carriesthe IP address of the hypervisor itself, and is also bound to a vEth port.


vEth Ports

A vEth port is a port on the Cisco AVS. A vEth port is allocated to a host that runs a VM and is assignedto a port group.

Local Virtual Ethernet ports (lvEth)

Each host has a number of lvEth ports. These ports are dynamically allocated as needed.

Three types of ports are on the physical side of the switch. From bottom to top, they are:

Virtual machine NICs

Each physical NIC in VMware is represented by an interface called a vmnic. The vmnic number isallocated during VMware installation (or when a new physical NIC is installed) and remains the samefor the life of the host.

Uplink ports

Uplink ports associate port configuration with vmnics. Each uplink port on the host represents a physicalinterface. Because physical ports do not move between hosts, there is a 1:1 mapping between uplinkports and vmnics. Uplink ports are managed entirely by VMware.

Physical ports

Each physical port added to a Cisco AVS appears as a physical Ethernet port, just as it would on ahardware-based switch.

Each interface, regardless of type, has the following characteristics:

Administrative configuration

You can set configuration attributes. The administrative configuration does not change unless youmodify it using the Application Policy Infrastructure Controller (APIC).

Operational state

The operational state includes such attributes as the interface speed. These values are read-only; theycannot be changed. Some values might not be valid when the interface is down (for example, theoperation speed).

Diagnosing Port ActivityYou can diagnose port activity by examining the following:

• Administrative state

• Speed

• Trunk VLAN status

• Number of frames sent and received


Ports, Endpoint Groups, and Layer 2Diagnosing Port Activity

• Transmission errors, including discards, errors, cyclic redundancy checks (CRCs), and invalid frames

Step 1 Verify that the host is connected to the fabric by entering the vemcmd show openflex command and examining its outputas described in Enabling OpFlex, on page 19.

Step 2 In the vSphere Web Client connected to the vCenter Server, verify that the correct port profiles are assigned to thephysical and virtual NICs.

Step 3 On an ESXi hypervisor console, verify that the ports have been created by entering the following command:vemcmd show port

Example:avs-instance# vemcmd show portLTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port20 Eth1/4 UP UP FWD - 1039 3 0 0 vmnic349 UP UP FWD - 0 2 0 0 vmk150 UP UP FWD - 0 2 0 0 orion3-vm2.eth151 UP UP FWD - 0 3 0 0 orion3-vm1.eth1

1039 Po1 UP UP FWD - 0 0 0The output of the command should look like the example, with Admin, Link, and State equal to UP, UP, and FWDfor each port.

Step 4 If the port states are incorrect, see Troubleshooting Unavailable Ports, on page 29.

Troubleshooting Unavailable PortsIf ports do not come up, enter the vemcmd show port command to diagnose the cause.

The following table lists the possible causes and solutions for troubleshooting unavailable ports.

SolutionPossible CauseOutput from vemcmd showport

Wait for the port to come up(approximately 10 minutes).

The port is coming up and iswaiting for the endpoint group(EPG) to be downloaded.

WAIT ACK or WAIT EPG

Troubleshoot the OpFlex controlchannel as described in EnablingOpFlex, on page 19.

The OpFlex control channel is notonline.

WAIT ACK or WAIT EPG(same as above)

Bring up the interface manually on theVM, and then edit the appropriateconfiguration files to make sure theinterface comes up during bootup (orafter a reboot).

The interface on the virtual machine(VM) is down.

Zero Mac


Ports, Endpoint Groups, and Layer 2Troubleshooting Unavailable Ports

SolutionPossible CauseOutput from vemcmd showport

The port will be down in an errordisabled state for 30 seconds but shouldcome up after that. To avoid this error,make sure that no BPDUs are sent fromthe VM or disable BPDU Guard.

Bridge Protocol Data Unit (BPDU)Guard feature is enabled, andBPDUs are received from the VM.

BPDU Viol

In all cases listed above, if the solution does not enable the port to come up, detach the VM from the CiscoAVS and then reattach it.

If the issue persists, contact the Cisco TAC.

Checking Port Synchronization Using Port CountersCounters can show synchronization problems by revealing a large disparity between received and transmittedframes.

Step 1 Create a baseline by clearing the counters by entering the command vemcmd clear stats.The values stored in the counters are meaningless for a port that has been active for an extended period. Clearingthe counters provides a profile of the actual link behavior over a known period of time.

Note

Step 2 Display the total number of packets sent and received by entering the command vemcmd show stats.

Example:avs-instance# vemcmd show statsLTL Received Bytes Sent Bytes Txflood Rxdrop Txdrop Name8 3 202 0 0 0 0 09 0 0 3 202 3 0 010 10 772 7 420 7 0 012 7 420 10 772 10 0 016 5 582 0 0 0 0 0 ar19 935 187513 456 48497 11 7 0 vmnic220 830 170397 355 37063 21 7 0 vmnic349 743 81212 714 137646 0 0 0 vmk150 60 4816 44 3856 0 0 0 orion3-vm2.eth151 45 3688 46 3748 0 0 0 orion3-vm1.eth11039 1004 201869 457 48557 32 13 0

Step 3 Display a breakdown of packets into unicast, broadcast, multicast, and flood by entering the command vemcmd showpackets.

Example:avs-instance# vemcmd show packetsLTL RxUcast TxUcast RxMcast TxMcast RxBcast TxBcast Txflood RxdropTxdrop RxJumbo TxJumbo Name

19 1033 596 835 3 104 445 18 94 0 0 0 vmnic220 0 0 835 0 104 11 9 0 0 0 0 vmnic349 588 1027 5 0 430 11 0 0 0 1 0 vmk150 8 6 0 0 8 107 107 0 0 0 1 orion3-vm4.eth151 0 0 0 0 8 107 107 0 0 0 0 orion3-vm3.eth11039 2066 1192 3337 6 416 912 54 188 0 0 0


Ports, Endpoint Groups, and Layer 2Checking Port Synchronization Using Port Counters

Troubleshooting Endpoint GroupsIn the Cisco AVS, an endpoint group (EPG) is an entity that is assigned multiple interfaces, giving them allthe same configuration. Changes to an EPG configuration are propagated automatically to all interfaces thatare assigned to it.

In vCenter Server, an EPG is represented as a port group. The virtual Ethernet (vEth) interfaces are assignedin vCenter Server to an EPG in order to:

• Define the port configuration by the policy.

• Apply a single policy across a large number of ports.

EPGs that are configured as uplinks can be assigned by the server administrator to physical ports (which canbe vmnics or PNICs). EPGs that are not configured as uplinks can be assigned to a virtual machine (VM)virtual port.

For more information about assigning EPGs, see your VMware documentation.

Step 1 Enter the following command:echo “dump profile_cfg” > /tmp/dpafifo

Step 2 Enter the following command and check the output of the file:vi /var/log/vemdpa.log

Example:Profile:

alias: dvportgroup-3228pp_id 3228 flags 0mode: Trunkadmin_state: no shutmtu 9000allowed_vlans: 1-4095EPP Switching mode: NSEPP Encap : VLAN 1EPP seg id 0EPP reused 0EPP seg arp flood 0chan: mode on sg_type mac-pinningPorts: cnt 319 20 561

Profile:alias: dvportgroup-3229pp_id 3229 flags 20mode: Accessadmin_state: no shutaccess_vlan 2EPP Switching mode: NSEPP Encap : VLAN 4093EPP seg id 4093EPP reused 0EPP seg arp flood 0Ports: cnt 149

Profilealias: dvportgroup-3230pp_id 3230 flags 0mode: Accessadmin_state: no shutaccess_vlan 3EPP Switching mode: LS


Ports, Endpoint Groups, and Layer 2Troubleshooting Endpoint Groups

EPP Encap : VLAN 39EPP seg id 15433636EPP reused 1EPP seg arp flood 1Ports: cnt 250 51

All the ports associated with a port group should belong to the same profile. In the first example above,dvportgroup-3228 is the port group, and 19, 20, and 561 are the ports.

Recovering from Endpoint Group Creation FailureAfter you create a virtual machine manager (VMM) domain on the Application Policy Infrastructure Controller(APIC) GUI, configuration mistakes can prevent endpoint groups (EPGs) from being created. If the EPGsthat you create on the APIC GUI do not appear under the Cisco AVS, use the following procedure:

Step 1 On the APIC GUI, ensure that the VMM domain that is created in vCenter is associated with the correct EPG.Step 2 Ensure that a large enough address pool exists to support all the EPGs that you defined.

a) In VLAN mode, ensure that the VLAN pool contains a number of VLANs equal to or greater than the number ofEPGs that you defined.

b) In Virtual Extensible LAN (VXLAN) mode, ensure that the multicast IP pool contains a number of multicast IPaddresses equal to or greater than the number of EPGs that you defined.

Debugging Isolated Endpoints Within an EPGIf you isolated endpoints within an EPG on Cisco AVS, you can use debugging commands to check whetherthe intra-EPG deny policy is enabled on the EPG and to track statistics.

Verifying That Policy to Deny Intra-EPG Communication Is EnabledYou can verify that the policy to deny communication between the endpoints is enabled for the EPG.

Enter the following command and examine its output: vemcmd dpa dump profile_cfg

Example:~ # vemcmd dpa dump profile_cfg=>dpa command is: dump profile_cfgvLeaf garbage collection count: 1...Profile:


Ports, Endpoint Groups, and Layer 2Recovering from Endpoint Group Creation Failure

alias: dvportgroup-798eppdn: uni/epp/fv-[uni/tn-T1/ap-AP1/epg-EPG-3]pp_id 4 flags 0mode: Accessadmin_state: no shutaccess_vlan 3EPP Switching mode: LSEPP Encap : VXLAN 8912896, 224.1.1.6EPP seg id 16613250EPP reused 0EPP pending 0EPP seg unk unicast 0EPP seg arp flood 1EPP seg intra-epg policy 1EPP IP/MAC profile FALSEMicrosegment table id 6441803785434561149Ports (using): 151

#byeBye#Ports (holding): 1In the preceding example, tn-T1/ap-AP1/epg-EPG-3 in the second line of the profile section is the EPG name.

In the fourteenth line of the profile, the 1 following EPP seg intra-epg policy indicates that the policy isenabled. A 0 would indicate that the policy is disabled.

Tracking Statistics for Isolated Endpoints in an EPGYou can track statistics for endpoints you isolated within an EPG using a vemcmd command. Alternatively,you can choose and view statistics in the advanced GUI. See the Cisco AVS chapter in the Cisco ACIVirtualization Guide for more information.

Enter the following command and examine its output: vemcmd show intra-epg-policy-stats

Example:~ # vemcmd show intra-epg-policy-statsLTL ucast-packets bumcast-packets VM-Name51 0 37 Ubuntu-2.eth1

~ #

Troubleshooting Layer 2 SwitchingYou can troubleshoot connections between two Layer 2 endpoints.

Step 1 Verify that the switch mode is correct by completing the following steps:a) Verify that OpFlex is online by entering the following command:

vemcmd show openflexA status of 12 (Active) indicates that OpFlex is online.


Ports, Endpoint Groups, and Layer 2Tracking Statistics for Isolated Endpoints in an EPG

b) Move profile information into an output file by entering the following command:echo “dump profile_cfg” > /tmp/dpafifo

c) Display the resulting output file by entering the following command and examine its output:vi /var/log/vemdpa.logThe output should show valid virtual extensible LAN (VXLAN) network IDs (VNIDs) and endpoint group (EPG)multicast addresses. The EPG multicast addresses should be from the pool to which they were assigned earlier in theApplication Policy Infrastructure Controller (APIC).

Step 2 Verify the configuration and status of the ports by completing the following steps:a) Display port information by entering the following command:

vemcmd show port

Example:avs-instance# vemcmd show portLTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port20 Eth1/4 UP UP FWD - 1039 3 0 0 vmnic349 UP UP FWD - 0 2 0 0 vmk150 UP UP FWD - 0 2 0 0 orion3-vm2.eth151 UP UP FWD - 0 3 0 0 orion3-vm1.eth1

1039 Po1 UP UP FWD - 0 0 0The output of the command should look like the example, with Admin, Link, and State equal to UP, UP, andFWD, respectively for each port.

b) On the Cisco AVS instances that are attached to the two endpoints, enter the following command:vemcmd show port vlansThe VLAN or VXLAN tags at either endpoint should match.

Step 3 For inter-EPG traffic, use the APIC GUI to ensure that contracts are correctly configured between the EPGs.See the Cisco APIC Getting Started Guide and the Cisco APIC Layer 4 to Layer 7 Services Deployment Guide forinformation about configuring contracts.

Step 4 For traffic issues involvingVXLAN encapsulation, ensure that the IGMP querier and IGMP snooping policy are configuredin the APIC GUI under tenant infra and infra bridge domain.See the Cisco Application Virtual Switch Configuration Guide for information about configuring IGMP querier andIGMP snooping policy.

Step 5 For VXLAN mode, verify that IGMP joins were sent out correctly from the Cisco AVS for the EPGs attached to theendpoints:a) Enter the following command:

vemcmd show epp multicast

Example:# vemcmd show epp multicastNumber of Group Additions 1Number of Group Deletions 0Multicast Address EPP Ref Count

225.2.1.92 1

b) If no group is present, ensure that the multicast pool for the virtual machine manager (VMM) domain contains atleast as many multicast addresses as there are EPGs.

The multicast-to-EPG association is one-to-one; therefore, a multicast pool that is too small prevents some EPGs frombeing created.

If a new multicast pool is created and associated with the VMM domain, all the EPGs must be disassociated from thatVMM domain and associated back.


Ports, Endpoint Groups, and Layer 2Troubleshooting Layer 2 Switching

Step 6 Verify that the port channel is up and that it matches the upstream port channel configuration by completing the followingsteps:a) Enter the following command:

vemcmd show pc

Example:# vemcmd show pcpce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs------- ---- ------ --------- ---- ----- -------------- ----

0 1 561 0 N 2 1 19,3* 2 20,

* denotes a designated sub-group

b) If the port channel type is static, make sure that all the interfaces that belong to the leaf switches that are associatedwith the interface policy group are added to the Cisco AVS.

c) If the problem persists or if there are upstream issues, see the Cisco ACI Troubleshooting Guide.





C H A P T E R 7Port Channels


• Port Channel Overview, page 37

• Verifying Port Channels, page 37

• Troubleshooting Port Channel Creation, page 41

Port Channel OverviewPort channels aggregate multiple physical interfaces into one logical interface to provide higher bandwidth,load balancing, and link redundancy.

Cisco AVS supports LACP, MAC pinning, and static port channels in standalone and virtual port channel(VPC) mode.

A port channel performs the following functions:

• Increases the aggregate bandwidth on a link by distributing traffic among all links in the channel.

• Maintains optimal bandwidth usage by load balancing across multiple links.

• Provides high availability. If one link fails, its traffic is switched to the remaining links. Higher-levelprotocols are unaware of the failed link, although bandwidth is diminished. The MAC address tablesare not affected by link failure.

Verifying Port Channels

Step 1 Display the ports on the Cisco AVS by entering the following command:vemcmd show port

Example:# vemcmd show portLTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port


21 Eth1/5 UP UP FWD - 1039 0 0 0 vmnic422 Eth1/6 UP UP FWD - 1039 0 0 0 vmnic523 Eth1/7 UP UP FWD - 1039 0 0 0 vmnic624 Eth1/8 UP UP FWD - 1039 0 0 0 vmnic749 UP UP FWD - 0 0 0 0 vmk150 UP UP FWD - 0 0 0 vmk21039 Po1 UP UP FWD - 0 0 0

Step 2 Verify that the uplinks are in the FWD state rather than the BLK state (or, in the case of Link Aggregation Control Protocol[LACP], that uplinks are not in the Suspended [s] state or the Individual [I] state).

Step 3 Gather information about the remote physical ports by completing the following steps:a) (For directly connected hosts) Display information about each local target logic (LTL) number with physical ports

(21 to 24 in the previous example) by entering the following command:vemcmd show lldp ltl

Example:# vemcmd show lldp 21Chassis Id = 7c:69:f6:df:e4:f2Port Id = Eth1/2Extras:3topology/pod-1/protpaths-101-102/pathep-[esx56-vpc]

leaf2topology/pod-1

b) For leaf switches connected to a Layer 2 cloud (either Cisco Nexus 5000 or fabric interconnect), enter the followingcommand on the Cisco Nexus 5000 or fabric interconnect console:show lldp neighbors interface <interface_id> detail

For Cisco UCSManager fabric interconnect nodes, youmust connect to Cisco NX-OS.Note

Example:AVS-N5K# show lldp neighbors interface ethernet 1/29 detailCapability codes:(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID Local Intf Hold-time Capability Port ID

Chassis id: 5087.89d3.ce3fPort id: Eth1/5Local Port id: Eth1/29Port Description: topology/pod-1/protpaths-105-106/pathep-[N5K-VPC1]System Name: scale-leaf4System Description: topology/pod-1/node-106Time remaining: 100 secondsSystem Capabilities: B, REnabled Capabilities: B, RManagement Address: 5087.89d3.ce3fVlan ID: not advertised

Total entries displayed: 1

Step 4 For each output above, note the remote physical port (Port Id; for example, Eth1/2 in substep a and Eth1/5 insubstep b of the preceding step), the policy group (for example, esx56-vpc and N5K-VPC1), and the remote switch(leaf) name (leaf2 and scale-leaf4).Note which physical port and policy group are associated with which leaf switch.

Step 5 Display VPC information by entering the following command on each leaf switch identified in Step 3a and Step 3b:show vpc

Example:leaf1# show vpcLegend:


Port ChannelsVerifying Port Channels

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1Peer status : peer adjacency formed okvPC keep-alive status : DisabledConfiguration consistency status : successPer-vlan consistency status : successType-2 inconsistency reason : Consistency Check Not PerformedvPC role : primary, operational secondaryNumber of vPCs configured : 4Peer Gateway : DisabledDual-active excluded VLANs : -Graceful Consistency Check : EnabledAuto-recovery status : Enabled (timeout = 240 seconds)Operational Layer3 Peer : Disabled

vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 up -

vPC status----------------------------------------------------------------------id Port Status Consistency Reason Active vlans-- ---- ------ ----------- ------ ------------1 Po3 up success success 4090

2 Po1 up success success 4090

3 Po4 up success success 4090

343 Po2 down* success success -

Step 6 In the previous command result, on the leaf and intermittent Layer 2 switches, verify that the VPC status is success.Step 7 On each leaf switch, enter the following command:

show vpc role

Example:scale-leaf3# show vpc role

vPC Role status----------------------------------------------------vPC role : secondaryDual Active Detection Status : 0vPC system-mac : 00:23:04:ee:be:02vPC system-priority : 32667vPC local system-mac : 50:87:89:a2:53:59vPC local role-priority : 106scale-leaf3#

scale-leaf4# show vpc role

vPC Role status----------------------------------------------------vPC role : primaryDual Active Detection Status : 0vPC system-mac : 00:23:04:ee:be:02vPC system-priority : 32667vPC local system-mac : 50:87:89:d3:ce:71vPC local role-priority : 105scale-leaf4#

Step 8 In the previous command result, verify that there is one primary and one secondary role per VPC, similar to the examplein Step 7.

Step 9 On each leaf switch, display port channel information by entering the following command:show port-channel summary



Example:leaf1# show port-channel summaryFlags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)M - Not in use. Min-links not met

--------------------------------------------------------------------------------Group Port- Type Protocol Member Ports

Channel--------------------------------------------------------------------------------1 Po1(SU) Eth LACP Eth1/1(P) Eth1/2(P)2 Po2(SU) Eth LACP Eth1/19(P) Eth1/20(P)3 Po3(SU) Eth LACP Eth1/33(P)4 Po4(SU) Eth LACP Eth1/34(P)leaf1#

Step 10 In the previous command result, verify that the protocol and the status of each member port (the flag, in parenthesesafter the port name) are correct for the port channel type:

Status FlagProtocolPort Channel Type

PNONEstatic

PLACPLACP

ILACPMACPIN

Any other protocol or status value indicates a configuration problem.

LACP automatically negotiates both ends of the port channel configuration, so correct protocol and statusindicate success. For static and MAC pinning port channels, it is possible to have a misconfiguration and stillshow the correct protocol and status. The SD (Down) status is the expected behavior for a port channel withMAC pinning.

The remaining steps describe how to identify misconfigured port channels.

Note

Step 11 Compare the outputs of the vemcmd show lldpltl (Step 4) and show port-channel summary commands (Step 9). Theoutputs should show the same physical ports for a given leaf.

Step 12 If you find misplaced physical ports in the diagnostic steps above, log in to the Cisco Application Centric Infrastructure(APIC) GUI and fix them, as follows:a) Go to Fabric > Access Policies > Interface Profiles > Policy Groups. Find the profile corresponding to the policy

groups found in the vemcmd show lldpltl command.b) Go to Fabric > Access Policies > Interface Profiles > Profiles.c) For each profile that corresponds to the policy group identified above, change the configuration so that the policy

reflects the actual port configuration.Alternatively, you can fix the misconfiguration by changing the physical port connections to agree with the connectionsthat are specified in the profile.



Troubleshooting Port Channel CreationIf port channel creation fails, you might have configured too many port channels or port channel members.

If necessary, reconfigure your system to require only eight port channel members.Only eight uplinks are supported on one system. Only one port channel is supported on one system.


Port ChannelsTroubleshooting Port Channel Creation


Port ChannelsTroubleshooting Port Channel Creation

C H A P T E R 8Switched Port Analyzer


• About the Switched Port Analyzer, page 43

• Viewing the Switched Port Analyzer Configuration, page 44

• Troubleshooting the Switched Port Analyzer, page 45

About the Switched Port AnalyzerThe Switched Port Analyzer (SPAN), sometimes called port mirroring or port monitoring, selects networktraffic for analysis by a network analyzer.

Two types of SPAN are supported:

• SPAN (local SPAN) that can monitor sources within a host.

• Encapsulated Remote SPAN (ERSPAN) that can send monitored traffic to an IP destination.

For detailed information about how to configure SPAN, see the chapter on configuring SPAN in the CiscoApplication Virtual Switch Configuration Guide.

The interfaces from which traffic can be monitored are called SPAN sources. Traffic can be monitored in thereceive direction, the transmit direction, or both directions for virtual Ethernet source interfaces (endpoints)or EPGs.

• Receive source (Rx)—Traffic that enters the switch through this source port is copied to the SPANdestination port.

• Transmit source (Tx)—Traffic that exits the switch through this source port is copied to the SPANdestination port.


Source Ports

The Cisco Application Virtual Switch (AVS) supports multiple source ports. A source port has thesecharacteristics:

• Cannot be a destination port.

• Can be configured to monitor the direction of traffic (receive, transmit, or both).

• (For local SPAN only) Must be on the same host as the destination port.

SPAN Destinations

The Cisco AVS supports only virtual Ethernet (vEth) interfaces (endpoints) as SPAN destinations.

Destination Ports

Each local SPAN session must have at least one destination port (also called a monitoring port) thatreceives a copy of traffic from the source ports. A destination port has these characteristics:

• Cannot be a source port.

• Receives copies of transmitted and received traffic for all monitored source ports. If a destinationport is oversubscribed, it can become congested. This congestion can affect traffic forwarding onone or more of the source ports.

• (For local SPAN only) Must be on the same host as the source port.

ERSPAN Destinations

ERSPAN destinations refer to an IP address to which the monitored traffic is sent. The destination IPshould be in overlay-1 (infra VRF) and be reachable through the configured ERSPAN-enabled vmknic(which is also the VXLAN tunnel endpoint [VTEP]) on the host. For detailed information about howto configure ERSPAN, see the Cisco Application Virtual Switch Configuration Guide.

SPAN Sessions

You can create up to 64 SPAN and ERSPAN sessions to define sources and destinations on the localdevice.

Viewing the Switched Port Analyzer Configuration

To display the SPAN information, enter the following command:show span

Example:The following example output shows the expected configurations for ERSPAN:# vemcmd show spanVEM SOURCE IP: 10.0.0.16

HW SSN ID ERSPAN ID HDR VER DST LTL/IP1 1 2 10.0.10.10


Switched Port AnalyzerViewing the Switched Port Analyzer Configuration

RX Ltl Sources :50,58,59,60,61,62,TX Ltl Sources :50,58,59,60,61,62,

Troubleshooting the Switched Port AnalyzerSwitched Port Analyzer Requirements

A running SPAN session must meet these requirements:

• A maximum of 64 SPAN sessions can run at one time.

• At least one operational source has been configured.

• At least one operational destination has been configured.

• The configured source and destination are on the same host (for local SPAN).

• A port cannot be configured as both a source port and a destination port.

• The static client endpoint (CEP) has been configured accordingly on the right leaf (for ERSPAN).

• The ERSPAN destination host is reachable through the host’s VXLAN tunnel endpoint (VTEP) (forERSPAN).

When a SPAN session contains multiple transmit source ports, packets that these ports receive can be replicatedeven though they are not transmitted on the ports. Some examples of this behavior on source ports are asfollows:

• Traffic that results from flooding.

• Broadcast and multicast traffic.

A session is stopped if any of the following events occur:

• All source ports are removed.

• All destination ports are removed.

• All source and destination ports are separated by a vMotion live migration.

After vMotion, the following might occur:

• A session is stopped if the source and destination ports are separated.

• A session resumes if the source and destination ports end up on the same host.

Troubleshooting

If you encounter problems with SPAN, consult the following table for symptoms and solutions.


Switched Port AnalyzerTroubleshooting the Switched Port Analyzer

SolutionPossible CauseSymptom

Verify that SPAN is configured properlyon the Cisco APIC. Verify that the portson which SPAN is enabled are UP and inthe Forwarding state as described inTroubleshooting Unavailable Ports, onpage 29.

Packets are not beingspanned to a localdestination or a remotedestination.

The vemcmd show span commanddoes not show the configuration.

Make sure that the ERSPAN destinationis hosted in the overlay-1 VRF. ERSPANis supported in this VRF only. If theERSPAN destination is a virtual machine(VM) on the Cisco AVS, make sure thatit is using the VTEP endpoint group(EPG).

The ERSPAN destination isnot on the overlay-1 virtualrouting and forwarding(VRF).

The ERSPAN session is configuredbut does not see packets at thedestination.

Complete the following actions:

1 Verify that the static CEP isconfigured with the following:

• The ERSPAN destination MACaddress

• The ERSPAN IP address

• The overlay-1 VLAN

• Type equal to tunnel endpoint(tep)

• The interface policy group thatidentifies the leaf switchesbehind which the ERSPANdestination VM is located

2 If the static CEP is configured but stillnot working, try deleting and readdingthe static CEP.

Static CEP is notconfigured.

Ensure that the ERSPAN destination VMhas an IP address in the same subnet asthat of the other VTEPs in the fabric.

The ERSPAN destinationVM has an IP address thatis not in the same subnet asthat of other VTEPs in thefabric.

After the static CEP is configured, pingany overlay-1 IP address (such as10.0.0.30) to force the fabric to learn theERSPAN destination IP address. Withoutthese initial pings, the tunnel goes downand ERSPAN fails.

The static CEP tunnel is notup on the leaf switches.

operSt does not show as up. (Login to visore on the leaf switch,find tunnelIf, and look for theERSPAN destination IP.)



SolutionPossible CauseSymptom

Configure the VM to add Option 61 to theDHCP requests.

The DHCP requests do notcontain Option 61.

The ERSPANdestinationVMdoesnot get an IP address throughDHCP on the VTEP port group.





C H A P T E R 9Distributed Firewall


• Distributed Firewall Flow Logging Troubleshooting, page 49

Distributed Firewall Flow Logging TroubleshootingYou can use vemcmd commands to troubleshoot issues with Distributed Firewall flow logging. This sectionlists the commands and their functions and provides sample output.

For general information about Distributed Firewall flow logging and how to configure it, see the section"Distributed Firewall Flow Logging" in the Cisco ACI Virtualization Guide.

For Distributed Firewall scalability information, see the Verified Scalability Guide for Cisco ACI.

vemcmd show dfw flows {all|unreported}Displays all or unreported permit flows.

The following example shows the output of the command vemcmd show dfw flows all:# vemcmd show dfw flows allFor ltl 8

---------------------------------------------------------------------------------------------------ACTIVE LIST:Failed to get DFW Entry

---------------------------------------------------------------------------------------------------ESTABLISHED-FREE LIST:Failed to get DFW Entry

---------------------------------------------------------------------------------------------------FREE LIST:For ltl 50



---------------------------------------------------------------------------------------------------FREE LIST:For ltl 51

---------------------------------------------------------------------------------------------------ACTIVE LIST:


http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html


V/D BUCK SIP DIP SP DP PRO TS EC SQ SQ-2 FGV 458 192.168.5.1 192.168.5.2 35110 5001 6 2657566 0 1 0 13V 1823 192.168.5.1 192.168.5.2 35108 5001 6 2657566 0 1 0 13V 2294 192.168.5.1 192.168.5.2 35109 5001 6 2657566 0 1 0 13V 3922 192.168.5.1 192.168.5.2 35111 5001 6 2657566 0 1 0 13V 3948 192.168.5.1 192.168.5.2 35107 5001 6 2657566 0 1 0 13

---------------------------------------------------------------------------------------------------ESTABLISHED-FREE LIST:

---------------------------------------------------------------------------------------------------FREE LIST:For ltl 52

---------------------------------------------------------------------------------------------------ACTIVE LIST:



---------------------------------------------------------------------------------------------------FREE LIST:

Number of Active Flows: 10

Number of Deleted Flows: 0

Number of Established Free Flows: 0

Number of Free Flows: 0

The following example shows the output of the command vemcmd show dfw flows unreported:# vemcmd show dfw flows unreportedFor ltl 8



---------------------------------------------------------------------------------------------------FREE LIST:For ltl 50



---------------------------------------------------------------------------------------------------FREE LIST:For ltl 51

---------------------------------------------------------------------------------------------------ACTIVE LIST:



---------------------------------------------------------------------------------------------------FREE LIST:For ltl 52

---------------------------------------------------------------------------------------------------ACTIVE LIST:



---------------------------------------------------------------------------------------------------


Distributed Firewallvemcmd show dfw flows {all|unreported}

FREE LIST:

Number of Active Flows: 10

Number of Deleted Flows: 0

Number of Established Free Flows: 0

Number of Free Flows: 0

vemcmd show dfwdenyflows {all|ltl_number}Displays all DFW deny flows or DFW deny flows for a particular LTL.

The following example shows the output of the command vemcmd show dfwdenyflows all:# vemcmd show dfwdenyflows allltl Vem Port Source IP Dest IP Source Port Dest Port Protocol Deny Reason Timestamp

51 UB4_sid.eth0 192.168.5.1 192.168.5.2 4546 500 TCP syn-ack ingress2016-06-20T08:21:10.42151 UB4_sid.eth0 192.168.5.1 192.168.5.2 4549 500 TCP syn-ack ingress2016-06-20T08:21:13.42251 UB4_sid.eth0 192.168.5.1 192.168.5.2 4545 500 TCP syn-ack ingress2016-06-20T08:21:09.42151 UB4_sid.eth0 192.168.5.1 192.168.5.2 4547 500 TCP syn-ack ingress2016-06-20T08:21:11.42251 UB4_sid.eth0 192.168.5.1 192.168.5.2 4548 500 TCP syn-ack ingress2016-06-20T08:21:12.422

The following example shows the output of the command vemcmd show dfwdenyflows 51 where 51 is theLTL number:# vemcmd show dfwdenyflows 51ltl Vem Port Source IP Dest IP Source Port Dest Port Protocol Deny Reason Timestamp

51 UB4_sid.eth0 192.168.5.1 192.168.5.2 4546 500 TCP syn-ack ingress2016-06-20T08:21:10.42151 UB4_sid.eth0 192.168.5.1 192.168.5.2 4549 500 TCP syn-ack ingress2016-06-20T08:21:13.42251 UB4_sid.eth0 192.168.5.1 192.168.5.2 4545 500 TCP syn-ack ingress2016-06-20T08:21:09.42151 UB4_sid.eth0 192.168.5.1 192.168.5.2 4547 500 TCP syn-ack ingress2016-06-20T08:21:11.42251 UB4_sid.eth0 192.168.5.1 192.168.5.2 4548 500 TCP syn-ack ingress2016-06-20T08:21:12.422

vemcmd show dfwslflows {all|ltl_number}Displays all entries or entries for a particular LTL in the short-lived flows table.

The following example shows the output of the command vemcmd show dfwslflows all:# vemcmd show dfwslflows allltl Vem Port Source IP Dest IP Source Port Dest Port Protocol Timestamp52 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35118 TCP 2016-06-20T08:11:34.68952 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35120 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35119 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35114 5001 TCP 2016-06-20T08:09:14.15752 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35116 TCP 2016-06-20T08:09:14.15852 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35115 TCP 2016-06-20T08:09:14.15851 UB4_sid.eth0 192.168.5.1 192.168.5.2 35116 5001 TCP 2016-06-20T08:09:14.15851 UB4_sid.eth0 192.168.5.1 192.168.5.2 35121 5001 TCP 2016-06-20T08:11:34.68952 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35114 TCP 2016-06-20T08:09:14.15752 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35113 TCP 2016-06-20T08:09:14.15051 UB4_sid.eth0 192.168.5.1 192.168.5.2 35112 5001 TCP 2016-06-20T08:09:14.14952 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35119 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35117 5001 TCP 2016-06-20T08:11:34.68952 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35112 TCP 2016-06-20T08:09:14.14951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35120 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35113 5001 TCP 2016-06-20T08:09:14.15051 UB4_sid.eth0 192.168.5.1 192.168.5.2 35118 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35115 5001 TCP 2016-06-20T08:09:14.15852 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35121 TCP 2016-06-20T08:11:34.68952 UB3_sid.eth0 192.168.5.2 192.168.5.1 5001 35117 TCP 2016-06-20T08:11:34.689~ #~ #

The following example shows the output of the command vemcmd show dfwslflows 51where 51 is the LTLnumber:# vemcmd show dfwslflows 51ltl Vem Port Source IP Dest IP Source Port Dest Port Protocol Timestamp


Distributed Firewallvemcmd show dfwdenyflows {all|ltl_number}

51 UB4_sid.eth0 192.168.5.1 192.168.5.2 35119 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35114 5001 TCP 2016-06-20T08:09:14.15751 UB4_sid.eth0 192.168.5.1 192.168.5.2 35116 5001 TCP 2016-06-20T08:09:14.15851 UB4_sid.eth0 192.168.5.1 192.168.5.2 35121 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35112 5001 TCP 2016-06-20T08:09:14.14951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35117 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35120 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35113 5001 TCP 2016-06-20T08:09:14.15051 UB4_sid.eth0 192.168.5.1 192.168.5.2 35118 5001 TCP 2016-06-20T08:11:34.68951 UB4_sid.eth0 192.168.5.1 192.168.5.2 35115 5001 TCP 2016-06-20T08:09:14.158

vemcmd show dfw globalsDisplays Distributed Firewall and logging state, total number of deny flows, permit flows, and short livedflows, respectively.

The following example shows the output of the command vemcmd show dfw globals:# vemcmd show dfw globalsShow DFW GLobals

DFW Feature Enable: ENABLEDDFW Total Flows : 10DFW Flows Allowed : 250000DFW Current Time : 2658561DFW Logging Enable: ENABLEDDFW Deny Logging Total Flows : 0Max DFW Deny Logging flows : 250000DFW Short Lived Total Flows : 0Max DFW Short lived flows : 5000

vemcmd show dfw globals ltl ltl_numberDisplays global statistics for a specified interface.

The following example shows the output of the command vemcmd show dfw globals ltl 51 where 51 is theLTL number:# vemcmd show dfw globals ltl 51Show DFW Port: 51 GLobals

DFW Feature Enable: ENABLEDDFW Total Flows : 10DFW Current Time : 2658777DFW Port Init : 1DFW Port Flows : 5DFW Free Flows : 0

vemcmd show dfw connection statsDisplays consolidated statistics per interface.

The following example shows the output of the command vemcmd show dfw connection stats:# vemcmd show dfw connection statsLTL CREATED DELETED AGED DENIED_GBL DENIED_PORT DENIED_NO_MEM REPLACED UNALIGNED

--- ------- ------- -------- ---------- ----------- ------------- -------- ---------50 0 0 0 0 0 0 0 051 14 0 10 0 0 0 0 052 5 0 0 0 0 0 0 0--- ------- ------- -------- ---------- ----------- ------------- -------- ---------Total 19 0 10 0 0 0 0 0

vemcmd show dfwflows ltl ltl_numberDisplays all permitted Distributed Firewall flows for a specified interface.


Distributed Firewallvemcmd show dfw globals

The following example shows the output of the command vemcmd show dfwflows ltl 51 where 51 is theLTL number:# vemcmd show dfwflows ltl 51Get DFWFLOW Table for ltl: 51

SIP DIP SP DP PRO State Age192.168.5.1 192.168.5.2 35110 5001 TCP ESTABLISHED 0192.168.5.1 192.168.5.2 35108 5001 TCP ESTABLISHED 0192.168.5.1 192.168.5.2 35109 5001 TCP ESTABLISHED 0192.168.5.1 192.168.5.2 35111 5001 TCP ESTABLISHED 0192.168.5.1 192.168.5.2 35107 5001 TCP ESTABLISHED 0

Number of Flows: 5

vemcmd dpa show dfwlog configDisplays configuration information received from APIC to assist with verification of the logging serverconfiguration.

The following example shows the output of the command vemcmd dpa show dfwlog config:# vemcmd dpa show dfwlog config=>dpa command is: show dfwlog configDFW-Log Config:

DFW Log Enable: enabledDFW Deny Logging Enable: enabledDFW Permit Logging Enable: enabledReporting Interval: 300 secSyslog Severity: information (6)Syslog Srvr 1: Enable: 1 IP: 10.197.138.81 Sev: information (6) Fac: local7 (7) Port: 514Syslog Srvr 2: Enable: 0 IP: 0.0.0.0 Sev: information (6) Fac: local4 (4) Port: 514Syslog Srvr 3: Enable: 0 IP: 0.0.0.0 Sev: information (6) Fac: local4 (4) Port: 514Syslog Srvr Name 1: 10.197.138.81Syslog Srvr Name 2:Syslog Srvr Name 3:

#byeBye#


Distributed Firewallvemcmd dpa show dfwlog config


Distributed Firewallvemcmd dpa show dfwlog config

C H A P T E R 10System Troubleshooting


• VEM Commands, page 55

• Cisco AVS Troubleshooting with vemcmd show Commands, page 57

• Cisco AVS Health Status, page 60

VEM CommandsThis section lists some common commands for diagnosing the Virtual EthernetModule (VEM) and controllingVEM kernel logging.

VEM Troubleshooting Commands

Use the following commands to display VEM information:

vem status

Collects status information.

vem version

Collects version information.

vem-support all

Collects support information.

vemcmd

Displays configuration and status information.


vemcmd dpa dump port ltl ltl_ID useg summary

Displays all information about a microsegment applied on the endpoint for a specific local target logic(LTL) number.[example@esx-console ~]# vemcmd dpa dump port ltl 61 useg summary=>dpa command is: dump port ltl 61 useg summaryLTL : 61Table ID : 15584356234873321078Port MAC : 00:50:56:8a:8f:a9VEM Port : AVS-CISCO-CL04-VM-14.eth0Admin : UPLink : UPState : FWDCause : -VLAN/VNID : 9338880Last IP : 0.0.0.0Parent EPG : dvportgroup-19629VM Attr EPG : VS-CISCO-CL04-VM-14-EPG]Effective EPG : CISCO-CL-04-NEW-MAC-EPG]Multicast IP : 225.1.31.2

vemcmd dpa dump port useg summary

Displays summary port Microsegmentation-related information. Combines various parameters fromdifferent commands: vemcmd show port, vemcmd show port vlans, vemcmd dpa dum useg, vemcmdshow portmac, and vemcmd show microsegmentation tables brief.[example@esx-console ~]# vemcmd dpa dump port useg summary=>dpa command is: dump port useg summaryLTL Table_id portmac vem-port state Cause vlan/vnid--------------------------------------------------------------------------------------------60 15584356234873321078 00:50:56:8a:48:84 AVS-CISCO-CL04-VM-02.eth0 FWD - 891291961 15584356234873321078 00:50:56:8a:9e:76 AVS-CISCO-CL04-VM-04.eth0 FWD - 891291962 15584356234873321078 00:50:56:8a:fb:6d AVS-CISCO-CL04-VM-24.eth0 FWD - 891291963 15584356234873321078 00:50:56:8a:56:74 AVS-CISCO-CL04-VM-29.eth0 FWD - 8912919--------------------------------------------------------------------------------------------LTL Parent-EPG VM-Attr-EPG Effective-EPG-----------------------------------------------------------------------51 /ap-AP1/epg-EPG100] None n-te2/ap-AP1/epg-EPG100]52 /ap-AP1/epg-EPG200] None n-te2/ap-AP1/epg-EPG200]60 e1/ap-AP1/epg-EPG3] -AVS-CISCo-VMM-DOM-NEW5] ap-AP1/epg-IP-FILTER-1H]61 e1/ap-AP1/epg-EPG7] -AVS-CISCo-VMM-DOM-NEW5] ap-AP1/epg-IP-FILTER-1H]62 e1/ap-AP1/epg-EPG9] -AVS-CISCo-VMM-DOM-NEW5] ap-AP1/epg-IP-FILTER-1H]63 1/ap-AP1/epg-EPG13] -AVS-CISCo-VMM-DOM-NEW5] ap-AP1/epg-IP-FILTER-1H]

vemcmd help

Displays the type of information you can display.[example@esx-console ~]# vemcmd helpshow card Show the card's global infoshow vlan [vlan] Show the VLAN/BD tableshow bd [bd] Show the VLAN/BD tableshow l2 <bd-number> Show the L2 table for a given BD/VLANshow l2 all Show the L2 tableshow port [priv|vsm] Show the port tableshow pc Show the port channel tableshow portmac Show the port table MAC entriesshow trunk [priv|vsm] Show the trunk ports in the port tableshow stats Show port stats

vemlog

Displays and controls VEM kernel logs. See the following section for specific logging commands.


System TroubleshootingVEM Commands

vemlog show info

Displays information about the log buffer setup.[example@esx-console ~]# vemlog show infoEnabled: YesTotal Entries: 1092Wrapped Entries: 0Lost Entries: 0Skipped Entries: 0Available Entries: 6898Stop After Entry: Not Specified

vemlog show last number-of-entries

Displays the specified number of log entries.[example@esx-console ~]# vemlog show last 5Timestamp Entry CPU Mod Lv MessageOct 13 13:15:52.615416 1095 1 1 4 Warning vssnet_port_pg_data_ …Oct 13 13:15:52.620028 1096 1 1 4 Warning vssnet_port_pg_data_ …Oct 13 13:15:52.630377 1097 1 1 4 Warning svs_switch_state …Oct 13 13:15:52.633201 1098 1 1 8 Info vssnet new switch …Oct 13 13:16:24.990236 1099 1 0 0 Suspending log

VEM Logging Commands

Use the following commands to control the VEM kernel log during troubleshooting:

vemlog clear

Clears the log.

vemlog resume

Starts the log but does not clear the stop value.

vemlog start number-of-entries

Starts the log and stops it after the specified number of entries.

vemlog stop

Stops the log.

vemlog stop number-of-entries

Stops the log after the next specified number of entries.

Cisco AVS Troubleshooting with vemcmd show CommandsBeginning in Cisco AVS Release 5.2(1)SV3(2.5), you can execute vemcmd show commands to troubleshootCisco AVS remotely through Cisco APIC NX-OS style CLI. Previously, the only method you could use toexecute vemcmd show commands was directly on the Cisco AVS host.

You can execute troubleshooting vemcmd show commands through Cisco APIC by entering the followingcommands at the Cisco APIC NX-OS style CLI prompt:apic# attach-avs IP address of hostapic (<host name>)# vemcmd show command


System TroubleshootingCisco AVS Troubleshooting with vemcmd show Commands

The first command above connects you remotely through Cisco APIC to the Cisco AVS host. The secondcommand executes the troubleshooting vemcmd.

You can enter vemcmd show commands only on one Cisco AVS at a time.Note

When you enter commands in Cisco APIC, you cannot use special characters, such as ";", "|", and "&".See the Cisco APIC NX-OS Style CLI Command Reference.

Note

vemcmd show Commands for TroubleshootingYou can view a list of the vemcmd commands available for troubleshooting the Cisco AVS by entering thefollowing command at the Cisco APIC NX-OS style CLI prompt:show cli list | grep vem

The following table lists the vemcmd commands for troubleshooting and a description of what they do.

Descriptionvemcmd command

Shows vLeaf MAC pinning.vemcmd show avs macpinning

Shows cdp information.vemcmd show cdp ltl cdp_ltl [details]

Shows cdp information.vemcmd show cdp neighbors [details]

Shows DFW port connection stats.vemcmd show dfw connection stats dfw-stats-ltl

Displays DFW flows.vemcmd show dfw flows all|unreported

Shows DFW port globals.vemcmd show dfw globals [ltl ltl]

Shows DFW interfaces.vemcmd show dfw interfaces

Shows DFW port drop stats.vemcmd show dfw port-drop stats ltl

Shows DFW port TCP session stats.vemcmd show dfw session stats ltl

Shows DFW deny flows.vemcmd show dfwdenyflows all|0-4096

Shows port DFW FTP flows.vemcmd show dfwflows ftp ltl num

Shows port DFW internal flows.vemcmd show dfwflows internal ltl num

Shows port DFW flows.vemcmd show dfwflows ltl num

Shows DFW short-lived flows.vemcmd show dfwslflows all|0-4096

Shows vLeaf EPP multicast address information.vemcmd show epp multicast


System Troubleshootingvemcmd show Commands for Troubleshooting



Shows the heap list.vemcmd show heap

Shows the host details.vemcmd show host

Shows the L2 table for a given VLAN.vemcmd show l2 l2_vlan

Show the LACP PDU cache.vemcmd show lacp lacp_ltl

Shows LLDP information.vemcmd show lldp ltl lldp-ltl [details]

Shows LLDP information.vemcmd show lldp neighbors [details]

Shows the memory pool list for a given VLAN.vemcmd show mempool

Shows microsegment EPG table information.vemcmd show microsegment tables brief

Shows vLeaf microsegment table information.vemcmd show microsegment tables info tbld-id

Shows vLeaf OpFlex information.vemcmd show opflex

Shows port packet stats.vemcmd show packets

Shows the port channel table.vemcmd show pc info pc-cookie

Shows port-channel mode on uplink.vemcmd show pc mode

Shows the platform-dependent (vssnet) port table.vemcmd show pd-port

Shows the platform-dependent (vssnet) port VLANtable.

vemcmd show port vlans internal|system|vsm

Shows the port table MAC entries.vemcmd show portmac

Shows proxy ARP info.vemcmd show proxy-arp

Shows switch opaque data.vemcmd show sod

Shows SPAN/ERSPAN information.vemcmd show span

Shows port stats.vemcmd show stats cookie stats-cookie

Shows all microsegment port details.vemcmd show useg all

Shows mircosegment EPG table information.vemcmd show useg tables brief

Shows corresponding microsegment table detailsvemcmd show useg tables info tbld-id

Shows unresolved microsegment ports.vemcmd show useg unresolved


System Troubleshootingvemcmd show Commands for Troubleshooting


Shows the VEM and VSM versions.vemcmd show version

Shows number of all valid VLANs and ports usingthem.

vemcmd show vlan vlan_num

Cisco AVS Health StatusThe Cisco ACI reports errors that occur on nodes in the fabric to the Cisco APIC as an aid to troubleshooting.Beginning with Cisco AVS Release 5.2(1)SV3(2.1), Cisco AVS faults are reported as well as faults for leafand spine switches in the ACI fabric.

Viewing the health status for the Cisco AVS can alert you to problems and let you know where they occur.For example, viewing health status can tell you when a port does not attach or when a policy fails to download.

The Cisco AVS monitors states of objects—an EPG, port, global policy, or Virtual Tunnel Endpoint(VTEP)—listed in a database.When an object undergoes a state change, that change is recorded. The databaseis polled every 10 seconds, and when the Cisco AVS detects an abnormal state, it reports the fault to CiscoAPIC. When the object returns to a normal state, the Cisco AVS clears the fault.

Faults Monitored for Cisco AVSThe Cisco AVS monitors two types of faults: host faults and port faults. For example, a host fault is raised ifthere are not enough VTEPs on the host, and a port fault is raised if OpFlex fails to download a base EPG fora port after one minute.

The following table lists the name, type, and description for each fault monitored for the Cisco AVS.

Fault DescriptionFault TypeFault Name

In VPC mode, one OpFlex channel is down.HostSwitch OpFlex Channel is down

The Cisco AVS hasn’t received the switchingmode, encap type, or NS GIPO from the fabric.

HostSwitchVMMDomain Config isn’tdownloaded

The Cisco AVS DPA process has crashed.HostHost Process has crashed

The ESXi kernel has crashed.HostHost Kernel has crashed

A PNIC link is down.PortPort Link is down

A VTEP hasn’t received an IP address from thefabric.

PortVTEP IP isn’t assigned

In VXLAN with MAC pinning mode (VXLANLB), the Cisco AVS has more VTEPs thanPNICs.

PortVTEP isn’t pinned to PNIC


System TroubleshootingCisco AVS Health Status

Fault DescriptionFault TypeFault Name

In VXLAN with MAC pinning mode (VXLANLB), the Cisco AVS has more PNICs thanVTEPs.

PortPNIC isn’t pinned to VTEP

AVMport hasn’t received an attach ack from thefabric.

PortPort Attach isn’t acked

A VM port hasn’t received a detach ack from thefabric.

PortPort Detach isn’t acked

A service VM port has been removed from itsservice chain by the fabric.

PortPort is quarantined

A VM port hasn’t received an EPG from thefabric.

PortPort EPG isn’t downloaded

See the section Viewing Cisco AVS Faults by Type, on page 62 in this guide.

Viewing Faults for Cisco AVSYou can view the status of Cisco AVS faults in Cisco APIC in two ways. You can view all the faults of aspecific domain or type through the System tab. Alternatively, you can navigate to and select an object andthen view the different faults for just that object.

For general information about monitoring network health—including viewing faults—in Cisco APIC, see theCisco APIC Troubleshooting Guide. For detailed information about faults, see the Cisco APIC Faults, Events,and System Messages Management Guide.

Viewing All Cisco AVS Faults by Domain or TypeIn the Cisco APIC, you can click different areas in the GUI to display all the faults of a specific domain ortype for the Cisco AVS and then detailed information about specific faults.

Step 1 Log in to the Cisco APIC, choosing Advanced or Basic mode.Step 2 If you are using Basic mode, choose System > System.

In Advanced mode, the System tab Dashboard opens by default.

Step 3 To view all the Cisco AVS faults, double-click the dark orange major fault icon in the Fault Counts By Domain areaor the Fault Counts By Type area.All Cisco AVS faults are classified as major in Cisco APIC. However, not all major faults are related to Cisco AVS.


System TroubleshootingViewing Faults for Cisco AVS

The list of major faults by domain or type opens in the Faults pane. Cisco APIC displays the faults by Severity, Domain,Type, Code, Count, Cause, and Sample Fault Description. All Cisco AVS faults are in the External domain.

Step 4 Read the Sample Fault Description to help determine whether the fault is related to Cisco AVS. For example, it mightrefer to a problem with OpFlex. You can hover your cursor over the Sample Fault Description for more information.

Step 5 To see detailed information about a particular fault, double-click the row with the fault.The Faults pane displays additional information about the fault: Severity, Acknowledged, Code, Cause, Creation Time,Last Transition, Affected Object, Lifecycle, and Description. Descriptions of Cisco AVS faults begin with the string"Virtual Switch fault."

Step 6 For an explanation of the fault and recommended action, double-click the row with the fault.The Fault Properties window for the fault opens. You can copy and paste information from the Fault Propertieswindow, but you cannot do so from the Faults pane.

What to Do Next

Once you have pinpointed the fault, follow procedures in this guide to resolve the problem. If the problempersists, contact Cisco Customer Support.

Viewing Cisco AVS Faults by TypeIf you think you know what kind of fault has occurred, you can navigate to a controller or EPG in Cisco APICand select it to view fault information. Port faults can appear under a controller or an EPG, depending on theindividual fault. Host faults appear under a controller.

Step 1 Log in to the Cisco APIC, choosing Advanced or Basic mode.Step 2 Take one of the following actions:

ThenTo look for...

ChooseVMNetworking > Inventory >VMware > VMM domain >Controllers> controller.

When you choose the controller, the work pane displays properties for the controller,including fault icons near the top of the pane.

A host fault under a controller

Choose Tenants > tenant > Application Profiles > Application EPGs > EPG.

When you choose the EPG, the work pane displays properties for the EPG, includingfault icons near the top of the pane.

A host or port fault under an EPG

Step 3 Hover your cursor over the dark orange major fault icon to see the number of faults for the controller or EPG.Step 4 Click the Faults tab.

The Faults window appears, listing information about the major faults for controller or EPG: Severity, Acknowledged,Code, Cause, Creation Time, Last Transition, Affected Object, Lifecycle, and Description.

Step 5 For an explanation of the fault and recommended action, double-click the row with the fault.


System TroubleshootingViewing Faults for Cisco AVS

The Fault Properties window for the fault opens. You can copy and paste information from the Fault Propertieswindow, but you cannot do so from the Faults pane.

Step 6 In the Faults window, click History to see a list of all the faults that have occurred for the controller or EPG and havenot yet been cleared.

What to Do Next

Once you have pinpointed the fault, follow procedures in this guide to resolve the problem. If the problempersists, contact Cisco Customer Support.

Troubleshooting Cisco AVS FaultsUse the procedures in this section to troubleshoot faults that occur on the Cisco AVS. Faults are listed accordingto their names in the GUI; each section describes the fault and provides steps you can take to address the fault.

In all cases, the user should collect Cisco AVS (vLeaf) and TOR (leaf) log files before and after debuggingthe fault.

Switch OpFlex Channel is downIn virtual port channel (VPC) mode, one OpFlex channel is down.

Step 1 Confirm that at least one OpFlex channel isn't active.

Example:# vemcmd show opflexStatus: 12 (Active)Channel0: 12 (Active), Channel1: 5 (Disconnected)

Step 2 Check that the port channel members are up.

Example:# vemcmd show port

LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port20 Eth1/3 UP UP FWD - 1040 2 0 0 vmnic221 Eth1/4 UP UP FWD - 1040 3 0 0 vmnic3

Step 3 Check that both leaves are operational.Step 4 Check the port channel and VPC configuration in Cisco APIC.Step 5 Restart the DPA.

Example:# vem restart


System TroubleshootingTroubleshooting Cisco AVS Faults

Switch VMM Domain Config isn't downloadedThe Cisco AVS hasn't received the switching mode, encapsulation type, or NS GIPO from the fabric.

Step 1 Confirm by checking that one or more VMM Domain configurations are unknown or " have a value of 0.0.0.0.

Example:# vemcmd show opflexSwitching Mode: unknownEncap Type: unknownNS GIPO: 0.0.0.0

Step 2 Check the VMM domain configuration in Cisco APIC.Step 3 Restart the DPA.


Host Process has crashedThe DPA process has crashed.

Step 1 Confirm by checking for a DPA core file.

Example:# ls -l /var/core/total 0-rwx------ 1 root root 6000000 Jun 19 16:35 vemdpa-zdump.000

Step 2 Collect the Cisco AVS and leaf switch log files and send them to Cisco Customer Support.Step 3

Host Kernel has crashedThe ESXi kernel has crashed.

Step 1 Confirm by checking for a kernel core file.



Example:# ls -l /var/core/total 0-rwx------ 1 root root 6000000 Jun 19 16:36 vmkernel-zdump.000

Step 2 Collect the Cisco AVS and leaf log files and submit them to Cisco Customer Support.

Port Link is downA PNIC link is down.

Step 1 Confirm by checking that a port channel member link is down.


LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port20 Eth1/3 UP UP FWD - 1040 2 0 0 vmnic221 Eth1/4 UP DOWN BLK - 0 3 0 0 vmnic3

Step 2 Check that both leaf switches are operational.Step 3 Check the port channel and VPC configuration in Cisco APIC.

VTEP IP isn't assignedA VTEP hasn't received an IP address from the fabric.

Step 1 Confirm that the VTEP's vmknic has a valid IP address.

Example:# esxcfg-vmknic -l | grep vmk1 (vmk2, etc)vmk1 10 IPv4 169.254.16.177 255.255.0.0 10.0.255.255 00:50:56:6e:7b:001500 65535 true DHCP

Step 2 Check that the port channel members' links are up.


LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port20 Eth1/3 UP UP FWD - 1040 2 0 0 vmnic221 Eth1/4 UP UP FWD - 1040 3 0 0 vmnic3

Step 3 Check that both leaf switches are operational.Step 4 Check the port channel and VPC configuration in Cisco APIC.Step 5 Remove the VTEP's vmknic.Step 6 Add a new vmknic to the VTEP port group.



VTEP isn't pinned to PNICIn VXLAN Mac Pinning mode (VXLAN LB), there are more VTEPs than PNICs. VXLAN load balancingcan't use the extra VTEPs.

Step 1 Confirm that the number of available VTEPs is greater than the number of available uplinks."

Example:# vemcmd show avs macpinningAvailable VTEPs: 2Available, Pending uplinks: 1, 0Usable VTEP-uplink pairs: 1

Step 2 Add PNICs or remove VTEPs until there is one VTEP per uplink.We recommend adding PNICs, which increasethe capacity of VXLAN load balancing.

PNIC isn't pinned to VTEPIn VXLAN Mac Pinning mode (VXLAN LB), there are more PNICs than VTEPs. VXLAN load balancingcan't use the extra PNICs.

Step 1 Confirm the VTEP-to-PNIC pinning.

Example:# vemcmd show avs macpinningAvailable VTEPs: 1Available, Pending uplinks: 2, 0Usable VTEP-uplink pairs: 1

Step 2 Add VTEPs or remove PNICs until there is one VTEP per uplink.We recommend that you add VTEPs, which increase the capacity of VXLAN load balancing.

Port Attach isn't ackedA VM port hasn't received an attach acknowledgment from the fabric. The port might not forward traffic ormight forward traffic with a stale EPG.

Step 1 Confirm that the port state is WAIT_ATT_ACK.



Example:# vemcmd dpa dump attach=>dpa command is: dump attachLTL Name EPG-Alias State Tries EPCP---------------------------------------------------------------------------51 ubuntu-vm-1 dvportgroup-7980 WAIT_ATT_ACK 0 N

Step 2 Check that at least one OpFlex channel is active.

Example:# vemcmd show opflexStatus: 12 (Active)Channel0: 12 (Active), Channel1: 12 (Active)

Step 3 Detach and reattach the port.Step 4 Restart the DPA.


Port Detach isn't ackedA VM port hasn't received a detach acknowledgment from the fabric. The port can't be reattached until itfinishes this detach.

Step 1 Confirm that the port state is DETACHED.

Example:# vemcmd dpa dump attach=>dpa command is: dump attachLTL Name EPG-Alias State Tries EPCP---------------------------------------------------------------------------51 ubuntu-vm-1 dvportgroup-7980 DETACHED 0 N



Step 3 Restart the DPA.




Port is quarantinedThe fabric has removed a service VM port from its service chain.

Step 1 Confirm that the port state is NACK_RCVD.

Example:~ # vemcmd dpa dump attach=>dpa command is: dump attachLTL Name EPG-Alias State Tries EPCP---------------------------------------------------------------------------52 service-vm-1 dvportgroup-7980 NACK_RCVD 0 N

Step 2 Detach the port.

Port EPG isn't downloadedAVM port hasn't received an EPG from the fabric. The port might not forward traffic or might forward trafficwith a stale EPG.

Step 1 Confirm that the port state is WAIT_FOR_EPP.

Example:# vemcmd dpa dump attach=>dpa command is: dump attachLTL Name EPG-Alias State Tries EPCP---------------------------------------------------------------------------51 ubuntu-vm-1 dvportgroup-7980 WAIT_FOR_EPP 0 N



Step 3 Detach and reattach the port.Step 4 Restart the DPA.




Documents

Cisco Application Virtual Switch Troubleshooting Guide ... · Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(2.x) First Published: 2016-07-01 Last Modified: