ASA-PIX/FWSM: Multicast Tips and Common ProblemsThe purpose of this document is to provide assistance to everyone in configuring and troubleshooting multicast through the firewall.

This document is meant to be interpreted with the aid of the official documentation from the configuration guide located here: ASA: http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/route_multicast.htmlFWSM: http://www.cisco.com/en/US/docs/security/fwsm/fwsm32/configuration/guide/ip_f.html#wp1041648PIX: http://www.cisco.com/en/US/docs/security/pix/pix63/configuration/guide/bafwcfg.html#wp1170913 The Cisco TAC has created another ASA Multicast Troubleshooting and Common Problems guide and posted it to Cisco.com:http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080befd2a.shtml Support:ASA supports igmp forwarding, sparse mode, and bidirectional mode. There is no sparse-dense-mode support. ASA forwards auto-rp packets (unless configured to not to). ASA itself requires static auto-rp config.

Configuring multicast:Pls. follow this link: http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a00807631d2.shtml

Shared tree and Source specific tree:I read once that Multicast is like dating service. Sender starts the stream and the DR on the local LAN registers the sender with the dating service (RP). Reciever sends IGMP report and the DR on that local LAN segment sends a PIM join towards the RP- shared tree. Once the dating service connects the two - sender and receiver, then they can talk via the shortest path and do not have to go through the dating service - shared tree anymore.

Where to add these commands:igmp join-group:This command when applied under the interface, sends a join for the group address. Think of this as a permanent receiver for the group. This comes in handy when the receivers off of the interface do not send igmp reports in a regular interval. This command makes the firewall to accept and forward the multicast packets. Configuring the firewall to join a multicast group causes upstream routers to

maintain multicast routing table information for that group and keep the paths for that group active. Where to add this command: This command needs to be configured under the interface config mode facing the receivers.hostname(config-if)# igmp join-group group-address igmp static-group:The firewall does not accept the multicast packets but rather forwards them to the specified interface. To configure the security appliance to forward the multicast traffic without being a member of the multicast group, use the igmp static-group command. When this command is added, the ASA sends out an IGMP report out the interface sourcing with its IP address on the interface. Where to add this command: This command needs to be applied under the interface config facing the receivers.hostname(config-if)# igmp static-group group-address

igmp forward interface:Where to add this command: This command when applied under the interface config facing the receivers. This command will forward all the igmp reports received on the interface towards the interface where the server is located. This command cannot be configured along with PIM.hostname(config-if)# igmp forward interface outside

Terminology:What is a first hop router:A first hop router is the router that is connected to the source which is responsible for registering the source with the RP. If there are two routers connected to the same segment as the source then the one that is the DR (designated router) for that LAN segment will take care of the registering process. Now, if one of the two is the firewall and you want the firewall to take care of the registration process because the RP is on the other side of the firewall then, the firewall should be DR for that LAN segment.

What is a last hop router?A last hop router is the one that is connected to the receiver. Again if there are two or more routers in the LAN connected to the receiver, the router which is the DR is the one that is resonsible to connect the receiver to the shared tree. If one of the devices is the firewall then, if we want the firewall to process the igmp reports then it has to be the DR with a higher priority.

OIL:OIL stands for Outgoing Interface List. OIL is always taken from the (*,G) and copied as the OIL for the (S,G).In coming interface and outgoing interface for (*, G):Incoming interface for the (*, G) is always towards the RP. Outgoing interface is towards the receiver.In coming interface and outgoing interface for (S, G):Incoming interface for (S,G) is always towards the source. Outgoing interface is towards the receiver.DR - How to increase the pim priority:The following example sets the DR (Designated Router) priority for the interface to 5:hostname(config-if)# pim dr-priority 5

PIM Chart:

(*,G) (S,G)

When it is created

By receipt of IGMP Membership report from directly-connected Receiver (host)

Dynamically created when an (S,G) entry must be created.

By receipt of (S,G) PIM join By receipt of Multicast

packet

OIL info

Interface that received IGMP membership report

Interface that received PIM (*,G) Join

Manually configured If none of the above,

"NULL"

Interface that received a PIM (S,G) join

Cop of (*,G) OIL except when matching (S,G) IIF

Otherwise "NULL"

RPF info

IP address of next-hop neighbor towards RP according to unicast routing table.

If on the RP itself then "NULL"

IP address of next-hop neighbor towards the Multicast Source

If directly connected to Multicast Source then 0.0.0.0

IIF Info

Interface that leads to RP according to unicast routing table

If on the RP itself then "NULL"

Interface that leads to Multicast Source according to unicast routing table.

When this when (S,G) entry expires After 210 sec. if no

entry is deleted?

When notified by the IGMP process that all members for a group are gone.

By receipt of PIM (*,G) Prune message from downstream neighbor.

multicast packets or PIM register messages received from SPT.

When (*,G) entry deleted.

How to interpret the "sh mroute x.x.x.x"Sample 1:Topology:Receiver--RP---(INT_SCI/8)ASA(INT_MANDAT/0)--(10.2.112.9)sourceGroup address: 239.252.1.10ASA#sh mroute 239.252.1.10 Multicast Routing TableFlags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected, L - Local, I - Received Source Specific Host Report, P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT Timers: Uptime/Expires(10.2.112.9, 239.252.1.10), 00:10:48/00:03:11, flags: SFT Incoming interface: INT-MANDAT RPF nbr: 10.2.112.9, Registering Outgoing interface list: INT-SCI, Forward, 00:10:48/00:03:29 Tunnel0, Forward, 00:10:48/never

There is no *, G which indicates that the receiver hasn't requested the feed yet. Only the source has started and the source lives behind the interface named INT-MANDAT. "F" - is the Register Flag. Meaning the firewall hasn't seen a register stop message from the RP yet. The "Tunnel0" interface is a transient state and goes away once the register stop is received. RP lives behind the INT-SCI interface.

Sample 2:RPF neighbor all zeros means that the ASA is the first hop router connected local to the sender.When you see the "T" flat you can jump in joy. This mean the shortest path tree has formed and multicast traffic is being received by the receiver. Topology:receiver--(inside)ASA(outside)--sender

ASA#sh mroute 230.0.0.10 (*, 230.0.0.10), 00:08:20/never, RP 0.0.0.0, flags: SCL (Connected Local LAN)Incoming interface: Null -----> receiver is directly connected RPF nbr: 0.0.0.0 Immediate Outgoing interface list: ------> receiver is off of the inside interface inside, Forward, 00:08:20/never (10.135.152.47, 230.0.0.10), 00:08:16/00:03:13, flags: SJT (shortest Path Tree - T) Incoming interface: outside ------> source is on the outside interface. RPF nbr: 0.0.0.0 Inherited Outgoing interface list: inside, Forward, 00:08:20/never ------> receiver is off the inside interface. Sample 3:Topology:sender(172.25.1.11)--RP(172.20.50.254)--vlan50--nonpci(0)ASA(0)WLAN--vlan30--Receiver(172.20.30.133) same security level - 0

Group add 225.4.5.7

ASA#sh mroute 225.4.5.7(*, 225.4.5.7), 01:08:42/never, RP 172.20.50.254, flags: SCLJ Incoming interface: nonpci ------> (routing table shows int nonpci towards the RP) RPF nbr: 172.20.50.254 Immediate Outgoing interface list: ------> (receiver is off the interface WLAN) WLAN, Forward, 01:08:42/never

(172.25.1.11, 225.4.5.7), 00:04:59/00:03:00, flags: SJT Incoming interface: nonpci ------>(routing table shows int nonpci for the source) RPF nbr: 172.20.50.254 Inherited Outgoing interface list: WLAN, Forward, 01:08:42/never------> (receiver is off the interface WLAN)

How to interpret the "sh mfib x.x.x.x"Sample 1:Topology: (10.204.125.81)R1-------(10.204.125.85/OUT)***ASA***(IN/172.25.250.11)-R2(172.25.250.1)--receiver. I --------------

| | Sender-10.29.95.133 RP-10.29.95.252Group address: 233.49.81.127

Source or source is on the lower security interface and the receiver is on the higher security interface. It is very important to see the"F" flag so indicate that the ASA is forwarding multicast traffic.

ASA5520-01# sh mfibEntry Flags: C - Directly Connected, S - Signal, IA - Inherit A flag, AR - Activity Required, K - KeepaliveForwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per secondOther counts: Total/RPF failed/Other dropsInterface Flags: A - Accept, F - Forward, NS - Negate Signalling IC - Internal Copy, NP - Not platform switched SP - Signal PresentInterface Counts: FS Pkt Count/PS Pkt Count

(*,233.49.81.127) Flags: C K Forwarding: 0/0/0/0, Other: 846486/2/846484 OUT Flags: A NS ----> We are accepting packets on the outside interface inside Flags: F NS ---> We are forwarding packets to the inside interface. Pkts: 0/0 (10.29.95.133,233.49.81.127) Flags: K Forwarding: 1/0/36/0, Other: 0/0/0 OUT Flags: A inside Flags: F NS Pkts: 0/1

Sample 2:Topology:

receiver--(inside)ASA(XETRA)--router--N/W-RP-Source

source: 193.29.93.62Group address: 224.0.46.0

ASA5520-01# sh conn

UDP XETRA 193.29.93.62:25100 inside 224.0.46.0:55199, idle 0:00:00, bytes 2649468, flags -

ASA5520-01# sh mfib 224.0.46.0 193.29.93.62

Entry Flags: C - Directly Connected, S - Signal, IA - Inherit A flag, AR - Activity Required, K - KeepaliveForwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per secondOther counts: Total/RPF failed/Other dropsInterface Flags: A - Accept, F - Forward, NS - Negate Signalling IC - Internal Copy, NP - Not platform switched SP - Signal PresentInterface Counts: FS Pkt Count/PS Pkt Count

(193.29.93.62,224.0.46.0) Flags: K Forwarding: 18629/10/1295/105, Other: 4716/0/4716 XETRA Flags: A F ---> XETRA interface is where the source is and we Accept packets from. Pkts: 0/0 inside Flags: F NS ---> inside interface is where the receiver is where we Forward the packets to. Pkts: 1598/2

Common Problems:Syslog 106010:Topology:receiver---(inside)FWSM(outside)---sender (10.11.21.10)group add: 239.226.16.3%FWSM-3-106010: Deny inbound udp src OUTSIDE:10.11.21.10/1450 dst DMZ-EMP90:239.226.16.3/30120

With nat-control enabled on the FWSM the above issue can be corrected with the following:

nat (inside) 0 access-list 101access-list 101 permit ip host 239.226.16.3 host 10.11.21.10

Think of the above as the source when sends out multicast packets the destination is the multicast address 239.226.16.3 and is destined towards the inside. Even though no traffic will be sourced from a multicast address, we just need to providetranslation for the reverse flow from high security to low security.

If this case were an ASA we would see the following in asp drop captures 1. ASA was dropping all the multicast packet for the reason below:union-asa# sh cap capasp | i 239.0.1.2 38: 12:07:56.782689 10.80.8.38> 239.0.1.2: icmp: echo request Drop-reason: (no-mcast-intrf)

2. nat 0 with acl was configured on this ASA and it didn't allow exemption for our group 239.0.1.2. Once I added that flow reached the other ASA2 and we saw *,g as well as s,g mfib limit (5000) reached:"sh mfib <group-address>" may not show any output. This may due to the fact that the mfib entries max limit may have been reached. This ENH request CSCtj22365 is filed so, that when resolved, the FWSM will send a syslog message indicating that the max mfib entry limit has been reached.

fwsm# sh mfib sumIPv4 MFIB summary: 5000 total entries [4974 (S,G), 23 (*,G), 3 (*,G/m)] 190440 total MFIB interfaces

Failed to locate egress interface:Oct 23 2010 21:40:44: %ASA-6-110002: Failed to locate egress interface for UDP from outside:10.135.152.47/1034 to 230.0.0.10/7060

Look for contradicting lines in the config. In this case where does the sender live on the inside or outside?

static (inside,outside) 10.135.152.47 10.135.152.47 net 255.255.255.255mroute 10.135.152.47 net 255.255.255.255 outside

In the above case we had to remove the static and "toggle" multicast-routing on the ASA.HSRP:Let us say that the RP is a loop back address configured on a pair of routers running HSRP. We need to make sure the mroute configured on the firewall points to one of the physical IP addresses and not the HSRP address. When we issue "sh pim int <name>" , the firewall only forms neighbor relationship with the physical IP addresses and not the HSRP address. Also, we cannot configure mroute to both the physical IP addresses. There is no redundancy when it comes to multicast. Pls. read this link for further explanation: http://www.cisco.com/en/US/tech/tk828/technologies_tech_note09186a0080094aab.shtml TTL (Time to Live Value) too low or set to 1:When a router forwards a multicast packet from one interface to another, it decrements the Time To Live (TTL) value in the IP header by one. The firewall does not decrement the TTL value in the IP header but, still it will drop the packet if the

TTL is set to 1. It sends the packet only if the resulting TTL value is not zero and is greater than the multicast TTL threshold value of the outgoing interface when configured. If a multicast application on the source allows the setting of a TTL value and results in not matching the mentioned criteria, the packets are dropped by the router. The multicast source needs to set the TTL value as there are number of hops between the source and the receiver.

Multicast TTL threshold:http://www.cisco.com/en/US/customer/tech/tk828/technologies_tech_note09186a0080094b55.shtml#ttlsettingMulticast TTL value too low or set to 1: http://www.cisco.com/en/US/customer/tech/tk828/technologies_tech_note09186a0080094b55.shtml#ttlthreshold Multlicast over VPN tunnel:ASA/PIX will not pass multicast traffic over IPsec VPN tunnels. Only unicast traffic over VPN tunnel is supported. The alternative option is to use GRE between two end points on either side of the tunnel and send multicast over GRE and encrypt that traffic and send it over the tunnel. FP no mcast output intrfR1(receiver)--(inside)ASA(outside)---R3(rp/sender) Group 239.1.1.1 "capture capasp typs asp-drop all" may show the following: ASA1# sh cap capasp | i 239.1.1.1 18: 14:52:02.979731 10.7.123.3.64281 > 239.1.1.1.5000: udp 16 Drop-reason: (no-mcast-intrf) FP no mcast output intrf Here is the command reference link to sh asp-drop:Name: no-mcast-intrfFP no mcast output intrf: All output interfaces have been removed from the multicast entry. - OR - The multicast packet could not be forwarded.Recommendation: Verify that there are no longer any receivers for this group. - OR - Verify that a flow exists for this packet. Syslogs:

None In this case R1 being the receiver was also the DR for that segment so, the ASA did not do anything with the IGMP reports that it received. In this case making the ASA as the DR resolved the issue. Debug pim group x.x.x.x shows Assert processing message winsTopology:Group address 239.1.1.247vlan13 - INSIDEvlan300 - OUTSIDE FWSM (10.20.213.1)----vlan 13----FWSM---vlan300--RTR----RP---Sender (172.16.41.205) | Receivers (10.20.213.204)----| | ASA(10.20.13.2)----| IGMP reports from the receivers arrive on the FWSM but, the FWSM doesn't send PIM join up to the RP. The egress multicast packets do not show the joins due to CSCsf31515Debug pim group 239.1.1.247 showed the following: IPv4 PIM: (172.16.41.205,239.1.1.227)RPT J/P adding Prune on OUTSIDEIPv4 PIM: (172.16.41.253,239.1.1.227)RPT J/P adding Prune on OUTSIDEIGMP: Received v2 Report on INSIDE from 10.20.213.201 for 239.1.1.227IGMP: Updating EXCLUDE group timer for 239.1.1.227IPv4 PIM: (172.16.41.205,239.1.1.227) Received [15/110] Assert from 10.20.13.2 on INSIDEIPv4 PIM: (172.16.41.205,239.1.1.227) Assert processing message winsIPv4 PIM: (172.16.41.205,239.1.1.227) INSIDE Update assert timer (winner 10.20.13.2) FWSM (backup site)ASA (production site) FWSM and ASA though configured to be on the same vlan, were not supposed to see each other. The problem is that the FWSM saw the presence of the ASA but the ASA didn't see the presence of the FWSM. FWSM was DR on VLAN 13 and the ASA was DR (since it did not see the FWSM) as well. Both tried to process multicast packets and ended in a race condition and the Assert messages and winner indicated that. Troubleshooting commands:On the Firewall:

sh pim neighborsh igmp interfacesh igmp trafficsh mroute x.x.x.xsh mfib x.x.x.xdebug pim group x.x.x.xdebug igmp group x.x.x.x

On the Router:sh ip pim interfacesh ip mroute x.x.x.xsh ip igmp groupdebug ip pim

---Doc Reference from https://supportforums.cisco.com/docs/DOC-12943

More Cisco and Networking Tips you can visit: http://blog.router-switch.com/category/networking-2/