DNS DDoS Analysis and Defense

2013. Sep. Sukbum Hong (antihong@gmail.com)

Please let me know, if there is any error, question, or comment.

Page 1

Contents • DNS Risks • DNS DDoS Attack History • DNS DDoS Attack Types and Defense

(1) Bandwidth Consuming attack (2) Massive A type Query attack (3) Massive other query type attack (4) Amplification attack open-resolver project (5) non-existant(NXDOMAIN) query attack (6) RRL defense

• Conclusion

Page 2

What’s happen if DNS was attacked? Why Risk?

• Every service(web,e-mail,intranet) will be shutdown if DNS is down

• All domains in DNS server will be impacted -. Generally One DNS server services several thousands of domains together

• Hard to change the IP address if get attacked -. generally, DNS TTL is 1Day or 2Days -. need to change the NAMEHOST(whois) as well as DNS A record

• Hard to defense attack as DNS packet is simple -. impossible to distinguish the legitimate traffic from illegitimate traffic

• Hard to defense attack as DNS packet is UDP -. No protocol based ACL

• Hard to block the source IP using rate-limit based

-. As attacker can spoof the Source IP address

Page 3

“Operation Global Blackout” on 31st March :: 2012/03

http://pastebin.com/NKbnh8q8 The principle is simple; a flaw that uses forged UDP packets is to be used to trigger a rush of DNS queries all redirected and reflected to those 13 IPs. The flaw is as follow; since the UDP protocol allows it, we can change the source IP of the sender to our target, thus spoofing the source of the DNS query. The DNS server will then respond to that query by sending the answer to the spoofed IP. Since the answer is always bigger than the query, the DNS answers will then flood the target ip. It is called an amplified because we can use small packets to generate large traffic. It is called reflective because we will not send the queries to the root name servers, instead, we will use a list of known vulnerable DNS servers which will attack the

root servers for us.

Page 4

GoDaddy Outage Takes Down Millions of customer Sites :: 2012/09/10

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/

http://www.wired.com/wiredenterprise/2012/09/godaddy-moves-to-verisign/

Amid Outage, GoDaddy Moves DNS

to Competitor VeriSign

Page 5

Knocked Spamhaus offline with 120G or 300Gbps attack :: 2013/03/19

http://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-offline-and-ho

Officially 120Gbps, Unofficially 300Gbps attack

Page 6

300Gbps almost broke the Internet?

http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet

there's an online attack underway.

The biggest in history??

Enough to slow down the internet……………….???

http://gizmodo.com/5992652/that-internet-war-apocalypse-is-a-lie

That Internet War Apocalypse Is a Lie

Just to put it in perspective the traffic estimates for the DDOS attack were as high as 300 Gbps at the target. That would easily overwhelm the average hosting center, but not a core component of the Internet. For example, DECIX, the German Internet exchange in Frankfurt, regularly handles 2.5 Tbps at peak on any given day: http://www.de-cix.net/about/statistics/

While it may have severely affected the websites it was targeted at, the global Internet as a whole was not impacted by this localized incident.

Page 7

NetworkSolutions outage because of DNS DDOS attack

http://blogs.cisco.com/security/hijacking-of-dns-records-from-network-solutions/

Over 5,000 domains including linkedin.com in NSI DNS customers were changed with Unknown DNS.

ns1624.ztomy.com(204.11.56.20) . It was in the process to move to Prolexic to defense the DDoS attack

July 16, 2013

http://blogs.cisco.com/security/network-solutions-customer-site-compromises-and-ddos/

Linkedin.com homepage was redirected to DomainSale Homepage

$ traceroute -q 1 ns1624.ztomy.com. ....... 15 209.200.136.34 (209.200.136.34) 118.578 ms 16 unknown.prolexic.com (72.52.18.126) 239.717 ms 17 204.11.56.20 (204.11.56.20) 235.350 ms

July 16, 2013

June 20, 2013

Service outage for 24 hours because of ddos attack

Page 8

Hosting Service outage :: June

:: Hosting Provider in Sydney

http://www.tppwholesale.com.au/support/service-alerts/more-information-recent-ddos-attacks

we took the drastic step of rate limiting DNS queries using the Arbor Pravail equipment to stem the flow of the attack. but due to the aggressive filtering nature there will be some false positives and some customers who will be denied services despite being legitimate users. This kind of rate limiting is not ideal or a long term solution and will result in some further inconvenience. Our long term strategy is to further cluster, load balance and segregate name services to provide greatly enhanced scale, fault tolerance and capacity. This had not been required prior to this attack.

:: DNS hosting provider based in Toronto

It was difficult to differentiate the real traffic from the DDoS traffic

“This is the ‘nightmare scenario’ for DNS providers, because it is not against a specific domain which we can isolate and mitigate, but it’s against easyDNS itself”

http://blog.easydns.org/2013/06/04/post-mortem-of-the-june-3-4th-ddos/

We can recommend DNSMadeEasy, DNSimple or No-Ip, then there's Route53 (our users had good results with easyRoute53 overnight). - See more at: http://blog.easydns.org/2013/06/04/post-mortem-of-the-june-3-4th-ddos/#sthash.wRlXg4iO.dpuf

:: DNS hosting Provider in Florida

“The attacker essentially flooded us with ‘ANY’ queries for a variety of domains managed by our DNS service, with the intention of amplifying these small queries into significantly larger responses aimed at a specific network.”

Page 9

.CN root DNS outage

8/25(Sun) :: 00:00 DDoS attack to .CN root DNS server

02:00 Mitigated the attack

04:00 doos attack again

01:00 service recovered

No information about Attack size, how to attack,etc.

cn. 172800 IN NS a.dns.cn. cn. 172800 IN NS b.dns.cn. cn. 172800 IN NS c.dns.cn. cn. 172800 IN NS d.dns.cn. cn. 172800 IN NS e.dns.cn. cn. 172800 IN NS ns.cernet.net.

google.cn. 86400 IN NS ns2.google.com. google.cn. 86400 IN NS ns3.google.com. google.cn. 86400 IN NS ns1.google.com. google.cn. 86400 IN NS ns4.google.com. ;; Received 109 bytes from 203.119.27.1#53(c.dns.cn) in 75 ms

Around ~30% .CN traffic was downed even long TTL cache.

TTL 1 hour 1D(24h)

1m 1680(1.7%) 70

2m 3360 140

30m 50,000(50%) 2100

1h 100,000(100%) 4200(4.2%)

How much LDNS can lose the DNS cache?

Assumption : # of LDNS is 0.1M

Page 10

DNS DDoS Attack Types and Defense :: Bandwidth Consuming attack

Packet size based Filtering at Network Level?

30Gbps

20Gbps

1Gbps based network

Solution1 :: PBR impossible as eating too much CPU Router(config)# access-list 111 remark "DNS PBR“ Router(config)# access-list 111 permit udp any host dns.ip.addr eq 53 Router(config)# route-map dnsddos permit 10 Router(config-route-map)# match ip address 111 Router(config-route-map)# match length 512 1500 Router(config-route-map)# set interface Null 0 Router(config-if)# ip route-cache policy Router(config-if)# ip policy route-map dnsddos

route 173.X.X.X/32-DNS-DROP { match { destination 173.X.X.X/32; port 53; packet-length [ 99971 99985 ]; } then discard;}

http://blog.cloudflare.com/todays-outage-post-mortem-82515

-. Direct attack from Zombies

-. Normal traffic should be UDP not TCP TCP :: Zone transfer, when response over 512byte

-. Defense :: distributing the DNS infra

-. Defense :: Packet size based filtering if within the Infra size

-. Defense :: efficient by filtering the fragmented packet in upstream ISP

Page 11

DNS DDoS Attack Types and Defense :: Massive A type Query Attack

1.2.3.4.59873 > 10.10.1.2.53: 53495+ A? www.example.com. (44)

2.3.4.5.46922 > 10.10.1.2.53: 20009+ A? www.example.com. (44)

3.4.5.6.59873 > 10.10.1.2.53: 33495+ A? www.example.com. (44)

4.5.6.7.46922 > 10.10.1.2.53: 40009+ A? www.example.com. (44)

............................?

-. A Kind of QPS attack

-. Direct attack from Zombies

-. If source ip is not spoofed, we can filter rate-limited based policy

-. How to filter ? If source ip is randomly changed? and if the packet is exactly same with normal query traffic?

Victim :: 1.1.1.1

www.example.com IP Address?

How to differentiate attack or normal query?

Page 12

DNS DDoS Attack Types and Defense :: other Query type Attack

$ dig anonsc.com any

anonsc.com. IN A 123.45.67.59

anonsc.com. IN A 123.45.67.60

anonsc.com. IN A 123.45.67.61

anonsc.com. IN A 123.45.67.62

……………….

;; MSG SIZE rcvd: 3271

Ex: Direct attack case

Ex: Amplification attack case

Page 13

tcpdump example when ANY query $ tcpdump -X port 53 -n (or tshark port 53 –n –x)

19:38:14.172255 IP 114.xx.xx.xx.60249 > 61.110.xxx.xxx.domain: 22765+ ANY? cdnetworks.co.kr. (34) 0x0000: 4500 003e 0000 4000 3311 9310 726f 3e14 E..>..@.3...ro>. 0x0010: 3d6e c6ad eb59 0035 002a fb7f 58ed 0100 =n...Y.5.*..X... 0x0020: 0001 0000 0000 0000 0a63 646e 6574 776f .........cdnetwo 0x0030: 726b 7302 636f 026b 7200 00ff 0001 rks.co.kr.....

[0001] means A record type

[0001] means IN

TYPE HEX

A 00010001

ANY 00ff0001

MX 000f0001

NS 00020001

PTR 000c0001

SOA 00060001

AAAA 001c0001

TXT 00100001

HINFO 000d0001

$iptables -A INPUT -p udp --dport 53 -m string --algo bm --hex-string '|00FF0001|' -m recent --set --name dnsany

$iptables -A INPUT -p udp --dport 53 -m string --algo bm --hex-string '|00FF0001|' -m recent --name dnsany --rcheck --seconds 60 --hitcount 5 -j DROP

DNS DDoS Attack Types and Defense :: other Query type Attack

Default # of ipt_recent is 100, so need to maxmize the value in advance $ rmmod ipt_recent modprobe $ ipt_recent ip_list_tot=4095

Page 14

Massive spoofed query as if source ip is Victim Source:: 1.1.1.1:53 or 1024: / Dst :: open Resolver:53

Massive response from open resolver Source:: Open_Resolver:53 / dst :: 1.1.1.1:53(or 1024~)

command

Distributed reflective, amplified attack

Prepare big size response packet in advance

$ dig re.vr.lt txt 60byte

;; MSG SIZE rcvd: 4000(byte)

x ~70 times Victim :: 1.1.1.1

re.vr.lt DNS server :: 2.2.2.2

Page 15

Distributed reflective, amplified attack

Victim IP

Resolver DNS

Packet generating from Zombie PC

Packet from Victim Side

Resolver DNS

Page 16

Distributed reflective, amplified attack

IP (tos 0x0, ttl 49, id 25778, offset 0, flags [DF], proto: UDP (17), length: 65) 96.31.66.143.80 > 61.110.198.173.53: [udp sum ok] 59637+ [1au] TXT? re.vr.lt. ar: . OPT UDPsize=4096 (37)

IP (tos 0x0, ttl 64, id 34118, offset 0, flags [+], proto: UDP (17), length: 1500) 61.110.198.173.53 > 96.31.66.143.80: 59637 q: TXT? re.vr.lt. 1/4/1 re.vr.lt. TXT[|domain]

IP (tos 0x0, ttl 64, id 34118, offset 1480, flags [+], proto: UDP (17), length: 1500) 61.110.198.173 > 96.31.66.143: udp

IP (tos 0x0, ttl 64, id 34118, offset 2960, flags [none], proto: UDP (17), length: 1039) 61.110.198.173 > 96.31.66.143: udp

# dig re.vr.lt txt +bufsize=4096

;; MSG SIZE rcvd: 3878

Page 17

Distributed reflective, amplified attack

# tcpdump -w dns.pcap -nn host 96.31.66.143

Only 1st response is DNS and the rests are Fragmented UDP packets

EDNS0 사용(Extension Mechanism for DNS) :: rfc2671

DNS 요청자는 RFC 2671에 정의된 EDNS0(DNS 확장 메커니즘)을 사용하여 UDP 패킷의 크기를 알리고 UDP 패킷 크기의 원래 DNS 제한(RFC 1035)인 512(8진수)보다 큰 패킷 전송을 이용할 수 있습니다. DNS 서버는 UDP 전송 계층에서 요청을 받으면 OPT RR(리소스 레코드)에서 요청자의 UDP 패킷 크기를 확인하고 요청자가 지정한 최대 UDP 패킷 크기에 허용되는 만큼 리소스 레코드가 포함되도록 응답의 크기를 조절합니다.

$ man dig

+bufsize=B

Set the UDP message buffer size advertised using EDNS0 to B bytes. The maximum and minimum sizes of this buffer are 65535 and 0 respectively. Values outside this range are rounded up or down appropriately.

Page 18

Distributed reflective, amplified attack

http://dns.measurement-factory.com/surveys/openresolvers/ASN-reports/latest.html

http://www.chaz6.com/files/resolv.conf :: list of public ipv4/ipv6 dns cache servers

152,600 x Open Resolver !!!

http://openresolverproject.org/breakdown.cgi

2013-09-01 results 27,166,819 gave the correct answer to the A? for the DNS name queried

152,600 x 4,000 byte x 8(bit) = 4.8Gbps??

http://openresolverproject.org/searchby-asn.cgi?search?asn=XXXX ASN

Assumption :: if one zombie can query 152,600 open resolver in a second if one open resolver can generate 4,000 byte answer then, one DNS query can be 4.8Gbps traffic

Page 19

How to do at each Backbone/Access level?

20Gbps

40Gbps

Access Control List How to Filter? BackBone NET Level Access NET Level

Filtering big size UDP packet against the DNS server

Access Control based on Source Port and Destination Port Src:53 / Dst:53 ??

Access Control Filtering for Fragmented packets

Source IP Validation

SRC based Ratelimit

Signature based Filtering

Auth. DNS Server(ns1/ns2..)

Page 20

Signature Based Filtering against Amplification

How to filter if we get massive response packets , i,e. amplification attack

According to below image, we can see that QUESTION means 00010000 which means Questions :1, ANSWER:0

$ iptables -A INPUT -p udp --dport 53 -m string --algo bm --from 31 --to 32 --hex-string ! '|00010000|' -j DROP

# iptables -m string -h

string match options:

--from Offset to start searching from

--to Offset to stop searching

--algo Algorithm

[!] --string string Match a string in a packet

[!] --hex-string string Match a hex string in a packet

Page 21

Massive QUERY for $RANDOM.domain.com :: Non-Existent host

Objective :: The DNS server spends its time searching for something that doesn't exist instead of serving legitimate requests.

The result is that the cache on the DNS server gets filled with bad requests, and clients can't find the servers they are looking for.

• source IP based rate-limit if the source ip is not spoofed

• query type(ANY,TXT,CNAME,etc) based rate-limit or filtering

• it maybe problem if standard A query type with spoofed random source IP

NXDOMAIN query attack

Nov 21 09:09:58 s332-kt9-sel named[4942]: client 170.160.126.199#1234: query (cache) 'www.ceyxyl.biz/A/IN‘ Nov 21 09:09:58 s332-kt9-sel named[4942]: client 172.105.101.71#1234: query (cache) 'www.tcgexy.org/A/IN' Nov 21 09:09:58 s332-kt9-sel named[4942]: client 177.112.102.240#1234: query (cache) 'www.etueqt.org/A/IN' Nov 21 09:09:58 s332-kt9-sel named[4942]: client 59.34.42.184#1234: query (cache) 'www.nisyjr.com/A/IN' Nov 21 09:09:58 s332-kt9-sel named[4942]: client 93.3.157.3#1234: query (cache) 'www.inrxpx.biz/A/IN'

Page 22

http://www.redbarn.org/dns/ratelimits :: Default function since centos 6.x since 2013.

http://ss.vix.su/~vjs/rl-arm.html

rate-limit {

[ responses-per-second number ; ]

[ referrals-per-second number ; ]

[ nodata-per-second number ; ]

[ nxdomains-per-second number ; ]

[ errors-per-second number ; ] // SERVFAIL, FORMERR excluding nxdomains

[ all-per-second number ; ] // normally at least 4~5 times bigger than other value

[ window number ; ]

[ log-only yes_or_no ; ]

[ qps-scale number ; ] // responses-per-second, errors-per-second, nxdomains-per-second ,all-per-second values are reduced by the ratio

[ ipv4-prefix-length number ; ] // default Is /24, need to change /32

[ slip number ; ]

}

e,g. qps-scale 250; responses-per-second 20; and a total query rate of 1000 queries/second for all queries from all DNS clients including via TCP,

then the effective responses/second limit changes to (250/1000)*20 or 5. Responses sent via TCP are not limited but are counted to compute the query per second rate.

RRL(Response Rate Limiting) defense

Page 23

--- named.conf --- rate-limit { nxdomains-per-second 1; ipv4-prefix-length 32; slip 2; };

RRL(Response Rate Limiting) defense

1 CLIENT -> SERVER DNS Standard query A 0.809928333227621.example.com 2 SERVER -> CLIENT DNS Standard query response, No such name 3 CLIENT -> SERVER DNS Standard query A 0.990417249591218.example.com 4 SERVER -> CLIENT DNS Standard query response 5 CLIENT -> SERVER TCP 41702 > 53 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=3124747279 TSER=0 WS=7 6 SERVER -> CLIENT TCP 53 > 41702 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 TSV=3737413786 TSER=3124747279 WS=6 7 CLIENT -> SERVER TCP 41702 > 53 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=3124747282 TSER=3737413786 8 CLIENT -> SERVER DNS Standard query A 0.990417249591218.example.com 9 SERVER -> CLIENT TCP 53 > 41702 [ACK] Seq=1 Ack=50 Win=14528 Len=0 TSV=3737413788 TSER=3124747282 10 SERVER -> CLIENT DNS Standard query response, No such name 11 CLIENT -> SERVER TCP 41702 > 53 [ACK] Seq=50 Ack=110 Win=5888 Len=0 TSV=3124747284 TSER=3737413788 12 CLIENT -> SERVER TCP 41702 > 53 [FIN, ACK] Seq=50 Ack=110 Win=5888 Len=0 TSV=3124747284 TSER=3737413788 13 SERVER -> CLIENT TCP 53 > 41702 [FIN, ACK] Seq=110 Ack=51 Win=14528 Len=0 TSV=3737413791 TSER=3124747284 14 CLIENT -> SERVER TCP 41702 > 53 [ACK] Seq=51 Ack=111 Win=5888 Len=0 TSV=3124747287 TSER=3737413791

3 way handshake

4 way handshake

Truncated: Message is truncated

Jul 1 14:11:10 SERVER named-sdb[15282]: limit responses to xx.xx.xx.xx/32 for xxxx.com IN A (00014672)

http://www.circleid.com/posts/20130820_a_question_of_dns_protocols/

17% of visible resolvers do not successfully followup with a TCP connection following the reception of a truncated UDP response. Also performance issue.

Page 24

Anycast based Distribution

http://www.root-servers.org/ j.root-servers.net(192.58.128.30)

Google case:8.8.8.8

On the Internet, anycast is usually implemented by using

BGP to simultaneously announce the same destination IP

address range from many different places on the Internet.

This results in packets addressed to destination addresses

in this range being routed to the "nearest" point on the net

announcing the given destination IP address.

excerpted from Wikipedia.org

Page 25

Conclusion :: Technical Requirements for DNS DEFENSE

Tolerant against Massive QPS attack (~Mqps)

pass only valid dns packet

rate-limit per query type

ip rate-limit based on source ip or query type,etc

filter bad flag combinations

filter multiple request type in a packet

filter based on packet size(length,range)

Source IP validation

Using Multiple DNS provider

No solution against the Massive standard Query with randomly spoofed IP Address