Upload
cosimo-streppone
View
1.597
Download
0
Embed Size (px)
DESCRIPTION
Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it is. This talk is about the story of our team’s first unprepared fight against a DDoS attack.
Citation preview
Our My first DDoS attack
Velocity Europe 2011 – BerlinCosimo Streppone
Operations Lead
<video of Mr. Wolf going to Jimmy's house in Pulp Fiction>this couldn't fit in the PDF... sorry.
http://www.youtube.com/watch?v=hsKv5d0sIlU
my.opera.com/Ao-Trang-Oi/blog/
nginx – secret sauces?
# Pavel's secret gzip tuning saucegzip on;gzip_disable msie6;gzip_min_length 1100;gzip_buffers 16 8k;gzip_comp_level 3;gzip_types text/plain application/xml application/x-javascript text/css;
nginx – secret sauces?
# Michael's secret file cache sauceopen_file_cache max=1000 inactive=20s;open_file_cache_valid 30s;open_file_cache_min_uses 2;open_file_cache_errors on;
nginx – antidos.conf
# More on https://calomel.org/nginx.htmlclient_header_timeout 5;client_body_timeout 10; ignore_invalid_headers on; send_timeout 10;
# To limit slowloris-like attacksclient_header_buffer_size 4k; large_client_header_buffers 4 4k;
# Cut abusive established connections,# forcing clients to reconnectlocation ~ ^/Ao-Trang-Oi/blog/ { return 444;}
nginx – drop client connections
nginx
backends
varnish
nginx – varnish caching
iptraf
GET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1
User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 […] __utmz=218314117.1286774593. […] utmcsr=google|utmccn= […] utmctr=cach%20de%20hoc%20mon […]
<... random high speed junk follows ...>
tcpdump of anomalous traffic
GET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: close
tcpdump of anomalous traffic
cosimo: we're seeing a pretty "interesting" problem within our nginx frontscosimo: there's a few hosts sending a legitimate HTTP GET requestcosimo: followed by a binary stream of random bytes that never endscosimo: this is just 1 request going on and oncosimo: is there some way to alter the nginx config to shut down these client connections?cosimo: the client is sending something like:cosimo: GET /blah HTTP/1.1cosimo: Host: ...cosimo: Etc: etc...cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: don't know if nginx can do thiscosimo: cool
OMGWTFBBQ!!!!11111“this is nkiller2”
#nginx, 14th October 2010
BLAH BLAH BLAH BLAH BLAH BLBLAH BLAH BLAH
PHRACK#66
tcp window zero?
iptables -A -m u32 --u32 “6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000” -j ZERO_WINDOW_RECENT
u32 zero window filter
6 &0xFF =0x6
4 &0x1FFF =0x0
u32 zero window filter
u32 zero window filter
0>>22 &0x3C ()12 &0xFFFF =0x0
0>>22 &0x3C ()12 &0xFFFF =0x0
??
u32 zero window filter
0>>22&0...@12&0xFFFF=0x0000
0>>22&0x3C@12&0xFFFF=0x0000
0>>22& [EMAIL PROTECTED] &0xFFFF=0x0000
0>>22&0x3C@12&0xFFFF=0x0000
0>>22 &0x3C @12 &0xFFFF =0x0
u32 zero window filter
iptables rules - logging
$ipt -N ZERO_WINDOW_RECENT
$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT
$ipt -A ZERO_WINDOW_RECENT -m recent --set --name ZERO_WINDOW
$ipt -A ZERO_WINDOW_RECENT -m recent --update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j LOG --log-level info --log-prefix "ZeroWindow"
~18k distinct IPs
iptables rules - blocking
$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent –set --name ZERO_WINDOW
$ipt -A ZERO_WINDOW_RECENT -m recent –update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j DROP
nginx
backends
varnish
shields-up.vcl
non-cacheable content
cacheable content
nginx
backends
varnish
HTTPS-only traffic
shields-up.vcl
all HTTP content
nginx feels better
10s
20s
0s
Pingdom response time
End 29-Oct-2010
Start 13-Oct-2010 End 29-Oct-2010
Packets/s seen by firewall
¿Questions?
What can we, as Ops, do better?
● Embrace failures and learn from them
● Be fast (no panic/blame, think Mr. Wolf)
● Coordinate (#ops, war rooms, ...)
● Take notes
● Learn TCP/IP
● Know your tools
(tcpdump, tcpflow, strace, nc, iptraf, …)
my base_packages puppet module
class base_packages {
$packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ]
package { $packagelist: ensure => "installed", }
}
Thanks to...
● ithilgore (sock-raw.org) for writing nkiller2
● @vr in #nginx for pointing us at nkiller2
● David Falloon for his great “untested” idea
● marc.info for correctly handling “@” in ml
● SANS Institute for the TCP/IP references
● My team at Opera
Danke!