Security & Risk ConferenceOctober 26th - 29th 2011Lucerne, Switzerland

Sniping Slowloris and Friends Taking out DDoS attackers with minimal harm to your users

Dr. Christian Folini - netnea AG - christian.folini@netnea.com

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 2

Christian Folini CV

IT Consultant at netnea, Berne.Working for Swiss Post, one or two banks etc.

Started to work on defense against Slowloristype DDoS in 2006

Specialised in Webserver Security and Web Application Security on Unix Servers and System Administrationin general

Speaker at Security Conferences (OWASP, Swiss Cyberstorm etc.)Developer of a ModSecurity Rule Editor named “REMO“Studies in History and Computer Science in Fribourg, Berne,

Bielefeld and BerlinPhD in Medieval History at Fribourg University, Switzerland

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 3

ModSecurity Mailinglist 2006

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 4

RSnake Announcing Slowloris

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 5

Swiss Post Press Release Dec 06, 2010

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 6

Being Attacked By Anonymous

„We are Anonymous. We are Legion. We do not forgive. We do not forget. Expect us.“

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 7

Quotes from the Attacker's Chat-Room

< machiavelli> again I think holding postfinance.ch down for several weeks would lead to cash in wikileaks' hands. Postfinance would be forced to actually release the wikileaks funds they've stolen or go out of business.

< biertrinker> paypal is wasting time. lets do postfinance.ch [...] to let them see that war is still not over

< RemmiDemmi> postfinance.ch would be good

< pride2> it would make a good statement

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 8

Incoming Internet IP Traffic During Attack

Two weeks of Internet traffic on one of the links of Swiss Post(Above the line are outgoing, below the line are Incoming PPS = IP Packets per Second)

● The attacks started within minutes after the press release.● The attacks were very chaotic and heterogeneous. Literally dozens of attacks in parallel.

● The attackers were able to fill the complete network bandwith for a certain time (even if 90% of the attack traffic was filtered by the ISP).

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 9

Request Delaying / Slowloris DDoS Explained

Imagine somebody paying a big bill with small coins ... producing them ... slowly ... one ... by ... one ...

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 10

Blindness / Apache mod_status screenshot

A standard webserver does not have a clue what is happening. The attack hits home before he has finished initializing the request, hence the ”..reading..“.

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 11

Defense: What Everybody Tells You

● Ivan Ristić: „With some exceptions, DoS are very difficult to defend against.“

● Official Response to bug report regarding Slowloris: „DoS attacks by tying up TCP connections are expected.“

● Apache documentation: „All network servers can be subject to denial of service attacks that attempt to prevent responses to clients by tying up the resources of the server. It is not possible to prevent such attacks entirely.“

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 12

It boils down to a problem of Strict Differentiation:

The defense is about telling good traffic from bad traffic when the bad traffic mimics good traffic and you are blind to start with.

Defense Strategy

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 13

What You Can Do : Build Up Knowledge

● Know your architecture So you know your weak spots● Know your protocols So you know the options of the attackers● Know your application So you know their behaviour under normal load and under attack● Know your customers So you are aware of their habits and tolerance levels● Know your allies and their phone number So you are not alone when the attackers hit you● Know your tools Because a combination of tools is your best defense● Know your defense plan So you are sure to react in a well-prepared manner● Know your enemies So you know their motivation, resources, location and knowledge level

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 14

What You Can Do: The Basics

Think about using an "event based" webserver (but they have other limits...)

Think about routing the traffic through an external specialist

Understand HTTP Keepalive and decide if you really need it

Lower your timeouts (3s sounds like a decent value in my eyes)

Use mod_reqtimeout

Look into mod_qos (by Pascal Buchbinder, Winterthur)

Use GeoIP

Use netstat

Use tcpdump

Use IP Blacklisting

Look into ModSecurity – there are a few useful directives

Look into mod_backdoor

Think about separating Uploads / big POSTs from the rest of the application

Forget mod_evasive

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 15

ModSecurity / modreqtimeout stuff

You work with the following directives:

ModSec: Limit number of connections:

● SecReadStateLimit 5

● SecWriteStateLimit 5

ModReqtimeout: Limit the total timeout per request phase:

● RequestReadTimeout header=1, body=30

Ryan Barnett put all the details on the ModSecurity Blog:

http://blog.spiderlabs.com/2011/07/advanced-topic-of-the-week-mitigating-slow-http-dos-attacks.html

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 16

What You Can Do: Combining the Tools

Slowloris type DDoS tools don't ever finish a request

-> comparing netstat output and the access log should be able to tell you more

Other DDoS tools do full requests, but they do not fetch follow up css, javascript and image files

-> the access log has the details

There is a typical median lifetime of a connection to your application

-> observing netstat output should give you an idea

And now the really advanced stuff:

Run an agent that supervises the connections and observes the access log and the login log (if that exists in your application):

Look for clients accessing the wrong URLs

Look for clients using the wrong method on the wrong URLs (i.e. Doing a POST on a page where POST is not expected)

Look for clients having an atypical order of requests

Look for clients with atypical request structure

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 17

Introducing: flying-frog.rb

Flying frog is a monitoring agent that hovers over the incoming traffic and the application log. It picks individual attackers, like a frog eats a mosquito.

Features:● Monitoring SYN requests as they come in (and add connections to

the its own connection table).● Check out the connections in the connection table and calculate their age.● Monitor authentication log and register authstate of client IPs

(Their ability to authenticate their session with the server). ● Report median connection lifetime of all connections

(this is the key variable that tells you if you are under an attack).● Report client IPs with more connections then the limit allows.● Report client IPs with a connection that is older than the limit allows.● Limits can be configured based on GeoIP country field, port and authentication

state: i.e. The maximum lifetime of a connection should be 10s, but authenticated customers from Switzerland on port 443 are allowed 60 seconds (so they can upload the files to the server)

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 18

Limit the individual number of connections: $params["basemaxconnection"] = 10 $params["maxconnectionconditions"] = <<EOF country:CH 20 auth:auth 100 EOF

Limit the individual timeouts:

 $params["basetimeout"] = 5 $params["timeoutconditions"] = <<EOF port:443 10 country:CH 20 auth:auth 50 country:CH auth:auth 100 EOF

Configuring flying-frog.rb

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 19

Downloading flying-frog.rb

CAVEAT: flying-frog.rb is provided as is with no warranty attached. It has been tested, but please consider it alpha code quality.

Download from https://www.netnea.com

#days Security Conference – October 26th – 29th 2011 / Dr. Christian Folini: Sniping Slowloris and Friends 20

Defense of Medieval Castles

Just a thought which might be new to you:

Footpaths around castles and staircases within the towers are almost always in clockwise direction. Who knows why?