Upload
lior-rotkovitch
View
895
Download
44
Embed Size (px)
Citation preview
BIG IP ASM V12 DDOS PROFILE
Lior Rotkovitch, NPI
ASM, L7 DDoS & Analytics
Global Service Tech Summit, Seattle Sep, 2015, v3
© F5 Networks, Inc 2
ASM – DDoS Profile
DDOS - HTTP FLOODS ATTACKS
• From single IP to single URL
• From multiple IP’s to single fixed URL
• From multiple IP’s to multiple fixed URL’s
• From multiple IP’s to multiple random URL’s
• From multiple IP’s from a specific country
• Fine Tune your Thresholds & Reporting
DDOS - BOTS
• Simple bots
• Impersonating Bots
• Bots with cookies & JS capabilities
• Bots acting as full browser
• Reporting
© F5 Networks, Inc 3
HTTP Floods facts:
Servers DatabaseHacktivism
Google Web Bot
Unidentified User
User
Source IP‘s
Users Or Bots
Web Site
Web Bot
• Legitimate Layer 7 requests
• Asking a web page thousands of time instead of one (few) times
• Exhausting backend servers resources: memory, CPU, Disk etc
• Relatively easy to execute with simple tools
• Not easy to detect the offending source nor to prevent it
• Wrong identification will prevent valid users from accessing the site (false positive )
© F5 Networks, Inc 4
HTTP Floods types
Hacktivism
Google Web Bot
Unidentified User
User
Source IP‘s
Users Or Bots
Web Bot
Requests increase from or/and to URL’s inside the web site
• From single IP to single URL
• From multiple IP’s to single fixed URL
• From multiple IP’s to multiple fixed URL’s
• From multiple IP’s to multiple random URL’s
• From multiple IP’s originating from a specific country
Web Site
Servers Database
© F5 Networks, Inc 5
ASM Detection & Mitigation concept - HTTP Floods
Latency App URL’s &
objects
Hacktivism
Google Web Bot
Unidentified User
User
RPSSource IP‘s
Users Or Bots
Web Bot
Source IP’s
ASM process:
1. Monitoring entities: RPS Latency IP’s URL’s
2. Detecting Increase
3. Activating Mitigation
Web Site
Servers Database
© F5 Networks, Inc 6
ASM Detection & Mitigation concept – DoS Profile
Location: Security ›› DoS Protection ›› DoS Profiles ›› dos
© F5 Networks, Inc 7
TPS Based Detection: Transaction Per Second based detection and mitigation
Client: Hey server, can I get the web page ?
ASM: no, you are sending too many requests.
You will have to
• Answer CSID
• Answer CAPTCHA
• Be Rate Limited / Blocked
Server
© F5 Networks, Inc 8
TPS Based DetectionMonitoring Request Per Second increase form source IP, Geo, URL, Site Wide.Then apply one of the mitigation policies: CSID, CAPTCHA, Rate limit
© F5 Networks, Inc 9
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 10
By Source IP: Detection Criteria
Detection
Detection: thresholds for determining DDoS attack - by source IP increase
Mitigation: which mitigation will apply on the offending source IP
Mitigation
© F5 Networks, Inc 11
Ratio thresholds - measuring ratio with two time interval:
• Long (History Interval): Measure the last 1 hour RPS averages every 10 seconds
• Short (Detection Interval): Measure the last 10 seconds RPS averages every 10 seconds
Detection – Ratio
By Source IP: Detection Criteria
© F5 Networks, Inc 12
TPS increased by: ((370 - 50) /50)*100 = 640%
640% > 500% = True
By Source IP: Detection Criteria
Detection – Ratio
Long (History Interval): 50 TPS
Short (Detection Interval): 370 TPS
Example:
© F5 Networks, Inc 13
By Source IP: Detection Criteria
TPS increased by % AND minimum fix number of transactions
640% AND 40 = True
Detection – Ratio
Example:
At least X Transactions:
A minimum condition to
prevent false positive
increase (source IP
starts browsing the site
and goes from 0 to 30
RPS)
© F5 Networks, Inc 14
By Source IP: Detection Criteria
(TPS increased by % AND minimum fix number of transactions) OR TPS reached
640% AND 40 OR 200 = True
Detection – Ratio
Fixed
Example:
TPS reached:
Ratio thresholds
OR’ed with fixed TPS
or
© F5 Networks, Inc 15
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 16
Client Side Integrity Defense – Concept
User
Web Bot
Client: Hey server, can I get the web page ?
ASM: no, you are sending too many requests. Are you a browser ?
if a browser:
Yes, I’m a browser
If a bot:
*^lkjdfg@#$
ASM: ok, you are allowed. Here is the web page you asked for.
ASM: Bye Bye – Blocked
Server
© F5 Networks, Inc 17
• Checking JavaScript capabilities
• A client is considered legitimate if it meets the following criteria:
• The client support JavaScript
• The client support HTTP cookies
• The client should calculate a challenge inside the JS
• If satisfied = legitimate client that can access the site
Client Side Integrity Defense – Concept
© F5 Networks, Inc 18
Client Side Integrity Defense - Flow
User Browser DoS Profile App
First main page access
HTTP Request (no cookie)
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie) Reconstruct request
Original HTTP Request
HTTP Response (main page)
HTTP Response (main page)
More object requests (cookie)
Validate cookie: format & time stamp
More object requests
More responsesMore responsesDeliver page
• This is the flow and timeline
of events.
• Transparent to the user,
done under the hood
• Note that request is held at
the ASM and not arriving the
app until checks are
satisfied
• Not all checks are described
here, some are internal IP.
Send JS test
© F5 Networks, Inc 19
Client Side Integrity Defense –JavaScript sample
• The JS is obfuscated
• From user perceptive this
is transparent action.
© F5 Networks, Inc 20
• If no reply – No problem for us
• If didn’t solve the challenge but still sending request – Block (RST)
• If did solve the challenge but:
• Cookie is wrong format – Block (RST)
• Time stamp expired – Block (RST)
• If client access a resource (image) without getting the cookie first – Block (RST)
Client Side Integrity Defense – Mitigation summary
© F5 Networks, Inc 21
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 22
CAPTCHA Challenge - Concept
User
Web Bot
Client: Hey server, can I get the web page ?
ASM: no, you are sending too many requests. Please answer this CAPTCHA challenge, show me your human !:
If a user:
OK, I answered
If none user:
Ha ? *^lkjdfg@#$
ASM: ok, you are allowed. Here is the web page you asked for.
ASM: Bye Bye – Block him dood !
Server
© F5 Networks, Inc 23
CAPTCHA Challenge
Ultimate solution for identifying human or bot
Send challenge to every IP that reached IP detection criteria thresholds
To CAPTCHA or not to CAPTCHA ?
Some argues that CAPTCHA is not a good usability because an innocent user gets CAPTCHA and he will not know why. So, remember that a valid user should pass browser tests. i.e. if a user is blocked (or gets a CAPTCHA) there is a reason and maybe he is not innocent (infected ?) .
“Completely Automated Public Turing test to tell
Computers and Humans Apart”
© F5 Networks, Inc 24
CAPTCHA – customize response
• Can be customize to the web site look and feel colors via css
• Failure Response page is served if the first attempted fails
© F5 Networks, Inc 25
CAPTCHA Challenge - Flow
User Browser DoS Profile App
Request mypage.php
GET /mypage.php (no cookie)
CAPTCHA HTML +JS response
Cookie with time stamp
Solve CAPTCHA
CAPTCHA rendered
Submit CAPTCHA
solution
GET /mypage.php + CAPTCHA
cookieVerify CAPTCHA solution
Validate cookie
GET /mypage.php
HTML of mypage.phpHTML of mypage.phpmypage.php
rendered
Send CAPTCHA• While the system is still in a
state of attack the offending source will be presented with another CAPTCHA every 5 min.
• Same as CSID, request is held at the ASM until CAPTCHA is solved
© F5 Networks, Inc 26
• If didn’t submit the challenge - no request DOSing us
• If didn’t solve the challenge but still sending us attacks – Blocked
• If did solve the challenge but:
• Cookie is wrong format – RST
• Time stamp expired 5 min– RST
CAPTCHA – mitigation summary
© F5 Networks, Inc 27
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 28
Request Blocking / Rate limit
Client: Hey server, can I get the web page ?
ASM: no, you are sending too many requests.
I’m limiting your requests sending rate
While CSID and CAPTCHA try to understand who is the offending source (bots or browsers)
request limiting is indifferent to the “identity” and limits the offending sources.
© F5 Networks, Inc 29
Request Blocking
Request Blocking:
• Blocking: block all IP’s from the offending source – if a source IP reached thresholds I don’t
want him on my site at this point
• Rate Limit: limit the amount of allowed request from the offending source – if reached
thresholds I can sustain only some of the traffic at this point
© F5 Networks, Inc 30
Request Blocking – Mitigation Summary
• Block all – blocking all traffic from the offending source (i.e. I don’t want to see any more traffic from this source)
• Rate Limit – rate limit the offending source
Example
If long was 50 TPS And increase in short is 150 TPS
Rate limit to 50 TPS
Rate limit will limit to long (history) TPS rate
© F5 Networks, Inc 31
TPS based: by source IP – Summary
Client Side Integrity Check
CAPTCHA Challenge
Request Blocking
• Measuring source IP increase
• All source IP’s that reached the thresholds will be presented with the enabled mitigation
• If still increasing , fall back according to the order in the GUI (switching mitigation)
© F5 Networks, Inc 32
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 33
HTTP Floods – Geolocation detection and Mitigation
Hacktivism
Google Web Bot
Unidentified User
User
Source IP‘s
Users Or Bots
Web Bot
Servers Database
Web Site
http floods type: From multiple source
IP’s originating from a specific country
© F5 Networks, Inc 34
• Geolocation – Relative to the whole traffic of the site:
500 % request increase of the whole site from a specific country
AND
At least 10 % of the whole site traffic
Geolocation - Detection
© F5 Networks, Inc 35
Geolocation – Mitigation
• Client Side Integrity Check
• CAPTCHA Challenge
• Request Blocking
All clients requests arriving from the specific country will be presented with mitigation:
(note that blocking will block
all users from this country)
© F5 Networks, Inc 36
Geolocation – Black n White listing
• Allows access to the web site regardless of
geolocation detection criteria thresholds only
i.e. other thresholds still apply
• Specifies the countries that the system always blocks
whenever the system is in a state of DDoS detection.
• Done regardless of the thresholds set in the DDoS
profile
© F5 Networks, Inc 37
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 38
HTTP Floods – URL Detection and Mitigation
App URL’s &
objects
Hacktivism
Google Web Bot
Unidentified User
User
RPSSource IP‘s
Users Or Bots
Web Bot
Source IP’s
http://site.com/sell.php
Servers Database
Web Site
• Measuring requests increase on a URL
• Floods types:
• From multiple IP’s to multiple fixed URL’s
• From multiple IP’s to multiple random URL’s
© F5 Networks, Inc 39
TPS increase by* AND at least X TPS **
OR TPS reached
URL Detection Criteria
Collecting RPS on URL’s
Calculation:
*Ration of long and short
**Minimum TPS thresholds for detection
© F5 Networks, Inc 40
URL Detection Criteria– Mitigation
• Client Side Integrity Check
• CAPTCHA Challenge
• Request Blocking – Rate limit (No block all)
All clients that access the URL:
© F5 Networks, Inc 41
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 42
HTTP Floods – Site Wide Detection and Mitigation
App URL’s &
objects
Hacktivism
Google Web Bot
Unidentified User
User
RPSSource IP‘s
Users Or Bots
Web Bot
Source IP’s
Monitoring: all entities
ServersDatabase
Web Site
Floods types:
• From multiple IP’s to multiple random URL’s
• Cases where DDoS attack is under the radar
© F5 Networks, Inc 43
*TPS increase by AND Minimum TPS thresholds for detection
OR TPS reached
Site-Wide Detection Criteria
• Collecting RPS on the entire website (all entities – URL’s, IP’s)
• In some cases the floods will avoid thresholds for IP based or URL based.
• Site wide provide another layer of detection and prevention
Detection: Ratio
Fixed
© F5 Networks, Inc 44
Site-Wide Detection Criteria – Mitigation
• Client Side Integrity Check
• CAPTCHA Challenge
• Request Blocking - only rate limit no blocking
All clients that access the site:
Prevention polices
© F5 Networks, Inc 45
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices)
2. Mitigation polices:
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
1
abc
3
4
5
2
6
© F5 Networks, Inc 46
Prevention duration
• Client Side Integrity Check
• CAPTCHA Challenge
• Request Blocking
De escalate - start from the top
Every 7200 seconds
Escalate top down every 120 second if thresholds are still increasing
© F5 Networks, Inc 47
Stress Based detection
• Predictive Latency – predict how long it will take to serve a new incoming request
Server: I’m fine, keep on sending them
ASM: Hey server, how many more requests can you handle ?
I’m the server
© F5 Networks, Inc 48
Stress Based Detection and prevention concept
Client: Hey server , can I get the web page ?
ASM: mmm let me check. The Server can take additional incoming requests. you are allowed
ASM: no, my backend latency is now too high and you are sending too many request.
You will have to:
• Answer CSID or
• Answer CAPTCHA or
• Be rate limit
Client: Hey server, can I get web pages again now ?
I‘m the server
…….
…….
After a while
© F5 Networks, Inc 49
Stress Based – GUI
• Same concept as TPS based: source IP, Geo, URL, Site wide and their mitigation policies.
• Addition condition of backend latency. i.e. only when the two conditions reach thresholds, then apply mitigation policy.
Note: Can work together (operate in
parallel) with TPS based and act as
layers of protection (e.g. TPS based
does only CSID in alert mode and
Stress based does request blocking in
case of latency increase)
© F5 Networks, Inc 50
Stress Based Detection & Mitigation
• Similar to TPS based,
Quiz yourself, what each item means ?
1. By Source IP
a) CSID
b) CAPTCHA
c) Request Blocking
2. By Geolocation
3. By URL
4. Site Wide
© F5 Networks, Inc 51
Stress Based Detection – thresholds condition
Latency threshold
exceeded?
TPS threshold
exceeded?AND
Then: Activate
Mitigation Policy
• Mitigation Is activated when two types of thresholds are reached :
Latency thresholds
AND
TPS thresholds
© F5 Networks, Inc 52
Stress Based Detection – thresholds condition
• in order to apply a prevention policy, both TPS and Latency thresholds must be exceeded, then the enabled prevention policy is activated.
• Latency thresholds are not visible in the GUI, they are part of automatic detection.
Example:
Automatic stress detection enters a
state of exceeding thresholds. This by
itself will not active the prevention.
Only when the TPS thresholds will
exceed then the prevention policy is
activated.
prevention
TPS thresholds
stress detection
© F5 Networks, Inc 53
TPS based VS Stress based
• Quick way to protect against DDoS. I’m in trouble
and I want to block now !
• Fixed number on the TPS reached is very easy
and useful. Also easy to detect offending sources
• Allows the option to activate the Mitigation only
when the backed experiencing latency AND RPS
increase (I only want to block when the attack is
causing backend latency)
• Provide Layers of defense and notify about backend
issues (not just DDoS)
Conclusion:
TPS based is quick while latency based
allows more granular approach
© F5 Networks, Inc 54
Heavy URL’s
Not all URL’s are equal
Some are more attractive than others
© F5 Networks, Inc 55
Heavy URL’s
• Heavy URL’s are URL’s that consume more processing resources
from the server
• Are good application DoS point - Even few requests can DoS the app
• Typical heavy URLs are search box, product ID’s
Heavy URL
Servers Database
http://site.com/serach.php?q=a
Ho wow, this will take a while…
……
Searching …
…..
hold on…
…..
Almost there….
© F5 Networks, Inc 56
Heavy URL’s concept
• Automatically measures latency on URL’s for 48 hours
and decide who is heavy
• When any URL based mitigation is active, the heavy URL’s
that was detected as heavy will also “get” the active
mitigation
© F5 Networks, Inc 57
Heavy URL’s concept
Heavy URL is another detection capability. Once it is reached the thresholds AND one of the By URL detection
thresholds are reached Then the URL’s that consider heavy URL’s will be applied with the active mitigation policy
© F5 Networks, Inc 58
Heavy URL’s configuration
Example: By URL TPS reached 1000 TPS and is currently applying CSID mitigation.
Heavy URL is enabled
© F5 Networks, Inc 59
Heavy URL’s Configuration
1. Automatic Detection - Automatically add URL that will be considered as heavy
2. Manual Heavy URLs – Manually add URL that will be considered as heavy
3. Ignored URL – Exclude those URL from “heaviness”
4. Latency Threshold – Above this threshold -> heavy URL
1
2
3
4
© F5 Networks, Inc 60
Heavy URL – Reporting
If search.php is defined as heavy and if index URL is currently being mitigated with CSID because it exceeded the
thresholds of URL reached (or fixed) then every source IP that is accessing search.php will also get the CSID check.
Security ›› Reporting ›› DoS ›› Application ›› URL Latencies
Example:
© F5 Networks, Inc 61
• Reporting first
• Fine Tune your thresholds
BeforeDDoS Attack
During DDoS Attack
Remember, security is a process
© F5 Networks, Inc 62
First rule of detection - AVR Reporting
• AVR graphs help you understand the site metrics:
Statistics ›› Analytics ›› Overview
Statistics ›› Analytics ›› Transactions ›› View by
• AVR graphs inside ASM tab:
Security ›› Reporting ›› DoS ›› Overview
Security ›› Reporting ›› DoS ›› Application ›› Transaction outcome
Know your web site metrics
• Sources
• IP’s
• URL’s
• Site Wide
• Geolocation
• RPS
• TPS
• Latency
L7 DDoS measurements
© F5 Networks, Inc 63
• Out of the box thresholds are good for most web sites
• Depending on the web site traffic fine tuning thresholds might be needed.
• Fine tuning thresholds can be divided into:
• Before DDoS Attack
• During DDoS Attack
Why Fine Tune Thresholds ?
Good for me ??
© F5 Networks, Inc 64
Process:
Pre requisite: Enable DDoS Profile on the desired virtual
1) White list IP’s, geolocations countries, URL’s (admin) etc
2) Get visibility with transparent mode – write down metrics*
3) Test and decide which prevention will apply thresholds exceeds (TPS bases/ Latency based , heavy URL config etc)
4) Fine tune thresholds for fixed and ratio based
5) Switch to blocking – When needed
Fine Tune Thresholds Before attack
*good list for L7 DDoS metrics
© F5 Networks, Inc 65
Fine Tune Thresholds Before DDoS for Source IP
• View by: Client IP address
• List top TPS Avg IP’s
Go to Statistics ›› Analytics ›› HTTP ›› Transaction
© F5 Networks, Inc 66
Fine Tune Thresholds Before DDoS for Source IP
• By examining the client IP Address you can
conclude the averages of “normal” traffic you
expect to see from the top source IPs.
• Knowing “normal” averages can help defining
the TPS increase by ratio.
• The idea is that you can determine how much
traffic is allowed until assumed a ddos attack.
© F5 Networks, Inc 67
Fine Tune Thresholds Before DDoS for Geolocation
The same concept works for the
geolocation thresholds graph.
From the drilldown choose Countries on
AVR reports
“Which countries you expect to see traffic ?
Go to Security ›› Reporting ›› dos ››Application ›› transaction outcome
© F5 Networks, Inc 68
Fine Tune Thresholds Before DDoS for URL
The same idea applies to URL’s.
Sort graph by URL’s
“Which URL should have to highest RPS ?
© F5 Networks, Inc 69
Fine Tune Thresholds Before DDoS for URL
© F5 Networks, Inc 70
Fine Tune Thresholds Before DDoS for Site Wide
On the drilldown choose
Virtual Server
“This will help us understand
the over all traffic load that we
have when there is no DDoS
attack.
© F5 Networks, Inc 71
Fine Tune Thresholds Before DDoS for Site Wide
The overall traffic should be much higher than the other thresholds.
The values reflect the total amount of TPS that the virtual can handle.
Site wide = Virtual server
© F5 Networks, Inc 72
Fine Tune Thresholds During attack
Process:
1) Fine tune white list source – if needed
2) Identify sources that exceed thresholds (source IP’s, URL’s, Geo, SiteWide) by looking at reporting.
3) Determine the attack type: from fixed/random source IP to fixed/ random URL. Conclude which of the detection types you need (source IP only ? Source IP and URL based only ? etc. )
4) Fine tune thresholds according to the exceeding sources (ratio / fixed)
5) Apply mitigation and decide what is working and what is not. Uncheck the mitigations that are not effective
6) Go to step 1 and repeat
© F5 Networks, Inc 73
Fine Tune Thresholds During attack – Source IP
• Security ›› Reporting ›› dos
››Application ›› transaction
outcome
• On the drilldown choose
Client IP Address
© F5 Networks, Inc 74
Fine Tune Thresholds During attack – Geolocation
• Security ›› Reporting ›› dos ››
Application ›› transaction
outcome
• On the drilldown choose
Countries
© F5 Networks, Inc 75
Fine Tune Thresholds During attack – URL’s
• Security ›› Reporting ›› dos ››
Application ›› transaction
outcome
• On the drilldown choose URL’s
© F5 Networks, Inc 76
Fine Tune Thresholds During attack - Site Wide
• Security ›› Reporting ›› dos ››
Application ›› transaction
outcome
• On the drilldown choose
Virtual Servers
© F5 Networks, Inc 77
AVR reports and graphs
Mitigation type – can help
understand which of the
mitigation is effective and
when switching mitigation
occurred
Time line Attack start / end
Host IP
Number
of TPS
Attack ID : Clicking
will show graph
Security ›› Event Logs ›› DoS ›› Application Events
© F5 Networks, Inc 78
AVR reports and graphs
impact is the latency
on the backend for all
entities. The higher
the latency the higher
the impact is.
High, medium and low
impact allow to filter
high impact attacks
and deal with them
first
Security ›› Reporting ›› DoS ›› Application ›› Transaction Outcomes
© F5 Networks, Inc 79
AVR reports and graphs
Start and End points - red flags indicate the start of an attack and the green flags indicate the end of
an attack. Switching mitigation can occure several time over the DDoS attack.
Security ›› Reporting ›› DoS ›› Application ›› Transaction Outcomes
© F5 Networks, Inc 80
AVR reports and graphs
• Incomplete – Indicates traffic that was dropped by the server because the connection was
incomplete or the server did not respond.
• Blocked – Indicates traffic that was blocked as a result of the mitigation policy (any of the
prevention policies including bots blocking)
• Proactive Mitigation – Indicates the amount of time that the proactive bot defense mechanism
was severed
• CAPTCHA mitigation – Indicates the amount of time that the CAPTCHA challenge was severed
to offending sources
• CS integrity mitigation –Indicates the amount of time that the client-side integrity defense
challenge was severed to offending sources
• BIG IP Response – Indicates traffic that is a response to the client from the BIG-IP system.
• Cache by BIG IP – Indicates traffic that is served from cache configured (WA, RamCache)
• Whitelisted – Indicates traffic from IP Address that are in the whitelist of the DoS profile
• Pass through - Indicates traffic that is pass to the application trough ASM to the server
© F5 Networks, Inc 81
AVR reports and graphs
The AVR DoS graph now
shows the thresholds
that are set in the TPS
detection tab.
The Display Thresholds
check box will display
them or clear them from
the graph.
© F5 Networks, Inc 82
Before DDoS:
• Write the “normal” thresholds for the web site: (IP’s, Geolocation, URL’s, Site Wide)
• Set the ratio and the fix threshold for each of the above detection criteria (How much the web site can take 2 times the traffic , 5 times etc…)
• Test the configuration and the prevention policy, then conclude which one is good for you
During DDoS:
• Identify the source IP’s, URL’s and entire site traffic increase and determine the attack type
• Set the fixed TPS number in each of the above criteria and apply mitigation
• Verify the results in the Transaction outcome graph
Fine Tune Thresholds– Summary
© F5 Networks, Inc 83
DDoS Bots - Detection & Mitigation
© F5 Networks, Inc 84
Layers of defense against Bots
Simple Bots
Impersonating Bots
Bots with cookies / JS capabilities
Bots acting as full browser
Gohogle
This bot section is mostly about
bots that DoS / DDoS. However,
Bots detection and prevention
can be used for various bot
problems the site is experiencing.
© F5 Networks, Inc 85
DDoS Bots
Servers Database
Google Web Bot
Unidentified User
User
Users Or Bots
Web Site
Web Bot
Bots can be classify in many ways, mostly there are:1. Simple bots
2. Impersonating Bots
3. Bots with cookies & JS capabilities
4. Bots acting as full browser
© F5 Networks, Inc 86
Enabling Bot signatures protection
© F5 Networks, Inc 87
Bots – Simple Bot
Server
ASM: yes, I have your signature. Sorry mate you are blocked.
I’m a simple Bot
Simple bot can be any command line tool such as: curl , wget , ab
© F5 Networks, Inc 88
Categorizing Bots
Bad Bots
Good Bots
Bad Bots aka Malicious are well know command
line tools – we want them out
Good Bots aka Benign are well know search engine and monitoring tools – we want
them in
© F5 Networks, Inc 89
Bot Signatures -
None
Report
Block
None
Report
Block
Each categories include:
• None – ignore
• Report – report only – used for monitoring
• Block – block
© F5 Networks, Inc 90
Excluding specific bot signatures from category settings
• Specific signature can be excluded from the
category setting
• Search the signature in Available signature list
and move it to the left pane.
• In this example ab tool will not be blocked
even if the category that includes it is in
blocking mode
© F5 Networks, Inc 91
First - White list good Bots
Web Server
I’m a google Bot
ASM: let’s see if you really are. I’m doing Reverse DNS lookup.
Yes, I see that, please continue.
DNS Server
Thanks
1
2
3
4
© F5 Networks, Inc 92
White list good Bots -with their domain name
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1. Request arrive with User Agent : Googlebot/2.1
DNS Server
Web Server
2. ASM search the google bot signature
3. The signature includes domain name. ASM issue Reverse DNS query to verify the origin of the request
4. Once approved ASM will allow the google bot to access the web site
© F5 Networks, Inc 93
Bot Signature Repository
• Bot Signature repository for the entire system is under Options.
• Bot signatures repository is update with the ASM signature update
Security ›› Options ›› DoS Protection ›› Bot Signatures List
© F5 Networks, Inc 94
Bot Signature List: general signatures repository
Signatures can be sort by:
• Signature Category
• Signature Type: Malicious / Benign
• User Define signatures Yes / No
• Partition: signature can be assign to a specific partition
Clicking on any of the sorting will change the order.
© F5 Networks, Inc 95
Sorting the Bot Signature Repository
Various filtering
Create new Bot Signature
© F5 Networks, Inc 96
Bot Signature Categories
Creating new category for Malicious or Benign
© F5 Networks, Inc 97
Create a new bot signature: simple edit mode
Simple edit mode: inside
a user agent header or
in a URL.
Category
Domain name – execute
reverser DNS query to
verify origin. Add the
domain if the Bot has one
Bot Signature name
Create when done
© F5 Networks, Inc 98
Create a new bot signature - advanced edit mode
Signature syntax example:
headercontent: "sample_text"; useragentonly;
Advanced Edit Mode - rule granularity
For full details consult F5 document
© F5 Networks, Inc 99
• Signatures associated with a domain name are validated with reverse DNS lookup.
• Blocking and reporting :
• Block flag - resets the connection and reports the action as "bot signature block" with the bot signature name.
• Report flag - Report bot name and categories (AVR)
• Updating bot signature as part of the ASM signature update
Bot signature facts
© F5 Networks, Inc 100
Bots – Impersonating Bot
Web Server
Gohogle
I’m a google Bot ha ha ha
ASM: let’s see if you are. I’m doing Reverse DNS lookup.
DNS Server
Gohogle
Bummer ASM: you are not google bot
Bye Bye -> block this creature !
ASM: Hey DNS, who’s this guy ?
DNS: no one important
© F5 Networks, Inc 101
Bots – Impersonating Bot
1 Request arrive with User Agent : Googlebot/2.1
DNS Server
Web Server
2 ASM search the google bot signature
3 The real google bot includes domain name. ASM issue Reverse DNS query to verify the origin of the request
4 If the source IP is not the expected one according to the DNS query ASM will block the impersonating bot
© F5 Networks, Inc 102
Bots with cookies & JS capability
Web ServersI’m a bot that can understand JS and
support cookies
ASM: prove it, answer my challenges
No you are not, bye bye -> block this bot.
Bummer
Ha ?
© F5 Networks, Inc 103
Proactive Bot Defense
PBD is good for:
• Bots that can handle JS
• Bots that can handle JS and cookies
• Bots floods
• Under the radar bots
• Block any bot accessing the site (humans only web site)
© F5 Networks, Inc 104
Proactive Bot Defense and Bot Signature
Proactive Bot
Defense is now
integrated with
the bot
signatures.
When enabling
proactive bot
defense the
bot signature
feature will be
enabled as
well
© F5 Networks, Inc 105
• Send Client side challenges to ALL client and thus mitigate bots all the time
• Various challenges are sent and then validate by PBD – blocked or allowed
Proactive Bot Defense
© F5 Networks, Inc 106
PBD - Client side integrity defense - flow User Browser DoS Profile App
First main page access
HTTP Request (no cookie)
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie) Reconstruct request
Original HTTP Request
HTTP Response (main page)
HTTP Response (main page)
More object requests (cookie)
Validate cookie: format & time stamp
More object requests
More responsesMore responsesDeliver page
© F5 Networks, Inc 107
• Always – sending CS all the time
• During attack – only if other component of the dos profile is in dos mode PBD will send the CS challenge (acting as two layers of mitigation)
This allows second layer of protection (rate limit and PBD)
• Grace period - cookie expiration time 300 = 5min
• White list – exclude PBD on those IP’s
Proactive Bot Defense – configuration
© F5 Networks, Inc 108
Bots acting as full browser
Web ServerI’m a Bot that
simulate browser
ASM: ok, what are your capability ? If you will not answer right you will have to answer a CAPTCHA
You are not human, byyyye -> block this unhuman !
DNS Server
Bummer
Capability ? CAPTCHA ?
© F5 Networks, Inc 109
PBD – Additional bots identification with capabilities script
Bots: Bots acting as full browsers - Browser Simulation
© F5 Networks, Inc 110
How bots that simulate browsers are evaluated ?
Block Suspicious Browsers – addition tests are done to understand if this is a bot or a browser. ASM
will evaluate the source and will give it a score:
if the score indicates that the source is a bot it will block it.
If the score indicate uncertainty and if CAPTCHA challenge is checked, then CAPTCHA will be presented
to the source. If answer it is a human if not, blocked.
© F5 Networks, Inc 111
Block Suspicious Browsers
• If Block Suspicious Browsers is unchecked send CS Challenge
• If Block Suspicious Browsers is checked and CAPTCHA is checked send Client
Capabilities challenge and give it a score: If score in doubt send a CAPTCHA for
human verification
• If Block Suspicious Browsers is checked but CAPTCHA Challenge is unchecked do
not send CAPTCHA and only block if the score is more than a human
© F5 Networks, Inc 112
Client Capabilities -challenge script flow
User Browser DoS Profile App
First request GET /sell.php
GET /sell.php (no cookie)
Client Capabilities Challenge response
Return Client Capabilities
verification
Reconstruct request
HTTP Response (cookie)
HTTP Response
GET /img.png (cookie)
Blank page & Set cookie
Original HTTP Request + cookie
1. Authenticate and decrypted JS results,
2. Verify capabilities and set a score
3. Determine an action based on score
GET /img.png (cookie)
Validate cookie: format
& time stamp
© F5 Networks, Inc 113
DoS Bots Reporting
© F5 Networks, Inc 114
Bot signatures simulation Reporting ›› dos ›› Application ›› Transaction outcomes
Transaction outcomes
is very useful for
monitoring traffic and
indicates various
measurements
© F5 Networks, Inc 115
Bot signatures simulation
Analytics ›› HTTP ›› throughput ›› request throughput
AVR will provide details on DoS bot signatures (use drill downs )
© F5 Networks, Inc 116
• Simple bots can easily be detected and blocked
• White listing of bots = visibility to bot access and keep other bots out
• Impersonating bots can be monitored / blocked
• Bots that support JavaScript and cookies can now be noticed and be blocked
• Reporting on the visiting bots to your web site is available via AVR
• Custom bots signature is powerful tool to deal with bots
• Bots signature is updating via the ASM signatures update
Summary
© F5 Networks, Inc 117
Resources
Our documentation is free for all. Read and learn more:
BIG-IP Application Security Manager: Getting Started
BIG-IP Application Security Manager Operations Guide
BIG-IP Application Security Manager: Implementations
BIG-IP Application Security Manager: Custom Signature Reference
BIG-IP Analytics: Implementations