Backtracking Algorithmic Complexity Attacks Against a NIDS
Randy Smith, Cristian Estan, Somesh Jha
University of Wisconsin–Madison
Algorithmic Complexity Attacks
Vulnerable algorithm: algorithm whose worst case differs from typical case. The larger the difference, the more vulnerable the algorithm.
Examples:
Algorithm Average Worst
Quicksort O(n log n) O(n2)
Hash lookup constant O(n)
Algorithmic Complexity Attacks
Algorithmic Complexity Attack – an attacker induces worst-case behavior in a vulnerable algorithm. Common observable effect is denial of service.
Crosby and Wallach: induced worst-case behavior in hash function implementations.
“Algorithms are now part of the attack surface” (Crosby and Wallach, 2003)
Are NIDS vulnerable? NIDS and IPS are ubiquitous, but…
Do they contain vulnerable algorithms? Can they be exploited?
YES! Only need 1 packet every 3 seconds.
Evading a NIDS
Attacker’s Goal: Evade NIDS Two attack vectors in an evasion attempt:
1st—alg. complexity attack targeting the NIDS
2nd—true attack targeting the network
Effect of an algorithmic complexity attack:(NIDS) Packets enter network unexamined(fail-closed IPS) Packets are dropped
Main results
In Snort, vulnerability in rule-matching worst-case vs. typical case: 6 orders of magnitude. “Backtracking Attack” Easily exploitable through packet payloads
Improved rule-matching algorithm limits running time differences to within 1 order of magnitude.
Outline
Snort rule matching Inducing backtracking attacks Countermeasures Measurement results Conclusion
Snort Rule Matching
content:”fmt=”; //P1
content:”player=”; //P3
content:”overflow”,relative; //P5
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”;
pcre:”/^(mp3|ogg)/”,relative; //P2
pcre:”/.exe|.com/”,relative; //P4
sid:5678)
Snort Rule Matching
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)
fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%
Rule matches!
Matching the packetP1
P5
P4
P3
P2P2
Rule matches!
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)
fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%
Inducing Backtracking attacks
P1,P2,P3,P4 match in 3 positions each
P5 never matches
fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%
fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”ReelAudio jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)
Leads to excessive packet traversals!
Matching the malicious packet
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)
fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe
P5 P5 P5
P2
P4
P5 P5 P5
P4
P5 P5 P5
P4
P3
P5 P5 P5
P4
P5 P5 P5
P4
P5 P5 P5
P4
P2
P3
P1
P5 P5 P5
P4
P5 P5 P5
P4
P5 P5 P5
P4
P2
P3
P1
Are real rules vulnerable?
Rule numberProcessing
(s/GB)
Slowdown
Same proto All traffic
3682 (SMTP) 30,933,874 232,936X 1,501,644X
2611 (Oracle) 6,220,768 56,296X 301,979X
1382 (IRC) 1,956,858 134,031X 94,993X
2403 (NetBIOS) 357,777 490X 17,368X
1755 (IMAP) 89,181 444X 4,329X
Safer backtracking
Memoization: maintain a table of subproblem “answers”; never evaluate a predicate twice at the same starting payload offset
alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)
Identify constrained predicate sequences Monotone memoization: don’t re-evaluate monotone
predicates that have been evaluated at lower offsets
Reductions in processing cost
fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe
P5 P5 P5
P2
P4
P5 P5 P5
P4
P5 P5 P5
P4
P3
P5 P5 P5
P4
P5 P5 P5
P4
P5 P5 P5
P4
P2
P3
P1
P5 P5 P5
P4
P5 P5 P5
P4
P5 P5 P5
P4
P2
P3
411
18
7 14 21
4650
54 4650
54 4650
54
28 35 42 28 35 42 28 35 42
Outline
Snort rule matching Inducing backtracking attacks Protecting against backtracking attacks Measurement results Conclusion
Measurement results
Rule number
Slowdown factor w.r.t. same protocol
Before w/ Memo+
3682 (SMTP) 232,936X 0.95X
2611 (Oracle) 56,296X 1.57X
1382 (IRC) 134,031X 6.00X
2403 (NetBIOS) 490X 0.17X
1755 (IMAP) 444X 0.46X
Live experiment topology
Background Traffic AC Attack True Attack
Live experiment
Background Traffic @ 10Mbps AC Attack
Targets Snort SMTP rule 3682Directed at sendmail server
True Attack: NIMDA300 exploit attempts, sent 1 byte per second.New exploit started every second.
Live experiment results
Attack DescriptionExploits
Detected
Required Rate (kbps)
Control (No attack) 300/300 --
2 packets every 60 s. 220/300 0.4
1 packet every 5 s. 4/300 2.4
1 packet every 3 s. 0/300 4.0
20 packets initially 0/300 0.8
1 packet every 3 s. 300/300 --
20 packets initially 300/300 --
Conclusions
NIDS operation is complex. Many opportunities for vulnerable algorithms.
In Snort, rule-matching is vulnerable and can be exploited by an attacker.
Memoization, along with other semantics-preserving operations, significantly reduces vulnerability.
Other vulnerable algoritms exist.
Backtracking Algorithmic Complexity Attacks Against a NIDS
Thank you.